Tagged Questions
0
votes
0answers
25 views
Statistics function fails with a converted pandas DataFrame
df is my dataframe and I use .apply() on it.
This does not work and produces Division by zero:
def power_divergence(x): # x is a series here
s = sp.stats.power_divergence(np.array(x), ...
1
vote
1answer
21 views
The precision of scipy.special.gammaln
The Problem
A lot of my programming involves the statistics functions in scipy.stats. A new problem required computing the pmf of the beta-binomial distribution. Because it has an analytic form, but ...
1
vote
0answers
21 views
use scipy.stats to automatically fit and use the parameter in pdf calculation
I would like my program to automatically choose the distribution that has the best fitness and use this distribution's probability density function to calculate the probability
Use ...
1
vote
1answer
49 views
How to visualise the number of dimensions in an array for python numpy
I am experimenting the different dimensions one can have in an array using ndim().
x=np.arange(0,100,1).reshape(1,20,5)
The shape is:
[[[ 0 1 2 3 4]
[ 5 6 7 8 9]
[10 11 12 13 14]
[15 ...
2
votes
0answers
22 views
Python: Compute Delaunay Triangulation based on arbitrary distance?
Right now I'm using scipy.spatial.Delaunay for construction of Delaunay triangulation and sub-graphs of it. As I suppose, euclidean distance is used as metric to determine the distance of the points. ...
0
votes
0answers
34 views
How to fit multiple curves into one function using python?
I have a function containing 2 dependent variables (x1 and x2), 2 parameters (a and b), producing 1 independent variable (y).
Below the link are experimenetal values in nested arrays where column 1 ...
5
votes
1answer
97 views
Why does X.dot(X.T) require so much memory in numpy?
X is a n x p matrix where p is much larger than n. Let's say n = 1000 and p = 500000. When I run:
X = np.random.randn(1000,500000)
S = X.dot(X.T)
Performing this operation ends up taking a great ...
2
votes
2answers
33 views
How to make a scipy array from custom data format?
Warning: Python newbie...
I have text that looks like this, which came from a database query:
2000;"SCHOOLS OF MEDICINE";416765.0
2000;"SCHOOLS OF ARTS AND SCIENCES";36000.0
2000;"SCHOOLS OF ...
0
votes
1answer
27 views
Difference between curve_fit and leastsq in python from scipy.optimize
I have a function containing:
Independent variable X,
Dependent variable Y
Two fixed parameters a and b.
Using identical experimental data, both the curve_fit and leastsq functions could be fitted ...
3
votes
3answers
38 views
Numpy savetxt heterogenous data
I have two arrays A and B such that:
A.shape is (N, 1) and dtype is int.
B.shape is (N, M) and dtype is float.
The array A contains the ids/serial no. info. I want to write A and B in a single file ...
0
votes
1answer
15 views
Install github clones of a python package to WinPython
I'm trying to install a package, pybrain, from its github website. It does not have nicely packaged archive files or executables, so I don't know how to install it via WPPM.
Do I still do python ...
0
votes
1answer
13 views
Why do scikit-learn regressors raise this shape error?
I have a matrix of data that I store in one of the scipy.sparse formats for sparse matrices, and a bunch of outcomes that I need to predict. Basically I want to fit a linear model for each one of the ...
2
votes
1answer
39 views
How to work around the ValueError: array is too big error?
I've got a scipy sparse matrix (csr:Compressed Sparse Row matrix). I'd like to use Orange's feature selection methods (Orange.feature.scoring.score_all (InfoGain/MDL)). However, from my understanding ...
0
votes
1answer
36 views
matlab operations of 2 numpy arrays of same shape in python
I have 2 numpy array with equal shape.
V = [[-1 -1 -1] [-2 -2 -2] [-3 -3 -3]]
U = [[1 2 3] [2 3 4] [3 4 5]]
I want to convert matlab to python for below line.
Ot = U*([V(:,1) V(:,2) -V(:,3)])';
...
0
votes
1answer
26 views
conversion of repmat in matlab to python
I have numpy array
X1 = [[-1 -2 -3 -4 -5] [ 1 2 3 4 5] [ 6 7 8 9 -1]]
with shape 3, 5.
I want to convert matlab repmat function to numpy
X1 = X1 - repmat(mean(X1')',1,size(X1,2));
How can I ...
0
votes
1answer
18 views
scipy wired output of scipy.stats.truncnorm
I am using scipy version of random generation for truncated normal distribution:
from scipy.stats import truncnorm;
truncnorm.rvs(8.26,1e10)
if I try multiple times, the supposed random generation ...
2
votes
0answers
35 views
memory error in numpy svd
I am performing numpy svd
U, S, V = np.linalg.svd(A)
shape of A is :
(10000, 10000)
Due to the large size, it gives me memory error :
U, S, V = np.linalg.svd(A, full_matrices=False) # ...
1
vote
2answers
62 views
Find global minimum of discretely defined function
I have a four parameter function for which I do not have the mathematical form because it is actually the result of several separate processes. In its simplest form it can be thought of as a black box ...
1
vote
2answers
19 views
Scipy.opimize.fmin_powell direc argument syntax
There is no information on how the direc argument of fmin-powell is supposed to be entered. All the scipy documentation for fmin_powell says is
direc : ndarray, optional
Initial direction ...
0
votes
2answers
35 views
Python - scipy fmin, giving the arguments to fmin
I'm a bit of a newbie in Python.
I'm writing a little piece of code in order to find the minimum of a function:
import os,sys,matplotlib,pylab
import numpy as np
from scipy.optimize import fmin
par ...
3
votes
1answer
33 views
Least-square fitting, confusing assignment query for python scipy
I'm new to this and have been looking at this for hours, comparing it with other least-square fitting examples, but the example code does not seem to click.
From ...
-1
votes
2answers
45 views
plotting graphs in numpy, scipy
I have x1, y1, z1 coordinated at time t1 and I have x2, y2, z2 coordinates at time t2. I want to plot the graph for them.
I have all numpy arrays.
A = [[44.254 44.114 44.353 44.899 45.082] ...
0
votes
1answer
14 views
How to know integration technique used by quadpack [on hold]
I have been learning about QUADPACK and I used scipy.integrate.quad to calculate an integral from 0 to infinite. It gave a very good result, but now I want to know which integration method (QAGI, ...
2
votes
1answer
20 views
linear interpolation in scipy
I want to interpolate in scipy. I have 3 coordinate values at specific time period t1.
x 44.254 44.114 44.353 44.899 45.082
y -0.934 0.506 1.389 0.938 0.881
z 44.864 45.225 44.005 ...
-2
votes
0answers
30 views
simple python minimize bfgs algorithm
'''Hi, first time learning python here!
I'm trying to maximize a sample log-likelihood function with respect to standard deviation to obtain a maximum likelihood estimator. The distribution is normal ...
0
votes
1answer
13 views
How can I set, rather than fit, the co-efficients of a spline interpolation using scipy?
I am trying to train a predictive model and want to use a spline-like interpolation to represent some function that forms part of the model. However, this is not a simple case of fitting some x,y data ...
3
votes
0answers
52 views
Auto-correlation measurement for spatial separation?
I have a three columns data, two columns spatial coordinates and in the third column, one property of my data which I am interested to compute the auto-correlation between this parameter according ...
0
votes
2answers
31 views
fmin_cg function usage for minimizing neural network cost function
I am trying to port some of my code from MatLab into Python and am running into problems with scipy.optimize.fmin_cg function - this is the code I have at the moment:
My cost function:
def ...
1
vote
1answer
46 views
SciPy interpolation ValueError: x and y arrays must be equal in length along interpolation axis
I'm trying to work with interp1d of SciPy.interpolate. I "plugged in" two arrays (filtered_mass and integrated_column), of same size, but it still give me ValueError that the sizes of the arrays must ...
2
votes
2answers
33 views
How do I slice array labels in python pandas?
pandas dataframe labels can be arrays, eg instead of ['a', 'b', 'c'] as columns I have [(0,10), (1,11), (2,12)].
My array is called df and I only have 2 rows, 0 and 1.
I would like to slice the ...
0
votes
0answers
16 views
How to create the scipy linkage matrix given the singletons and their cophenetic distance?
Given the cophenetic distances (corresponds to Z[:, 2]) and the singletons involved in a fusion:
How do I create the linkage matrix Z as created by scipy.cluster.hierarchy.linkage? My specific problem ...
0
votes
0answers
8 views
Cluster samples below a threshold distance in a custom distance matrix using SciPy
I would like to do clustering in SciPy using a custom distance matrix that clusters my data using a threshold value.
To illustrate what I want to get, I used vmin in matplotlib's pcolor...
This is ...
0
votes
3answers
64 views
Fitting gaussian to a curve in Python II
I have two lists .
import numpy
x = numpy.array([7250, ... list of 600 ints ... ,7849])
y = numpy.array([2.4*10**-16, ... list of 600 floats ... , 4.3*10**-16])
They make a U shaped curve.
Now I ...
3
votes
2answers
88 views
Dot product of csr_matrix causes segmentation fault
I have two (scipy) CSR sparse matrices:
A (12414693, 235470)
B (235470, 48063)
Performing:
A.dot(B)
causes a segmentation fault.
What am I doing wrong?
EDIT
I've submitted a bug to the scipy ...
0
votes
1answer
19 views
Scipy multidimensional kernal density estimate
I've been trying to get a kernal density estimate for a 30x30 array. A short example is below,
from scipy.stats import gaussian_kde
x = arange(-0.5,0.51,1/29.)
y = arange(-0.5,0.51,1/29.)
z = ...
0
votes
2answers
34 views
How does l_bfgs optimization method approximate the gradient
I'm using scipy's fmin_l_bfgs_b optimization method on a 2-dimensional function available as a black box. Gradients cannot be evaluated directly, so I'm asking the method to approximate the gradients ...
2
votes
2answers
50 views
Scipy Sparse - distance matrix (Scikit or Scipy)
I am trying to compute nearest neighbour clustering on a Scipy sparse matrix returned from scikit-learn's DictVectorizer. However, when I try to compute the distance matrix with scikit-learn I get an ...
0
votes
2answers
52 views
linear interpolation in numpy
I have 2 numpy arrays
X = [[2 3 6], [7 2 9], [7 1 4]]
a = [0 0.0005413307 0.0010949014 0.0015468832 0.0027740823 0.0033288284]
b = [0 0.0050251256 0.0100502513 0.0150753769 ...
0
votes
1answer
27 views
Taking reciprocal of each elements in a sparse matrix?
I have a sparse matrix below.
a = sparse.diags([1,2,3],[-1,0,1],shape=(6,6),format ="csr")
I want to take the reciprocal of each of the elements in the sparse matrix.
I search it on the internet ...
2
votes
2answers
56 views
How could this matrix / vector dot multiplication be possible in numpy.linalg module?
The code as below...
>>> A
array([[1, 2],
[3, 4]])
>>> b
array([5, 6])
>>> A.dot(b) # <------- this is not possible in math
array([17, 39]) # <------ and the ...
0
votes
2answers
65 views
Installation of numpy in PyCharm on Windows
I keep getting errors when I try to install numpy in Pycharm (Windows)
This is the error I get
C:\Python27\lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'define_macros'
...
4
votes
2answers
35 views
Why does from scipy import spatial work, while scipy.spatial doesn't work after import scipy?
I would like to use scipy.spatial.distance.cosine in my code. I can import the spatial submodule if I do something like import scipy.spatial or from scipy import spatial, but if I simply import scipy ...
1
vote
2answers
46 views
sqrt for element-wise sparse matrix
I have a sparse matrix:
from scipy import sparse
a = sparse.diags([1,4,9],[-1,0,1],shape =(10,10),format ="csr")
I want to take the square root of each of the elements in the sparse matrix
I look ...
-1
votes
1answer
35 views
Fitting a weighted histogram with a normal distribution
I know how to fit the data entering an histogram with a normal distribution using the SCipy library (Fitting a histogram with python) but how could I do the same if on top of having data I have an ...
0
votes
1answer
27 views
Submatrix in scipy
I have a sparse matrix A and a column vector a,
In[1]: A
Out[1]: <256x256 sparse matrix of type '<type 'numpy.float64'>'
with 512 stored elements (blocksize = 2x2) in Block Sparse Row ...
0
votes
1answer
47 views
scipy.optimize.fmin_bfgs optimization gives different result from simple function call
I am exploring some of the numpy/scipy functions and I noticed that scipy.optimize.fmin_bfgs requires a change in the function being called to give correct results compared to a straight function ...
4
votes
0answers
69 views
What libraries are people using with scipy to build a scientific computing stack comparable to matlab? [closed]
This is domain specific, of course, so let me list the toolboxes that matter to me (economics and some macro-finance): "Symbolic Math, PDEs, Statistics, Curve Fitting, Optimization, Global ...
1
vote
1answer
36 views
correct usage of scipy.optimize.fmin_bfgs required versus R code
I am used to doing all my statistics in R and python for all the peripheral tasks. Just for fun I attempted a BFGS optimization to compare it to the ordinary LS result - both in python using ...
1
vote
1answer
33 views
numpy.loadtxt - negated usecols?
I would like to read a CSV file with numpy.loadtxt. I know that I can specify the columns I want to read with the usecols parameter. However, what I actually want to do is to specify a list of columns ...
0
votes
0answers
19 views
scipy.optimize.fmin_slsqp does nothing if the starting values have too many decimal places?
I am experimenting with fmin_slsqp.
If I use starting values with many decimal places, then it only does 4 function evaluations, using the same parameter values every time(my script prints them out), ...
