Tagged Questions
0
votes
0answers
10 views
Chi squared test in Python
I'd like to run a chi-squared test in Python. I've created code to do this, but I don't know if what I'm doing is right, because the scipy docs are quite sparse.
Background first: I have two groups ...
0
votes
1answer
6 views
Expanding a Pandas.dataframe after encoding
My actual problem need to encode strings in a data frame, as I do in the following step:
import pandas as pd
df = pd.DataFrame({"cool": list("ABC"), "not_cool": list("CBA")})
encoding = {"A": [0, 0, ...
0
votes
0answers
5 views
save/load object with large numpy matrices
I have a dataset class where some of the members are matrices that can get very large; sometimes too large to keep in RAM. Therefore, I allocate these matrices as numpy.memmaps.
I would like to save ...
0
votes
1answer
9 views
Update numpy version in spyder
I've searched around but I can't seem to find any clear instructions. How can I update the version of Numpy provided in an already installed Spyder environment? I used Anaconda to install Spyder, ...
0
votes
0answers
16 views
Large-scale semantic similarity between text documents
I currently have 360.000 text documents (Wikipedia articles) for which I want to calculate semantic similarity between them. I would prefer to calculate all-pairs similarity, but this does not seam ...
2
votes
0answers
14 views
How to force python print numpy datetime64 with specified timezone?
I want to see numpy datetime64 objects by my specified timezone.
>>> import numpy as np
>>> np.datetime64('2013-03-10T01:30:54')
numpy.datetime64('2013-03-10T01:30:54+0400')
...
2
votes
1answer
20 views
Making ROC curve using python for multiclassification
Following up from here: Converting a 1D array into a 2D class-based matrix in python
I want to draw ROC curves for each of my 46 classes. I have 300 test samples for which I've run my classifier to ...
0
votes
1answer
21 views
Python: raw image data to grayscale
I have raw image data stored in a text file, in a format like this and I would like to make it a 2D array and convert it to grayscale.
0
0
0
0
24
243
227
223
224
225
227
228
228
What I would like ...
0
votes
0answers
23 views
How to remove jumps by scale factors of 10 in Pandas?
I have a series in Pandas. It sometimes jumps by some factor of 10 due to poor data. What is the most efficient way to do this? I have written the attached code, but feel that it is inefficient and ...
0
votes
0answers
24 views
plotting a scatter plot for list/array in matplotlib
It seems very trivial ,but I am not able to get past this simple problem. I fetch the first row of a very huge dataframe.
appleProd1 appleProd2 appleProd3 appleProd4 ..... 70 ...
0
votes
1answer
48 views
Why seems Python to crash when I create a nested list of objects?
I'm new to Python and just doodled around to get to know the language. Basically, I wanted produce random walks and have them analyzed. However, I was not able to accomplish a list of fit objects with ...
0
votes
1answer
18 views
Trouble writing sparse matrix to csv
I'm a newbie to Python, and I'm trying to write the data in a matrix to a CSV file. The variable is defined as:
(Pdb) trainFeatures
<1562936x312116 sparse matrix of type '<type ...
0
votes
1answer
19 views
How to access an element in a Numpy array
So I have this list of Numpy arrays:
import numpy as np
from numpy import array
m = [array([0, 64]), array([ 0, 79]), array([0, 165]), array([0, 50])]
How do I index the number 50 from the ...
0
votes
0answers
24 views
how get the right numpy
I am trying to install bcolz package for python 2.7 on Mac OS X 10.8.
If I use
python setup.py install
The result is:
* Found Cython 0.20.2 package installed.
* Found numpy 1.8.1 package ...
2
votes
2answers
35 views
Change default arguments of function in python
I have a function that converts a numpy array to a array containing True or False based on a condition then groups the True or False entries that are adjacent to one another and calculates the length ...
0
votes
1answer
34 views
Improving memory efficiency of a vectorised function
I have nine large float arrays (3000 by 3000). The arrays are called g_pro_X_array etc.
The vectorised function checks through the cells in each array adding them together and once they exceed it ...
0
votes
1answer
7 views
Trying to install pymc on Anaconda(python) in Windows 7 and getting weird error?
I want to run some data science algorithms using Markov Chain Monte Carlo for Bayesian analysis and am trying to install PyMC but am frustratingly getting this error...
File ...
1
vote
1answer
30 views
Converting a 1D array into a 2D class-based matrix in python
I'm using a multi-class classifier, so in order to evaluate it after testing, I need the predictions from the classifier (y_pred)to be compared against the true class values (y_test).
But I have ...
0
votes
1answer
14 views
Plotting ROC curve - too many indices error
I'm picking up the ROC code straight from here: http://scikit-learn.org/stable/auto_examples/plot_roc.html
I've hard coded my number of classes as 46, in the for loop, as you can see, however even if ...
0
votes
1answer
15 views
array.shape() giving error tuple not callable
I have a 2D numpy array called results, which contains its own array of data, and I want to go into it and use each list:
for r in results:
print "r:"
print r
y_pred = np.array(r)
...
0
votes
2answers
25 views
Using scipy.stats library to generate data in a specific boundary
I want to sample with scipy.stats library, using an upper and a lower boundary for the sampled data. I am interested to use scipy.stats.lognorm and scipy.stats.expon and set a constrain ...
1
vote
0answers
23 views
Python parallel computing using joblib
I am having issues trying to get my parallel processing to work in a function that can be called from the command line. The function downloads data from a website, does some processing and saves the ...
0
votes
1answer
14 views
Numpy matrix operations on custom C structures with overloaded operators
I'm working on a project which requires handling matrices of custom C structures, with some C functions implementing operations over these structures.
So far, we're proceeding as follows:
Build ...
1
vote
1answer
33 views
Parsing colon seperated sparse data with pandas and numpy
I would like to parse data file with the format col_index:value in pandas/numpy. For example:
0:23 3:41
1:31 2:65
would correspond to this matrix:
[[23 0 0 41]
[0 31 65 0]]
It seems like a ...
1
vote
1answer
35 views
Writing both float and integer on a text file from one numpy array
I have a nxm numpy array that either has positive values with 8 decimal points like 0.02113342 or NoValue data that is -9999. I am using the line below to make a text file from the numpy array
...
0
votes
0answers
12 views
add a space using np.savetext format (python) [on hold]
Are there any specifiers that allow me to add 2 spaces before floats?
I tried using delimiter, but it didn't work.
I'd like the dat.txt file to look like this:
1.50
2.30
1.30
instead of
1.50
...
0
votes
0answers
24 views
Speed up chunked matrix multiplication with transpose in Python
Currently trying to multiply a matrix with its transpose. The matrix is a csr_matrix -- i.e., A=csr_matrix() and B=A.T.tocsc(). I am using a chunked algorithm for several rasons in the following way:
...
0
votes
1answer
17 views
looking for a 3D version of numpy.linalg.norm
I'm looking for a build-in function in python. It should compute the frobenius norm of a 3D array.
My current approach is:
np.sqrt(np.sum(np.square(x[:,:,:])))
but this is too slow for the size ...
0
votes
1answer
28 views
TypeError: only length-1 arrays can be converted to Python scalars (for loop)
I am trying to code an analytical solution to a dam break in a rectangular channel. The idea is to have water on one side of the dam at 4m and no water on the downstream side of the dam, then to have ...
0
votes
1answer
8 views
Interpolating array columns with PiecewisePolynomial in scipy
I'm trying to interpolate each column of a numpy array using scipy's PiecewisePolynomial. I know that this is possible for scipy's interp1d but for piecewise polynomial interpolation it does not seem ...
0
votes
1answer
25 views
List comprehension & error: only length-1 arrays can be converted to Python scalars
I am trying to create equally sized samples between the values of x below which are greater or equal to 0.45:
x
array([ 0. , 0.05, 0.1 , 0.15, 0.2 , 0.25, 0.3 , 0.35, 0.4 ,
0.45, ...
0
votes
2answers
33 views
understanding numpy's dstack function
I have some trouble understanding what numpy's dstack function is actually doing. The documentation is rather sparse and just says:
Stack arrays in sequence depth wise (along third axis).
...
-2
votes
0answers
28 views
Transfer function estimation with python [on hold]
Is there any way in python to estimate transfer function from frequency response data?
I want equivalent of the tfest function available in matlab System Identification Toolbox
...
0
votes
2answers
40 views
How to see that a numpy array of zeros is not empty?
Here is my problem:
I use numpy any() function to check if my array is empty or not.
a = numpy.array([1., 2., 3.])
a.any()
#True
a = numpy.array([0., 0., 0.])
a.any()
#False
I would think that, ...
1
vote
0answers
33 views
How to remove Barrel Distortion in Python? [on hold]
I am working in python and trying to remove the Barrel distortion caused in image due to Wide angle lens.
I have tried using the matlab file ...
0
votes
1answer
20 views
Python - 'numpy.float64' object is not callable using minimize function for alpha optimization for Simple Exponential Smoothing
I'm getting the TypeError: 'numpy.float64' object is not callable error for the following code:
import numpy as np
from scipy.optimize import minimize
def ses(data, alpha):
fit=[]
...
1
vote
2answers
19 views
Numpy array index out of range with Genetic Algorithm
I wrote a script to generate an image based from a source, with randomized ellipses using genetic algorithms. I keep receiving this error (the length of seeds is different every time, this is just an ...
2
votes
2answers
18 views
Acces all off diagonal elements of boolean numpy matrix
Suppose there is a diagonal matrix M:
#import numpy as np
M = np.matrix(np.eye(5, dtype=bool))
Does anybody know a simple way to access all off diagonal elements, meaning all elements that are ...
0
votes
0answers
54 views
SciPy Optimization: Matrix Operations
I have an optimization problem I wish to solve that has some special characteristics. I have been trying to figure out how to fit it into the mold that SciPy optimize wants, but have been having some ...
1
vote
1answer
38 views
Markov Chain Monte Carlo (python, numpy)
I am doing some research in physics, for which I need to analyze some data using a Markov Chain Monte Carlo (MCMC). I tried to just write one myself but I keep coming across bugs when python/numpy ...
0
votes
1answer
41 views
fastest way to get lookup table indices with numpy
This question follows this other one, and aims at accelerating the following code. I built (with help) some code that takes pixel values from a m x n x 3 numpy.ndarray (an RGB image), compares the ...
3
votes
0answers
43 views
Python defaultdict for large data sets
I am using defaultdict to store millions of phrases, so my data structure looks like mydict['string'] = set(['other', 'strings']). It seems to work ok for smaller sets but when I hit anything over 10 ...
0
votes
2answers
51 views
Estimate formants using LPC in Python
I'm new to signal processing (and numpy, scipy, and matlab for that matter). I'm trying to estimate vowel formants with LPC in Python by adapting this matlab code:
...
0
votes
2answers
28 views
getting indices when comparing multidimensional arrays
I have two numpy arrays, one an RGB image, one a lookup table of pixel values, for example:
img = np.random.randint(0, 9 , (3, 3, 3))
lut = np.random.randint(0, 9, (1,3,3))
What I'd like is to know ...
0
votes
3answers
70 views
How to efficiently calculate huge matrix multiplication (tfidf features) in Python?
I currently want to calculate all-pair document similarity using cosine similarity and Tfidf features in python. My basic approach is the following:
from sklearn.feature_extraction.text import ...
1
vote
0answers
43 views
Import errors after upgrading to sklearn 0.15
Using Ubuntu 13.10 64 bit and python 2.7.5. I've been using sklearn 0.14 for quite some time. After upgrading to version 0.15 via:
pip install --upgrade scikit-learn
I've encountered the following:
...
-1
votes
2answers
35 views
why this a python script works on OSX and not on Windows?
I have a python script which runs perfectly on OSX.
After installing python, numpy and matplotlib on windows I cannot understand why the same does not run on windows.
Do you have any idea?
import ...
0
votes
2answers
30 views
python - RGB matrix of an image
Taking an image as input, how can I get the rgb matrix corresponding to it?
I checked out the numpy.asarray function. Does that give me the rgb matrix or some other matrix?
1
vote
1answer
18 views
Faster/Better way to modify (tuple defined) parts of a Numpy array in Python?
I have a numpy pixel array (either 0 or 255), using .where I pull the tuples of where it is > 0. I now want to use those tuples to add 1 to a separate 2D numpy array. Is the best way to just use a for ...
0
votes
1answer
65 views
Synchronizing Data in Python (or Excel)
I frequently use Python (and occasionally Excel) to process and compare test data between multiple experiments. In some cases the data might be out of sync which makes direct comparisons difficult. ...
