current community

your communities

more stack exchange communities

Stack Exchange
tour help
careers 2.0
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 1 down vote favorite
1

I have a list named datas, it's come from datas.append(data).

And array data which size is 29*44100 (29 seconds .wav).

Now I have 903 waveform songs.

after load 293 into datas(use append() function),

I convert them from list to array by vstack(datas)

and error happend: ValueError: array is too big.

Is any other way that I can do the same thing but not cause this error?

Thanks.

for more code information:

rates = []
datas = []
labels = []
count = 0

filepath = glob.glob('*.wav')

for fp in filepath:

    if (count<293):

        count +=1            
        rate, data0 = read(fp)
        data = numpy.asarray(data0,dtype=theano.config.floatX)
        data /= numpy.max(numpy.abs(data),axis=0)#normalize to +1..-1            
        length = data.size     

        for index in range(0,length,44100):

            if (index+44100) < length:

                datas.append(data[index:index+44100])
                labels.append(random.randint(1,5))

train_set = numpy.vstack(datas)
share|improve this question
    
you shouldnt delete the same post and than repost it just because you got a "-2" before you edited your code in, people can undo the -1 after you edited in the code, –  usethedeathstar Oct 10 '13 at 6:56
    
sorry for that, I'm a stranger here.I won't do it next time. –  user2858910 Oct 10 '13 at 7:03
add comment

1 Answer

up vote 1 down vote

I tried to reproduce your problem by this piece of code:

import numpy as N

datas = []
for i in range(293):
    datas.append( N.random.rand(44100) )        
res = N.vstack(datas)

which runs fine for me, also for quite large ranges. Does this work for you? If yes, the problem might be somewhere else. You might want to create a large, empty array with e.g. N.zeros((44100,293)) and write into the array directly to avoid memory issues that I sometimes run into when working with big lists:

res = N.zeros((44100,293))
for i in range(293):
    res[:,i] = N.random.rand(44100)

Edit from discussion in comments:

As far as I understand, the amount of data cannot be allocated next to each other in RAM. See this thread for details. You must down-sample or slice your data in an appropriate way to not exceed your memory. Furthermore, consider less big data formats for your array, like int8 instead of the standard float64.

As you mentioned, that you want do do a linear regression, I would like to highlight, that the linear regression e.g. by least squares method can work on segmented data. As you vary a set of fitting variables, you will evaluate the deviation at each point and finally look at the sum of deviations. You have to write a fitting routine, as it usually applies a matrix based method. This just came up to my mind on my way to work - maybe it helps you out.

share|improve this answer
    
one song has 29 arrays representing 29 seconds, each array contain 44100 values. so if I loading 290 songs , the list length will come to 8420. so the total array length is 44100*29*290 ,it's about 107683380000. –  user2858910 Oct 10 '13 at 9:52
    
could I do vstack serval times, to put the arrays not all a time, but maybe just one hundred a time, so that my memory will enough to afford it? –  user2858910 Oct 10 '13 at 9:58
    
maybe there is no way to load the big data to my computer, is it a hardware limit. Or I have to reduce the data dimension from 44100 to 11025. But I still wish there is the better way exist.Thank you. –  user2858910 Oct 10 '13 at 10:14
    
So the problem I found with list.append() is, that at for each element a pointer to this element is saved internally and the space for those pointers is limited (at least, that is how I understood it). That is, why I try to avoid big lists with lots of data and go for arrays instead. Thus, I would really recommend using an array and never use a list in your code. Try running the latter code with 290*29 entries and see if it works out. To save memory you might want to set the data type of the array (dtype=...) to something smaller than float64 if you do not need the precision? –  Faultier Oct 10 '13 at 10:44
    
This does not work on my machine, too. I checked this mail list, there is no way to detect the maximum array size beforehand: mail.scipy.org/pipermail/numpy-discussion/2011-January/… Could you maybe work on your data piece by piece? If you plan e.g. a FFT, or other linear functions, this is viable. –  Faultier Oct 10 '13 at 10:52
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.