current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Join the Stack Overflow Community
Stack Overflow is a community of 4.7 million programmers, just like you, helping each other.
Join them; it only takes a minute:
Sign up
up vote 0 down vote favorite

I've got a problem on creating a numpy array of numpy arrays. I would create it in a loop:

a=np.array([])
while(...):
   ...
   b= //a numpy array generated
   a=np.append(a,b)
   ...

Desired result:

[[1,5,3], [9,10,1], ..., [4,8,6]]

Real result:

[1,5,3,9,10,1,... 4,8,6]

Is it possible? I don't know the final dimension of the array, so I can't initialize it with a fixed dimension.

share|improve this question
up vote 4 down vote accepted

Never append to numpy arrays in a loop: it is the one operation that NumPy is very bad at compared with basic Python. This is because you are making a full copy of the data each append, which will cost you quadratic time.

Instead, just append your arrays to a Python list and convert it at the end; the result is simpler and faster:

a = []

while ...:
    b = ... # NumPy array
    a.append(b)
a = np.asarray(a)

As for why your code doesn't work: np.append doesn't behave like list.append at all. In particular, it won't create new dimensions when appending. You would have to create the initial array with two dimensions, then append with an explicit axis argument.

share|improve this answer
    
Thanks for the reply. The big problem is that I need about 100 arrays each one with a very big dimension, each one has got 1440000 integers inside. With list my program is killed by OS. Any suggestions? – Stefano Sandonà Jul 6 '15 at 16:10
    
Can you guess how many elements you will have, and preallocate a big enough array? If not you might have to get very clever, like building the output array in chunks or something. – nneonneo Jul 6 '15 at 16:16
    
You should also be able to preallocate using a larger array, then use a view of that array at the end once you know how many elements are actually present. – nneonneo Jul 6 '15 at 16:17
    
Finally I followed your suggestions! Thanks! – Stefano Sandonà Jul 7 '15 at 8:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.