current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
stack overflow careers
Sign up ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free.
up vote 4 down vote favorite
1

br is the name of a list of strings that goes like this:

['14 0.000000 -- (long term 0.000000)\n',
 '19 0.000000 -- (long term 0.000000)\n',
 '22 0.000000 -- (long term 0.000000)\n',
...

I am interested in the first two columns, which I would like to convert to a numpy array. So far, I've come up with the following solution:

x = N.array ([0., 0.])
for i in br:
    x = N.vstack ( (x, N.array (map (float, i.split ()[:2]))) )

This results into having a 2-D array:

array([[  0.,   0.],
       [ 14.,   0.],
       [ 19.,   0.],
       [ 22.,   0.],
...

However, since br is rather big (~10^5 entries), this procedure takes some time. I was wondering, is there a way to accomplish the same result, but in less time?

share|improve this question

3 Answers 3

up vote 3 down vote accepted

This is dramatically faster for me:

import numpy as N

br = ['14 0.000000 -- (long term 0.000000)\n']*50000
aa = N.zeros((len(br), 2))

for i,line in enumerate(br):
    al, strs = aa[i], line.split(None, 2)[:2]
    al[0], al[1] = float(strs[0]), float(strs[1])

Changes:

  • Preallocate the numpy array (this is big). You already know you want a 2-dimensional array with particular dimensions.
  • Only split() for the first 2 columns, since you don't want the rest.
  • Don't use map(): it's slower than list comprehensions. I didn't even use list comprehensions, since you know you only have 2 columns.
  • Assign directly into the preallocated array instead of generating new temp arrays as you iterate.
share|improve this answer
4  
aa = numpy.array([x.split(' ',2)[0:2] for x in br], dtype='float') – steabert Aug 31 '11 at 17:55
    
Good to know about enumerate: I wasn't aware of it! Also thanks @steabert to his contribution. The speeds of both solutions seem quite similar to me. – Jir Sep 1 '11 at 8:23
up vote 1 down vote

Changing

map (float, i.split()[:2])

to

map (float, i.split(' ',2)[:2])

might result in a slight speedup. Since you only care about first two space-separated items in each line there is no need to split the entire line. The 2 in i.split(' ',2) tells split to just make a maximum of 2 splits. For example,

In [11]: x='14 0.000000 -- (long term 0.000000)\n' 

In [12]: x.split()
Out[12]: ['14', '0.000000', '--', '(long', 'term', '0.000000)']

In [13]: x.split(' ',2)
Out[13]: ['14', '0.000000', '-- (long term 0.000000)\n']
share|improve this answer
    
Thanks for the explanation of the second argument of split! – Jir Sep 1 '11 at 8:21
up vote 1 down vote

You can try to preprocess (with awk for exemple) the list of strings if they come from a file, and use numpy.fromtxt. If you can't do anything about the way you get this list, you have several possibilities:

  • give up. You will run this function once a day. You don't care about speed, and your actual solution is good enough
  • write an IO plugin with cython. You have a big potential gain because you will be able to do all the loops in c, and affects directly the values in a big (10^5, 2) numpy ndarray
  • try another language to fix your problem. If using languages such as c or haskell, you may use ctypes to call the functions compiled in a dll from python

edit

maybe this approach is slightly faster:

def conv(mysrt):
    return map(float, mystr.split()[:2])

br_float = map(conv, br)
x = N.array(br_float)
share|improve this answer
    
Liked the 'out-of-the-box' thinking! – Jir Sep 1 '11 at 8:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.