current community

your communities

more stack exchange communities

Stack Exchange
tour help
stack overflow careers
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 16 down vote favorite
8

I am using a Python (via ctypes) wrapped C library to run a series of computation. At different stages of the running, I want to get data into Python, and specifically numpy arrays.

The wrapping I am using does two different types of return for array data (which is of particular interest to me):

  • ctypes Array: When I do type(x) (where x is the ctypes array, I get a <class 'module_name.wrapper_class_name.c_double_Array_12000'> in return. I know that this data is a copy of the internal data from the documentation and I am able to get it into a numpy array easily:

    >>> np.ctypeslib.as_array(x)

This returns a 1D numpy array of the data.

  • ctype pointer to data: In this case from the library's documentation, I understand that I am getting a pointer to the data stored and used directly to the library. Whey I do type(y) (where y is the pointer) I get <class 'module_name.wrapper_class_name.LP_c_double'>. With this case I am still able to index through the data like y[0][2], but I was only able to get it into numpy via a super awkward:

    >>> np.frombuffer(np.core.multiarray.int_asbuffer(
    ctypes.addressof(y.contents), array_length*np.dtype(float).itemsize))

I found this in an old numpy mailing list thread from Travis Oliphant, but not in the numpy documentation. If instead of this approach I try as above I get the following:

>>> np.ctypeslib.as_array(y)
...
...  BUNCH OF STACK INFORMATION
...
AttributeError: 'LP_c_double' object has no attribute '__array_interface__'

Is this np.frombuffer approach the best or only way to do this? I am open to other suggestions but must would still like to use numpy as I have a lot of other post-processing code that relies on numpy functionality that I want to use with this data.

share|improve this question
    
Do you have control over the C lib? Could you change the API of the library? –  Sven Marnach Dec 4 '10 at 20:03
    
Yes - I have the source. I'm not sure which way to go though, as the pointer approach allows Python to directly act on the data which I suppose in some cases could be an advantage. In my case though, yes it would be an advantage to have everything come out as a ctype array. Any recommendations? –  dtlussier Dec 4 '10 at 20:09
    
I would suggest to make the library use an (NumPy-) array you allocate in Python and pass on to the library. That way, you can act on the same memory, but you don't have to bother to do any awkward conversions. You already have a NumPy array, and passing it to a library is well-supported by using numpy.ctypeslib.ndpointer as argument type to the ctypes wrapper of your function. (If this is not clear, just ask...) –  Sven Marnach Dec 4 '10 at 20:46

2 Answers 2

up vote 14 down vote accepted

Creating NumPy arrays from a ctypes pointer object is a problematic operation. It is unclear who actually owns the memory the pointer is pointing to. When will it be freed again? How long is it valid? Whenever possible I would try to avoid this kind of construct. It is so much easier and safer to create arrays in the Python code and pass them to the C function than to use memory allocated by a Python-unaware C function. By doing the latter, you negate to some extent the advantages of having a high-level language taking care of the memory management.

If you are really sure that someone takes care of the memory, you can create an object exposing the Python "buffer protocol" and then create a NumPy array using this buffer object. You gave one way of creating the buffer object in your post, via the undocumented int_asbuffer() function:

buffer = numpy.core.multiarray.int_asbuffer(
    ctypes.addressof(y.contents), 8*array_length)

(Note that I substituted 8 for np.dtype(float).itemsize. It's always 8, on any platform.) A different way to create the buffer object would be to call the PyBuffer_FromMemory() function from the Python C API via ctypes:

buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory
buffer_from_memory.restype = ctypes.py_object
buffer = buffer_from_memory(y, 8*array_length)

For both these ways, you can create a NumPy array from buffer by

a = numpy.frombuffer(buffer, float)

(I actually do not understand why you use .astype() instead of a second parameter to frombuffer; furthermore, I wonder why you use np.int, while you said earlier that the array contains doubles.)

I'm afraid it won't get much easier than this, but it isn't that bad, don't you think? You could bury all the ugly details in a wrapper function and don't worry about it any more.

share|improve this answer
    
That's great - thanks for the overview of the pros and cons. the .astype() call was just an accidental copy and past error. I've pulled it out of my question now. Thanks for picking up on that. –  dtlussier Dec 4 '10 at 20:42
up vote 3 down vote

Another possibility (which may require more recent versions of libraries than is available when the first answer was written -- I tested something similar with ctypes 1.1.0 and numpy 1.5.0b2) is to convert from the pointer to the array.

np.ctypeslib.as_array(
    (ctypes.c_double * array_length).from_address(ctypes.addressof(y.contents)))

This seems to still have the shared ownership semantics, so you probably need to make sure that you free the underlying buffer eventually.

share|improve this answer
2  
Or without special support from numpy: you could convert y pointer to a pointer to an array type: ap = ctypes.cast(y, ctypes.POINTER(ArrayType)) where ArrayType = ctypes.c_double * array_length and create numpy array from that: a = np.frombuffer(ap.contents). See How to convert pointer to c array to python array –  J.F. Sebastian Jan 19 '13 at 18:28
    
I was trying this, but the ap object don't have a member, contents. –  Totte Karlsson Mar 7 '14 at 22:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.