current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Join the Stack Overflow Community
Stack Overflow is a community of 6.5 million programmers, just like you, helping each other.
Join them; it only takes a minute:
Sign up
up vote 6 down vote favorite
1

Problem:

  • Binary data of fixed size records
  • Want to use struct.unpack_from and struct.pack_into to manipulate the binary data
  • Want no copies of the data
  • Want multiple views into the memory to simply offset calculations etc.
  • Data could be in an array.array bytearray or ctypes string buffer

What I tried to do:

part1 = buffer(binary_data, 0, size1)
part2 = buffer(binary_data, size1, size2)
part3 = buffer(binary_data, size1 + size2) # no size is given for this one as it should consume the rest of the buffer
struct.pack_into('I', part3, 4, 42)

The problem here is that struct.pack_into complains about the buffers being read only. I have looked into memoryviews as they can create a read/write view however they don't allow you to specify the offset and size like the buffer function does.

How can I accomplish having multiple zero-copy views into a buffer of bytes that is readable,writable and can be accessed/modified using struct.unpack_from and struct.pack_into

share|improve this question
    
I can do that but I was hoping for a better way. I was hoping to be able to create several buffers with fixed sizes and offsets. That way the pack_into calls wouldn't have to compute the total offset just the offset into the particular structure/piece I am writing into. If I pack into the structures directly all my calls will look like pack_into(fmt, buffer, part_offset + offset, v1, v2 ...) as opposed to pack_into(fmt, buffer, offset, v1, v2, ...) I was really trying to avoid computing the total offset to the data every time. – Matt Sep 19 '13 at 13:22
up vote 5 down vote accepted

In 2.6+, ctypes data types have a from_buffer method that takes an optional offset. It expects a writable buffer and will raise an exception otherwise. (For readonly buffers there's from_buffer_copy.) Here's a quick translation of your example to use ctypes char arrays:

from ctypes import *
import struct

binary_data = bytearray(24)
size1 = size2 = 4
size3 = len(binary_data) - size1 - size2

part1 = (c_char * size1).from_buffer(binary_data)
part2 = (c_char * size2).from_buffer(binary_data, size1)
part3 = (c_char * size3).from_buffer(binary_data, size1 + size2)
struct.pack_into('4I', part3, 0, 1, 2, 3, 4)

>>> binary_data[8:]
bytearray(b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00')

>>> struct.unpack_from('4I', part3)
(1, 2, 3, 4)
share|improve this answer
    
Thanks, that works perfectly. – Matt Sep 19 '13 at 16:23
2  
This was tagged 2.7, so I didn't mention memoryview. In 3.x memoryview objects work with struct.pack_into. The ability to create new views using slices is more convenient than this ctypes approach. – eryksun Sep 19 '13 at 16:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.