up vote 3 down vote favorite
1

I have a cpu intensive code which uses a heavy dictionary as data (around 250M data). I have a multicore processor and want to utilize it so that i can run more than one task at a time. The dictionary is mostly read only and may be updated once a day.
How can i write this in python without duplicating the dictionary?
I understand that python threads don't use native threads and will not offer true concurrency. Can i use multiprocessing module without data being serialized between processes?

I come from java world and my requirement would be something like java threads which can share data, run on multiple processors and offers synchronization primitives.

share|improve this question
feedback

3 Answers

up vote 1 down vote accepted

You can share read-only data among processes simply with a fork (on Unix; no easy way on Windows), but that won't catch the "once a day change" (you'd need to put an explicit way in place for each process to update its own copy). Native Python structures like dict are just not designed to live at arbitrary addresses in shared memory (you'd have to code a dict variant supporting that in C) so they offer no solace.

You could use Jython (or IronPython) to get a Python implementation with exactly the same multi-threading abilities as Java (or, respectively, C#), including multiple-processor usage by multiple simultaneous threads.

share|improve this answer
feedback
up vote 0 down vote

Take a look at this in the stdlib: http://docs.python.org/library/multiprocessing.html There are a bunch of wonderful features that will allow you to share data structures between processes very easily.

share|improve this answer
feedback
up vote 1 down vote

Use shelve for the dictionary. Since writes are infrequent there shouldn't be an issue with sharing it.

share|improve this answer
from the docs it seems that a value fetch will return a copy of data using pickle? a task for me typically access 1/3 of dict which would mean lots of temp objects in this approach – TigrisC Jan 26 '10 at 1:05
Then there's no good solution in CPython for your issue. Only a series of not-quite-awful ones, all involving some sort of database. – Ignacio Vazquez-Abrams Jan 26 '10 at 1:11
feedback

Your Answer

 
or
required, but never shown
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.