Why parallel for loop fails in python

Question

I want to do a for loop parallel:

def multiple(a, b):
    return a * b
Parallel(n_jobs=2)(delayed(multiple)(a=i, b=j) for i in range(1, 6) for j in range(11, 16))

But it gives me an error message:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Users/tiantian/Library/Python/2.7/lib/python/site-packages/joblib/parallel.py", line 758, in __call__
    while self.dispatch_one_batch(iterator):
  File "/Users/tiantian/Library/Python/2.7/lib/python/site-packages/joblib/parallel.py", line 603, in dispatch_one_batch
    tasks = BatchedCalls(itertools.islice(iterator, batch_size))
  File "/Users/tiantian/Library/Python/2.7/lib/python/site-packages/joblib/parallel.py", line 127, in __init__
    self.items = list(iterator_slice)
  File "<input>", line 1, in <genexpr>
  File "/Users/tiantian/Library/Python/2.7/lib/python/site-packages/joblib/parallel.py", line 183, in delayed
    pickle.dumps(function)
TypeError: expected string or Unicode object, NoneType found

What is wrong with the code?

Answer 1

disclaimer: I would comment, but not enough rep ;/

First of all you should clearly mention that you are using joblib,

Second, I tested this code using joblib==0.10.3 in interactive interpreter with python 2.7.12 and 3.5.2 - works perfectly.

Error you are getting is thrown when joblib is testing if the function you are trying to use can be pickled (only pickable objects can be multiproccessed), is this full code besides imports? Aren't you messing with multiple somewhere on the way?

It also reminds me some case about multiprocessing not working properly when you're using interactive interpreter, maybe write it as a script if you haven't already?