parallel processing - Python multiprocessing map function error -

i have simple multiprocessing example i'm trying create. ordinary map() function version works, when changed pool.map, i'm getting strange error:

from multiprocessing import pool functools import partial x = [1,2,3] y = 10 f = lambda x,y: x**2+y  # ordinary map works: map(partial(f,y=y),x) # [11, 14, 19]  # multiprocessing map not p = pool(4) p.map(partial(f, y=y), x) exception in thread thread-2: traceback (most recent call last):   file "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner     self.run()   file "/usr/lib/python2.7/threading.py", line 504, in run     self.__target(*self.__args, **self.__kwargs)   file "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks     put(task) picklingerror: can't pickle <type 'function'>: attribute lookup __builtin__.function failed 

pickling error? exactly?

the arguments pool.map must picklable. module-level functions picklable, partial(f, y=y) not defined @ module-level , not pickable.

there simple workaround:

def g(x, y=y):     return f(x, y)  p.map(g, x) 

---

functions made functools.partial used unpickable. however, python2.7 or better, can define g (at module level) using functools.partial:

import multiprocessing mp import functools  def f(x, y):     return x**2 + y  x = [1,2,3] y = 10  g = functools.partial(f, y=y)  if __name__ == '__main__':     p = mp.pool()     print(p.map(g, x)) 

yields [11, 14, 19]. note result f had defined def rather lambda. think because pickle relies on "fully qualified" name references function object values.