Parallel computation in Python is an important aspect to learn for new and pro programmers. Learn more about this function in this guide below.
Parallel computation in Python: Using the multiprocessing module to parallelise tasks
import multiprocessing
def fib(n):
"""computing the Fibonacci in an inefficient way
was chosen to slow down the CPU."""
if n <= 2:
return 1
else:
return fib(n-1)+fib(n-2)
p = multiprocessing.Pool()
print(p.map(fib,[38,37,36,35,34,33]))
Out: [39088169, 24157817, 14930352, 9227465, 5702887, 3524578]
As the execution of each call to fib happens in parallel, the time of execution of the full example is 1.8× faster than if done in a sequential way on a dual processor.
Python 2.2+
Parallel computation in Python: Using a C-extension to parallelize tasks
The idea here is to move the computationally intensive jobs to C (using special macros), independent of Python, and have the C code release the GIL while it’s working.
include “Python.h”
…
PyObject *pyfunc(PyObject *self, PyObject *args) {
…
Py_BEGIN_ALLOW_THREADS
Threaded C code
…
Py_END_ALLOW_THREADS
…
}
Using Parent and Children scripts to execute code in parallel
child.py
import time def main(): print "starting work" time.sleep(1) print "work work work work work" time.sleep(1) print "done working" if name == 'main': main()
parent.py
import os
def main():
for i in range(5):
os.system("python child.py &")
if name == 'main':
main()
This is useful for parallel, independent HTTP request/response tasks or Database select/inserts. Command line arguments can be given to the child.py script as well. Synchronization between scripts can be achieved by all scripts regularly checking a separate server (like a Redis instance).
Using PyPar module to parallelize
PyPar is a library that uses the message passing interface (MPI) to provide parallelism in Python. A simple example in PyPar (as seen at https://github.com/daleroberts/pypar) looks like this:
import pypar as pp
ncpus = pp.size()
rank = pp.rank()
node = pp.get_processor_name()
print 'I am rank %d of %d on node %s' % (rank, ncpus, node)
if rank == 0:
msh = 'P0'
pp.send(msg, destination=1)
msg = pp.receive(source=rank-1)
print 'Processor 0 received message "%s" from rank %d' % (msg, rank-1)
else:
source = rank-1
destination = (rank+1) % ncpus
msg = pp.receive(source)
msg = msg + 'P' + str(rank)
pypar.send(msg, destination)
pp.finalize()