Advanced and Parallel Python

Size: px

Start display at page:

Download "Advanced and Parallel Python"

Gary Greer
6 years ago
Views:

1 Advanced and Parallel Python December 1st, By: Bart Oldeman and Pier-Luc St-Onge 1

2 Financial Partners 2

3 Setup for the workshop 1. Get a user ID and password paper (provided in class): ##: usernm XXXXXXXXXX ********** 2. Access to local computer (replace ## and with appropriate values, is provided in class): a. User name: csuser## b. 3. HTTPS connection to Colosse (replace **********): a. b. User name: usernm c. Password: ********** d. If requested: i. click Start Server button, set walltime 8 3

4 Select Modules Change Notebook Kernel In the Software tab, select: compilers/llvm/3.7.1 compilers/gcc/4.8.5 Open notebooks/01-stack.ipynb File -> Save and Checkpoint 4

5 Import Examples and Exercises In case the cq-formation-advanced-python folder is not in your home directory, open a Terminal and type: module load apps/git/ # If on Colosse git clone -b ulaval \ cd cq-formation-advanced-python 5

6 Outline Revisiting the Scientific Python Stack Why (and What) is Python? Accelerating Python code: PyPy and Numpy Using C code from Python code Finding Bottlenecks - Profiling code Compiling Python Code Using Cython and Numba Parallelizing Python Programs Parallel Programming Concepts The multiprocessing Module MPI for Python (mpi4py) 6

7 The Scientific Python stack 7

8 Scientific Python stack In the introductory workshop we looked at: Python itself Numpy, for numerical array objects Scipy, for higher level routines IPython, an advanced Python shell Matplotlib, for plotting On top of that we introduce some new components, for example: Cython, for speed and interfacing mpi4py for using MPI in Python 8

9 Speeding up Python programs 9

10 Speeding up Python Central example: approx_pi.c / approx_pi.py: // approx_pi.c double approx_pi(int intervals) { double pi = 0.0; # approx_pi.py def approx_pi(intervals): pi = 0.0 int i; for (i = 0; i < intervals; i++) { pi += (4 - ((i % 2) * 8)) / (double)(2 * i + 1); } for i in range(intervals): pi += (4-8 * (i % 2)) / (float)(2 * i + 1) return pi return pi; } 10

11 Speeding up Python Compile: $ gcc -O2 pi_collect.c approx_pi.c -o pi_collect $./pi_collect Time = 0.88 sec Python run (example on Guillimin): $ module load iomkl/2015b Python/3.5.0 $ python pi_collect.py approx_pi The compiled C code runs almost 100 times faster than the Python code (0.88 vs. 66 seconds with intervals = ). Note that approx_pi is the module to import for pi_collect.py. 11

12 Speeding up Python How to speed up: two approaches 1. Make Python go faster a. Use the PyPy just-in-time compiler b. Use Numpy with vectorized code c. Use Cython 2. Call C code from Python a. b. c. d. e. Manually Use SWIG Use Ctypes Use Cython... 12

13 Speeding up Python using PyPy How to speed up: use PyPy: $ module add pypy/ $ pypy3 pi_collect.py approx_pi gives 2.2 seconds (30 times faster) An alternative to PyPy is Numba (not installed on Guillimin). 13

14 Speeding up with numpy How to speed up: use vectorized code: from future import division # only needed for Python 2.x def approx_pi(intervals): pi1 = 4/numpy.arange(1, intervals*2, 4) pi2 = -4/numpy.arange(3, intervals*2, 4) return numpy.sum(pi1) + numpy.sum(pi2) $ python3 pi_collect.py approx_pi_numpy gives 1.4 seconds (47 times faster). Drawback: extra memory use. How to speed up: Cython: see later 14

15 Interfacing with C/C++/Fortran 15

16 Interfacing with C and C++ There are at least 14 different ways to do it: By hand using the Python API (*) Pyrex Cython (**) SWIG (*) SIP Boost.Python PyCXX CTypes (*) Py++ f2py (*) PyD Interrogate Robin (*) Quick introduction Pybind11 (**) Most popular now, more thorough introduction 16

17 Using the Python API Pros: no extra dependencies Cons: a lot of boilerplate code, which can change between Python version /* Example of wrapping approx_pi() with the Python-C-API. */ #include <Python.h> #include "approx_pi.h" static PyObject* approx_pi_func(pyobject* self, PyObject* args) // wrapped approx_pi() { int value; double answer; if (!PyArg_ParseTuple(args, "i", &value)) // parse input, python float to c double return NULL; /* if the above function returns -1, an appropriate Python exception will * have been set, and the function simply returns NULL */ answer =approx_pi(value); /* construct the output from approx_pi, from c double to python float */ return Py_BuildValue("f", answer); } 17

18 Using the Python API /* define functions in module */ static PyMethodDef PiMethods[] = { {"approx_pi", approx_pi_func, METH_VARARGS, "approximate Pi"}, {NULL, NULL, 0, NULL} }; static struct PyModuleDef PiModule = { PyModuleDef_HEAD_INIT, "approx_pi_pyapi", NULL, -1, PiMethods, NULL, NULL, NULL, NULL }; /* module initialization */ PyMODINIT_FUNC PyInit_approx_pi_pyapi(void) { (void) PyModule_Create(&PiModule);} Compile using $ python3 setup_approx_pi_pyapi.py build_ext --inplace from distutils.core import setup, Extension # define the extension module module = Extension('approx_pi_pyapi', sources=['approx_pi_pyapi.c', 'approx_pi.c']) setup(ext_modules=[module]) # run the setup 18

19 Using CTypes Pros: the ctypes package is in Python by default, pure Python solution Cons: wrapped code in shared lib, interface not fast First compile approx_pi_ctypes.so: $ gcc -fpic -shared -O2 approx_pi.c -o approx_pi_ctypes.so # approx_pi_ctypes.py """ Example of wrapping approx_pi using ctypes. """ import ctypes approx_pi_dll = ctypes.cdll.loadlibrary('./approx_pi_ctypes.so') # find and load the library approx_pi_dll.approx_pi.argtypes = [ctypes.c_int] # set the argument type approx_pi_dll.approx_pi.restype = ctypes.c_double # set the return type def approx_pi(arg): ''' Wrapper for approx_pi ''' return approx_pi_dll.approx_pi(arg) 19

20 Using SWIG Mature solution Wrapper file is autogenerated from interface file. /* approx_pi_swig.i */ /* Example of wrapping approx_pi using SWIG. */ %module approx_pi_swig %{ /* the resulting C file should be built as a python extension */ #define SWIG_FILE_WITH_INIT /* Includes the header in the wrapper code */ #include "approx_pi.h" %} /* Parse the header file to generate wrappers */ %include "approx_pi.h" 20

21 Using SWIG Use distutils as before (python3 setup_approx_pi_swig.py build_ext --inplace) but mention the interface file in the setup script. from distutils.core import setup, Extension approx_pi_module = Extension("_approx_pi", sources=["approx_pi.c", "approx_pi.i"]) setup(ext_modules=[approx_pi_module]]) This generates three files: approx_pi_swig.py, approx_pi_swig_wrap.c, and _approx_pi_swig*.so 21

22 Using f2py Fortran version: approx_pi.f90 subroutine approx_pi(intervals, pi) integer, intent(in) :: intervals double precision, intent(out) :: pi integer i pi = 0 do i = 0, intervals - 1 pi = pi + (4 - (mod(i,2) * 8)) / dble(2 * i + 1) enddo end subroutine approx_pi Compile using f2py3 -c -m approx_pi_f2py approx_pi.f90 Then do python3 pi_collect.py approx_pi_f2py

23 Cython 23

24 Cython Cython compiles from Python (with extensions) to C. Based on Pyrex Goals: faster execution (especially with those extensions) and easier interoperability with other C code. Cython files use the.pyx extension. 24

25 Cython Example: approx_pi_cython1.pyx (same as approx_pi.py) def approx_pi(intervals): pi = 0.0 for i in range(intervals): pi += (4-8 * (i % 2)) / (float)(2 * i + 1) return pi Executing python3 setup_cython.py build_ext --inplace from distutils.core import setup from Cython.Build import cythonize setup(ext_modules = cythonize("*.pyx")) turns all.pyx files into.c files and.so modules Run python3 pi_collect.py approx_pi_cython seconds: the C code uses only Python objects. 25

26 Cython: declare variables Need to declare variables using cdef to make it fast Example: approx_pi_cython2.pyx def approx_pi(int intervals): cdef double pi cdef int i pi = 0.0 for i in range(intervals): pi += (4-8 * (i % 2)) / (float)(2 * i + 1) return pi Execute python3 setup_cython.py build_ext --inplace Run python3 pi_collect.py approx_pi_cython seconds: almost as fast as native C. 26

27 Cython: division Inspecting approx_pi_cython2.c we found it uses Pyx_mod_long( pyx_v_i, 2) instead of a plain pyx_v_i % 2. This is because in C, -1%10=-1 but in Python, -1%10=9. Here we can ignore this and tell Cython to use C behaviour, by adding a line #cython:cdivision=true Execute python3 setup_cython.py build_ext --inplace Check that approx_pi_cython3.c uses %. Run python3 pi_collect.py approx_pi_cython seconds: the same as native C. Note: use Cython in IPython/Jupyter using %load_ext cythonmagic and %%cython in a cell. 27

28 Cython: wrapping C code Last but not least: interfacing with C code: # approx_pi_cython4.pyx cdef extern from "approx_pi.h": double c_approx_pi "approx_pi" (int intervals) # C name: approx_pi, Cython name: c_approx_pi def approx_pi(int intervals): return c_approx_pi(intervals) Plus special setup_cython4.py script from distutils.core import setup, Extension from Cython.Distutils import build_ext setup(cmdclass={'build_ext': build_ext}, ext_modules=[extension("approx_pi_cython4", sources=["approx_pi_cython4.pyx", "approx_pi.c"])]) Execute python3 setup_cython4.py build_ext Run python3 pi_collect.py approx_pi_cython4 --inplace

29 Parallel Programming Concepts 29

30 Vocabulary Serial tasks Any task that cannot be split in two simultaneous sequences of actions Examples: starting a process, reading a file, any communication between two processes Parallel tasks Data parallelism: same action applied on different data. Could be serial tasks done in parallel. Process parallelism: one action on one set of data. Action split in multiple processes or threads. Data partitioning: rectangles or blocks 30

31 Parallel tasks Parallel efficiency (scaling) Amdahl s law: how long does it take to compute a task with an infinite number of processors? Gustafson's law: what size of problem can we solve in a given time with N processors? Shared memory Multiple threads share the same memory space in a single process: full read and write access. Distributed memory Each process has its own memory space Information is sent and received by messages 31

32 Distributed Memory Model Process 1 Different variables! Network A(10) A(10) Process 2 32

33 Serial Code Parallelization Implicit Parallelization - minimum work for you Threaded libraries (MKL, ACML, GOTO, etc.) Compiler directives (OpenMP) Good for desktops and shared memory machines Explicit Parallelization - work is required! You tell what should be done on what CPU Solution for distributed clusters (shared nothing!) Hybrid Parallelization - work is required! Mix of implicit and explicit parallelization Vectorization and parallel CPU instructions Good for accelerators (CUDA, OpenCL, etc.) 33

34 The multiprocessing Module 34

35 The multiprocessing Module Because of the implementation of CPython, only one thread at a time can execute Python code This avoids common issues with the shared memory model: race condition,... There is a threading module, but it is no longer recommended Solution: the multiprocessing module! 35

36 Pool of Workers For embarrassingly parallel tasks, the Pool class allows the creation of worker processes. Each process will compute different data. Warning: only works in a script! from multiprocessing import Pool def prod(values): return values[0] * values[1] if name == ' main ': N = 12 values = [(i + 1, N - i) for i in range(0, N)] print(values) workers = Pool(processes=4) results = workers.map(prod, values) print(results) 36

37 Pool of Workers Run: python script.py What happens with 4 workers: 37

38 Pool of Workers Asynchronous map calls can be used in order to do something else in the main process. The map_async() method returns an AsyncResult object which can wait until all workers are done. from multiprocessing import Pool import time def prod(values): time.sleep(1) return values[0] * values[1] if name == ' main ': N = 12 values = [(i + 1, N - i) for i in range(0, N)] print(values) workers = Pool(processes=4) results = workers.map_async(prod, values) print('waiting...') print(results.get(timeout=10)) 38

39 Pool of Workers Asynchronous map calls can use a callback function. Then, the main thread has to wait by first closing the access to workers, and by joining the pool of workers. def printres(results): print(results) if name == ' main ': N = 12 values = [(i + 1, N - i) for i in range(0, N)] print(values) workers = Pool(processes=4) results = workers.map_async(prod, values, callback=printres) print('waiting...') workers.close() workers.join() 39

40 Pool of Workers class Pool([processes[,...]]) processes: number of worker processes. If None, processes=multiprocessing.cpu_count() Methods: map(func, iterable[,...]): returns results map_async(func, iterable[,...]): returns an AsyncResult object close(): closes access to worker processes join(): waiting for all workers to exit. Must call close() before. 40

41 Pool of Workers class AsyncResult Methods: get([timeout]): blocking, get results as soon as they are available. In case of error, get wait([timeout]): blocking, waits until the call is done ready(): non-blocking, returns a boolean indicating if the call has completed. successful(): non-blocking, returns a boolean indicating if the call has succeeded. 41

42 Exercise - Baby Genomic Edit baby-genomic.py Use a pool of 4 workers Use the asynchronous map function Provide a callback function that will print results at the end Tip: use the edproxy() function in order to call the real editdistance() function. Run: time -p python baby-genomic.py 42

43 The Process class The Process class: manually spawn and control each process Process(target=fct, args=(arg1,arg2)).start() Communication channels: The Pipe class: to communicate between two processes, one sends data, one receives data The Queue class: a shared pipe managed with locks and semaphores, one puts data, one gets data Synchronization: The Lock class: one acquires lock, one releases lock 43

44 MPI for Python (mpi4py) 44

45 MPI for Python The mpi4py package provides bindings from Python to MPI (Message Passing Interface). MPI functions are then available in Python but with some simplifications: MPI_Init() and MPI_Finalize() are done automatically The bindings can auto-detect many values that need to be specified as explicit parameters in the C and Fortran bindings. Example: dest = 1; tag = 54321; MPI_Send( &matrix, count, MPI_INT, dest, tag, MPI_COMM_WORLD ) becomes MPI.COMM_WORLD.Send(matrix, dest=1, tag=54321) 45

46 MPI for Python Import as from mpi4py import MPI Then often use comm = MPI.COMM_WORLD Two variations for most functions: a. all lowercase, e.g. comm.recv() works on general Python objects, using pickle (can be slow) received object (value) returned: matrix = comm.recv(source=0, tag=mpi.any_tag) b. capitalized, e.g. comm.recv() works fast on numpy arrays & other buffers received object given as parameter: comm.recv(matrix, source=0, tag=mpi.any_tag) Specify [matrix, MPI.INT], or [data, count, MPI.INT] if autodetection fails. 46

47 Conclusions Main techniques covered: Speeding up: PyPy, Numba, CTypes, Cython Parallel programming: multiprocessing, mpi4py Useful links: erfacing_with_c.html

48 Questions? Calcul Quebec support team: Specific site support teams: 48

Practical Introduction to Message-Passing Interface (MPI)

1 Practical Introduction to Message-Passing Interface (MPI) October 1st, 2015 By: Pier-Luc St-Onge Partners and Sponsors 2 Setup for the workshop 1. Get a user ID and password paper (provided in class):