An introduction to scientific programming with. Session 5: Extreme Python

Size: px

Start display at page:

Download "An introduction to scientific programming with. Session 5: Extreme Python"

Cecil Hensley
6 years ago
Views:

1 An introduction to scientific programming with Session 5: Extreme Python

2 Managing your environment Efficiently handling large datasets Optimising your code Squeezing out extra speed Writing robust code Graphical interfaces

3 Managing your environment

4 Some good things about Python lots of modules from many sources ongoing development of Python and modules Some bad things about Python lots of modules from many sources ongoing development of Python and modules A solution Maintain (or have option to create) separate environments (or manifests) for different projects

5 virtualenv general Python solution modules are installed with pip $ pip install virtualenv # install virtualenv $ virtualenv ENV1 # create a new environment ENV1 $ source ENV/bin/activate # set PATH to our environment (ENV1)$ pip install emcee # install modules into ENV1 (ENV1)$ pip install numpy==1.8.2 # install specific version (ENV1)$ python # use our custom environment (ENV1)$ deactivate # return our PATH to normal

6 virtualenv can record current state of modules to a 'requirements' file using that file can always recreate the same environment (ENV1)$ pip freeze > requirements.txt $ cat requirements.txt emcee==2.1.0 numpy==1.8.2 $ deactivate $ virtualenv ENV2 $ source ENV2/bin/activate (ENV2)$ pip install -r requirements.txt

7 conda specific to the Anaconda Python distribution similar to 'pip', but can install binaries (not just python) avoid using pip within conda environment (although possible) $ conda create -n ENV1 # create a new environment ENV1 $ source activate ENV1 # set PATH to our environment $ conda install numpy # install modules into ENV1 $ conda install -c thebamf emcee # install from binstar $ source deactivate # return our PATH to normal $ conda list n ENV1 -e > requirements.txt # clone ENV1 $ conda create -n ENV2 --file requirements.txt # to ENV2

8 Updating packages $ conda update --all $ conda update scipy emcee OR $ pip install --upgrade $ pip install --upgrade scipy emcee

9 Efficiently handling large datasets

10 Python has tools for accessing most (all?) databases e.g. MySQL, SQLite, MongoDB, Postgres, Allow one to work with huge datasets Data can be at remote locations However, most databases are designed for webserver use Not optimised for data analysis

11 For creating, storing and analysing datasets from simple, small tables to complex, huge datasets standard HDF5 file format incredibly fast even faster with indexing uses on the fly block compression designed for modern systems fast multi-code CPU; large, slow memory "in-kernel" data and algorithm are sent to CPU in optimal way "out-of-core" avoids loading whole dataset into memory

12 Can store many things in one HDF5 file (like FITS) Tree structure Everything in a group (starting with root group, '/') Data stored in leaves Arrays (e.g. n-dimensional images) >>> from tables import * >>> h5file = openfile("test.h5", mode = "w") >>> x = h5file.createarray("/", "x", arange(1000)) >>> y = h5file.createarray("/", "y", sqrt(arange(1000))) >>> h5file.close()

13 Tables (columns with different formats) described by a class accessed by a row iterator >>> class MyTable(IsDescription): z = Float32Col() >>> table = h5file.createtable("/", "mytable", MyTable) >>> row = table.row >>> for i in xrange(1000): row["z"] = i**(3.0/2.0) row.append() >>> table.flush() >>> z = table.cols.z

14 Expr enables in-kernel & out-of-core operations >>> r = h5file.createarray("/", "r", np.zeros(1000)) >>> xyz = Expr("x*y*z") >>> xyz.setoutput(r) >>> xyz.eval() /r (Array(1000,)) '' atom := Float64Atom(shape=(), dflt=0.0) maindim := 0 flavor := 'numpy' byteorder := 'little' chunkshape := None >>> r.read(0, 10) array([ 0., 1., , , 64., , , , , 729. ])

15 where enables in-kernel selections >>> r_bigish = [ row['z'] for row in table.where('(z > 1000) & (z <= 2000)' ] >>> for big in table.where('z > 10000;'):... print('a big z is {}'.format(big['z']) There is also a where in Expr

Python Data Analysis Library http://pandas.pydata.

16 Python Data Analysis Library Easy-to-use data structures DataFrame (more friendly recarray) Handles missing data (more friendly masked array) read and write various data formats data-alignment tries to be helpful, though not always intuitive Easy to combine data tables Surprisingly fast!

17 1D Series, 2D DataFrame, 3D Panel Various ways of creating these structures >>> import pandas as pd >>> t = np.arange(0.0, 1.0, 0.1) >>> s = pd.series(t) >>> x = t + np.random.normal(scale=0.1, size=t.shape) >>> y = x**2 + np.random.normal(scale=0.5, size=t.shape) >>> df1 = pd.dataframe({ 'x' : x, 'y' : y}, index = t) >>> df1.plot() >>> df1.plot('x', 'y', kind='scatter')

18 Always indexed Indices used when joining tables >>> t2 = t[::2] >>> z = t2**3 + np.random.normal(scale=1.0, size=t2.shape) >>> df2 = pd.dataframe({ 'z' : z, 'z2' : z**2}, index = t2) >>> df3 = df1.join(df2) >>> df3.sort('y') Handling data fairly intuitive, similar to numpy slices Powerful and flexible Good documentation

19 DataFrame can be created directly from a recarray Pandas data structures can be efficiently stored on disk based on PyTables data = pyfits.getdata('myfunkydata.fits') df = pd.dataframe(data) # store DataFrame to HDF5 store = pd.hdfstore('myfunkydata.h5', mode='w', complib='blosc', complevel=5) store['funkydata'] = df # some time later store = pd.hdfstore('myfunkydata.h5', mode='r') df = store['funkydata']

20 Graphical interfaces

21 Several modules to construct GUIs in Python wxpython is one of the most popular E.g.,

22 Django a high-level Python Web framework that encourages rapid development and clean, pragmatic design. and many others, e.g. Zope (massive), web2py (light), Give your scientific code a friendly face! easy configuration monitor progress particularly for public code, cloud computing, HPC An (unsicentific) example, followed by another one

23 Optimising your code

24 timeit use in interpreter, script or command line python -m timeit [-n N] [-r N] [-s S] [statement...] Options: -s S, --setup=s statement to be executed once initially (default pass) -n N, --number=n how many times to execute 'statement' (default: take ~0.2 sec total) -r N, --repeat=n how many times to repeat the timer (default 3) ipython magic version %timeit # one line %%timeit # whole notebook cell

25 # fastest way to calculate x**5? $ python -m timeit -s 'from math import pow; x = 1.23' 'x*x*x*x*x' loops, best of 3: usec per loop $ python -m timeit -s 'from math import pow; x = 1.23' 'x**5' loops, best of 3: usec per loop $ python -m timeit -s 'from math import pow; x = 1.23' 'pow(x, 5)' loops, best of 3: usec per loop

26 Understand which parts of your code limit its execution time print summary to screen, or save file for detailed analysis From shell $ python -m cprofile o program.prof my_program.py From ipython %prun -D program.prof my_function() %%prun # profile an entire notebook cell Lots of functionality see docs

Nice visualisation with snakeviz http://jiffyclub.github.

27 Nice visualisation with snakeviz $ conda install c thebamf snakeviz OR $ pip install snakeviz In ipython: %load_ext snakeviz %snakeviz my_function() %%snakeviz # profile entire cell

28 Writing robust code

29 Several testing frameworks unittest is the main Python module nose is a third-party module that nicely automates testing

30 Squeezing out extra speed

31 Python includes modules for writing "parallel" programs: threaded limited by the Global Interpreter Lock multiprocessing generally more useful from multiprocessing import Pool def f(x): return x*x pool = Pool(processes=4) # start 4 worker processes z = range(10) print pool.map(f, z) # apply f to each element of z in parallel

32 from multiprocessing import Process from time import sleep def f(name): print('hello {}, I am going to sleep now'.format(name)) sleep(3) print('ok, finished sleeping') if name == ' main ': p = Process(target=f, args=(lock, 'Steven')) p.start() # start additional process sleep(1) # carry on doing stuff print 'Wow, how lazy is that function!' p.join() # wait for process to complete $ python thinking.py Hello Steven, I am going to sleep now Wow, how lazy is that function! OK, finished sleeping (Really, should use a lock to avoid writing output to screen at same time)

33 Cython is used for compiling Python-like code to machine-code supports a big subset of the Python language conditions and loops run 2-8x faster, overall 30% faster for plain Python code add types for speedups (hundreds of times) easily use native libraries (C/C++/Fortran) directly Cython code is turned into C code uses the CPython API and runtime Coding in Cython is like coding in Python and C at the same time! Some material borrowed from Dag Sverre Seljebotn (University of Oslo) EuroSciPy 2010 presentation

34 Use cases: Performance-critical code which does not translate to array-based approach (numpy / pytables) existing Python code! optimise critical parts Wrapping existing C/C++ libraries particularly higher-level Pythonised wrapper for one-to-one wrapping other tools might be better suited

35 Cython code must be compiled. Two stages: A.pyx file is compiled by Cython to a.c file, containing the code of a Python extension module The.c file is compiled by a C compiler Generated C code can be built without Cython installed Cython is a developer dependency, not a build-time dependency Generated C code works with Python 2.3+ The result is a.so file (or.pyd on Windows) which can be imported directly into a Python session

36 Ways of building Cython code: Run cython command-line utility and compile the resulting C file use favourite build tool for cross-system operation you need to query Python for the C build options to use Use pyximport to importing Cython.pyx files as if they were.py files; building on the fly (recommended to start). things get complicated if you must link to native libraries larger projects tend to need a build phase anyway Write a distutils setup.py standard way of distributing, building and installing Python modules

37 Cython supports most of normal Python Most standard Python code can be used directly with Cython typical speedups of (very roughly) a factor of two should not ever slow down code safe to try name file.pyx or use pyimport = True >>> import pyximport >>> pyximport.install() >>> import mypyxmodule # converts and compiles on the fly >>> pyximport.install(pyimport = True) >>> import mypymodule # converts and compiles on the fly # should fall back to Python if fails

38 Big speedup from defining types of key variables Use native C-types (int, double, char *, etc.) Use Python C-types (Py_int_t, Py_float_t, etc.) Use cdef to declare variable types Also use cdef to declare C-only functions (with return type) can also use cpdef to declare functions which are automatically treated as C or Python depending on usage Don't forget function arguments (but note cdef not used here)

39 Efficient algorithm to find first N prime numbers def primes(kmax): p = [] k = 0 n = 2 while k < kmax: i = 0 while i < k and n % p[i]!= 0: i = i + 1 if i == k: k = k + 1 p.append(n) n = n + 1 return p primes.py $ python -m timeit -s 'import primes as p' 'p.primes(100)' 1000 loops, best of 3: 1.35 msec per loop

40 def primes(kmax): p = [] k = 0 n = 2 while k < kmax: i = 0 while i < k and n % p[i]!= 0: i = i + 1 if i == k: k = k + 1 p.append(n) n = n + 1 return p cprimes.pyx $ python -m timeit -s 'import pyximport; pyximport.install(); import cprimes as p' 'p.primes(100)' 1000 loops, best of 3: 731 usec per loop 1.8x speedup

41 xprimes.pyx def primes(int kmax): # declare types of parameters cdef int n, k, i # declare types of variables cdef int p[1000] # including arrays result = [] # can still use normal Python types if kmax > 1000: # in this case need to hardcode limit kmax = 1000 k = 0 n = 2 while k < kmax: i = 0 while i < k and n % p[i]!= 0: contains only C-code i = i + 1 if i == k: p[k] = n 40.8 usec per loop k = k + 1 result.append(n) 33x speedup n = n + 1 return result # return Python object

for implementing fast specific algorithms Also provides way to create

42 Cython provides a way to quickly access Numpy arrays with specified types and dimensionality! for implementing fast specific algorithms Also provides way to create generalized Ufuncs Can be useful, but often using functions provided by numpy, scipy, numexpr or pytables will be easier and faster

0 for i in range(m): for j in range(n): result += arr[i,j] return result a = np.arange(10000).

43 JIT: just in time compilation of Python functions Compilation for both CPU and GPU hardware from numba import def sum2d(arr): M, N = arr.shape result = 0.0 for i in range(m): for j in range(n): result += arr[i,j] return result a = np.arange(10000).reshape(1000,10) %timeit sum2d(a) 1000 loops, best of 3: 334 µs per loop %timeit sum2d(a) # 1 loops, best of 3: 2.15 µs per loop

44 An introduction to scientific programming with The End

An introduction to scientific programming with. Session 5: Extreme Python

An introduction to scientific programming with Session 5: Extreme Python Outline Managing your environment Efficiently handling large datasets Optimising your code Squeezing out extra speed Writing neat