Packaging Python code and Sphinx

Transcription

1 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine MCS 507 Lecture 18 Mathematical, Statistical and Scientific Software Jan Verschelde, 7 October 2013 Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

2 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

3 extending Python with your own package A package is a directory containing an init.py file; and also modules or other packages. The code in a Python package can be a pure Python module, defined in a single.py file; or a shared object.so file with compiled C code. The directory site-packages contains installed packages, such as numpy, scipy, sympy, etc. On Mac OS X, the directory site-packages can be found in the subdirectory lib/python2.7/site-packages of /Library/Frameworks/Python.framework/Versions/2.7/. On Red Hat Linux: /usr/lib/python2.6/site-packages. The Hitchiker s Guide to Packaging 1.0 documentation is at Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

4 preparing a package for distribution Distutils is a standard and basic package in the Python standard library, used for creating distributions, imported in the setup.py file. Following the Hitchhiker s Guide to Packaging: 1 Define the directory structure: Make a directory with LICENSE.txt, MANIFEST.in, README.txt, setup.py. The name for this directory could be the package name, followed by the version number, e.g.: ourfirstpackage In this directory is a subdirectory with the package name. The directory with the package name contains the init.py and the module with the code. 2 python setup.py sdist makes a file MANIFEST and a directory dist with a gzipped tar file. 3 As superuser we install with python setup.py install. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

5 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

6 making ourfirstpackage We use ourfirstmodule.py of Lecture 7. 1 New directory: mkdir ourfirstpackage In ourfirstpackage-0.0.1, do mkdir ourfirstpackage and copy ourfirstmodule.py into ourfirstpackage. 3 As LICENSE.txt, we copy the text of GNU GPL version 3. 4 The content of README.txt (start documenting in rest): =========================== Welcome to ourfirstpackage! =========================== It contains ourfirstmodule of lecture 7 of MCS 507. ourfirstmodule This module exports x**2*cos(y) + 4*exp(z)*sin(x). Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

7 the file MANIFEST and init.py The contents of the file MANIFEST: include *.txt So all.txt files are included in the distribution. The contents of the file init.py: """ ourfirstpackage =============== Exports the function x**2*cos(y) + 4*exp(z)*sin(x) in the module ourfirstmodule. """ # Definition of the version number version = Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

8 the file setup.py from distutils.core import setup from distutils.sysconfig import get_python_lib setup( name = ourfirstpackage, author = MCS 507, author_ = jan@math.uic.edu, description = our first package, url = version = 0.0.1, packages = [ ourfirstpackage ], py_modules = [ ourfirstpackage/ourfirstmodule ], license = GNU GENERAL PUBLIC LICENSE version 3, platforms = [ linux2 ], long_description=open( README.txt ).read() ) Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

9 preparing the distribution $ python setup.py sdist running sdist running check reading manifest template MANIFEST.in writing manifest file MANIFEST creating ourfirstpackage creating ourfirstpackage-0.0.1/ourfirstpackage making hard links in ourfirstpackage hard linking LICENSE.txt -> ourfirstpackage hard linking README.txt -> ourfirstpackage hard linking setup.py -> ourfirstpackage hard linking ourfirstpackage/ init.py -> ourfirstpackage hard linking ourfirstpackage/ourfirstmodule.py -> ourfirstp creating dist Creating tar archive removing ourfirstpackage (and everything under it) $ ls dist ourfirstpackage tar.gz $ Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

10 installing the package $ sudo python setup.py install running install running build running build_py creating build creating build/lib creating build/lib/ourfirstpackage copying ourfirstpackage/ourfirstmodule.py -> build/lib/\ ourfirstpackage copying ourfirstpackage/ init.py -> build/lib/\ ourfirstpackage running install_lib running install_egg_info Removing /Library/Frameworks/Python.framework/Versions/\ 2.7/lib/python2.7/site-packages/ourfirstpackage py2.7 Writing /Library/Frameworks/Python.framework/Versions/2.7/\ lib/python2.7/site-packages/\ ourfirstpackage py2.7.egg-info $ Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

11 using ourfirstpackage $ python Python (v2.7.3:70274d53c1dd, Apr , 20:52:43) [GCC (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more i >>> import ourfirstpackage >>> help(ourfirstpackage) >>> from ourfirstpackage import ourfirstmodule >>> help(ourfirstmodule) >>> from ourfirstpackage.ourfirstmodule import main >>> main() v = x**2*cos(y) + 4*exp(z)*sin(x) give x : 1 give y : 2 give z : 3 v = >>> Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

12 help on ourfirstpackage Help on package ourfirstpackage: NAME ourfirstpackage FILE /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-pa DESCRIPTION ourfirstpackage =============== Exports the function x**2*cos(y) + 4*exp(z)*sin(x) in the module ourfirstmodule. PACKAGE CONTENTS ourfirstmodule DATA version = VERSION Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

13 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

14 PHCpack PHCpack is a software package to solve polynomial systems with polyhedral and numerical methods. PHCpack consists of 1 open source code in Ada with interfaces to C and Python, compiles with gcc, available as a software package; 2 an executable program phc for various platforms with interfaces for Maple, MATLAB (Octave), Macaulay 2, and Sage (PHCpack is one of the optional packages). ACM TOMS archived version 1.0 of PHCpack as Algorithm 795. The current version is , available at: jan/download.html and at Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

15 symbolic-numeric view on polynomial homotopies Our goal is to solve a polynomial system f(x) = 0. A polynomial homotopy continuation method consists in two parts: 1 definition of a family of systems, a homotopy, e.g.: h(x, t) = (1 t) g(x) + t f(x) = 0, t [0, 1], where g(x) = 0 is a generic system with known solutions; 2 path tracking or continuation methods using predictor-corrector to approximate solution paths x(t) defined by h(x(t), t) = 0. A homotopy is optimal if no paths diverge. We have optimal homotopies for several classes of systems, e.g.: polyhedral homotopies for sparse systems with fixed supports. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

16 software integrated in PHCpack To compute mixed volumes fast: T. Gao, T.Y. Li, and M. Wu: Algorithm 846: MixedVol: A software package for mixed volume computation. ACM Trans. Math. Softw. 31(4): , For fast extended multiprecision arithmetic: Y. Hida, X.S. Li, and D.H. Bailey: Algorithms for quad-double precision floating point arithmetic. In 15th IEEE Symposium on Computer Arithmetic pages IEEE, Software at dhbailey/mpdist/qd tar.gz. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

17 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

18 Newton polytopes The support A of a polynomial f is a finite set of exponent vectors which models the sparse structure of f : f(x) = a A c a x a, c a C, x a = x a 1 1 x a 2 2 x an n. The convex hull of A is the Newton polytope of f, denoted by Q = conv(a). Mostly we work in the plane (n = 2) and stick to polygons. Today we consider systems f(x) = 0 where the equations all share the same support A, or the same Newton polytope Q. For generic choices of the coefficients, the monomials whose exponent vector is not a vertex do not have an influence on the volume of Q and may be omitted. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

19 regular triangulations With a triangulation, we compute the volume: vol(q) = S vol(s) S = conv(c), #C = n + 1. The cells C in a triangulation span simplices S. For Newton polytopes, the unit simplex has volume 1. Our triangulations are induced by a lifting function ω: ω : Z n Z : a ω(a), which embeds the support A into Z n+1, Â = ω(a). Accordingly: Q = conv(â). A triangulation is regular if there is a lifting function so there is a 1-to-1 mapping of the facets of the lower hull of the lifted point configuration and the simplices in the triangulation. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

20 polyhedral homotopy To construct the homotopy that starts at system supported at the other cell of the triangulation, we look at the inner normal of the lifted cell, i.e.: v = ( 1, 1,+1). This inner normal defines the change of coordinates x 1 = y 1 t 1 and x 2 = y 2 t 1. After multiplication by t, this change of coordinates yields { c1,11 y ĝ(y, t) = 1 y 2 t 0 + c 1,10 y 1 t 0 + c 1,01 y 2 t 0 + c 1,00 t 1 = 0 c 2,11 y 1 y 2 t 0 + c 2,10 y 1 t 0 + c 2,01 y 2 t 0 + c 2,00 t 1 = 0. At ĝ(y, t = 0) = 0 there is another solution which contributes one path. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

21 Kushnirenko s theorem Theorem (Kushnirenko (1976)) Consider the system f(x) = 0 and let A be the support of every polynomial in f. Then the number of isolated solutions of f(x) = 0 in (C ) n cannot exceed the volume of the polytope spanned by A. Polyhedral homotopies provide a constructive proof: 1 Any regular triangulation defines a polyhedral homotopy with exactly as many paths as the volume. 2 There are no more solutions than the volume of the Newton polytope, following a refined concept of. Kushnirenko s theorem was generalized by Bernshteǐn who showed that the mixed volume of Newton polytopes of the system provide a generically share upper bound for the isolated solutions. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

22 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

23 a session with phcpy >>> import phcpy >>> p = [ x*y + 2*x - 3;, x**2 + y; ] >>> phcpy.mixed_volume(p) 3 >>> s = phcpy.solve(p) ROOT COUNTS : total degree : 4 2-homogeneous Bezout number : 3 with partition : {x }{y } general linear-product Bezout number : 3 based on the set structure : { x }{ y } { x y }{ x } mixed volume : 3 stable mixed volume : 3 Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

24 the solutions >>> print s[0] t : E E+00 m : 1 the solution for t : x : E E-01 y : E E+00 == err : 4.154E-16 = rco : 1.496E-01 = res : 1.665E-16 = >>> print s[1] t : E E+00 m : 1 the solution for t : x : E E-122 y : E E+00 == err : 2.428E-16 = rco : 2.059E-01 = res : 2.220E-16 = >>> print s[2] s[2] is the complex conjugate of s[1]. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

25 representations of polynomials and solutions Polynomials are mapped to file similar to plain string conversion in computer algebra and can be parsed in a direct manner. Symbols not to use as variable names: i, I, e, E. A solution is represented by 1 a value for the continuation parameter t; 2 a multiplicity flag; 3 real and imaginary parts as values for the variables; 4 three numerical attributes for an approximate zero z: 1 err z, or estimate for forward error; 2 rco: estimate for the inverse of the condition number; 3 res: f(z), or estimate for backward error. Newton s method with extended precision refines a regular solution. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

26 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

27 Sphinx generates documentation What is Sphinx? From Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the new Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well. Sphinx uses restructuredtext as its markup language, and many of its strengths come from the power and straightforwardness of restructuredtext and its parsing and translating suite, the Docutils. The same documentation is generated in several formats, including latex, pdf, html. Why Sphinx? Used for the python language, numpy, sympy, matplotlib. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

28 step-by-step documentation generation Given Python code with documentation strings, generate documentation with Sphinx. There are a few steps to take: 1 Make a new directory doc and do "cd doc". 2 Run sphinx-quickstart in the doc directory. Following default options, doc contains a "Makefile" and two directories: "build" and "source". 3 Edit conf.py in source as follows: sys.path.insert(0, os.path.abspath( <path> )) where <path> is the directory where our Python code is. 4 Edit index.rst with ".. automodule::" lines. 5 Type "make latexpdf" or "make html" to generate. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

29 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

30 running sphinx-quickstart The doc directory is in PHCv2/Python. Running sphinx-quickstart, some items: The root path is the current directory [.] if we run sphinx-quickstart in the doc directory. We want separate source and build directories. We can enter project name, author, version, and release numbers. Source files have suffix.rst. The master document is "index". We answer y to "autodoc" because we want automatically insert docstrings from modules. We answer y to "viewcode" to include links to the source code of documented Python objects. We also answer y to "Create Makefile?" Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

31 editing conf.py The source directory contains a file "conf.py" which we have to edit. On Mac OS X: sys.path.insert(0, \ os.path.abspath( /Users/jan/PHCv2/Python/PHCpy/phcpy )) On Linux: sys.path.insert(0, \ os.path.abspath( /home/jan/phcv2/python/phcpy/phcpy )) Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

32 editing index.rst Downloading PHCpack The source for PHCpack can be downloaded from < The Python interface phcpy to PHCpack is a programmer s interface. You will need to compile the source code of PHCpack with the GNU Ada compiler, available at < Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

33 use of automodule in index.rst the functions of phcpy The module phcpy depends on the shared object file phcpy2c.so. It exports the blackbox solver of PHCpack, a fast mixed volume calculator, and a path tracker. The test function of the module generates two trinomials (a polynomial with three monomials) with randomly generated complex coefficients... automodule:: phcpy :members: Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

34 documenting examples some important examples PHCpack has been tested on many examples of polynomial systems taken from the research literature. The module examples exports some of those examples... automodule:: examples :members: Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

35 documenting phcwulf the module phcwulf The file phcwulf defines a simple client/server interaction to solve many random trinomials... automodule:: phcwulf :members: Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

36 documenting phcsols the module phcsols Solutions of phcpy.solve are returned as lists of PHCpack solution strings. The phcsols module contains functions to parse a PHCpack solution string into a dictionary... automodule:: phcsols :members: Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

37 Packaging Python code and Sphinx 1 Python packages extending Python with your own package making ourfirstpackage 2 the software PHCpack a package for Polynomial Homotopy Continuation polyhedral homotopies the Python interface phcpy 3 Documenting Software with Sphinx Sphinx generates documentation generating documentation for phcpy 4 Interface Design PHCpack as a state machine Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

38 PHCpack as a state machine phcpy is build on top of the C interface to PHCpack. The C interface was developed for the parallel versions of the path trackers in PHCpack. The main program, written in C, does the scheduling of the path tracking jobs, using the Message Passing Interface (MPI). Every job makes calls to the path trackers of PHCpack, the code written in Ada. The C interface should remain light, without having to duplicate the data structures for polynomials and solutions. Idea: we can build up a polynomial adding one term at a time, with calls to an interface routine. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

39 code of phcpy.solver.solve def solve(pols, silent=false): """ Calls the blackbox solver of PHCpack. On input in pols is a list of strings. By default, the solver will print to screen the computed root counts. To make the solver silent, set the flag silent to True. """ from phcpy.phcpy2c \ import py2c_syscon_clear_laurent_system from phcpy.phcpy2c \ import py2c_syscon_initialize_number_of_laurentials from phcpy.phcpy2c import py2c_syscon_store_laurential from phcpy.phcpy2c import py2c_solcon_clear_solutions from phcpy.phcpy2c import py2c_solve_laurent_system py2c_syscon_clear_laurent_system() py2c_solcon_clear_solutions() Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

40 code of phcpy.solver.solve continued dim = len(pols) py2c_syscon_initialize_number_of_laurentials(dim) for ind in range(0, dim): pol = pols[ind] nchar = len(pol) py2c_syscon_store_laurential(nchar, dim, ind+1, \ pol) py2c_solve_laurent_system(silent) return load_solutions() Advantage: the Python code deals only with strings. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41

41 Summary and Exercises We do not release source code without documentation. Sphinx generates documention automatically. Exercises: Use Sphinx in your computer projects! Take the Python scripts you used in your first computer project and make a Python package. Document your Python package with restructured text and use Sphinx to generate documentation in html and latex-pdf formats. Scientific Software (MCS 507 L-18) Packaging Python code and Sphinx 7 October / 41