Python on GACRC Computing Resources

Size: px

Start display at page:

Download "Python on GACRC Computing Resources"

Samuel Taylor
5 years ago
Views:

1 Python on GACRC Computing Resources Georgia Advanced Computing Resource Center EITS/University of Georgia Zhuofei Hou, 1

2 Outline GACRC Python Overview Python on Clusters Python Packages on Clusters Run Python Interactively on Clusters Run Python Batch Job on Clusters 2

3 GACRC Who Are We? Georgia Advanced Computing Resource Center Collaboration between the Office of Vice President for Research (OVPR) and the Office of the Vice President for Information Technology (OVPIT) Guided by a faculty advisory committee (GACRC-AC) Why Are We Here? To provide computing hardware and network infrastructure in support of highperformance computing (HPC) at UGA Where Are We? (Web) (Web Help) (Wiki Help) (Wiki) 3

4 Python Overview Language Open source general-purpose scripting language ( Working with procedural, object-oriented, and functional programming Glue language with Interfaces to C/C++ (via SWIG), Object-C (via PyObjC), Java (Jython), and Fortran (via F2PY), etc. ( Mainstream version is 2.7.x; new version is 3.5.x (as to March 2016) 4

5 Python Overview Modules Python has a large collection of built-in modules included in standard distributions: Many third-party packages for scientific modules: NumPy Sympy SciPy Biopy Matplotlib 5

6 Python Overview Scientific Modules NumPy: Matlab-ish capabilities, fast N-D array operations, linear algebra, etc. ( SciPy: Fundamental library for scientific computing ( Sympy: Symbolic mathematics ( matplotlib: High quality plotting ( Biopy: Phylogenetic exploration ( A scientific Python distribution may include all those packages for you! 6

7 Python Overview Scientific Distributions Anaconda A Python distribution including ~200 of the most popular Python packages for science, math, engineering, and data analysis. Supports Linux, Mac and Windows ( Python(x,y) Windows only ( WinPython Windows only ( 7

8 Python on Clusters Python Anaconda Python 8

9 Python on zcluster Version Installation Path Invoke Command (default) /usr/bin python 2.7.2* /usr/local/python/2.7.2 python /usr/local/python/2.7.8 python /usr/local/python/3.3.0 python /usr/local/python/3.4.0 python3.4 * Most Python site-packages GACRC installed are for the version of on zcluster 9

10 Python on Sapelo Version Installation Path Module Load Invoke Command (default) /usr/bin /usr/local/apps/python/2.7.8 module load python/2.7.8 python /usr/local/apps/python/3.4.3 module load python/3.4.3 python3 10

11 Anaconda Python on zcluster Version Installation Path Python Version Exports Invoke Command /usr/local/anaconda/ export PATH=/usr/local/anaconda/2.3.0/bin:$PATH python /usr/local/anaconda/ export PATH=/usr/local/anaconda/ /bin:$PATH 11

12 Anaconda Python on Sapelo Version Installation Path Python Version Module Load Invoke Command /usr/local/apps/anaconda/ module load anaconda/ /usr/local/apps/anaconda/ module load anaconda/2.5.0 python /usr/local/apps/anaconda/ module load anaconda/

13 Python Packages on Clusters Python Packages Anaconda Python Packages

14 Python Packages on Clusters How to know if the package you need is already installed on clusters? 1. python c import pkgname; print pkgname version 2. conda list pkgname 3. python -m pip list grep pkgname Examples: Next page! 14

15 Python Packages on Clusters 1. python2.7 c import numpy; print numpy. version zcluster 2. python2.7.8 m pip list grep numpy 3. export PATH=/usr/local/anaconda/2.3.0/bin:$PATH conda list numpy 1. module load python/2.7.8 Sapelo python c import numpy; print numpy. version 2. python -m pip list grep numpy 3. module load anaconda/ conda list numpy 15

16 Common Python Packages on zcluster Package python2.7 (python2.7.2) python2.7.8 (python2.7.8) python3.4 (python3.4.0) Anaconda2.3.0 (python2.7.11) Anaconda (python3.4.3) Numpy Scipy n/a Biopython n/a Matplotlib Cython Pandas n/a n/a Scikit-image n/a n/a Scikit-learn n/a Networkx 2.0.dev Requests n/a

17 Common Python Packages on Sapelo Package python (python2.7.8) python3 (python3.4.3) Anaconda2.5.0 (python2.7.11) Anaconda (python3.4.3) Numpy Scipy n/a Biopython 1.66 n/a Matplotlib Cython Pandas Scikit-image n/a n/a Scikit-learn n/a Networkx n/a n/a Requests n/a

18 Python Package Paths on zcluster Version Python Package Path Python Shared Library Path /usr/local/python/2.7.2/lib/python2.7 /usr/local/python/2.7.2/lib/python2.7/site-packages /usr/local/python/2.7.8/lib/python2.7 /usr/local/python/2.7.8/lib/python2.7/site-packages /usr/local/python/3.3.0/lib/python3.3 /usr/local/python/3.3.0/lib/python3.3/site-packages /usr/local/python/3.4.0/lib/python3.4 /usr/local/python/3.4.0/lib/python3.4/site-packages N/A /usr/local/python/2.7.8/lib N/A N/A python can find those packages automatically! To be exported in LD_LIBRARY_PATH 18

19 Python Package Paths on Sapelo Version Python Package Path Python Shared Library Path /usr/local/apps/python/2.7.8/lib/python2.7 /usr/local/apps/python/2.7.8/lib/python2.7/site-packages /usr/local/apps/python/2.7.8/lib /usr/local/apps/python/3.4.3/lib/python2.7 /usr/local/apps/python/3.4.3/lib/python3.4/site-package /usr/local/apps/python/3.4.3/lib python can find those packages automatically! To be exported in LD_LIBRARY_PATH 19

20 Anaconda Python Package Paths on zcluster Version Installation Path Python Version Python Package Path /usr/local/anaconda/ /usr/local/anaconda/2.3.0/lib/python2.7 /usr/local/anaconda/2.3.0/lib/python2.7/sitepackages /usr/local/anaconda/ /usr/local/anaconda/ /lib/python3.4 /usr/local/anaconda/ /lib/python3.4/sitepackages Python Shared Libraries were built for all versions in lib python can find those packages automatically! 20

21 Anaconda Python Package Paths on Sapelo Version Installation Path Python Version Python Package Path /usr/local/apps/anaconda/ /usr/local/apps/anaconda/ /usr/local/apps/anaconda/ /usr/local/apps/anaconda/2.2.0/lib/python2.7 /usr/local/apps/anaconda/2.2.0/lib/python2.7/sit e-packages /usr/local/apps/anaconda/2.5.0/lib/python2.7 /usr/local/apps/anaconda/2.5.0/lib/python2.7/sit e-packages /usr/local/apps/anaconda/ /lib/python3.4 /usr/local/apps/anaconda/ /lib/python3.4/ site-packages Python Shared Libraries were built for all versions in lib python can find those packages automatically! 21

22 Run Python Interactively on Clusters Run Python Interactively Run Anaconda Python Interactively DO NOT run jobs from login node; Run interactive tasks from interactive node: zcluster.rcc.uga.edu (zcluster) sapelo1.gacrc.uga.edu (Sapelo) qlogin interactive node - Download 22

23 Run Python Interactively on Clusters Running on zcluster interactive node: python Python (#1, Oct , 22:02:41) [GCC (Red Hat )] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> a = 7 >>> e = 2 >>> a**e 49 >>> Running on Sapelo interactive node: [zhuofei@n15 ~]$ python Python (r266:84292, Jan , 09:42:36) [GCC (Red Hat )] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> a = 7 >>> e = 2 >>> a**e 49 >>> 23

24 Run Python Interactively on Clusters /usr/local/python/2.7.8/bin/python myscript.py (default, Jan , 15:33:35) [GCC (Red Hat )] 49 [zhuofei@n15 ~]$ module load python/2.7.8 [zhuofei@n15 ~]$ python myscript.py (default, Sep , 07:26:46) [GCC (Red Hat )] 49 import sys print sys.version a = 7 e = 2 print a**e 24

25 Run Python Interactively on Clusters chmod u+x myscript.py (default, Jan , 15:33:35) [GCC (Red Hat )] 49 [zhuofei@n15 ~]$ chmod u+x myscript.py [zhuofei@n15 ~]$./myscript.py (default, Sep , 07:26:46) [GCC (Red Hat )] 49 #!/usr/local/python/2.7.8/bin/python #!/usr/local/apps/python/2.7.8/bin/python2.7 import sys print sys.version a = 7; e = 2 print a**e tells exec where python is on zcluster import sys print sys.version a = 7; e = 2 print a**e tells exec where python is on Sapelo 25

26 Run Python Interactively on Clusters chmod u+x myscript.py export PATH=/usr/local/python/2.7.8/bin:$PATH (default, Jan , 15:33:35) [GCC (Red Hat )] 49 [zhuofei@n15 ~]$ chmod u+x myscript.py [zhuofei@n15 ~]$ module load python/2.7.8 [zhuofei@n15 ~]$./myscript.py (default, Sep , 07:26:46) [GCC (Red Hat )] 49 #!/usr/bin/env python import sys print sys.version a = 7; e = 2 print a**e env tells exec where python is by searching PATH 26

27 Run Anaconda Python Interactively on Clusters chmod u+x myscript.py export PATH=/usr/local/anaconda/2.3.0/bin:$PATH Anaconda (64-bit) (default, Dec , 18:08:32) [GCC (Red Hat )] 49 [zhuofei@n15 ~]$ chmod u+x myscript.py [zhuofei@n15 ~]$ module load anaconda/2.5.0 [zhuofei@n15 ~]$./myscript.py Anaconda (64-bit) (default, Dec , 18:08:32) [GCC (Red Hat )] 49 #!/usr/bin/env python import sys print sys.version a = 7; e = 2 print a**e 27

28 Run Python Batch Job on Clusters Run Python Batch Job on zcluster Run Python Bach Job on Sapelo Note: zcluster Job Working Space: /escratch4/username/ Sapelo Job Working Space: /lustre1/username/ - Download 28

29 Run Python Batch Job on zcluster #!/bin/bash cd working_directory export PATH=/usr/local/python/2.7.8/bin:$PATH export PYTHONPATH=/usr/local/python/2.7.8/lib/python2.7:/usr/local/python/2.7.8/lib/python2.7/site-\ packages:$pythonpath time python myscript.py [options] qsub -q rcc-30d sub.sh optional qsub options, e.g., -pe threads 4 -l mem_total=20g 29

30 Run Anaconda Python Batch Job on zcluster #!/bin/bash cd working_directory export PATH=/usr/local/anaconda/2.3.0/bin:$PATH export PYTHONPATH=/usr/local/anaconda/2.3.0/bin:/usr/local/anaconda/2.3.0/lib/python2.7:$PYTHONPATH time python myscript.py [options] qsub -q rcc-30d sub.sh optional qsub options, e.g., -pe threads 4 -l mem_total=20g 30

31 Run (Anaconda) Python Batch Job on Sapelo #PBS -S /bin/bash #PBS -q batch #PBS -N PythonJob1 #PBS -l nodes=1:ppn=4:amd #PBS -l walltime=48:00:00 #PBS -l mem=10gb cd $PBS_O_WORKDIR module load python/3.4.3 time python3 myscript.py [options] #PBS -S /bin/bash #PBS -q batch #PBS -N PythonJob1 #PBS -l nodes=1:ppn=4:amd #PBS -l walltime=48:00:00 #PBS -l mem=10gb cd $PBS_O_WORKDIR module load anaconda/ time python myscript.py [options] qsub sub.sh 31

32 Thank You! 32

Python Language Basics I. Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer

Python Language Basics I. Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer Python Language Basics I Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer zhuofei@uga.edu 1 Outline What is GACRC? Hello, Python! General Lexical Conventions Basic