Linux Clusters Institute:

Transcription

1 Linux Clusters Institute: 3 rd Party Software Management Instructor: Timothy Bouvet Title: System Engineer NCSA tbouvet@illinois.edu 3 rd Party Software Management Topics: Best Practices Software Request Environment Modules Advanced Topics: Software Management Tools Software Maintenance 4-8 August

2 Best Practices Logbook - Maintain on your management node. Documentation - User pages, admin only. Test System - Create a test node or vm. Installs - User vs Root. 3rd Party Software - Deploy on a global filesystem. Sources - Keep in a common source directory. Build Directories keep around for reference. 4-8 August Online Logbook 4-8 August

3 User Documentation 4-8 August Admin Documentation 4-8 August

4 Software Request Software Request 1. Staff or User requests a package. Evaluate the request 1. Useful for many users or a few? 2. Used where? Compute, login, mom or a combination? 3. Do we want to deploy and support this software? 4. Yes forward to admin, No assist user deployment Admin rpm vs source install 1. Is an rpm available from repos or web search? 2. Yes Update/Install rpm(easiest to maintain). 3. No Source install in a global file system. Note: Develop a process for your site or institution 4-8 August Environment Modules What are modules? - dynamic way to support multiple versions of software like compilers. Why use modules? - easiest way to select versions of software and set user default environment. Who uses modules? - CRAY, Large HPC Centers, DOE to name a few. Uses tcl - tool command language for modulefiles. Managed Software - software installed in non default locations (source packages, relocatable rpms, srpms) Installing - Source vs RPM 4-8 August

5 Source Install As Root: mkdir /global mode 775; chgrp to your $USER group As $USER(SAFER): mkdir /global/src ; cd /global/src/ wget tar -xf modules tar.gz; cd modules /configure --prefix=/global --with-etc-path=/global/modules/etc make; make install (does not copy all needed files to prefix) mkdir /global/modules/etc; mkdir /global/modulefiles cp /global/modules/3.2.10/modulefiles/null /global/modulefiles/ cp /global/src/modules /etc/global/profile.modules and csh.modules to /global/modules/etc/ (location for testing) 4-8 August Source Install Cont. cd /global/modules/; ln -sf default (Required to work) To test add to your.bashrc or.cshrc: $HOME/.bashrc:. /global/modules/etc/profile.modules $HOME/.cshrc: source /global/modules/etc/csh.modules Enable for all as root: (On all systems and images, note rename) cp /global/modules/etc/profile.modules /etc/profile.d/modules.sh cp /global/modules/etc/csh.modules /etc/profile.d/modules.csh Note: INSTALL guide - ignore run add.modules as each user to setup $USER/.dot files and as root for default /etc/skel/.dot files to enable modules. Modifying user.dot files is a bad practice. 4-8 August

6 Source Modules.sh # # # system-wide profile.modules # # Initialize modules for all sh-derivative shells # # # trap "" #Exempt root env from global file system if [ `id -u`!= 0 ] ; then case "$0" in -bash bash */bash). /global/modules/default/init/bash ;; -ksh ksh */ksh). /global/modules/default/init/ksh ;; -zsh zsh */zsh). /global/modules/default/init/zsh ;; *). /global/modules/default/init/sh ;; # sh and default for scripts esac module use /global/modulefiles #add our search path module load modules #add module man pages etc. module load null #add default modules fi trap August Source Modules.csh # # # system-wide csh.modules # # Initialize modules for all csh-derivative shells # # # # Exempt root env from global file system if ("`id -u`"!= "0") then if ($?tcsh) then source /global/modules/default/init/tcsh else source /global/modules/default/init/csh endif module use /global/modulefiles #add our search path module load modules #add module man pages etc. module load null #add default modules endif 4-8 August

7 Source Module versioning 4-8 August RPM Install Check repos for environment modules/modules rpm Install the rpm on all systems and images Installs /etc/profile.d/module.sh module.csh If this is an upgrade rpm install it should save module.sh & modules.csh to x.rpmsave in /etc/profile.d Default module search path: /usr/share/modules/init/.modulespath (don t use, lost on rpm update) mkdir /global/modulefiles - for our modulefiles cp /usr/share/modules/modulefiles/null /global/modulefiles/ 4-8 August

8 RPM Modules.sh # # # system-wide profile.modules # # Initialize modules for all sh-derivative shells # # # trap "" case "$0" in -bash bash */bash). /usr/share/modules/init/bash ;; -ksh ksh */ksh). /usr/share/modules/init/ksh ;; -zsh zsh */zsh). /usr/share/modules/init/zsh ;; *). /usr/share/modules/init/sh ;; # sh and default for scripts esac #Exempt root env from global file system if [ `id -u`!= 0 ] ; then module use /global/modulefiles #add our search path module load null #add default modules fi trap August RPM Modules.csh # # # system-wide csh.modules # # Initialize modules for all csh-derivative shells # # # if ($?tcsh) then source /usr/share/modules/init/tcsh else source /usr/share/modules/init/csh endif # Exempt root env from global file system if ("`id -u`"!= "0") then module use /global/modulefiles #add our search path module load null #add default modules endif 4-8 August

9 How does the module command work? The command "module" is an alias for something like: sh: module () { eval `/some/path/modulecmd sh $*` } csh: eval `/some/path/modulecmd csh!* where the modulecmd outputs valid shell commands to stdout which manipulates the shell's environment. Any text that is meant to be seen by the user must be sent to stderr. Source: August Module Commands add/load - adds a module to your shell rm/unload - removes module from your shell avail - lists available module list - currently loaded modules display/show - what a modules will do to shell help - modulefile or module help information switch/swap - swap one module for another use/unuse - add/remove modulefile search path 4-8 August

10 How to capture the module command output? Since the module command is essentially an eval the visible output to the screen must necessarily be sent to stderr. Output to file: sh: module avail 2> spoolfile csh: module avail >&! spoolfile Pipe output: sh: module avail 2>&1 grep gcc csh: ( module avail ) & grep gcc NOTE: module avail gcc will list only gcc versions available Source: August Module Load tbouvet@h2ologin4:~> which gcc /usr/bin/gcc (native gcc) tbouvet@h2ologin4:~> module load gcc/4.8.2 tbouvet@h2ologin4:~> which gcc /opt/gcc/4.8.2/bin/gcc tbouvet@h2ologin4:~> echo $PATH /opt/gcc/4.8.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/x11:/ usr/x11r6/bin:/usr/games:/usr/lib64/jvm/jre/bin:/usr/lib/mit/ bin:/usr/lib/mit/sbin:/sbin:/usr/sbin NOTE: Loads in front of shell search path if the modulefile uses prepend-path directives. 4-8 August

11 which gcc /opt/gcc/4.8.2/bin/gcc Module Unload module unload gcc/4.8.2 echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/bin/x11:/usr/x11r6/bin:/usr/game s:/usr/lib64/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/sbin:/usr /sbin which gcc /usr/bin/gcc (native gcc) NOTE: When a modulefile is unloaded, append-path and prepend-path become remove-path. 4-8 August Module Avail 4-8 August

12 Module List Currently loaded modules 4-8 August Module Show module show gcc /opt/modulefiles/gcc/4.8.2: conflict gcc prepend-path PATH /opt/gcc/4.8.2/bin prepend-path MANPATH /opt/gcc/4.8.2/snos/share/man prepend-path INFOPATH /opt/gcc/4.8.2/snos/share/info prepend-path LD_LIBRARY_PATH /opt/gcc/4.8.2/snos/lib64 prepend-path LD_LIBRARY_PATH /opt/gcc/gmp/4.3.2/lib prepend-path LD_LIBRARY_PATH /opt/gcc/mpfr/2.4.2/lib prepend-path LD_LIBRARY_PATH /opt/gcc/mpc/0.8.1/lib setenv GCC_PATH /opt/gcc/4.8.2 setenv GCC_VERSION setenv GNU_VERSION August

13 Module Help module help modules Module Specific Help for 'modules' modules - loads the modules software & application environment This adds /global/modules/3.2.10/* to several of the environment variables. Version August Module Swap tbouvet@h2ologin4:~> module list Currently Loaded Modulefiles: 1) java/jdk1.7.0_45 3) swtools 5) gsissh/6.2p2 2) globus/ ) gcc/4.8.2 tbouvet@h2ologin4:~> module swap gcc/4.8.2 gcc/4.9.0 tbouvet@h2ologin4:~> module list Currently Loaded Modulefiles: 1) java/jdk1.7.0_45 3) swtools 5) gsissh/6.2p2 2) globus/ ) gcc/ August

14 Module Use env grep MODULEPATH MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefile s tbouvet@bwbsles:~> module use /global/modulefiles tbouvet@bwbsles:~> env grep MODULEPATH MODULEPATH=/global/modulefiles:/usr/share/Modules:/usr/share/ Modules/modulefiles tbouvet@bwbsles:~> module avail /global/modulefiles null test2/2.4.3(default) test2/2.4.4 NOTE: Adds to $MODULEPATH search path 4-8 August Module Unuse tbouvet@bwbsles:~> env grep MODULEPATH MODULEPATH=/global/modulefiles:/usr/share/Modules:/usr/share/ Modules/modulefiles tbouvet@bwbsles:~> module unuse /global/modulefiles tbouvet@bwbsles:~> env grep MODULEPATH MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefile s NOTE: Removes from $MODULEPATH search path 4-8 August

15 Modulefile Structure 4-8 August Modulefile GCC example from the web #%Module1.0 proc ModulesHelp { } { global dotversion puts stderr "\tgcc (gcc, g++, gfortran) } module-whatis "GCC (gcc, g++, gfortran)" conflict gcc (if true modulefile will not load) prepend-path PATH /packages/gcc/4.6.2/bin prepend-path LD_LIBRARY_PATH /packages/gcc/4.6.2/lib64 prepend-path LIBRARY_PATH /packages/gcc/4.6.2/lib64 prepend-path MANPATH /packages/gcc/4.6.2/man setenv CC gcc setenv CXX g++ setenv FC gfortran setenv F77 gfortran setenv F90 gfortran 4-8 August

16 Writing Modulefiles Use modules modulefile as a template Include build directories in modulefile: $PATH /bin /sbin, $MANPATH /man /share/man, $LD_LIBRARY_PATH $LD_RUN_PATH /lib /lib64, $INFOPATH /share/info Man modulefile - helpful man page Web search writing modulefiles Use system with modulefiles as examples Test modulefile for deployment functionality 4-8 August Module load use.own tbouvet@h2ologin4:~> module load use.own tbouvet@h2ologin4:~> module avail null /u/staff/tbouvet/privatemodules null NOTE: Will create privatemodules directory and add null modulefile. An empty modulefile directory will not show up. Use this directory to create and test modulefiles. 4-8 August

17 Hiding Modulefiles Incorrect Method: h2ologin4:/opt/cray/modulefiles/perftools-lite # chmod tbouvet@h2ologin4:~> module avail perftools-lite utility.c(2173):error:50: Cannot open file '/opt/cray/modulefiles/perftoolslite/6.1.0' for 'reading Correct Method: h2ologin4:/opt/cray/modulefiles/perftools-lite # mv tbouvet@h2ologin4:~> module avail perftools-lite /opt/cray/modulefiles perftools-lite/6.1.1 perftools-lite/6.1.2 perftools-lite/6.1.3(default) perftools-lite/6.2.0 perftools-lite/6.1.4 NOTE: perftools-lite/6.1.0 is now invisible 4-8 August Modulefile Default tbouvet@h2ologin4:~> module avail gcc /opt/modulefiles gcc/4.6.3 gcc/4.7.3 gcc/4.8.2(default) gcc/4.9.0 tbouvet@h2ologin4:~> cat /opt/modulefiles/gcc/.version #%Module set ModulesVersion "4.8.2 NOTE:.version file in modulefiles/gcc/ controls default gcc that will be loaded by module load gcc. Edit.version file to change the default. Place a.version file in all your modulefile deployment directories as a good practice. 4-8 August

18 Metamodules Loads a suite of modules 4-8 August Module Questions? 4-8 August

19 Software Management Tools Structure for 3 rd party software installs, rebuilds, utilize environment modules Swtools - product of the National Center for Computational Sciences (NCCS) at Oak Ridge National Laboratory (ORNL) (Used by NCSA:Blue Waters for 3 rd Party Software Management) Smithy - product of the National Center for Computational Sciences (NCCS) at Oak Ridge National Laboratory (ORNL) Based on swtools and homebrew PMS for Mac OS X Generates modulefile for your builds. Easybuild - software build and installation framework written in Python that allow you to install software in a structured, repeatable and robust way. Generates modulefile for your builds. 4-8 August SWTools Python code developed to manage third-party software installations and rebuilds in a structured manner. Used at large sites with many software Installers For global installs by $USER in a common group Ensures consistency of installations Generates html files suitable for web pages Provides for quick rebuild or upgrade of package by group Requires modulefiles to be generated by hand for builds 4-8 August

20 Software Maintenance Recommend maintaining five versions of packages. one production, one testing, three down-rev Need to determine 3 rd party software usage? Poll Users with questionnaire or Install altd library tracking utility (ORNL) Analyze usage for licensing, removal, upgrade decisions. Focus user support on widely used modules. Use MOTD or to notify users of software changes. 4-8 August ALTD Is a wrapper around the linker and job launcher. Tracks library usage by job and user. We module load altd into users PATH. It Populates a MySQL database. Python script used to mine data in MySQL for reports. Blue Waters uses ALTD to monitor module usage. Mainly used on CRAY systems at HPC centers. Can be installed on any machine. 4-8 August

21 Altd Report ALTD Usage from to ======= Top 10 Library usage, ranked by number of linking instances mpt/ libsci/ cudatoolkit/ fftw/ papi/ hdf5/ acml/ tpsl/ mpt/ hdf5-parallel/ ======= Top 10 Compiler usage, ranked by number of linking instances gcc/ gcc/ cce/ gcc/ gcc/ pgi/ pgi/ cce/ pgi/ pgi/ August Questions? Thank You! 4-8 August