Introduction to NCAR HPC. 25 May 2017 Consulting Services Group Brian Vanderwende

Transcription

1 Introduction to NCAR HPC 25 May 2017 Consulting Services Group Brian Vanderwende

2 Topics we will cover Technical overview of our HPC systems The NCAR computing environment Accessing software on Cheyenne and Yellowstone Compilers MPI/Parallelism Submitting batch jobs with the PBS and LSF job schedulers Data storage Q&A 2

3 Overview of NCAR systems HPC (simulation) Yellowstone 4536 dual-socket nodes 8-core Sandy Bridge 32 GB memory/node Cheyenne 4032 dual-socket nodes GB mem GB mem. 18-core Broadwell Data Analysis and Visualization Geyser 16 quad-socket nodes 10-core Westmere 1 TB memory/node Caldera 30 nodes (16 w/gpus) Tesla K20X 64 GB memory/node 1 Yellowstone core-hour = 0.82 Cheyenne core-hours 3

4 File storage at NCAR GLADE parallel spinning-disk storage Uses IBM GPFS/Spectrum Scale technology Optimized for parallel I/O operations Simulation and analysis I/O occurs here HPSS tape archive POSIX-like interface for long-term archival and data backups Data must be sent and received to tape - not available on demand All storage is shared between YS and CH! 4

5 Logging into the HPC systems Use your authentication token (yubikey) along with your username to login ssh -X -l username cheyenne.ucar.edu You will then be on one of six login nodes Your default shell is tcsh, but you can switch your default shell at sam.ucar.edu Standard UNIX commands are provided by Red Hat on Yellowstone and SUSE on Cheyenne 5

6 The login nodes are a shared resource - use them lightly! The six login nodes (yslogin#/cheyenne#) are accessed by all HPC users Your programs compete with those of s of other users for processing and memory Therefore, limit your usage to: Reading and writing text/code Compiling programs Performing small data transfers Interacting with the job scheduler Programs that use excessive resources on the login nodes will be terminated 6

7 CISL builds software for users to load with environment modules We build programs and libraries that you can access by loading an environment module Compilers, MPI, NetCDF, MKL, Python, etc. Modules configure your computing environment so you can find binaries/executables, libraries and headers to compile with, and manuals to reference Modules are also used to prevent conflicting software from being loaded You don t need to use modules, but they simplify things greatly, and we recommend their use 7

8 Note that Yellowstone and Cheyenne each have their own module/software tree! 8

9 Cheyenne module tree adds choice and improves clarity Yellowstone Cheyenne Compiler Intel GNU Software built with Intel MKL netcdf pnetcdf Compiler Intel Intel GNU Intel MKL netcdf MPI SGI MPT 2.15 Intel MPI Intel MPT 2.15 pnetcdf OpenMPI

10 Some useful module commands module add/remove/load/unload <software> module avail - show all community software installed on the system module list - show all software currently loaded within your environment module purge - clear your environment of all loaded software module save/restore <name> - create or load a saved set of software module show <software> - show the commands a module runs to configure your environment 10

11 Compiling software We will support Intel, GCC, and PGI Wrapper scripts are loaded by default (ncarcompilers module) which make including code and linking to libraries much easier Building with netcdf without the wrappers: setenv NETCDF /path/to/netcdf ifort -I${NETCDF}/include model.f90 -L${NETCDF}/lib -lnetcdff -o model Building with netcdf using the wrappers: ifort model.f90 -o model Do not expect a parallel program compiled with one MPI library to run using a different library! 11

12 Where you compile code depends on where you intend to run it Cheyenne has newer Intel processors than Yellowstone and Caldera, which in turn have newer chips than Geyser CISL recommends compiling separate versions of applications for Yellowstone and Cheyenne If you must build across systems: 1. Compile for the oldest system you want to use, to ensure that results are consistent 2. Build libraries and binaries statically to avoid conflicts between dependencies 12

13 To access compute resources, prepare batch job scripts LSF (Yellowstone) PBS (Cheyenne) #!/bin/bash #BSUB -J WRF_PBS #BSUB -P <project> #BSUB -q regular #BSUB -W 30:00 #BSUB -n 144 #BSUB -R span[ptile=16] #BSUB -o log.oe #BSUB -e log.oe # Run WRF with IBM MPI mpirun.lsf./wrf.exe #!/bin/bash #PBS -N WRF_PBS #PBS -A <project> #PBS -q regular #PBS -l walltime=00:30:00 #PBS -l select=4:ncpus=36:mpiprocs=36:ompthreads=1 #PBS -j oe #PBS -o log.oe # Run WRF with SGI MPT mpiexec_mpt./wrf.exe 13

14 Pinning tasks to CPUs with SGI MPT s dplace/omplace commands By default, tasks can migrate across available CPUs throughout execution Sometimes it is advantageous to pin tasks to a particular CPU (e.g., OpenMP across a socket) SGI provides two helper commands for pinning #PBS -l select=2:ncpus=36:mpiprocs=2:ompthreads=18 # Run program with one MPI task and 18 OpenMP threads per socket # (two per node with two nodes) mpiexec_mpt omplace./executable_name 14

15 Using threads/openmp to exploit shared-memory parallelism #!/bin/tcsh #PBS -l select=1:ncpus=10:ompthreads=10 # Run program with 10 threads omplace./executable_name Only OpenMP #!/bin/tcsh #PBS -l select=2:ncpus=36:mpiprocs=1:ompthreads=36 # Run program with one MPI task and 36 OpenMP # threads per node (two nodes) mpiexec_mpt omplace./executable_name Hybrid MPI/OpenMP If ompthreads is not specified, number of threads will default to value of ncpus - be careful! 15

16 Run serial code on multiple data files using command file jobs./cmd1.exe < input1 > output1./cmd2.exe < input2 > output2./cmd3.exe < input3 > output3./cmd4.exe < input4 > output4 cmdfile contents #!/bin/tcsh #PBS -l select=1:ncpus=4:mpiprocs=4 # This setting is required to use command files setenv MPI_SHEPHERD true PBS Job script mpiexec_mpt launch_cf.sh cmdfile Optimal if each command has similar runtime 16

17 Submitting jobs to and querying information from PBS To submit a job to PBS, use qsub: Script: qsub myjob.pbs Interactive: qsub -I -l select=ncpus:36:mpiprocs:36 -l walltime=10:00 -q share -A <project> qinteractive <project> - get one core job for one hour qstat <job_id> - query information about the job qstat -u $USER - summary of your active jobs qstat -Q <queue> - show status of specified or all queues qdel <job_id> - delete and/or kill the specified job 17

18 Submitting jobs to and querying information from LSF To submit a job to LSF, use bsub: Script: bsub < myjob.lsf Interactive: bsub -Is -n 16 -W 0:10 -q small -P <project> $SHELL execgy/execca - interactive job on geyser/caldera bjobs <job_id> - query information about the job bjobs - summary of your active jobs bqueues - show status of specified or all queues bkill <job_id> - delete and/or kill the specified job 18

19 Selected job submission queues PBS Queue Priority Wall clock Nodes Description capability 2 12 h Execution window: Midnight Friday to 6 a.m. Monday share N/A 6 h 0.5 Interactive use for debugging and other tasks on a single, shared, 128-GB node. regular 3 12 h 1152 LSF Queue Priority Wall clock Cores Description geyser 2 24 h 1-40 Shared nodes - interactive and batch bigmem 1 6 h Exclusive - daytime limit of 4 nodes caldera 2 24 h 1-16 Shared nodes - interactive and batch gpgpu 2 6 h Exclusive - daytime limit of 4 nodes 19

20 Current and future access for the DAV systems Geyser and Caldera Geyser and Caldera will continue to serve as data analysis and visualization machines Currently, they are only accessible from Yellowstone using LSF scheduler The current plan is for half of each machine to be switched to Cheyenne this summer For licensing reasons, it is likely that these transferred nodes will run an open-software stack: Slurm for scheduling jobs Open MPI as the default message passing library More details to come... 20

21 Managing your compute time allocation After compiling a program, try running small test jobs before your large simulation For single core jobs, use the share queue, to avoid being charged for unused core-hours: Exclusive: wall-clock hours nodes used 36 cores per node Shared: core-seconds / 3600 Use the DAV clusters for NCL, Python, MATLAB, and R scripts and interactive visualization (VAPOR) 21

22 How to store data across NCAR HPC systems File space Quota Data Safety Description Home /glade/u/home/$user 25 GB Backups & Snapshots Store settings, code, and other valuables Work /glade/p/work/$user 512 GB Stable but no backups Good place for keeping run directories and input data Project /glade/p/project Varies Stable but no backups HPSS hsi -> /home/$user TB/yr Charge Stable but no backups Storage limits depend on your allocation, data cannot be used interactively Scratch /glade/scratch/$user 10 TB At-risk! Purged! Use as temporary data storage only; manually back up files (e.g., to HPSS) Keep track of your usage with gladequota 22

23 Storage tips Archive large numbers of small files to limit wasted space and quota on GLADE Files occupy sub-blocks, and used space will round up to nearest sub-block (e.g., 256 KB on scratch) If data will be needed for a long time, consider moving to the HPSS tape archive: hsi cput <filename> hsi cget <filename> Large collections of files can be combined while transferring to HPSS using HTAR. Efficient! htar -cvf <archive.tar> <directory> 23

24 Considerations when using the shared file spaces Data written on either system contributes to the same file space quota Shared access to file systems from both clusters should make data management easier, but pay attention to where you have compiled programs! Similarly, organize job scripts carefully (e.g., use.lsf and.pbs extensions) Startup files are also shared, so machine specific commands require conditional statements to ensure they run on the correct system... 24

25 How to make.tcshrc/.profile machine specific ~/.tcshrc ~/.profile (bash) tty > /dev/null if ( $status == 0 ) then alias rm rm -i set prompt = "%n@%m:%~" if ( $HOSTNAME =~ yslogin* ) then # Yellowstone settings alias bstat bjobs -u all else # Cheyenne settings alias qjobs qstat -u $USER endif endif alias rm= rm -i PS1="\u@\h:\w> " if [[ $HOSTNAME == yslogin* ]]; then # Yellowstone settings alias bstat bjobs -u all source.profile-ys else fi # Cheyenne settings alias qjobs= qstat -u $USER source.profile-ch 25

26 Important dates in June 2017 The old scratch space from Yellowstone will disappear - copy files to new scratch before then! The 45-day purge policy starts on new scratch 15 July 2017 First day in which new-scratch files may be purged July 2017 Target for Cheyenne integration of DAV systems End of 2017 Decommissioning of Yellowstone begins 26

27 CISL Helpdesk/Consulting Walk-in: ML 1B Suite 55 Phone: Specific questions from today and/or feedback: For science questions (e.g., running CESM/WRF), consult relevant support resources 27