Guillimin HPC Users Meeting March 16, 2017

Transcription

1 Guillimin HPC Users Meeting March 16, 2017 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

2 Please be kind to your fellow user meeting attendees Limit to two slices of pizza per person to start please And please recycle your pop cans. Thank you! 2

3 Outline Compute Canada News System Status Software Updates Training News Special Topic Best Practices for Job Submission 3

4 Compute Canada News 2017 Resource Allocation Competitions Scientific reviews completed Announcement of Awards: Soon! Implementation of Awards: Mid April 2017 Important notes: There will be some small number of migrations of allocations, either partial or full, from Guillimin to other Compute Canada systems (such as the new Cedar and Graham) Once announced, we can help answer any questions regarding migration if your allocation is on a different system 4

5 Compute Canada News 2017 High Performance Computing Symposium HPCS June 5th 9th Queen's University, Kingston (Ontario) Call for papers and posters: submissions due April 17 5

6 System Status March 6 - GPFS unresponsive on login nodes Caused by long waiters in GPFS GPFS communications unable to complete their actions Source: Infiniband communications issues between some worker nodes and the rest of the cluster, which can have adverse effects on general GPFS functions Problematic nodes were identified and removed from the cluster network GPFS waiters were cleaned and regular access restored in the afternoon of March 6 6

7 System Status Upcoming scheduled power maintenance: April Precise dates within the week still to be confirmed Major power maintenance by ETS to upgrade 25kV feeds to campus and therefore to the HPC Centre Impact Significantly reduced or no access to worker nodes To be Confirmed: Guillimin storage and login nodes may be placed on generator so as to enable access to data during that week Recommendations Attempt to complete any important project beforehand If you need to work on any code or data during that week, make sure to keep a copy at another site, when feasible More details to be announced soon 7

8 New Software Installations Please use module spider modulename for load instructions. Java/1.6.0_24 - Programming language (old version for compatibility) Python/{2.7.12, 3.5.2} - Programming language NAMD/2.9-PACE - Molecular dynamics code with PACE force field support Stacks/ Pipeline for building loci from short-read sequences SAMtools/ Manipulates alignments in the SAM format HDF5/ serial - Library for storing and managing data (no-mpi version) Bazel/ Build tool for Tensorflow Tensorflow/{ Python , Python , Python , Python-3.5.2} - Package for machine learning Vim/8.0 - The ubiquitous text editor (Vi Improved) tmux/2.3 - Terminal multiplexer RandomLib/ Library for random numbers 8

9 Training News All upcoming events: calculquebec.eventbrite.ca March 23 - Introduction to OpenMP (McGill) Apr. 4 - Analyse de données massives avec Spark (U. Laval) Apr. 6 - Programmation en R intermédiaire (UdeM) May Recently completed: Feb Data Analysis in Ecology, R/Python (UQAM) Mar. 2, 9 - Software Carpentry, Python (McGill) Mar. 7, 8 - Software Carpentry, Python (U. Laval) Mar Introduction a OpenMP (U. Sherbrooke) All materials from previous workshops are available online: wiki.calculquebec.ca/w/formations/en All user meeting presentations online at 9

10 User Feedback and Discussion Questions? Comments? We value your feedback. Contact us at: Guillimin Operational News for Users Status Pages (all CQ systems) Follow us on Twitter 10

11 Best Practices for Job Submission March 16, 2017 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

12 The Scheduler is Playing Tetris Lower priority Time Low priority High priority (reservation) Unused cores Nodes 12

13 The Scheduler is Playing Tetris Backfill (small, low priority job can run when higher priority jobs can't) Time Unused cores Nodes 13

14 Hardware Resources Available on Guillimin Partition Count (W, SB) Memory per core on Westmere nodes (ppn=12) Memory per core on Sandy Bridge nodes (ppn=16) Debug (SW2) - for short test jobs 0, 3 4 GB Serial Workload (SW, SW2) - for serial jobs and "light" parallel jobs 576, GB 4 GB High Bandwidth (HB) - for massively parallel jobs 384, 0 2 GB Large Memory (LM, LM2) - for jobs, requiring large memory footprint 192, GB 8 GB Extra Large Memory (XLM2) - limited selection of extra large memory nodes 0, 12 12, 16 or 32 GB Accelerated Workload (AW) - nodes with GPUs and Xeon Phis 0, or 8 GB 14

15 Let the Scheduler Choose the Right Queue Description (queue name) nodes=1:ppn<12 ppn=12 ppn=16 procs=n, n 12 Default (metaq) SW, SW2, AW SW, HB, LM SW2, LM2, XLM2 SW, HB, LM, SW2, LM2, XLM2 High Bandwidth (hb) Serial Workload (sw) Large Memory (lm) Accelerated Workload (k20, phi) Debug (debug) NOT ALLOWED HB SW2 HB SW, SW2 SW SW2 SW, SW2 NOT ALLOWED LM LM2 LM, LM2 AW AW AW AW SW2 SW2 SW2 SW2 15

16 Let the Default Queue Route Your Serial Job PBS -l value where n<12 walltime 36h (serial-short) walltime > 36h (sw-serial) #PBS -l nodes=1:ppn=n SW, SW2, AW SW #PBS -l nodes=1:ppn=n:westmere SW SW #PBS -l nodes=1:ppn=n:sandybridge SW2, AW SW2 Serial: Default memory: pmem=2700m (2.7G per core) Recommended: n 6, or n=12 otherwise (full node) Serial (Sandy Bridge): Optional memory: pmem=3700m (3.7G per core) Recommended: n 8, or n=16 otherwise (full node) 16

17 How to Pack Serial Jobs The Linux operating system can run your process in the background so that your script continues without waiting for it to finish Use the ampersand symbol, & The wait command says to wait for all background processes to finish #!/bin/bash #PBS -l walltime=30:00:00 #PBS -l nodes=1:ppn=12 SRC=$HOME/program_dir cd $SCRATCH/dir1 ; $SRC/prog > output & cd $SCRATCH/dir2 ; $SRC/prog > output & cd $SCRATCH/dir3 ; $SRC/prog > output &... cd $SCRATCH/dir12 ; $SRC/prog > output& wait #!/bin/bash #PBS -l walltime=30:00:00 #PBS -l nodes=1:ppn=12 SRC=$HOME/program_dir for i in $(seq 12) do cd $SCRATCH/dir$i $SRC/prog > output & done wait 17

18 How to Pack Thousands of Serial Tasks GNU Parallel is an easy-to-use tool for launching processes in parallel Example: testing all combinations of two parameters: {1, 2, 3} x {94, 95, 96} $ parallel echo {1} x {2} ::: $(seq 1 3) ::: $(seq 94 96) 1 x 94 1 x 95 1 x 96 2 x 94 2 x 95 2 x 96 3 x 94 3 x 95 3 x 96 18

19 GNU Parallel Run different commands in parallel parallel ::: hostname date 'echo hello world' Input sources from a file parallel -a input-file echo Input sources from the command line parallel echo ::: A B C Input sources from STDIN cat input-file parallel echo Input from multiple sources parallel -a abc-file -a def-file echo cat abc-file parallel -a - -a def-file echo # Will operate on each pair of inputs 19

20 Let the Default Queue Route Your Parallel Job pmem value walltime <= 72h walltime > 72h ppn=12 ppn=16 procs ppn=12 ppn=16 procs 1700m(*) HB, SW, LM SW2, LM2 HB, SW, LM, SW2, LM2 2700m( ) SW, LM SW2, LM2 SW, LM, SW2, LM2 3700m( ) LM SW2, LM2 LM, SW2, LM2 HB SW2 HB SW SW2 SW LM SW2 SW2 5700m LM LM2 LM, LM2 LM LM2 LM, LM2 7700m N. A. LM2 LM2 N. A. LM2 LM2 >7800m XLM2 XLM2 XLM2 XLM2 XLM2 XLM2 (*) pmem=1700m is default if procs>12 or nodes>1 ( ) pmem=2700m is default if procs=12 or nodes=1:ppn=12 ( ) pmem=3700m is default if ppn=16 20

21 Let the Default Queue Route Your Parallel Job Parallel (ppn=12, Westmere): #PBS -l nodes=n:ppn=12 Default pmem: 2700m if n=1, 1700m otherwise Parallel (ppn=16, Sandy Bridge): #PBS -l nodes=n:ppn=16 Default pmem: 3700m Parallel (procs=k, k>11, multiples of 48 are best): #PBS -l procs=k Default pmem: 2700m if k=12, 1700m otherwise 21

22 Submission styles (accelerators, debug) GPUs: #PBS -l nodes=2:ppn=16:gpus=2 #PBS -l pmem=123200m Reserves two full nodes with 2 GPUs each pmem is per node for GPUs! Xeon Phi: #PBS -l nodes=1:ppn=8:mics=1,pmem=29600m Public Queues: Default queue: metaq, generally no need to specify queue name Exception: debug queue: #PBS -q debug, for test jobs (default walltime 30 mins, max 2 hours) 22

23 Private Queues pmem value walltime <= 72h walltime > 72h ppn=12 ppn= m hbplus hb sw2-parallel 2700m swplus sw-parallel sw2-parallel 3700m sw2plus sw2-parallel sw2-parallel 5700m lm lm lm 7700m lm2 N. A. lm2 >7800m xlm2 xlm2 xlm2 Other queues: k20 phi debug 23

24 How to Monitor Your Job in Queue Idle queues with partitions for accurate priority: showq -i -p gm-1r16-n04 showq -i -p k20 showq -i -p phi Idle queue for your account: showq -i -w acct=abc-123-ax -v Idle queue for serial jobs: showq -i -p gm-1r16-n04 -w qos=serial Idle queue for any queue. For example, debug: showq -i -w class=debug Note: priority ranking goes by QOS = Quality of Service: serial,normal,avx(sw2),lm,xlm2,aw This way, lm jobs get priority over normal jobs on LM nodes. 24

25 Conclusion Any question? For other questions: 25