Using Compute Canada Masao Fujinaga Information Services and Technology University of Alberta
Introduction to cedar batch system jobs are queued priority depends on allocation and past usage
Cedar Nodes "Base" compute nodes: 576 nodes 128 GB of memory, 32 cores "Large" compute nodes: 128 nodes 256 GB of memory, 32 cores "Bigmem500" 24 nodes 0.5 TB (512 GB) of memory, 32 cores "Bigmem1500" nodes 24 nodes 1.5 TB of memory, 32 cores "GPU base" nodes: 114 nodes 128 GB of memory, 24 cores, 4 GPUs "GPU large" nodes. 32 nodes 256 GB of memory, 24 cores, 4 GPUs "Bigmem3000" nodes 4 nodes 3 TB of memory, 32 cores
Software for connecting to Compute Canada Windows MobaXterm http://mobaxterm.mobatek.net Mac XQuartz http://xquartz.macosforge.org/landing/
Install MobaXterm (Windows)
Start MobaXterm
Start local terminal
Connect with MobaXterm
Transfer files with MobaXterm
Connect with Terminal (mac or linux)
Transfer files mac/linux scp myfile mycomputecanadaid@cedar.westgrid.ca: FileZilla windows MobaXterm WinSCP
Connect to cedar Logging in ssh -Y mycomputecanadaid@cedar.computecanada.ca Interactive use compiling testing, debugging use less than 1 hour and 2 cores longer interactive use: salloc
Untar workshop files home directory cp ~masao/workshopsept2017.tar.gz. tar -xf workshopsept2017.tar.gz cd workshopsept2017 current directory
Setting up the environment Modules module avail Organized according to compiler, MPI, cuda.
Finding modules Module spider module spider blast+ You will need to load all module(s) on any one of the lines below before the "blast+/2.6.0" module is available to load. nixpkgs/16.09 gcc/5.4.0 module load gcc blast+
Software Compute Canada website https://docs.computecanada.ca/wiki/available_software
Other software Install own https://docs.computecanada.ca/wiki/installing_software_in_your_home_directory https://docs.computecanada.ca/wiki/python python27-scipy-stack/2017a, python35-scipy-stack/2017a python27-mpi4py/2.0.0, python35-mpi4py/2.0.0 https://docs.computecanada.ca/wiki/r r-bundle-bioconductor/3.4 (gcc) https://docs.computecanada.ca/wiki/perl Request installation support@computecanada.ca
Compiling compile icc -o hello hello.c ifort -o hello hello.f90 run./hello
Man pages man icc space bar to get a page /mysearch to search for mysearch q to quit man -k mytopic get pages related to mytopic
Editing graphical editors nedit, gedit non-graphical editors vi, emacs editing through Winscp
Batch job #!/bin/bash #SBATCH --reservation=masao_10 #SBATCH --account=def-myaccount #SBATCH --ntasks=1 #SBATCH --mem-per-cpu=2000mb #SBATCH --time=00:05:00 #SBATCH --mail-type=all #SBATCH --mail-user=myemail@emailaddress./hello
Job submission sbatch sbatch job-cedar.slm Submitted batch job 1136843 squeue squeue -u masao JOBID USER ACCOUNT 1136843 masao def-masao_cp NAME job-cedar.slm ST START_TIME PD N/A TIME_LEFT NODES CPUS GRES MIN_MEM NODELIST (REASON) 5:00 1 1 (null) 2000M (Priority) NAME job-cedar.slm ST START_TIME R 2017-09-19T08:31 TIME_LEFT NODES CPUS GRES MIN_MEM NODELIST (REASON) 4:55 1 1 (null) 2000M cdr310 (None) running JOBID 1136843 USER ACCOUNT masao def-masao_cp
Job output slurm-<jobnumber>.out Common errors out of walltime out of memory batch script in DOS format No end-of-line character at the end
pi program compile icc -o pi pi.c interactive run./pi input from terminal
pi job batch run create pi.in./pi < pi.in > pi.out no input file./pi << EOF-pi > pi.out 100 1000 0 EOF-pi
MPI compile mpicc -o hello-mpi hello-mpi.c mpif90 -o hello-mpi hello-mpi.f90 Run ulimit -l 2097152 mpiexec -np 2./hello-mpi
Interactive batch job Submitting an interactive batch job salloc --time=6:0:0 --ntasks=2 --account=def-myaccount -reservation=masao_10 no batch script may have to wait a long time depending on how busy the system is
Benchmarking Timing reproducible, without other users time command walltime
Parallel benchmark Time the real time it takes to run the program using various number of processors For any parallel program, it is important to know the parallel efficiency of the program
benchmark exercise compile sample1.f capital oh zero mpif90 -o sample1 sample1.f90 -fixed -O0 submit script1.sbatch. This will run the program five times, changing -n to 1, 2, 4, 8, and 12 look at the real time ( grep real out.time ) plot using www.wolframalpha.com Replace t1,t2,t4,t8,t16 with your values plot time plot{{1,t1},{2,t2},{4,t4},{8,t8},{16,t16}} plot speedup plot{{1,1},{2,t1/t2},{4,t1/t4},{8,t1/t8},{16,t1/t16}}
benchmark results
Optimizing compiler optimization default: -O2 higher optimizations: -O3, -axcore-avx2 Exercise icc -o sample2 sample2.c -O1 time./sample2 time runs of sample2 after having compiled with -O0 -O1 -O2 -O3 -O3 -axcore-avx2 -O3 -axcore-avx2 -fp-model precise Note the checksum
Math Library Intel Math Kernel Library (MKL) http://software.intel.com/en-us/mkl_11.1_ref -mkl=sequential
Filesystems home directory Project Default project Projects Links to other project directories scratch Use for running jobs quotas diskusage_report
Moving data in Compute Canada Globus https://www.computecanada.ca/research-portal/national-services/globus-portal/ web-based fault tolerant can transfer to your desktop
Upcoming ARC Training Sessions October 3 10:00 AM 11:00 AM PDT October 5 11:00 AM 1:00 PM PDT October 24-26 1:00-3:00 PM MDT (each day) November 1 11:00 AM 1:00 PM PDT November 21 11:00 AM 1:00 PM PST Data Visualization on Compute Canada's Supercomputers Intro to Parallel Programming for Shared Memory Machines Scheduling & Job Management: How to Get the Most from a Cluster Introduction to Classical Molecular Dynamics Simulations Exploring Containerization with Singularity
Getting Help https://docs.computecanada.ca use the search support@computecanada.ca specify user ID which system job ID error messages script location