Slurm and Abel job scripts. Katerina Michalickova The Research Computing Services Group SUF/USIT November 13, 2013

Transcription

1 Slurm and Abel job scripts Katerina Michalickova The Research Computing Services Group SUF/USIT November 13, 2013

2 Abel in numbers Nodes Cores (1 node->2 processors->16 cores) Total memory - ~40 TB (typically 64 GB per node) Total storage - ~400TB InfiniBand interconnect on all nodes (FDR) # 96 at top500.org in 2012 Read more at

3 Topics queuing system job administration user administration software modules job scripts examples simple scratch arrayrun parallel jobs OpenMP MPI

4 Queuing system Lets you specify resources for your computation Keeps track of which resources are available on which nodes, and starts your job when the requested resources are available On Abel, we use the Simple Linux Utility for Resource Management - SLURM A job is started by sending a shell-script to slurm with the command sbatch. Resources are requested by special comments in the shell-script (#SBATCH --)

5 Interactive use of Abel Abel is used through the queuing system. It is not allowed to run jobs directly on the login nodes (the nodes you find yourself on when you ssh abel.uio.no). The login nodes are just for logging in, copying files, editing, compiling, running short tests (no more than a couple of minutes), submitting jobs, checking job status, etc. If interactive login is needed, use qlogin.

6 Ask SLURM for the right resources Project Memory Time Queue Disk CPUS Nodes Combination thereof Logging

7 sbatch - project #SBATCH --account=project Specify the project to run under. Every Abel user is assigned a project. Use command projects to find out which project you belong to. UiO scientists/students can use the uio project It is recommended to seek additional resources if planning intensive work. Application for compute hours and data storage can be placed with the Norwegian metacenter for computational science (NOTUR) #SBATCH --job-name=jobname Job name

8 sbatch - memory #SBATCH --mem-per-cpu=size Memory required per allocated core (format: 2G or 2000M) How much memory should one specify? The maximum usage of RAM by your program (plus some). Exaggerated values might delay the job start. #SBATCH --partition=hugemem If you need more than 61.5GB of RAM on a single node. Currently not many nodes available with this feature.

9 mem-per-cpu - top maximum usage of virtual RAM by your program

10 sbatch - time #SBATCH --time=hh:mm:ss Wall clock time limit on the job Some prior testing is necessary. One might, for example, test on smaller data sets and extrapolate. As with the memory, unnecessarily large values might delay the job start. Maximum time for a job is 1 week (168 hours). If more needed, use --partition=long (to a maximum 4 weeks)

11 sbatch CPUs and nodes Does your program support more than one CPU? If so, do they have to be on a single node? How many CPUs will the program run efficiently on? #SBATCH cpus-per-task Number of cores on one node #SBATCH nodes Number of nodes to allocate #SBATCH --ntasks-per-node Number of cores to allocate within each allocated node #SBATCH --ntasks=cores Number of cores to allocate

12 sbatch CPUs and nodes If you need cpus on one node (e.g. threading or OpenMP) #SBATCH cpus-per-task=8 If you just need some cpus, no matter where (e.g. MPI): #SBATCH --ntasks=17 If you need a specific number of cpus on each node #SBATCH --nodes=8 --ntasks-per-node=4 If you need the cpu's on a single node #SBATCH --nodes=1 --ntasks-per-node=8 If you need/prefer a specific type of cpus

13 sbatch - files #SBATCH --output=file Send 'stdout' (and stderr) to the specified file (instead of slurmxxx.out) #SBATCH --error=file Send 'stderr' to the specified file #SBATCH --input=file Read 'stdin' from the specified file

14 sbatch low priority #SBATCH --qos=lowpri Run a job in the lowpri queue Even if all of your project's cpus are busy, you may utilize other cpus Such a job may be terminated and put back into the queue at any time. If possible, your job should ensure its state is saved regularly, and should be prepared to pick up on where it left off.

15 sbatch GPU partition #SBATCH --partition=gpu gres=gpu:1 for one GPU card #SBATCH --partition=gpu gres=gpu:2 for two GPU cards

16 Inside the job script All jobs must start with the bash-command: source /cluster/bin/jobsetup A job-specific scratch-directory is created for you on /work partition. The path is in the environment variable $SCRATCH. We recommend using this directory especially if your job is IO intensive. You can copy results back to your home-directory when the job exits using chkfile (or cleanup) in your script. The SCRATH directory is removed when the job finishes.

17 Environment variables SLURM_JOBID job-id of the job SCRATCH name of job-specific scratch-area SLURM_NPROCS total number of cpus requested SLURM_CPUS_ON_NODE number of cpus allocated on node SUBMITDIR directory where sbatch were issued TASK_ID task number (for arrayrun-jobs)

18 Job administration cancel a job see job details see the job queue see the projects

19 Cancel a job - scancel scancel jobid # scancel --user=me Cancel a job Cancel all your jobs

20 Job details - scontrol show job

21 See the queue - squeue [-j jobids] show only the specified jobs [-w nodes] show only jobs on the specified nodes [-A projects] show only jobs belonging to the specified projects [-t states] show only jobs in the specified states (pending, running, suspended, etc.) [-u users] show only jobs belonging to the specified users All specifications can be comma separated lists Examples: squeue j 4132,4133 shows jobs 4132 and 4133 squeue -w compute shows jobs running on compute squeue -u foo -t PD shows pending jobs belonging to user 'foo' squeue -A bar shows all jobs in the project 'bar'

22

23 Checking a job - squeue PD R S CG CD CF CA F TO PR NF Pendind Running Suspended Completing Completed Configuring Cancelled Failed Timeout Preemepted Node failed

24 --nonzero --pe --memory --group See the projects - qsumm only show accounts with at least one running or pending job show processor equivalents (PEs) instead of CPUs show memory usage instead of CPUs do not show the individual Notur and Grid accounts --user=username only count jobs belonging to username --help show all options

25

26 User administration - project and cost

27 Interactive use of Abel - qlogin Send request for a resource Join the queue Work on command line when resource becomes available Book one node (or 16 cores) on Abel for your interactive use for 1 hour: qlogin --account=your_project --ntasks-per-node=16 --time=01:00:00 Run source /cluster/bin/jobsetup after receiving allocation For a more info, see:

28 Software on Abel Available on Abel: Software on Abel is organized in modules. List all software (and version) organized in modules: module avail Load software from a module: module load module_name If you cannot find what you looking for: ask us

29 Job script Your program joins the queue via a job script Job script - shell script with keywords in comments read by the queuing system Compulsory keywords: #SBATCH --account #SBATCH --time #SBATCH --mem-per-cpu Setting up a job environment source /cluster/bin/jobsetup

30 Minimal job script #!/bin/bash # Job name: #SBATCH --job-name=jobname # Project: #SBATCH --account=uio # Wall time: #SBATCH --time=hh:mm:ss # Max memory #SBATCH --mem-per-cpu=max_size_in_memory # Set up environment source /cluster/bin/jobsetup # Run command./executable > outfile

31 Use of the SCRATCH area #!/bin/sh #SBATCH --job-name=yourjobname #SBATCH --account=yourproject #SBATCH --time=hh:mm:ss #SBATCH --mem-per-cpu=max_size_in_memory source /cluster/bin/jobsetup ## Copy files to work directory: cp $SUBMITDIR/YourDatafile $SCRATCH ## Mark outfiles for automatic copying to $SUBMITDIR: chkfile YourOutputfile ## Run command cd $SCRATCH executable YourDatafile > YourOutputfile

32 Strength of cluster computing Large problems (or parts of) can be divided into smaller tasks and executed in parallel Types of parallel applications: Divide input data and execute your program on all subsets (array run) Execute parts of your program in parallel (MPI or OpenMP programming)

33 Arrayrun To run many instances of the same job, use an arrayrun command All jobs are submitted from the same directory, one must organize input and output files for each run Use TASK_ID varible in file names

34 TASK_ID variable TASK_ID is an environment variable, it can be accessed by all scripts during the execution of arrayrun 1 st run TASK_ID = 1 2 nd run TASK_ID = 2 N th run TASK_ID = N TASK_ID is used to organize input and output files Accesing the value of TASK_ID variable: In shell script : $TASK_ID In perl script: ENV{TASK_ID}

35 Arrayrun job script worker script #!/bin/sh #SBATCH --account=yourproject #SBATCH --time=hh:mm:ss #SBATCH --mem-per-cpu=max_size_in_memory #SBATCH --partition=lowpri source /cluster/bin/jobsetup DATASET=dataset.$TASK_ID OUTFILE=result.$TASK_ID cp $SUBMITDIR/$DATASET $SCRATCH chkfile $OUTFILE cd $SCRATCH executable $DATASET > $OUTFILE

36 Arrayrun job script submit script #!/bin/sh #SBATCH --account=yourproject #SBATCH --time=hh:mm:ss (longer than worker script) #SBATCH --mem-per-cpu=max_size_in_memory (low) source /cluster/bin/jobsetup arrayrun workerscript 1,4,42 1, 4, , 2, 3, 4, :2 0, 2, 4, 6, 8, 10 32,56, , 56, 100, 101, 102,..., 200!no spaces, decimals, negative numbers

37 Example of arrayrun simple program Print out TASK_ID variable together with current time

38 Example of arrayrun worker script

39 Example of arrayrun submit script

40 Array run example BLAST - sequence similarity search program Input biological sequences ftp://ftp.ncbi.nih.gov/genomes/influenza/influenza.faa Database of sequences ftp://ftp.ncbi.nih.gov/blast/db/

41 Array run example 2 Output sequence matches probabilistic scores sequence alignments

42 Parallelizing BLAST Split the query file Perl fasta splitter from

43 Abel worker script

44 Abel submit script

45 Abel in action

46

47 Parallel jobs on Abel start Two kinds of parallel jobs Single node OpenMP serial Init parallel env. Terminate parallel env. Multiple nodes MPI serial end

48 Single node Shared memory is possible Threads OpenMP Message passing MPI

49 OpenMP job script OpenMP]$ cat hello.run #!/bin/bash #SBATCH --account=staff #SBATCH --time=00:01:00 #SBATCH --mem-per-cpu=100m #SBATCH --ntasks-per-node=4 --nodes=1 source /cluster/bin/jobsetup export OMP_NUM_THREADS=$SLURM_CPUS_ON_NODE./hello.x

50 Multiple nodes Distributed memory Message passing, MPI

51 MPI on Abel we support Open MPI module load openmpi use mpicc and mpif90 as compilers use the same MPI module for compilation and execution read jobs specifying more than one node automatically get #SBATCH --constraint=ib

52 MPI job script #!/bin/bash #SBATCH --account=staff #SBATCH --time=0:01:0 #SBATCH --mem-per-cpu=100m #SBATCH --ntasks-per-node=1 #SBATCH --nodes=4 source /cluster/bin/jobsetup module load openmpi mpirun./hello.x

53 Thank you.