Minnesota Supercomputing Institute Regents of the University of Minnesota. All rights reserved.

Transcription

1 Minnesota Supercomputing Institute

2 Introduction to Job Submission and Scheduling Andrew Gustafson

3 Interacting with MSI Systems

4 Connecting to MSI SSH is the most reliable connection method Linux and Mac users can use the terminal command: ssh login.msi.umn.edu Windows users will need to use an ssh capable program, like PuTTY or Cygwin. SSH connections must first connect to login.msi.umn.edu. From there you can connect to other systems. For graphical connections use NICE:

5 MSI Computing Environment MSI systems are primarily Linux compute clusters running CentOS. Home directories are unified across systems. Software is managed via a module system. Jobs are scheduled via a queueing system.

6 Machine Architecture: Cluster Source:

7 Clusters at MSI Mesabi Itasca About 17,700 total cores, on Intel Haswell processors. 24 cores and 62gb per node in the large primary queues. Special queues with large memory (up to 1TB), and GPUs. Allows node sharing: good for both small and large jobs. mesabi.msi.umn.edu About 9,000 total cores, on Intel Nehalem processors. 8 cores and 22gb per node in the large primary queue. Special queues with larger memory and 16 cores per node. itasca.msi.umn.edu Lab Server About 500 total cores, on older hardware. For interactive, or small single node jobs. 8 cores and 15gb per node in the primary queue. lab.msi.umn.edu

8 Clusters at MSI Mesabi Itasca Login Lab First connect to login.msi.umn.edu, then connect to a cluster. Must be on-campus, or using the VPN.

9 Home Directories Home directories are unified across all Linux systems. Each group has a disk quota which can be viewed with the command: groupquota Panasas ActivStor 14: 3.01PB storage, capable of 30 GB/sec read/write, and 270,000 IOPS

10 Loading Software Software modules are used to alter environmental variables, in order to make software available. MSI has hundreds of software modules. Module Commands: Description Command Example See all available modules: module avail module avail Load a module: module load module load matlab/2015a Unload a module: module unload module unload matlab/2015a Unload all modules: module purge module purge See what a module does: module show module show matlab/2015a List currently loaded modules: module list module list

11 Job Scheduling On MSI systems, calculations are performed within jobs. A job is a planned calculation that will run for a specified time length on a specified set of hardware. There are two types of job: 1. Non-interactive (vast majority) 2. Interactive The job scheduler front-end is called the Portable Batch System (PBS). Jobs start in your home directory with no modules loaded.

12 Job Scripts To submit a non-interactive job, first make a PBS job script. Example: #!/bin/bash -l #PBS -l walltime=8:00:00,nodes=3:ppn=8,pmem=1000mb #PBS -m abe #PBS -M sample_ @umn.edu cd ~/program_directory module load intel module load ompi/intel mpirun -np 24 program_name < inputfile > outputfile

13 Job Submission To submit a job script use the command: qsub -q queuename scriptname A list of queues available on different systems can be found here: Submit jobs to a queue which is appropriate for the resources needed. Resources to consider when choosing a queue: Walltime Total cores and cores per node Memory Special hardware (GPUs, etc)

14 Job Submission To view queued jobs use the commands: qstat -u username showq -w user=username For detailed information: checkjob -v jobnumber To cancel a submitted job use the command: qdel jobnumber

15 Interactive Jobs Nodes may be requested for interactive use using the command: qsub -I -X -l walltime=1:00:00,nodes=1:ppn=8,mem=2gb The job waits in the queue like all jobs, and when it begins the terminal returns control.

16 Service Units (SUs) Jobs on the high performance computing (HPC) systems consume Service Units (SUs), which roughly correspond to processor time. Each research group is given a service unit allocation at the beginning of the year. To view the number of service units remaining use the command: acctinfo If a group is using service units faster than the "fairshare target", then the group's jobs will have lower queue priority.

17 Simple Parallelization: Backgrounding Most easily done with single node jobs. #!/bin/bash -l #PBS -l walltime=8:00:00,nodes=1:ppn=8,pmem=1000mb #PBS -m abe #PBS -M cd ~/job_directory module load example/1.0./program1.exe < input1./program2.exe < input2./program3.exe < input3./program4.exe < input4./program5.exe < input5./program6.exe < input6./program7.exe < input7./program8.exe < input8 wait > > > > > > > > output1 output2 output3 output4 output5 output6 output7 output8 & & & & & & & &

18 Simple Parallelization: Job Arrays Works best on Mesabi. Template Job Script, template.pbs: #!/bin/bash -l #PBS -l walltime=8:00:00,nodes=1:ppn=8,pmem=1000mb #PBS -m abe #PBS -M cd ~/job_directory module load example/1.0./program.exe < input$pbs_arrayid > output$pbs_arrayid Submit an array of 10 jobs: qsub -t 1-10 template.pbs

19 Minnesota Supercomputing Institute Web: Telephone: (612) The University of Minnesota is an equal opportunity educator and employer. This PowerPoint is available in alternative formats upon request. Direct requests to Minnesota Supercomputing Institute, 599 Walter library, 117 Pleasant St. SE, Minneapolis, Minnesota, 55455,