Introduction to High Performance Computing at UEA. Chris Collins Head of Research and Specialist Computing ITCS

Transcription

1 Introduction to High Performance Computing at UEA. Chris Collins Head of Research and Specialist Computing ITCS

2 Introduction to High Performance Computing High Performance Computing at UEA This presentation is available to download and look over at your own pace at: The exercises are available to download at: Introduction to Linux: There is a brief getting started guide to Linux available, which also includes some more advanced commands that may be useful for working on the cluster available at:

3 Introduction to High Performance Computing By the end of this course you should: Know why we have a HPC cluster and how it can benefit research Review the components of the HPC cluster Understand the process of submitting jobs to the HPC cluster Be able to connect to the HPC cluster and copy files and data to the HPC cluster Work interactively on the cluster Prepare job scripts and submit jobs to the appropriate queues requesting the appropriate resources

4 What is High Performance Computing? The goal of High Performance Computing is to reduce the execution time of a compute intensive or data intensive problem. HPC facilities are designed to deal with the grand challenges of science: Complex problems Large data sets Large memory requirements HPC systems are comprised of the fast processors, large memory capacities and fast interconnection networks. HPC designed to provide quick access to, and facilities for processing large amounts of data Problems are often sped up by dividing the work of the program up; so that many different processors work on parts of the problem concurrently. Two main uses of HPC facilities: High Throughput Computing High Performance Computing

5 HPC Usage at UEA Science Medicine and Health Sciences Social Sciences Climatic research statistical and dynamic climatology environmental modelling ocean biogeochemistry seismology fluid dynamics chemical modelling molecular dynamics crystallography and NMR protein sequence analysis evolutionary biology phylogenetics epidemiology evolutionary genomics large-scale RNA sequence analysis speech recognition knowledge discovery and data mining computer security statistical economics.

6 HPC Usage at UEA Publication list: Case studies:

7 Why use the UEA HPC? Compared to desktop processing: Number of nodes means larger number of concurrent jobs Processing power Faster multicore processors Larger memory systems available Nodes with between 24G and 192G of RAM Low latency / high speed interconnect for parallel processing systems Storage Enterprise level storage, secure, high availability and resilience Applications and libraries SAS 9.3, Stata 13-MP16, R v2.12 v2.15.2, Matlab, Maple. NetCDF, UV-CDAT, Ferret, Gromacs, IDL, HTK... Parallel libraries Parallel NetCDF, Parallel HDF, MPI and MPICH What can you do: Run applications as you would on your desktop (Matlab, Stata, SAS etc) in interactive sessions with graphical interfaces Submit jobs that automatically start when resources are available (can run multiple jobs at a time) Run batch and array jobs with different inputs/data to the same scripts concurrently

8 Introduction to High Performance Computing Cluster components

9 What a cluster looks like

10 Login Introduction to High Performance Computing cn001 cn003 cn005 cnxxx cn331 cn333 cn002 cn004 cn006 cnyyy cn332 cn334 Login node 1Gb/s network Processing nodes 334 Total computational cores 4784 Memory 24/32/48/64/128/192 GB High performance parallel GPFS storage 40Gb/s Infiniband low latency network

11 Login Introduction to High Performance Computing

12 Login Introduction to High Performance Computing

13 Login Introduction to High Performance Computing X=1 X=2 X=3 X=4 X=5 X=6

14 Login Introduction to High Performance Computing X=1 X=2 X=3 X=4 X=5 X=6

15 Login Introduction to High Performance Computing X=1 X=2 X=3 X=4 X=5 X=6

16 Login Introduction to High Performance Computing X=1 X=2 X=3 X=4 X=5 X=6

17 Cluster components Parallel Computing Faster than standard Ethernet with a lower latency, Quad Data Rate Infiniband infrastructure connects some nodes upon which parallel jobs can be run. Ethernet QDR IB Bandwidth 1Gbit/s 40Gbit/s Latency 40µs 1.1µs Communication latency = Sender overhead + Transmission time + Receiver overhead HPL Problem Size Number of Nodes Ethernet Runtime Infiniband Runtime

18 Cluster components Processing nodes 334 Total computational cores 4784 CPU Memory Infiniband interconnect Individual theoretical peak performance Theoretical peak performance (cluster) Dual x 6 core 2.66GHz Dual x 8 core 2.6GHz Dual x 10 core E GHz 24 / 32 / 48 / 64 / 128 / 192 GB 160 nodes - 40Gb/s Quad Data Rate GFLOP/s (X5650) GFLOP/s (E5-2670) 400 GFLOP/s (E v2) 79 TFlops

19 Introduction to High Performance Computing Connecting to the cluster

20 Connecting to the cluster To connect with SSH use PuTTY in Windows Start > All Programs > UEA Network Applications > SSH Login (PuTTY) To transfer files to the cluster use WinSCP in Windows Start > All Programs > UEA Network Applications > WinSCP To display graphics on your local PC from the cluster in Windows (VcXsrv, Xming or Exceed) Start > All Programs > UEA Network Applications > Xming > Xming

21 Using SSH to connect SSH (Secure Shell) creates a secure channel between your local machine and a remote machine (i.e. the cluster). This channel can also be used to tunnel data such as graphical data and files. On Windows we use PuTTY: Start > All Programs > UEA Network Applications > SSH Login (PuTTY) 1. Host name (or IP address): grace.uea.ac.uk 2. Saved sessions: grace 3. Go to Connection > SSH > X11 4. Check Enable X11 forwarding 5. X display location: localhost:0 6. Session > Save 7. Open From Linux or Mac we can connect from the command line: ssh X username@grace.uea.ac.uk Depending on where you connect from you may be asked for your username and password, this will be your standard UEA credentials

22 Using SCP to transfer files Based on SSH, we use SCP (Secure Copy) to transfer files between computers. On Windows we use WinSCP: Start > All Programs > UEA Network Applications > WinSCP 1. New 2. Host name: grace.uea.ac.uk 3. User name: <your username> 4. Password: You can enter a password or leave it blank to be prompted on access 5. Save 6. Login From Linux or Mac we can connect from the command line: scp -r directory username@grace.uea.ac.uk:/gpfs/dep/username/

23 Introduction to High Performance Computing Cluster storage

24 Cluster storage Path Use for HPC jobs Backed up Default quota Notes /gpfs/dep/username 30G Secure, high availability high resilience /gpfs/scratchdep/username 100G data for driving models; output prior to analysis /local - /gpfs/esarchive/dep/username - /home/dep/username - - At least 180G available on each compute node as temporary scratch Long term storage of rarely used data UEA central filestore

25 Cluster storage File and Data Management Do not run HPC jobs from your UEA central filestore Use filesystems appropriately: Scratch for temporary data or data that can be easily downloaded or reproduced Home for important data that you need to keep Be aware of the data your job uses for input and produces as output If using /local, copy data on at the start of the job, and move data off at the end. Don t leave any data on the compute nodes Remove old unneeded files, e.g.: Job error and out files Old job log file Old temporary files Core dump files (e.g. core.13417) The command quotacheck will report your usage on all filesystems (note: while you have quota on other department filesystems you cannot write to them)

26 Introduction to High Performance Computing HPC Software

27 Modules Module files are used to dynamically update user s environments to provide access to different applications. Module files allow us to support multiple versions of applications. module list List modules currently loaded in your environment module avail Lists all the modules available module show packagename Shows information about the package module module add packagename Adds modulename to your environment module del packagename Removes modulename to your environment module switch packagenamea packagenameb Unloads first module and replaces with second module module purge Removes all modules from your environment Modules must be added to job submission scripts to provide access to applications Modules can be automatically loaded by adding module add commands to your.bashrc file

28 Modules applications and libraries module avail will show you the complete set of applications, tools and libraries available. Some of the applications available include: MrBayes R , , Revolution R Gromacs ; GPU Gromacs HTK Matlab 2010a a CDAT 5.2 ; UV-CDAT 2.0 Python , Ruby 1.8.7, SAS 9.3 Stata 13 MP 16 NetCDF HDF5

29 Modules compilers and parallel communication Compiler modules icc/intel/11.1 Intel C/C++ and Fortran (F77/F90) compilers pgi/7.0.7 Portland Group compilers version pgi/10.9 Portland Group compilers version 10.9 GCC (available by default) GNU Compiler Collection Each compiler builds binaries that perform differently, Intel tends to result in the fastest performing binaries, then PGI Some code may have difficulties with certain compilers Parallel Communication There are various MPI (Message Passing Interface) libraries available. The main libraries are: mpi/platform/intel/8.1 mpi/platform/pgi/8.1 mpi/platform/gcc/8.1 mpi/openmpi/gcc/1.4.1 Platform MPI for Intel compilation Platform MPI for PGI compilation Platform MPI for GCC compilation OpenMPI for GCC compilation Platform MPI compiled code works on both standard Ethernet and Infiniband systems

30 Introduction to High Performance Computing Job submission

31 Job Scheduler Grace uses First-Come, First-Served scheduler which assesses job priority, resource requirements, host availability and user and queue job slot limits Queues based on resource and run time: Queue Resource Time Limit interactive Standard 12 hours interactive-lm Large memory 12 hours short Standard 5 hours medium Standard 24 hours long Standard 5 days short-ib Infiniband 5 hours medium-ib Infiniband 24 hours long-ib Infiniband 5 days medium-lm Large memory 24 hours long-lm Large memory 5 days

32 Working interactively Grace uses Platform LSF (Load Sharing Facility) a batch queuing system that controls the allocation of jobs to nodes. A suite of commands are available for submitting, monitoring and managing jobs and managing queues. interactive Start an interactive session (alias of bsub -L /bin/bash -q interactive $@ -cwd "$HOME" -Is /bin/bash) Xinteractive Start a graphical interactive session (alias of bsub -XF -L /bin/bash -q interactive $@ -cwd "$HOME" -Is /bin/bash)

33 Working interactively - STATA

34 Working interactively - Matlab

35 Job Submission To submit a job to run automatically on the cluster, a job submission script is required The job submission script is a file that contains: A set of directives which define how and where the job should run Information to set up job environment including module commands to provide access to the applications required Run the application The base directives that should be set include: Select queue Set job name Set any additional resource requirements (memory, parallel) Jobs are submitted with the following command: bsub < JobSubmission.bsub

36 BSUB batch jobs #!/bin/sh #BSUB -q medium #BSUB -J RTestJob #BSUB -oo R-%J.out #BSUB -eo R-%J.err. /etc/profile module add R/ R CMD BATCH Random.R

37 BSUB parallel jobs #!/bin/sh #BSUB -q medium-ib #BSUB -n 96 #BSUB -R 'select[ib]' #BSUB -R 'span[ptile=12]' #BSUB -R 'cu[maxcus=1]' #BSUB -oo ParTest-%J.log #BSUB -eo ParTest-%J.log #BSUB -J ParallelTest". /etc/profile module add mpi/platform/pgi/8.1 mpirun MyParallelBinary

38 BSUB directives #BSUB -q QueueName Run this job on QueueName.q e.g. short, long, long-ib #BSUB -J JobName Name this job JobName #BSUB -J JobName[StartNum-EndNum] Creates array jobs, individual jobs between StartNum and EndNum, i.e #BSUB -n NumOfSlots Set parallel job number of slots required e.g. 16 #BSUB -B Get when job starts #BSUB -N Get when job ends #BSUB -o output.log Write job output to output.log (appends if file exists -oo overwrites if file exists) #BSUB -e error.log Write job errors to error log (appends if file exists, -oe overwrites if file exists) For both output and error logs, the following options can be used: %J can be used to include JOBID %I for array JOBINDEX If o and e are omitted, then LSF will job output

39 Introduction to High Performance Computing Resources

40 Job Resources CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11

41 Job Resources CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11

42 Job Resources CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11

43 Job Resources CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11

44 Job Resources CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 #BSUB n 4 #BSUB R 'span[ptile=4]'

45 Job Resources CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 #BSUB R "rusage[mem=10000] #BSUB M 12000

46 Job Resources CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 #BSUB x

47 Introduction to High Performance Computing Sources of help

48 High Performance Computing Website - Connecting to Grace - Using Grace + Jobs + Queues + Software + Gaussian + Matlab + R + Stata + SAS + Memory + Storage - FAQ + HPC Basics + Cluster Components + Basic Linux - Case Studies - Publications

49 Troubleshooting and reporting problems You can help us provide a faster solution by providing us with the following information when reporting issues: Login Issues Name and location of machine connecting from Confirmation you are trying to connect to grace.uea.ac.uk OS and connection method (Putty or Exceed etc) Confirmation you are using your standard UEA username and password General job issues Application being used A general description of the problem Output of which application Output of module list Any error or warning messages Location and name of job submission script and output files Location of input and output data Have you run this application before on the cluster successfully? Have you used this input data successfully on the cluster? Node name It is also worth checking Google, the application website or mailing list or with colleagues

50 High Performance Computing Chris Collins Head of Research and Specialist Computing Leo Earl Systems developer (HPC) Julie Harold (0.4FTE) Systems developer (HPC/Linux) IT Linux Mike Salmon Linux systems developer Julie Harold (0.4FTE) - Systems developer (Linux/HPC) Jimmy Cross Linux systems developer What does the IT Linux team support? The team primarily supports the UEA Ubuntu Linux desktops (~150 computers) Provide expertise for ad hoc scientific projects requiring Linux systems SVN - Subversion and Trac are provided at UEA to enable the research community to share version controlled files within a research group