Our Workshop Environment

Transcription

1 Our Workshop Environment John Urbanic Parallel Computing Scientist Pittsburgh Supercomputing Center Copyright 2015

2 Our Environment Today Your laptops or workstations: only used for portal access Blue Waters is our HPC platform We will here briefly go through the steps to login, edit, compile and run before we get into the real materials We want to get all of the distractions and local trivia out of the way here. Everything after this talk applies to any HPC environment you will encounter.

3 Our Environment The Blue Waters system is a Cray XE/XK hybrid machine composed of AMD 6276 "Interlagos" processors (nominal clock speed of at least 2.3 GHz) and NVIDIA GK110 "Kepler" accelerators all connected by the Cray Gemini torus interconnect. Total Cabinets 288 Total Peak Performance PF Total System Memory PB XE Compute Cabinets 237 XE Peak Performance 7.1 PF XE Compute Nodes 22,640 XE Bulldozer Cores* 362,240 XE System Memory PB XK Compute Cabinets 44 XK Peak Performance (CPU+GPU) 7.18 PF XK Compute Nodes 4,224 XK Bulldozer Cores* (CPU) 33,792 XK Kepler Accelerators (GPU) 4,224 XK System Memory (CPU) 135 TB XK Accelerator Memory (GPU) 25 TB Architecture Topology 3D Torus 24x24x24 Compute nodes per Gemini 2 Peak Node Injection Bandwidth 9.6 GB/s

4 XK Compute Node Blue Waters is equipped with three node types: traditional compute nodes (XE6), accelerated compute nodes (XK7), and service nodes. AMD 6276 Interlagos Processors 1 Bulldozer Cores 8 Integer Scheduling Units 16 Memory / Bulldozer Core 4 GB Node System Memory 32 GB GPU Memory 6 GB Peak CPU Performance GF CPU Memory Bandwidth 51.2 GB/s CUDA cores 2688 Peak GPU Performance (DP) 1.32 TF GPU Memory Bandwidth 200 GB/s Two nodes per Gemini

5 Getting Connected We will be connecting through the bwedge.ncsa.illinois.edu hub. From anywhere from which you can ssh (using a Putty terminal, for example), ssh to the Hub with your password sheet credentials. Here is what it looks like using ssh from a unix command line. It may take a few seconds, but nothing was typed after the ssh command. The first time you login, you may be asked a few extra questions. urbanic@tg-login1:~> ssh instr003@bwbay.ncsa.illinois.edu Password: Last login: Mon Oct 28 01:13: from pool pitbpa.fios.verizon.net Access status: User Access Unrestricted. See for detailed information. Access by OTP or Two Factor Certificate Authority only. Use myproxy-logon -s tfca.ncsa.illinois.edu -p 7512 for gsissh access. gsissh or ssh -o PreferredAuthentications=keyboard-interactive for otp access. Last login: Sun Oct 27 01:47: from bwedge.ncsa.illinois.edu / _ )/ / / / / / / / _ / / // / -_) / / / _ `/ / -_) (_-< / /_/\_,_/\ / / /\_,_/\ /\ /_/ / / ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ Batch and Scheduler configuration. Queues: normal (default), high, low Features: "xe" (default), "xk", "x" (xe or xk non-specific) 30 min default wall time, -lnodes=x:ppn=y syntax supported. All SSH traffic on this system is monitored :36 Login node resource protection policy enabled. Processes with excessive cputime consumption will be terminated. notification will be sent to the process owner if this occurs :16 Blue Waters will be undergoing maintenance on Thu, October 31, 2:00pm through Fri, November 1, 3:00am. Queues will be drained. The scheduler, filesystems, and the near-line storage will be unavailable during this period. We also have another maintenance scheduled on Nov 18. A notification will be sent one week prior to that maintenance period :18 All filesystems have been returned for use. The Blue Waters system has resumed production. Questions? Mail help+bw@ncsa.illinois.edu to create a support ticket. For known issues: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ instr003@h2ologin1:~>

6 Editors For editors, we have several options: emacs vi jpico : use this if you aren t familiar with the others

7 Compiling We will be using standard Fortran and C compilers this week. They should look familiar. cc for C ftn for Fortran We will slightly prefer the PGI compilers (the Cray ones would also be fine). Note that on Blue Waters you would normally have to enable this compiler with module unload PrgEnv-cray module load PrgEnv-pgi I have put that in the.bashrc file that we will all start with.

8 PBS (You may already know this) No, we don t just do it to be control freaks Parallel machines are big 4D jigsaws that we need to fit together. If we didn t annoy you with this, other users would annoy you more. This will be in place on every serious GPU resource that you don t own.

9 PBS Interactive Mode To spare you the minor inconvenience of submitting jobs, we will use a feature of batch systems that is normally only used for debugging the interactive mode. We still need to request specific resources from the system scheduler, so our command would look like this: qsub -I -q training -l nodes=1:ppn=32 -l walltime=04:30:00 qsub Usually indicates a jobscript. -I This requests interactive mode. -q training Asks for the training queue. -I nodes Asks for 1 node with 16 cores. -l walltime.. Asks for 4.5 hours.

10 PBS Interactive Mode Because life is so very short, I have condensed this into a command called idev (really just an alias in your custom.bashrc). So, most of your interactions with the system today will look like: instr003@h2ologin1:~> idev Job submitted to account: jpk qsub: waiting for job nid11293 to start qsub: job nid11293 ready instr003@nid25357:~> Note that it may take a minute or two for this to happen. Once it does, things will be much more responsive. If you get confused as to whether you are on the login node or the compute node (this can easily happen), just look at your command prompt. Is it something like instr003@h2ologin2:~> or instr003@nid25357:~> The..login... or..nid give it away.

11 Running in Interactive Mode The great convenience of interactive mode is that you can just run your executable at will. You do not have to submit a job and wait for output. For the duration of the workshop we are basically giving you your own GPU node. You still need to run your executables with a platform specific command: aprun. It is quite simple in the manner that we will use it. aprun a.out This says that we wish to run the executable a.out on 1 node. and for OpenMP codes export OMP_NUM_THREADS=8 aprun d8 a.out This requests that we use 8 CPU threads on that node.

12 Multiple Sessions You are limited to one interactive mode session. As it takes a few minutes to start, you probably want to leave this session going. Under normal, non-workshop, circumstances, this would be expensive. However, there is no reason not to open other sessions (windows) to the login nodes for compiling and editing. You may find this convenient. Feel free to do so.

13 Our Setup For This Workshop After you copy the files from the instructor directory, you will have: /Exercises /Test /OpenMP laplace_serial.f90/c /Solutions /Examples /Prime

14 Let s get the boring stuff out of the way now. Preliminary Exercise 1) Log on as described earlier (use your password sheets). ssh username@bwbay.ncsa.illinois.edu 1) Copy the training exercise and solutions directories from the instructor directory to your home directory, and then copy the workshop shell script into your home directory. cp -r ~instr007/exercises. cp ~instr007/.bashrc. 2) Logout and back on again to activate this script. 3) Edit a file to make sure you can do so. Use emacs, vi or jpico (if the first two don t sound familiar). 4) Start an interactive session. idev 6) cd into your exercises/test directory and compile (C or Fortran) cd Exercises/Test cc test.c ftn test.f90 7) Run your program aprun a.out You should get back a message of Congratulations! (and some timing details)