Guillimin HPC Users Meeting July 14, 2016

Transcription

1 Guillimin HPC Users Meeting July 14, 2016 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

2 Outline Compute Canada News System Status Software Updates Training News Special Topic Debugging Tools Available on Guillimin 2

3 Compute Canada News 2016 C.C. Awards of Excellence Results Three members of the Calcul Québec team were honored at the C. C. Awards ceremony during the CANHEIT HPCS 2016 conference on June 20th Maxime Boissonneault (Team Choice Award), Minh Nghia Nguyen (Quality of the support) and David Morais (Galaxy integration) Visit the CC page Visit the CQ page 3

4 Storage Status File-system performance issues on /sb on June 21: Related to an IO module in one specific enclosure Problem started at 10:00am Normal performance restored at 2:00pm We temporarily limited the job activity in the cluster until the morning of June 22, to reduce load on /sb Replacement of faulty IO module on June 28: No new jobs intentionally started: 8:00 am - 5:15pm IO module can be changed live, but caused issues with access to some file system drive pools Problem accessing login nodes and /sb Many dependent VMs file systems became read-only Storage issues resolved, back to healthy state at 2:00pm Full performance restored June 29 at 1:30pm 4

5 Storage Status Space Management /gs is quite full: 94% used, 210 TB left (as of July 13) For better space management we continue to progressively migrate cold data from disk to tape Metadata remains on disk Users can still access their files through usual methods, but with an increased latency *New* prquota version: reports data migrated to tape and total disk+tape usage. Groups should not go beyond 100% of their allocation Storage space is a precious resource - manage it wisely! delete temporary files, compress large files not frequently accessed, tar many smaller files into collections, 5

6 Scheduler Status Very long list of jobs for SW2 nodes Usually: -l nodes=n:ppn=16 -l pmem=3700m If walltime > 72 hours: You have to wait for SW2 nodes Or you may resubmit with ppn=12 If walltime <= 72 hours: Any ppn=16 job will run on either SW2 or LM2 nodes Except, of course, for k20 and phi jobs You may consider using: -l procs=48 With procs=p, jobs will run on either SW2, LM or LM2 nodes 6

7 Software Update About the Lmod/EasyBuild based module structure: Default since March 22, 2016 New Installations MATLAB Distributed Computing Server (MDCS) license file (for all users, 64 licenses): R201{2, 3, 4, 5}{a, b} cudnn/5.0-ga (CUDA Deep Neural Network library) Eigen/3.2.8 (C++ templates for linear algebra) OpenCV/ (iomkl/2015b) (Computer Vision/Machine Learning) AllineaForge/6.0.6 (Allinea DDT debugger (full) and Map profiler (trial)). 7

8 Training News See Training and Outreach at for our calendar of training and workshops for 2016 and for links to registration pages Upcoming events: calculquebec.eventbrite.ca Suggestions for training? Please let us know! Training break, July and August Recently completed: June 21 - Introduction to GPU (CUDA) (McGill) June 22 - Simplifying Life with GPU Libraries (U. de Montréal) All materials from previous workshops are available online: wiki.calculquebec.ca/w/formations/en 8

9 User Feedback and Discussion Questions? Comments? We value your feedback. Contact us at: Guillimin Operational News for Users Status Pages (all CQ systems) Follow us on Twitter 9

10 Debugging Tools Available on Guillimin Based on DDT workshop, Dan Mazur, March 2015 July 14, 2016 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

11 Outline Introduction and motivation Guillimin login and DDT configuration Compiling for a debugger Launching debugging jobs Controlling program execution Stepping, breakpoints, watchpoints, tracepoints Viewing variable contents Debugging parallel programs 11

12 Why use a debugger? We want to debug efficiently to reduce the total development time: Inefficient Coding Debugging More efficient Coding Debugging Time 12

13 Why use a debugger? Greatly reduces the amount of time required for debugging For some projects, the time spent debugging can be greater than the time spent coding! If you program, you should learn how to debug effectively Why not just use printf() statements? May create much more data than is useful (parallel applications) Debugging is fixed at compile time Frequent task switching (thinking, editing, compiling, running, repeat) Might introduce bugs (constantly changing/recompiling code) Might forget to remove debugging output Slower (typing, compiling, runtime) Misleading (buffered output, changes timing and memory) 13

14 What does it do? Examples: GDB (command line and TUI), ddd, idb (GUI) HPC-oriented: DDT (Guillimin), TotalView (other clusters) Other useful tool: valgrind (memory leaks, races) All symbolic debuggers have these basic features: Step through code line-by-line Inspect variables Run and pause the code according to specified conditions (e.g. line numbers, function calls, variable changes) Catch signals such as SIGSEGV (Segmentation Fault) 14

15 What does it not do? Debuggers do not: Turn defective code into working code Find or fix bugs Understand the programmers' intentions Debugger Better name: 'Program Inspector' 15

16 Login and Configuration Login to Guillimin with X11 forwarding $ ssh -X username@guillimin.hpc.mcgill.ca Copy the workshop materials to your home directory $ mkdir ~/ddt $ cd ~/ddt $ cp -R /software/workshop/ddt/*./ Load the environment modules $ module load AlineaForge/6.0.6 iomkl/2015b Launch DDT $ ddt & 16

17 Compiling for Debuggers Most programs Binary format Optimized by compiler No longer described by original code Programs for debugging Special compiler flag (-g) Programs keep a link to their source code Debugger functions are limited without this link Example: segmentation fault in crash.c int main(void) { int a[1000]; a[ ] = 0; } icc -g crash.c -o crash;./crash; gdb crash 17

18 Launching a Job 18

19 Launching a Job 19

20 Launching a Job 20

21 Compile and Launch Compile the serial program serial.c for debugging $ icc -g -o serial serial.c Launch the program in DDT with the integer argument 0 (zero) Please follow along as I discuss the various functions 21

22 Template File DDT uses a template file for job submission default: /software/tools/ddt/templates/guillimin.qtf To have more control over qsub script Copy this file to your ~/.allinea directory Modify the file Point DDT to your own copy in options>job submission 22

23 Breakpoints Breakpoints indicate where the program should pause execution for inspection At a certain line of code or function call Select a line of code, right click, select 'add breakpoint' Or, click on the 'add breakpoint' icon Right click on a breakpoint to select 'edit breakpoint' or 'delete breakpoint' Conditional breakpoints (break 'if' some condition) 23

24 Breakpoints Where to break? Which processes? Useful for inspecting large loops Conditional breakpoints 24

25 Watchpoints Watchpoints pause execution when a certain memory location is changed Right click on a variable in the variable monitoring area to set a watchpoint 25

26 Tracepoints Log variables without pausing execution Like printf(), but more flexible Right-click on a line of code to set Right-click again to edit or delete 26

27 Tracepoint When to log? Which processes? Which variables? Useful for inspecting large loops Conditional tracepoints 27

28 Stack Trace Displays sequence of function calls How did we get here? 28

29 Multi-threaded Debugging Compile omp_mm.c/.f90 with openmp for debugging $ icc -g -fopenmp -o mm omp-mm.c Launch with DDT Make sure to select OpenMP with 4 threads 29

30 Multi-threaded Debugging 30

31 Multi-threaded Debugging Select which thread you are debugging (focus) - Variable output will be thread specific - Program execution can be per thread or all together Stack trace shows you where each thread is in the program execution 31

32 32

33 Multi-process Debugging Compile pi.c $ mpicc -g -o pi pi.c Launch in DDT with 4 mpi processes Use command line argument: 100 Hint: use qsub -X -I -q debug -l nodes=1:ppn=4 for interactive session on debug queue. 33

34 Conclusion Documentation: Goes into more detail (attaching, signals, exercises) For any other question: 34