Allinea DDT Debugger. Dan Mazur, McGill HPC March 5,

Transcription

1 Allinea DDT Debugger Dan Mazur, McGill HPC March 5,

2 Outline Introduction and motivation Guillimin login and DDT configuration Compiling for a debugger Launching debugging jobs Controlling program execution Stepping, breakpoints, watchpoints, tracepoints Viewing variable contents Debugging parallel programs Effective debugging strategies Practice 3

3 Why use a debugger? We want to debug efficiently to reduce the total development time: Inefficient Coding More efficient Coding Debugging Debugging Time 4

4 Why a debugger? Greatly reduces the amount of time required for debugging For some projects, the time spent debugging can be greater than the time spent coding! If you program, you should learn how to debug effectively Why not just use printf() statements? May create much more data than is useful (parallel applications) debugging is fixed at compile time frequent task switching (thinking, editing, compiling, running, repeat) might introduce bugs (constantly changing/recompiling code) might forget to remove debugging output slower (typing, compiling, runtime) misleading (buffered output, changes timing and memory) 5

5 What does it do? All symbolic debuggers have these basic features: Step through code line-by-line Inspect variables Run and pause the code according to specified conditions (e.g. line numbers, function calls, variable changes) Catches signals such as SIGSEGV (Segmentation Fault) 6

6 What does it not do? Debuggers do not: Turn defective code into working code Find or fix bugs Understand the programmers' intentions Debugger Better name: 'Program Inspector' 7

7 Exercise 1: Login and configuration Login to Guillimin with X11 forwarding $ ssh -X class##@guillimin.hpc.mcgill.ca ## same as csuser## numbers password is written on slips of paper handed out by instructor Copy the workshop materials to your home directory ~/. Load the environment modules $ cp -R /software/workshop/ddt/* $ module load DDT ifort_icc openmpi Launch DDT $ ddt & 8

8 Compiling for debuggers Most programs Binary format Optimized by compiler No longer described by original code Programs for debugging Special compiler flag (-g) Programs keep a link to their source code Debugger functions are limited without this link 9

9 Launching a Job 10

10 Launching a Job 11

11 Launching a Job 12

12 Exercise 2: Compile and launch Compile the serial program serial.c for debugging $ icc -g -o serial serial.c Launch the program in DDT with the integer argument 0 (zero) Please follow along as I discuss the various functions 13

13 Template File DDT uses a template file for job submission default: /sb/software/tools/ddt/templates/guillimin.qtf To have more control over qsub script Copy this file to your ~/.allinea directory Modify the file Point DDT to your own copy in options>job submission 14

14 Source Code Variable Monitoring File Information Execution Controls Process Focus Process Information 15

15 Breakpoints Breakpoints indicate where the program should pause execution for inspection Select a line of code, right click, select 'add breakpoint' At a certain line of code or function call Or, click on the 'add breakpoint' icon Right click on a breakpoint to select 'edit breakpoint' or 'delete breakpoint' Conditional breakpoints (break 'if' some condition) 16

16 Breakpoints Where to break? Which processes? Useful for inspecting large loops Conditional breakpoints 17

17 Watchpoints Watchpoints pause execution when a certain memory location is changed Right click on a variable in the variable monitoring area to set a watchpoint 18

18 Tracepoints Log variables without pausing execution Like printf(), but more flexible Right-click on a line of code to set Right-click again to edit or delete 19

19 Tracepoint When to log? Which processes? Which variables? Useful for inspecting large loops Conditional tracepoints 20

20 Stack Trace Displays sequence of function calls How did we get here? 21

21 Execution Controls A B C D E F G A - Play (F9), Continue until next breakpoint or pause B - Pause (F10), Interrupt program execution C - Add breakpoint D - Step into (F5), Enter next function and pause inside E - Step over (F8), Execute the next function and pause outside F - Step out (F6), Exit the current function and pause outside G - Run to line, Play but pause on a certain line 23

22 Exercise 3: Basic Functions Try each basic function at least once in serial.c and understand how it works Breakpoints Watchpoints Tracepoints Play Step into Step over Step out 24

23 Exercise 4: Inspecting Variables Compile variables.c and launch with ddt $ icc -g -o variables variables.c Inspect the contents of each variable Change the contents of one of the variables during execution (array, for example) 25

24 Exercise 4: Inspecting Variables 26

25 Exercise 4: Inspecting Variables 27

26 Exercise 4: Inspecting Variables 28

27 Exercise 4: Inspecting Variables 29

28 Exercise 4: Inspecting Variables Evaluate combinations of variables Change a variable 30

29 Exercise 5: Multithreaded Debugging Compile omp_mm.c/.f90 with openmp for debugging Launch with DDT $ icc -g -openmp -o mm omp-mm.c Make sure to select OpenMP with 4 threads 31

30 Multi-threaded Debugging 32

31 Multi-threaded Debugging Select which thread you are debugging (focus) -Variable output will be thread specific -Program execution can be per thread or all together Stack trace shows you where each thread is in the program execution 33

32 34

33 Multi-process Debugging Compile pi.c $ mpicc -g -o pi pi.c Launch in DDT with 4 mpi processes Use command line argument:

34 Process Groups Create groups of threads Step by group 36

35 Passwordless Login DDT can attach to running processes requires communication between nodes asks for password each time We will set up ssh keys to avoid the passwords 37

36 Public-key Authentication Public Key ~/.ssh/id_rsa.pub Analogous to the lock Private Key ~/.ssh/id_rsa Analogous to the key 38

37 SSH Keys Local system Remote system Step 1: On the local system, we generate a pair of keys, one private (id_rsa) and one public (id_rsa.pub). 39

38 SSH Keys Local system ~/.ssh/id_rsa ~/.ssh/id_rsa.pub Remote system Step 1: On the local system, we generate a pair of keys, one private (id_rsa) and one public (id_rsa.pub). 40

39 SSH Keys Local system ~/.ssh/id_rsa ~/.ssh/id_rsa.pub Remote system Step 1: On the local system, we generate a pair of keys, one private (id_rsa) and one public (id_rsa.pub). Step 2: We copy the public key to ~/.ssh/authorized_keys on the remote server 41

40 SSH Keys Local system ~/.ssh/id_rsa ~/.ssh/id_rsa.pub Remote system Copy ~/.ssh/authorized_keys Step 1: On the local system, we generate a pair of keys, one private (id_rsa) and one public (id_rsa.pub). Step 2: We copy the public key to ~/.ssh/authorized_keys on the remote server 42

41 SSH Keys Local system ~/.ssh/id_rsa ~/.ssh/id_rsa.pub Matching pair? Remote system ~/.ssh/authorized_keys Step 1: On the local system, we generate a pair of keys, one private (id_rsa) and one public (id_rsa.pub). Step 2: We copy the public key to ~/.ssh/authorized_keys on the remote system Step 3: For authorization, the public key on the remote system is matched against the private key on the local system to ensure they are a valid pair 43

42 SSH Keys Local system ~/.ssh/id_rsa ~/.ssh/id_rsa.pub Matching pair? Remote system ~/.ssh/authorized_keys We can send our public key to any computer we want to log in to. We do not need to keep it secret because ownership of the matching private key is needed for authentication. Use a passphrase. Otherwise, if your local system is compromised, all of your accounts on remote systems are also compromised. The private key and the passphrase are secret, like passwords. Never share your private key with anyone. 44

43 Setting up ssh keys ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/class59/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/class59/.ssh/id_rsa. Your public key has been saved in /home/class59/.ssh/id_rsa.pub. The key fingerprint is: 6a:c1:fe:42:dd:43:56:78:4c:c3:67:45:6d:69:e6:97 The key's randomart image is:... ssh-copy-id -i ~/.ssh/id_rsa.pub eval $(ssh-agent -s) ssh-add ~/.ssh/id_rsa Generate a pair (public, private) of keys Use default location Copy your public key into authorized_keys file Save your passphrase for this session Enter passphrase for /home/class59/.ssh/id_rsa: Identity added: /home/class59/.ssh/id_rsa (/home/class59/.ssh/id_rsa) [class59@lg-1r17-n01.ssh]$ ddt & 45

44 Exercise 6: Attaching Compile and launch the job mpi_hang $ mpicc -g -o mpi_hang mpi_hang.c $ qsub mpi_hang.sh Bug! This program hangs until it runs out of walltime When the job is running, put the list of nodes into ~/.allinea/nodes (e.g. JobID=123456) (Re-)launch DDT (DDT loads the node list at launch) $ ddt_gen_nodelist ddt & Select Attach to a running program Goal is to inspect the bug, actually fixing it is bonus 46

45 Exercise 6: Attaching 47

46 Exercise 6: Attaching 48

47 Exercise 6: Attaching 49

48 Exercise 6: Attaching The reason this code hangs is the line: tag = rank; rank is the MPI rank The tag must match between MPI_Send() and MPI_Recv() Both processes have sent messages and are stuck waiting in MPI_Recv() for a matching tag that will never come 50

49 Exercise 7: Signals Compile and run cstart.c on the command line $ icc -g -o cstart cstart.c $./cstart one two three Will crash (Segmentation Fault) if any of the command line arguments are: crash memcrash overflow Inspect the code for each case 51

50 Effective Debugging 0: Planning and proper development strategies (future workshop?) 1: Know exactly i) what it is supposed to do, ii) what it actually does, and iii) how you know that it is broken 2: Look for simple answers: Is your computer plugged in? turned on? Correct version? Correct input data? 3: Focus on the problem Isolate the faulty subsystem precisely Reduce the problem to the simplest possible test case 52

51 Effective Debugging 4: Scientific Method Invent a hypothesis to explain fault Design an experiment to test the hypothesis Repeat 5: Change one variable at a time (experimental controls) 6: Keep a good lab book (what results did each experiment produce?) 7: Avoid: Guesswork (do the experiments) Random trial and error Laying blame (libraries, compilers, OS, collaborators) Band-aids and workarounds Quitting as soon as its 'working' (If you didn't fix it, it isn't fixed) 53

52 MPI Debugging Common MPI Deadlocks Only one process calls a collective communication function All processes blocking receive before any process sends A message never gets sent Process tries to receive data from itself 54

53 MPI Debugging Other common MPI bugs Send/Recv type mismatch (eg. MPI_INT matched with MPI_FLOAT) Mixed-up parameters to MPI functions 55

54 MPI Debugging Collective communications are less error-prone than point-to-point Start with smallest possible number of processes (i.e. 1) and scale up slowly Start with smallest possible problem size and scale up slowly printf output is chronological for a process, but not between processes Compare sent messages and received messages 56

55 Getting Help 57

56 Exercise 8: Debugging Practice Please debug as many of the bugged practice problems as time allows If you have your own code you would like to debug, or inspect you may do that now Practice problems: InsLotsOfErrors.c ( Lnk.c ( programs in mpi_bug folder (Lawrence Livermore MPI Tutorial) programs in omp_bug folder (Lawrence Livermore OpenMP Tutorial) 58