Blue Gene/Q User Workshop. Debugging

Transcription

1 Blue Gene/Q User Workshop Debugging

2 Topics GDB Core Files Coreprocessor 2

3 GNU Debugger (GDB)

4 The GNU Debugger (GDB) The Blue Gene/Q system includes support for running GDB with applications that run on compute nodes. IBM provides a debug server called gdbserver. Each running instance of GDB is associated with one process or rank. GNU Debugger Interactive command line debugger A graphic interface also exists: the Data Display Debugger Pre-requisite Have to integrate symbols in your code Compile with flag -g Invocation Command Serial gdb <Binary> Distributed <MPIRUN Command> xterm -e gdb <Binary> Open as many Xterm windows as MPI jobs Reserved for small number of MPI tasks 4

5 GDB Optmized Code Optimization can change sequence of instructions, add or remove code, change variable data locations These changes make it difficult to associate the generated code with the original source statements. There are two options when debugging optimized code -g [level] O2 10 levels: lower preserve optimisation, sacrifice optimization At O2 -g default to -g 2 At O0 -g is 9 At O2 all levels provide full capability (although O3 without qhot may also work) At higher opt levels debug capability becomes limited -qoptdebug Produces files with optimized pseudocode that can be read by a debugger Extension *.optdebug Only at O3 or higher Still specify -g The debug information in the compiled file will point to the optdebug source 5

6 GDB how-to Overall process 1. Compile the program with option -g Additional option -qfullpath is recommended 2. Start the gdbtool server Option #1: launch user program with specific runjob command option --start-tool /sbin/gdbtool Option #2: launch additional gdbtool with running user program start_tool command 3. Display IP address of given rank with runjob or dump_proctable Debugging process listening on port by default To change port --listen_port=10001 in runjob "--rank=4 to specify a rank 4. Start GDB client powerpc64-bgq-linux-gdb Specify same binary as executed 5. Attach client remotely to selected process (gdb) target remote :10000 ( IP:port_number) 6

7 GDB how-to Overall process 6. Optional: To debug another rank in the same process, use the start_tool command to start another session to attach to the same process. Use the list_jobs command to find the job ID for the process. Use the dump_proctable command to find the IP address that is associated with the rank to be debugged /bgsys/drivers/ppcfloor/bin/dump_proctable --id job_id --rank 21 Run the start_tool command to start another gdbtool server /bgsys/drivers/ppcfloor/bin/start_tool --tool /sbin/gdbtool args "--rank=21 --listen_port=10002" --id Start another gdb client session to connect to the gdbtool session The tool_status command can be used to provide additional information when using the start_tool command 7

8 GDB how-to Start GDB client session Teacher]$ powerpc64-bgq-linux-gdb./poisson-default.exe GNU gdb (GDB) 7.2 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=powerpc64-linux-gnu --target=powerpc64-bgqlinux". For bug reporting instructions, please see: < Reading symbols from /gpfs/home/nt05984s/public/blue_gene_q_beta_testers_workshop/03- Hands_On/Poisson/Teacher/poisson-default.exe...done. (gdb) target remote :10000 Remote debugging using :10000 [Switching to Thread 1] 0x d0 in._start () (gdb) 8

9 The GNU Debugger (GDB): GDB Internal Commands Command Argument Explanation b <Function> Set a breakpoint to the specified function b <File>:<Line> Set a breakpoint on the specified source file / line run [<Binary Args>] Launch program execution p <Variable Name> Display the value or the specified variable n - Execute the next instruction c - Continue the program execution thread <no> Switch to thread <no> bt - Print a stack trace frame <no> Move to frame <n> in the stack info threads/locals/.. Print information on the given topic e.g. Current threads list (LINENUM) Show 10 lines of source around/after last list quit Quit the debugger and release the program 9

10 Core files

11 Core Files By default BG/Q core files are lightweight text files A lightweight core file contains a section for each thread Provide process information (torus coordinates, rank etc.) Details the state of each thread when the application exited Gives the stack trace of the thread The thread that crashed the program will have a line starting ***FAULT after the id section The stack contains a series of hexadecimal address describe function call chain until program exception +++STACK Frame Address Saved Link Reg fffff5ac c fffff5bc b fffff5c d fffff5d a fffff5e ec fffff5f a4d fffff c8 ---STACK The Saved Link Reg column contains the addresses 11

12 addr2line utility addr2line retrieves source code location from hexadecimal address Standard Linux command Usage: powerpc64-gnu-linux-addr2line -e <binary> <hexadecimal address> The application red book includes a script for extracting and formatting the stack-traces in core files Page 113 bgqtranslate.pl 12

13 Core Files Environemnt Variables BG_COREDUMPONEXIT=1 Always create a core file on program exit Useful for debugging application hangs BG_COREDUMPONERROR=1 Creates a corefile when an application calls exit() with a value other than 0 Useful if the value does not indicate the location or type of error BG_COREDUMPRANKS=[list] List of ranks that will produce core files when a job ends BG_COREDUMPBINARY= Create binary core-files (gdb) instead of lightweight core-files Specifying * - all ranks produce a binary core file Specifying a comma separated list of the ranks the given ranks create core files There are also numerous other less useful variables which control the content of lightweight files and their names 13

14 Allstack Based on Binary File Descriptor library (BFD) Usage allstacks -e your.exe [-c core.n] Extracts and translates the stack-trace in all lightweight core-files in a directory Optional you can specify one When run on multiple core files provides information on how many instances of each stack there are Can help with thread-synchronization and deadlock issues e.g. identify processes which exited in unexpected places 14

15 BG/Q Variable Monitoring Tool

16 Variable Monitoring Variable Monitoring Tool uses the debugging facility on BGQ to intercept preset target addresses. When any of the target addresses is accessed, the tool prints out information about the access. to diagnose memory access problems (e.g., memory leaks) There are four data-tier application (DAC) registers on one BGQ core with location number 0-3. Each DAC register can track one exact target address. Alternatively, two DAC registers (location 0 & 1, location 2 & 3) can be combined to track one target address range (i.e., target address with an address mask). init_monitor(); #pragma omp parallel private (i,j), shared (s,a) { set_monitor((uint64_t)&s[0],~0x7ful,0); start_monitor(); for(i=0;i<m;i++) { printf("i=%d\n",i); s[i]=0.0; for(j=0;j<n;j++) s[i]=s[i]+a[i][j]; } stop_monitor(); } Output: stdout[0]: WRITE action (DAC 0: ADDR: 0x1bffffba40, MASK: 0xffffffffffffff80) at instruction 0x10005a8 by OpenMP thread 3 stdout[0]: WRITE action (DAC 0: ADDR: 0x1bffffba40, MASK: 0xffffffffffffff80) at instruction 0x10005a8 by OpenMP thread 1 stdout[0]: WRITE action (DAC 0: ADDR: 0x1bffffba40, MASK: 0xffffffffffffff80) at instruction 0x10005a8 by OpenMP thread 2. 16

17 Coreprocessor

18 Coreprocessor Definition Basic parallel debugger that enables parallel debug of problems Uses the low-level hardware JTAG interface to read and organize hardware information instruction address registers (IAR), general purpose registers (GPR), special purpose registers (SPR), device control registers (DCR) Two features Processing of Compute Node core files Connection to running job No dependencies on the application code that is running on the node No special calls to be made / libraries to link to 18

19 Coreprocessor Features Functionalities Examine compute node core files Textual core files format is not understood by all debuggers Coreprocessor tool points out the node that is acting in a suspicious manner Probably caused the partition to dump Connect to Compute Nodes whenever nodes are non-responsive Other debuggers are not useful when a node is non-responsive Coreprocessor tool does not require a functional connection to a compute node through an I/O node Uses the JTAG connection to a node to collect debug information Features Can sort nodes based on their stack traceback and kernel status Helps isolate a failing or problem node quickly Supports stack dumping on a per processor basis Advantages Operating system can be completely dead and can still handle debug Quick isolation of nodes that are abnormal Scales to a full Blue Gene system 19

20 Coreprocessor GUI Requires X11 Forwarding Views to show commonality among state of nodes 20

21 Coreprocessor How-To Launch Coreprocessor GUI: /bgsys/drivers/ppcfloor/coreprocessor/bin/coreprocessor.pl Possible command-line options -a <block ID> Attach to running block -c <Core files directory> Load core files from the specified directory -b <ELF images for Compute Nodes> -b=<user Application Binary>:/bgsys/drivers/ppcfloor/cnk/bin/bgq_kernel.elf Specifies application binary and kernel image for symbols 2 options Inspection of Core Files (post-mortem) File >> Load Core Inspection of running jobs File >> Attach to Block Specify Block / User = job user / CNK Binary = <CNK Binary>:<Program Binary> 21

22 Coreprocessor How-To Main Features Condensed Traceback Select Grouping Mode >> Start Traceback (condensed) GUI Information Left number = depth within the stack Number in parentheses at the end of each line = number of nodes sharing the same stack frame Upper-right pane (Common nodes) = list of nodes that share that stack frame 22