IBM PSSC Montpellier Customer Center. Content

Transcription

1 Content IBM PSSC Montpellier Customer Center Standard Tools Compiler Options GDB IBM System Blue Gene/P Specifics Core Files + addr2line Coreprocessor Supported Commercial Software TotalView Debugger Allinea DDT 1 PSSC Montpellier Deep Computing Team

2 Preamble IBM PSSC Montpellier Customer Center Blue Gene specific environment makes it sometimes a bit harder to debug than on a regular platform Cross-compilation Workload Scheduler for job submission No interactive access to computation nodes Potential alternative Compile, run and debug on Frontend Node Same compilation environment 2 PSSC Montpellier Deep Computing Team

3 Compiler Options -C, -qcheck Checks each reference to an array element, array section, or character substring to ensure the reference stays within the defined bounds of the entity -g, -qdbg Generates debug information for use by a symbolic debugger -qdpcl Generates symbols that tools based on the IBM Dynamic Probe Class Library (DPCL) can use to see the structure of an executable file -qextchk Generates information to perform type-checking at both compile and link time to ensure consistency between declarations and references -qflttrap Determines what types of floating-point exception conditions to detect at run time -qformat (XLC) Warns of possible problems with string input and output format specifications -qinitauto Initializes uninitialized automatic variables to a specific value, for debugging purposes -qkeepparm When used with -O2 or higher optimization, specifies whether function parameters are stored on the stack -qobject Specifies whether to produce an object file or to stop immediately after checking the syntax of the source files -qoptdebug When used with high levels of optimization, produces files containing optimized pseudocode that can be read by a debugger -qxflag=dvz Causes the compiler to generate code to detect floating-point divide-by-zero operations 3 PSSC Montpellier Deep Computing Team

4 GNU Project Debugger (GDB) Simple GNU GPL debug server Debugging Session How-To Binary must be compiled and linked with options: -g -qfullpath GDB server must be started before the application mpirun command has a special option: -start_gdbserver Execution will pause once user program has been loaded onto the Compute Nodes Limitations Interactive mode One GDB Instance per MPI Task Multiple GDB clients required to debug multiple CNs at the same time Limited subset of primitives (however enough to be useful) Standard Linux GDB client Not aware of Power 450 Double FPU architecture Standard System Location /bgsys/drivers/ppcfloor/gnu-linux/bin 4 PSSC Montpellier Deep Computing Team

5 Using GDB: Example GDB Server mpirun -exe /home/bgpuser/hello_world/hello_world -cwd /home/bgpuser/hello_world -mode SMP -np 64 -start_gdbserver /bgsys/drivers/ppcfloor/ramdisk/sbin/gdbserver -verbose 1 Dump all the node IP Addresses / Ports per MPI rank dump_proctable OR Type directly the rank you want to attach to GDB client 2 MPI Rank 2: Connect to :7302 > press enter after entering the target command in the second window GDB Client (One client instance per Compute Node) cd /home/bgpuser/hello_world /bgsys/drivers/ppcfloor/gnu-linux/bin/gdb fonction (gdb) > target remote <IP Address>:<Port> (gdb) > step 5 PSSC Montpellier Deep Computing Team

6 Core Files Principles Standard Blue Gene Core Files On Blue Gene/P, core files are lightweight text files by default For performance reason Hexadecimal addresses in section STACK describe function call chain until program exception Section delimited by tags: +++STACK / ---STACK Binary Core Files Producing binary core files is nevertheless possible New feature in V1R4M0 Through environment variable BG_COREDUMP_BINARY BG_COREDUMP_BINARY = <MPI Rank> Binary core files can be used with GDB 6 PSSC Montpellier Deep Computing Team

7 Core Files Production At Program Exit Time Environment variables allow core files creation BG_COREDUMPONEXIT Creates a core file when the application exits Useful when the application performed an exit() operation and the cause and location of the exit() is not known BG_COREDUMPONERROR Creates a core file when the application exits with a non-zero exit status Useful when the application performed an exit(1) operation and the cause and location of the exit(1) is not known Compilation with g for addresses further interpretation Many environment variables available for core files control Cf. Redbook Application Development / Compute Node Kernel environment variables 7 PSSC Montpellier Deep Computing Team

8 addr2line Utility addr2line retrieves source code location from hexadecimal address Standard Linux command Usage addr2line -e <Binary> <Hexadecimal Address> Standard System Location /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-addr2line Example addr2line -e tokenizer 0x010015c8 /gpfs/home/ibm/codes/tokenizer/tokenizer.c:24 8 PSSC Montpellier Deep Computing Team

9 Coreprocessor Definition Basic parallel debugger that enables parallel debug of problems Uses the low-level hardware JTAG interface to read and organize hardware information instruction address registers (IAR), general purpose registers (GPR), special purpose registers (SPR), device control registers (DCR) Two features Processing of compute node core files Connection to running job No dependencies on the application code that is running on the node No special calls to be made / libraries to link to 9 PSSC Montpellier Deep Computing Team

10 Coreprocessor Features Functionalities Examine compute node core files Textual core files format is not understood by all debuggers Coreprocessor tool points out the node that is acting in a suspicious manner Probably caused the partition to dump Connect to Compute Nodes whenever nodes are non-responsive Other debuggers are not useful when a node is non-responsive Coreprocessor tool does not require a functional connection to a compute node through an I/O node Uses the JTAG connection to a node to collect debug information Features Can sort nodes based on their stack traceback and kernel status Helps isolate a failing or problem node quickly Supports stack dumping on a per processor basis Advantages Operating system can be completely dead and can still handle debug Quick isolation of nodes that are abnormal Scales to a full Blue Gene/P system 10 PSSC Montpellier Deep Computing Team

11 Coreprocessor GUI Requires X11 Forwarding Views to show commonality among state of nodes 11 PSSC Montpellier Deep Computing Team

12 Coreprocessor How-To Launch Core Processor GUI: /bgsys/drivers/ppcfloor/tools/coreprocessor/coreprocessor.pl Possible command-line options -a <Block ID> Attach to running block -c <Core Files Directory> Load core files from the specified directory -b <ELF Images for Compute Nodes> -b=<user Application Binary>:/bgsys/drivers/ppcfloor/cnk/bgp_kernel.cnk.elf Specifies application binary and kernel image for symbols 2 options Inspection of Core Files (post-mortem) File >> Load Core Inspection of running jobs File >> Attach to Block Specify Block / User = job user / CNK Binary = <CNK Binary>:<Program Binary> 12 PSSC Montpellier Deep Computing Team

13 Coreprocessor How-To Main Features Condensed Traceback Select Grouping Mode >> Start Traceback (condensed) GUI Information Left number = depth within the stack Number in parentheses at the end of each line = number of nodes sharing the same stack frame Upper-right pane (Common nodes) = list of nodes that share that stack frame 13 PSSC Montpellier Deep Computing Team

14 Traceback Purpose Produce a traceback of the code at exit time Traceback Interpretation Routine and source line addresses can be decoded with addr2line utility Usage Exit Wrapper Wraps standard exits functions in order to produce traceback Integrated into binary at link time #include <execinfo.h> #define MAX_CALL_DEPTH 50 void traceback() { void * addresses[max_call_depth] int depth = backtrace(addresses, MAX_CALL_DEPTH) for int level = 1; level < depth 1; level++) printf( %#10.8x\n, addresses[level]); } 14 PSSC Montpellier Deep Computing Team

15 Commercial Software Allinea DDT DDT = Distributed Debugging Tool Link TotalView Sophisticated software debugger product of TotalView Technologies Used for debugging and analyzing both serial and parallel programs Especially designed for use with complex, multi-process and/or multi-threaded applications Probably the most popular HPC debugger to date Link 15 PSSC Montpellier Deep Computing Team