Hands-on Workshop on How To Debug Codes at the Institute H. Birali Runesha, Shuxia Zhang and Ben Lynch (612) 626 0802 (help) help@msi.umn.edu October 13, 2005 Outline Debuggers at the Institute Totalview Starting TotalView The TotalView GUI and basic windows Data management and visualization Breakpoints and Action points Parallel debugging: MPI, OpenMP, etc. 1
Debuggers at the Institute Write statements dbx pdbx Xpdbx xldb Pedb GNU gdb CVD Intel IDB DDD Totalview Other tools: Compiler options, profilers, hardware counters dbx dbx is a command line debugger for serial and parallel Fortran and C programs respectively Most useful Commands: stop at linenumber # set a breakpoint at the given linenumber stop in myroutine # set a breakpoint at the given routine run < infile # run your program, reading from the file infile next # to execute the next step of your code print var # see the value of a given variable whatis var # determine the data type of a given variable cont # continue execution from where it stopped Compile your program with g option Start dbx with or without the executable as an argument. More info: man dbx, on dbx prompt use help Available: unix systems: IBM, SGI and SUN 2
pdbx Command-line parallel debugger built on dbx IBM debugger for parallel programs on AIX systems and accepts all flags supported by POE Supports C, C++, Fortran 77, Fortran 90 programs Compile the code with -g flag Setup POE environment variables, e.g.: setenv MP_BUFFER_MEM 28MB setenv MP_EAGER_LIMIT 4090kb Run pdbx: pdbx a.out [program_options] [poe options] or pdbx -a poe process id [limited poe options] Available: IBM Power3 and Power4 More information: man pdbx xpdbx Parallel debugger with graphical interface Load executable after starting xpdbx. Use the File menu Use X resources to customize the xpdbx window The default resources for xpdbx are listed in /usr/lib/x11/app-defaults/xpdbx Needs the rmpool option on the IBM SP/Power4 for an interactive run Can set break/trace points 3
xldb Serial and multi-thread debugger with graphical interface xldb a.out Window pops up with source, etc Group of blue bars at the top right Click on bar to open window To minimize window, click on bar at top to get menu, click on minimize To set breakpoint, click on source line To navigate, see commands window pedb Graphical user interface based parallel debugger for Fortran and C programs Same as xpdbx pedb a.out <poe options> Window pops up with source, etc To set breakpoint, double-click on source line. To delete breakpoint, right-click icon in left margin, select delete To navigate, use buttons below source listing Step over step into step return Continue halt play stop tasks (processes for MPI) are chosen using buttons in task window Use Find to find a specific variable 4
GNU gdb gdb is a command line debugger for serial programs Support C, C++, and Fortran77 programs Available: linux and unix systems Compile the code : g77 g my.f or gcc g my.c Start debugging: gdb./a.out or gdb./a.out core # to include the core or gdb./a.out 1234 # to attach GDB to process 1234 Most useful Commands: quit: Exit from GDB. break [file:]function: Set a breakpoint at function in file. run: Start your program bt: Backtrace: display the program stack. print expr: Display the value of an expression. c: Continue running your program (after stopping) stop:set a breakpoint at line number or routine Getting help: (gdb) help http://sources.redhat.com/gdb/onlinedocs/gdb _toc.html man gdb 5
CVD CVD - Invokes the WorkShop Debugger a source-level debugging tool with GUI set various types of breakpoints, watchpoints, and so on. view variables, expressions, structures, arrays, call tree debug Ada, C, C++, Fortran 77, and Fortran 90 programs. Available on SGI system running Irix Operating systems (SGI workstations and Origin) To invoke the Debugger for a Fortran 77 program: $ f77 -g -o myprog myprog.f $ cvd myprog # serial job or $ cvd myprog core # serial job with core dump or $ cvd mpirun -args -np 2 a.out # For an mpi job More information: Manpage: man cvd http://techpubs.sgi.com/library/tpl/cgi-bin/init.cgi CVD and Other Workshop Tools: Build Analyzer (cvbuild) - Invokes the Build Analyzer that provides a graphical display of build dependency information. Parallel Analyzer (cvpav) - Invokes the Parallel Analyzer that presents information about MP Fortran programs. Performance Analyzer (cvperf) - Invokes the Performance Analyzer that lets you specify the name of a performance task to enable proper data collection for your experiment. Static Analyzer (cvstatic) - Invokes the Static Analyzer that lets you display source code information about your code and facilitates the setting of breakpoints. Tester (cvxcov) - Invokes the test coverage tool that lets you perform dynamic test coverage over any set of tests. 6
Intel IDB Intel debugger are part of the compiler installation Support C, C++, Fortran 77/90 Both dbx (default) and gdb like interfaces Documentation: http://www.intel.com/software/products/compilers/docs/linux/idb_m anual_l.html Debugging of optimized code generated by intel compilers does not work Does not support multithreaded/multiprocess applications. claims do support mpich programs (untested) Available on the SGI Altix and systems with Intel compilers DDD Graphical debugger Part of Redhat distribution Works best with gdb Can hook in idb via: ddd --debugger idb -- dbx./a.out Multiple panes for Data/array browsing Source code Assembler code Command line interface Use help menu for details 7
Other Debuggers Portland Group PGDBG graphical debugger LF95: Fujitsu debugger FDB Write statements! Editors Totalview 8
Introduction Debugger for Unix/Linux platform Supports Fortran, C/C++ programs Serial, multi-threaded, parallel programs employing multiple processors MPI, MPICH, OpenMP, PVM, SHMEM, Global arrays, and UPC User interface and debugging style remain uniform and consistent across platforms and programming models. Starting TotalView Compile with g option Starting Totalview on MSI machines: module load totalview On a new process: % totalview a.out -a <arguments to a.out> On a core file: % totalview a.out core To attach to a running process: % totalview 9
o File > New Program dialog Box Short cut: ctrl-n F1 function key tells Totalview to display help info about the window TotalView Windows Root Window Unattached Processes Window Process Window Data Windows Test drive: Menu driven Alternative keyboard shortcuts for those who prefer them All 3 mouse buttons are used 10
Root Window Process/thread status Process name Number of threads Expand list Process Window Stack Trace pane Local variables for the selected frame Source pane Thread pane Action Points pane 11
Source Pane Gridded box is a possible site for a breakpoint Current function and source file Select to set onecurrent point of execution Dive on a source word to get more information Select a line to use Run to selection command Select or dive on a line number to set an action point Stack Trace Pane Select a line to choose a new stack frame Just dive on a routine and TotalView shows the routine in the Source Pane and its variables in the Stack Frame Pane. Stack frame value 12
Stack Frame Pane Names of arguments Names of function s local variables Stack Frame pane in the process window contains current call stack Values of variables Dive on a variable to bring up a data pane (and follow a pointer) Starting, Stopping, & Restarting Your Program Method 1 Set a breakpoint Select Go on the icon bar The program starts executing and stop when it reaches the breakpoint Method 2 Select step Totalview starts and then stops it immediately before the first statement in your main function To stop a running program, select halt To restart a program, select the Group > Restart command 13
Stepping through a Program Use the Step and Next commands. Step and Next both execute the current line. The difference between them is that if the line has a function call, Step dives into the function while Next executes it. If you want to get to a line without individually stepping each intervening line, select the line, and then select Run To. Stepping Out of a Function If you've stepped into a function and want to pop out to the statement that called it, select the Out command. Data Management 14
Displaying Data Seeing a value of a variable Method 1 Dive on a variable displayed in the Source Pane. The variable appears in a Variable Window. Method 2 Scroll to the variable in the Stack Frame Pane. If you want to display a compound variable like an array or structure, dive on it. TotalView responds by displaying information in a Variable Window. Method 3 Use the View > Lookup Variable command. The variable can be global or local. If you've defined the same variable in more than one routine, TotalView displays the variable contained in the current stack frame. Variable Window Array Data Window Array has an actual type as well as a declaration type Arrays have a slice field that you can edit to specify the dimensions to display You can update data by Selecting (left mouse button) the value and editing it 15
Array Slice An array slice indicates which portion of an array you want to see. I.e.: see elements 3 to 8 Fortran example- TotalView displays a one-dimensional array by using (:). It uses (:,:) if it is displaying a two-dimensional array, and so on Data Array Filtering Go to the Filter line and type a filter expression You can specify the elements of an array you re interested in This example requests all elements with values >0.8 16
Data Array Filtering (cont.) Filter: 5:998 will display elements between 5 and 998 Filter:== $nan Filter:== $inf In an array data pane select Visualize from the menu The Visualizer is automatically started Arrays of rank up to two may be visualized Unlike data panes visualizations are not automatically updated $visualize in EVAL points for animated display Visualization 17
Action Points Action Point Types Action point is a generic name for several types of breakpoints: Breakpoint stops execution of processes and threads that reach it Barrier breakpoint holds each process that reaches it until all processes in the group reach it Conditional breakpoint stops code if condition is met Evaluation point is point at which user-defined code fragments get executed Data watchpoint 18
Setting a Breakpoint at a Function Method 1: If you can see the function, click on a line number within the function. Method 2: Use the View > Lookup Function command to find the function. After the function displays, select a line number. Method 3: Select Action Points > At Location and then type the function's name in the dialog box. Setting a Data Watchpoint A data watchpoint is a type of breakpoint that monitors a variable's value. Unlike other action points, it isn't set on a source line. Instead, it's set on the memory location where your program stores the variable. When the contents of this memory location change, TotalView stops execution and displays the source line that changed it. This means you should only set watchpoints on global variables or on malloc'd memory. Set a watchpoint by first diving on the variable to display its Variable Window, and then select the Tools > Watchpoint command. 19
Printing Something at a Breakpoint After creating a breakpoint, right-click on the sign, and then select Properties from the popup menu. In the Properties dialog box, select Evaluate and then type a print statement. For example: printf("the value of foo divided by 20 is: %d\n",foo/20); When code is associated with a breakpoint, the breakpoint is called an eval point. Stopping Your Program Using a Condition Create an eval point (this was discussed in the previous section), entering a condition instead of a statement. For example, here's a condition that stops execution when the value of the variable i in the eval point's line is equal to 30: if (i == 30) $stop One of the big differences between an eval point and a breakpoint is that execution doesn't stop unless you tell it to. That is, if you don't add a $stop to your eval point, TotalView just continues executing. 20
Stopping Your Program Every "x" Times TotalView Executes a Line Create an eval point that uses the $count statement to tell TotalView how many times the line should execute. For example: $count 20 $count is another of TotalView's built-in functions. $stop is implied when you use this statement. Saving Your Action Points You usually want this to happen automatically and, unless someone has changed things, TotalView automatically saves them. If it isn't saving them, go to the Options Tab in the File > Preferences dialog box and select Save preferences file on exit. While you're looking at preferences, you might want to look at the other things you can set. 21
Enabled breakpoint Disabled breakpoint Action Point List Pane Breakpoint location Select on action point icon enables/disables it Dive on action point refocuses source pane to location; this behaves like a bookmark Parallelism in TotalView 22
Supported Parallelism TotalView supports all major parallel programming paradigms MPI OpenMP Pthreads PVM HPF Shmem TotalView Multi-Process Functionality TotalView performs actions on groups of processes Start / stop Share breakpoints across executables Take action on multiple processes when at breakpoint Attach to multiple running processes Single step groups of processes Set barrier breakpoints Easily view and manipulate variables processes Laminated variables for threads and processes Distributed HPF arrays 23
Debugging MPI Programs on the IBM SP with Totalview % totalview poe -a a.out [program_args][pe_args] Automatically attaches to all of the MPI processes TotalView can display MPI message queues Displaying a variable in all processes or threads Need to view or update the value of a variable in all of the processes or threads at once 1. First display an instance of the variable in a Variable Window 2. View > Laminate > Process, or View > Laminate > Thread 3. View > Laminate > None, delaminates the window Example on the right is for an OpenMP 24
Other tools Other Tools Profiler: Xprofiler: GUI based IBM performance profiling tool Graphical function call tree Flat report: line by line CPU time Compile and link your program with pg Run your code => produce gmon.out Invoke xprofiler xprofiler executable [-s] gmon.out.<processor-id> 25
Hardware counter: Hpm on the IBM systems Hardware performance counters Cycles Instructions Float point and fixed point operations Loads/stores Cache misses TLB misses Etc Supports MPI, OpenMP and threaded applications Multiple instrumentations and derived metrics MSI www.msi.umn.edu Help: help@,msi.umn.edu Totalview 612 626 0802 Visit www.etnus.com Online documentation at www.etnus.com/support/docs/index.html 26