The Art of Debugging: How to think like a programmer. Melissa Sulprizio GEOS-Chem Support Team

The Art of Debugging: How to think like a programmer Melissa Sulprizio GEOS-Chem Support Team geos-chem-support@as.harvard.edu Graduate Student Forum 23 February 2017

GEOS-Chem Support Team Bob Yantosca (Harvard) Melissa Sulprizio (Harvard) Lizzie Lundgren (Harvard) Mike Long (Harvard) Colin Thackray (Harvard) Junwei Xu (Dalhousie) Yanko Davila (CU Boulder)

Overview 1) Good coding practice 2) Debugging tips 3) Validation tips 4) Profiling to optimize speed 5) Demos (if time)

Good Coding Practice

Good Coding Practice Use copious comments Comments will make sure that other users will be able to understand the modifications that you have added Writing comments before actually writing the code may help in your planning Well documented code is as important as properly working code Tips for adding comments Flag your updates with your initials and the date, e.g. (mps, 2/23/17) Separate sections of the code with one or more divider comments When citing papers, make sure to include the complete citation at the top of the routine and use shorthand to reference that paper within the code Limit line length to avoid wrapping the text on the page

Good Coding Practice Use a text editor with syntax highlighting Use ProTeX headers ProTeX headers are specially formatted comments at the top of each subroutine, function, and module that list arguments, modification history, and references Documentation is automatically generated when you compile with make doc Follow these tips for writing effective code Indent new blocks of code within loops and if statements The exception to this is nested loops (e.g. looping over lat, lon, lev) Use lots of white space to separate code Use all capitals for Fortran reserve words You may at your discretion mix capitals and lowercase in variable names Structure loops efficiently In Fortran loop over arrays using N, L, J, I order Loops in the wrong order can dramatically slow down your code

General Coding Tips Understand the problem and break it down into small steps Write pseudocode to focus on the logic without being distracted by syntax Syntax varies between languages, but the thought process is the same Look for relevant examples in preexisting code and modify Don t start from scratch if you can help it Use Google, Stack Overflow, fellow group members Compile/run your code often Comment out sections or routine calls to locate problematic code Use print statements to check variables, determine where code stops, etc. Fix bugs before writing new code Use version control Lets you track your files over time so you can revert to a previous working version if necessary

Git Version Control Git version control software is a distributed source code management system Offers many improvements over other source code management software (e.g. CVS, Subversion) Allows you to keep an identical copy of the source code repository on your own system Includes the option to create branches for different streams of development Lets you easily apply and merge code from several developers Contains easy-to-use graphical tools Git is already installed on AS and Odyssey Online services include Bitbucket, GitHub, GitLab, etc. GEOS-Chem repositories are stored on Bitbucket: https://bitbucket.org/gcst/

Git Graphical Tools gitk This command opens the GitK window where you can browse the complete revision history For each commit, you can see the changes made to each file

Git Graphical Tools git gui This command opens the Git GUI, a graphical user interface that can be used for most Git commands Create new branches of development Commit changes to the repository Merge branches back into the master branch We recommend using the Git GUI rather than typing commands at the Unix prompt. The Git GUI greatly simplifies things.

Debugging Tips

General Debugging Tips Here are some tips to isolate the source of errors: 1) Look at the log file(s) 2) Make sure you didn t max out the allotted time or memory 3) Check if someone else has already reported the bug 4) Rerun with debug flags and verbose options turned on 5) Identify whether the error happens consistently 6) Use a debugger to locate the source of the error 7) Isolate the problem to a particular area of the code 8) Revert to a clean copy of the code 9) Focus on areas of the code that you modified 10) Check for math errors (divide by zero, log of negative, etc.) 11) When in doubt, print it out!

GEOS-Chem Debugging Tips Check the GEOS-Chem log file and the HEMCO log file Turn on ND70 in input.geos to print debug output to the log file Set Verbose and Warnings to 3 in HEMCO_Config.rc Compile with debug options turned on: BOUNDS=yes turns on array out-of-bounds error checking TRACEBACK=yes returns a list of routines that were called when the error occurred (now on by default) FPE=yes checks for floating-point exceptions (div-by-zero, NaN, etc.) DEBUG=yes compiles GEOS-Chem for use in a debugger Isolate the error to a particular operation (e.g. transport, chemistry, wet deposition) by turning on one at a time If stuck, send the GCST an email including the log files Please report any bugs in the standard code to the GCST and post on the GEOS-Chem wiki for other users to see!

Fortran Debuggers Using a debugger can save you a lot of time and hassle Debuggers allow you to: Examine data when a program stops Navigate the stack when a program stops Set break points Common debuggers include TotalView, IDB, GDB, DBX For GEOS-Chem, add DEBUG=yes to the make command make -j4 mpbuild DEBUG=yes # With parallelization make -j4 spbuild DEBUG=yes # Without parallelization DEBUG=yes evokes -g for symbolic debug information and -O0 to switch off compiler optimization

TotalView TotalView is a GUI-based source code and memory debugging tool for parallel applications written in C, C++, and Fortran To use type totalview geos.mp

Validation Tips

GEOS-Chem Unit Tester The GEOS-Chem Unit Tester is a package that will compile and run several simulations with a standard set of debugging flags A unit test will run simulations for a given combination of: 1. Meteorology field 2. Horizontal grid 3. Simulation type The steps of a unit test are: 1. Single processor (sp) phase: Run with OpenMP parallelization turned OFF 2. Multi processor (mp) phase: Run with OpenMP parallelization turned ON 3. Compare output of the sp and mp phases. If the outputs differ, this indicated a parallelization error Strict debugging flags are used for all simulations to look for floating-point math exceptions, array-out-of-bounds errors, and creation of array temporaries

GEOS-Chem Unit Tester Download the GEOS-Chem Unit Tester using Git: git clone https://bitbucket.org/gcst/geos-chem-unittest Navigate to the perl subdirectory Modify the UnitTest.input file for your desired settings Start the GEOS-Chem Unit Tester using./gcunittest UnitTest.input Check output generated by the GEOS-Chem Unit Tester and saved to the logs/{version} directory {VERSION}.results.txt: A text file summarizing results Evaluate Unit Test results Green = unit test passed Yellow = requires further investigation (diagnostic output differs) Red = unit test failed

Difference Tests A difference test validates two versions of a model (Ref and Dev) You can only expect a difference test to pass if the Dev run should produce identical results to the Ref run A difference test will not pass if the Dev code contains scientific changes, but this may still be useful for validation Difference tests with GEOS-Chem are set up using the GEOS-Chem Unit Tester

GEOS-Chem Difference Tests Navigate to the perl subdirectory of the GEOS-Chem Unit Tester Modify the DiffTest.input file for your desired settings Create a difference test run directory using./gccreatedifftest DiffTest.input Navigate to the run directory created by gccreatedifftest Start a difference test using make -j4 difftest Check results of the difference test in logs/log.*.results Explore differences between Dev and Ref using summarizediff.sh: provides overview of differences in BPCH files locatediff.sh: provides detailed list of differences in BPCH files plots/plot_diffs.pro: generate difference and ratio maps

Profiling

Profiling If you think your code is taking too long to run, you can use profiling tools to generate a list of time spent in each routine Profilers can help identify the source of computational bottlenecks caused by poorly written or parallelized code

GNU profiler The GNU profiler (gprof) is a free, open source package For GEOS-Chem, add GPROF=yes to the make command GPROF=yes adds the following compiler switches: -p for Intel Fortran compiler (ifort) -pg for GNU Fortran compiler (gfortran) -pg for PGI Fortran compiler (pgfortran) Run the program as usual A binary file called gmon.out will be produced Generate readable output using gprof program gprof geos.mp gmon.out > GC_Profile.txt

GNU profiler

TAU profiler TAU Performance System is the Tuning and Analysis Utilities supported by ParaTools, Inc. TAU is a profiling tool for performance analysis of parallel programs in Fortran, C, C++, Java, and Python Load TAU on Odyssey using: module load tau/2.24.1-fasrc01 # Options for TAU profiler export TAU_OPTIONS="-optRevert -optverbose..." NOTE: TAU on Odyssey is currently built for IFORT 15. Use the alias load_15 to load both IFORT 15 compiler and TAU. TAU uses a visualization tool called ParaProf to create graphical displays of the performance analysis results

TAU profiler For GEOS-Chem, add TAU_PROF=yes to the make command IMPORTANT: Compile on a single processor to allow TAU to properly instrument the code (i.e. don t pass -j4 to the make command) TAU_PROF=yes compiles with TAU using tau_f90.sh instead of ifort, gfortran, or pgfortran Run the program as usual One or more profile.* files will be produced, depending on the number of CPUs that you use Pack the profiling data into a single file using paraprof --pack Profile_Results.ppk Run ParaProf on the packed format (.ppk) file using paraprof Profile_Results.ppk

TAU Profiler: ParaProf ParaProf will open two windows:

TAU Profiler: ParaProf ParaProf will open two windows 1. The ParaProf Manager Window This window is used to manage profile data (e.g. upload/download profile data, edit metadata, launch visual displays, export data) 2. The Main Data window This window shows statistics and each thread as a combined bar graph Each routine is represented by a different color Clicking on the label for a bar (e.g. thread 0 ) will generate a histogram

TAU Profiler: ParaProf

Resources GEOS-Chem wiki: http://wiki.geos-chem.org Quick links: GEOS-Chem basics GEOS-Chem coding style guide GEOS-Chem coding and debugging Using Git with GEOS-Chem Debugging with the GEOS-Chem Unit Tester Performing difference tests with GEOS-Chem Profiling GEOS-Chem with gprof Profiling GEOS-Chem with TAU TotalView quick start guide GNU profiler documentation TAU Performance System documentation