Unit Testing, Difference Testing, Profiling, and Musings on Software Engineering

Size: px

Start display at page:

Download "Unit Testing, Difference Testing, Profiling, and Musings on Software Engineering"

Pauline Barton
5 years ago
Views:

1 Unit Testing, Difference Testing, Profiling, and Musings on Software Engineering Bob Yantosca Senior Software Engineer Jacob Group Meeting 06 August 2014

2 Contents Unit testing, revisited Update since Nov new features! Difference testing How it differs from Unit Testing Profiling GEOS-Chem with TAU Examples where TAU helped us speed up GC code Musings on software engineering Can say GC is a successful software product? Do we use industry best practices?

3 Unit Testing, Revisited

4 What is Unit Testing? Unit Tests are short simulations designed to reveal common flaws in software. Unit Tests can be automated and run daily. Automated is key Let the computer do multiple steps for you Errors caused by human forgetfulness less likely

5 We already do benchmarks... 1-month and 1-year benchmarks See Budgets Concentrations, differences, ratios Frequency distribution histogram Emission ratios Aerosol scatter plots J-value ratios Seasonal profiles (vertical and surface)

6 But sometimes it's not enough G-C benchmarks are designed to reveal shortcomings in model science algorithms Is chemistry too hot? Do we have too much deposition? Is there a bias w/r/t observations?, etc. Bugs or numerical issues in the code may not always manifest themselves in the benchmarks Effects may be too small to show up on the plots But the code may be still doing something wrong!!!

7 Motivation to be bug-free Bugs waste time and time is $$$ Every minute we spend fixing bugs is a minute that we are not working on something else GC in GEOS-5-DAS GC as a HPC standalone A bug in any one of the various GC lines of development can cause the entire schedule to slip! HEMCO emissions Other science updates

8 Motivation to be bug-free Bugs can damage our reputation Worst case scenario: Having to issue a retraction because of a bug.

9 GEOS-Chem Unit Tester G-C Unit Tester is an external package Downloadable via Git git clone git://git.as.harvard.edu/bmy/geos-chem-unittest UT The Unit Tester package has several subdirs: perl: Contains scripts & input files runs: Contains GEOS-Chem run directories logs: where log file output will get sent jobs: where job files will be created doc: where documentation gets built

10 GEOS-Chem Unit Tester You can perform several G-C unit tests at once Define UT options in a input file Simple to start: gcunittest UnitTest.input Each unit test validates a combination of met field + horizontal grid + simulation type geos5_2x25_co2 merra_4x5_hg geosfp_4x5_fullchem geos5_4x5_soa, etc.

11 Each UT is designed to reveal: 1. Floating-point math exceptions Infinity, NaN, division-by-zero 2. Array out-of-bounds errors e.g. ARRAY(4) = x; but ARRAY only has 3 elements 3. Inefficient subroutine calls Array Temporary duplicate copy of data being passed 4. Parallelization errors Different results when OpenMP parallelization is on vs. off 5. Missing or incomplete input files / data files

12 Error and warning messages The Unit Tester will send error / warning messages to a log file: Floating point math error: > forrtl: error (65): floating invalid Array out-of-bounds error: > main.f(12): error #5561: Subscript #1 of the array ARRAY has value 4 which is greater than the upper bound of 3 ARRAY(4) = 4.0 Inefficient subroutine call: > forrtl: warning (402): fort: (1): In call to MY_ROUTINE, an array temporary was created for argument #1

13 UT results get sent to the results log file ############################################################################### ### VALIDATION OF GEOS-CHEM OUTPUT FILES ### In directory: geos5_4x5_hg ### ### File 1 : trac_avg.geos5_4x5_hg sp ### File 2 : trac_avg.geos5_4x5_hg mp ### Sizes : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ### ### File 1 : trac_rst.geos5_4x5_hg sp ### File 2 : trac_rst.geos5_4x5_hg mp ### Sizes : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ############################################################################### ############################################################################### ### VALIDATION OF GEOS-CHEM OUTPUT FILES ### In directory: geosfp_4x5_rnpbbe ### ### File 1 : trac_avg.geosfp_4x5_rnpbbe sp ### File 2 : trac_avg.geosfp_4x5_rnpbbe mp ### Sizes : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ### ### File 1 : trac_rst.geosfp_4x5_rnpbbe sp ### File 2 : trac_rst.geosfp_4x5_rnpbbe mp ### Sizes : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ###############################################################################... etc for all of the unit tests that you have scheduled... Diagnostics (aka bpch ) outputs sp = parallelization OFF mp = parallelization ON Restart files at end of run sp = parallelization OFF mp = parallelization ON

14 Unit Testing: New Features since Bob Y's last group meeting in Nov 2013

15 TOMAS aerosol microphysics: geos5_4x5_tomas40 Tagged CO simulation geosfp_4x5_tagco; geos5_4x5_tagco, merra_4x5_tagco Tagged O3 simulation geosfp_4x5_tago3; geos5_4x5_tago3; merra_4x5_tago3; geos4_4x5_tago3 POPs simulation geosfp_4x5_pops; geos5_4x5_pops; merra_4x5_pops; geos4_4x5_pops CH4 simulation We added several new unit tests geosfp_4x5_ch4; geos5_4x5_ch4; merra_4x5_ch4 UCX (strat-trop chemistry) geosfp_4x5_ucx; geos5_4x5_ucx

16 UT results get sent to the results log file ############################################################################### ### VALIDATION OF GEOS-CHEM OUTPUT FILES ### In directory: geos5_4x5_hg ### ### File 1 : trac_avg.geos5_4x5_hg sp ### File 2 : trac_avg.geos5_4x5_hg mp ### Sizes Bummer! : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ### ### File 1 : trac_rst.geos5_4x5_hg sp ### File 2 : trac_rst.geos5_4x5_hg mp ### Sizes : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ############################################################################### ############################################################################### log file to find out if any unit tests ### VALIDATION OF GEOS-CHEM OUTPUT FILES ### In directory: geosfp_4x5_rnpbbe ### failed. ### File 1 : trac_avg.geosfp_4x5_rnpbbe sp ### File 2 : trac_avg.geosfp_4x5_rnpbbe mp ### Sizes : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ### Is there a better way to display this ### File 1 : trac_rst.geosfp_4x5_rnpbbe sp ### File information? 2 : trac_rst.geosfp_4x5_rnpbbe mp ### Sizes : IDENTICAL ( and ) ### Checksums : IDENTICAL ( and ) ### Diffs : IDENTICAL ############################################################################### If you scheduled many unit tests, then you have to scroll down this whole long... etc for all of the unit tests that you have scheduled...

17 Unit test results are posted online at:

18 G-C Unit Tester: Summary Unit Tests are now part of the standard G-C benchmarking protocol. We perform a full set of unit tests along with each 1-month benchmark G-C Unit Tester is also being adapted for use with HEMCO (in v10-01e and higher) For more info, see: wiki.geos-chem.org/geos-chem_unit_tester

19 Difference Testing

20 Difference testing Sometimes we need to test structural updates Example: the fixes that were submitted during the v9-02 public comment period from Jan-Mar 2014 Most updates were for the specialty simulations or TOMAS simulations; should not affect full-chem sim But we still need to make sure that these updates did not break the full-chemistry simulation Cannot do this with a unit test Unit tests look at the same model version

21 Difference test compares 2 snapshots of GEOS-Chem at different points along the development process Development Code aka Dev Does the Dev code give us identical results in the full-chemistry simulation compared to the Ref code? Reference Code aka Ref

22 How to set up a difference test Code.Dev GEOS-Chem Source Code Dir at the point in history labeled v9-02-public-release Code.Ref GEOS-Chem Source Code Dir at the point in history labeled v9-02-provisional-release Run.Dev Run directory with the proper input and restart files corresponding to Code.Dev Run.Ref Run directory with the proper input and restart files corresponding to Code.Ref Initial conditions must be identical Steps: 1. Compile Code.Ref and copy the executable file (geos.ref) to Run.Ref 2. Run geos.ref for a short model simulation (e.g. 1-3 model hours) 3. Compile Code.Dev and copy the executable file (geos.dev) to Run.Dev 4. Run the geos.dev executable for the same amt of time as for geos.ref 5. Compare (w/ diff) the restart and bpch files in Run.Dev and Run.Ref. 6. If restart and bpch files are identical, then Code.Dev passes the test.

23 Difference testing: Summary Difference tests can be automated Makefiles can do all of the steps listed on the previous slide; All you have to do is start the test. Difference tests can be frequent Can run a test each time you add code that shouldn't impact any scientific results Difference tests can be varied You can set up difference tests for full-chem, SOA, TOMAS, Rn-Pb-Be, Hg, CH4, etc. simulations

24 Profiling GEOS-Chem with TAU

TAU: Tuning and Analysis Utilities http://tau.uoregon.edu John Linford of ParaTools, Inc. has installed TAU on the AS cluster. Many of you were here for John's talk / workshop in March.

25 TAU: Tuning and Analysis Utilities John Linford of ParaTools, Inc. has installed TAU on the AS cluster. Many of you were here for John's talk / workshop in March. TAU can be used to profile GEOS-Chem. Profiling means that we can find out how much time each GEOS-Chem subroutine (or section of a subroutine) takes to execute. This helps us to identify computational bottlenecks.

26 G-C v10-01 has TAU support built in To activate TAU in GEOS-Chem, compile with: make -j4 MET=geosfp GRID=4x5 TAU_PROF=y etc. TAU saves profiling info to disk TAU also provides visualization routines: paraprof: Plots execution times into barcharts pprof: Lists execution times in text format We will show a couple of examples...

27 Example #1: Olson land map code Barchart output generated by paraprof

28 Example #2: TPCORE speedup Barchart output generated by paraprof

29 Profiling w/ TAU: Summary TAU is going to be an important part of G-C development going forward As G-C gets more complex, we will need to make sure that it is running efficiently We have just started to scratch the surface with what we can do with TAU

30 Musings on Software Engineering

31 Is there a way to test how good your software team is? Are you using best practices? I use Joel Spolsky's 12 Steps to Better Code as my guide. It is a quick test that you can use to get an easy yes or no to each question. Joel Spolsky is the founder of Fog Creek Software in NYC. His company has produced several innovative software packages, such as: FogBugz: Bug-tracking system for software developers Trello.com: Free and easy-to-use performance management system (Tack virtual cards on a virtual bulleting board). The GCST uses Trello.com for workflow management. For more info, see Joel's blog: joelonsoftware.com

32 Joel Spolsky's 12 Steps to Better Code 1. Do you use source control? Joel says: But if you don't have source control, you're going to stress out trying to get programmers to work together. Programmers have no way to know what other people did. Mistakes can't be rolled back easily. YES! We use the Git source code management system for GEOS-Chem. Git is the gold standard for version control.

33 Joel Spolsky's 12 Steps to Better Code 2. Can you make a build in one step? Joel says: If the process takes any more than one step, it is prone to errors...if it takes 20 steps to compile the code, run the installation builder, etc., you're going to go crazy and you're going to make silly mistakes. YES! You can compile GEOS-Chem into an executable file with a single command. We also provide a separate installer to build the netcdf libraries in one step (for first-time setup).

34 Joel Spolsky's 12 Steps to Better Code 3. Do you make daily builds? Joel says: When you're using source control, sometimes one programmer accidentally checks in something that breaks the build...breaking the build is so bad (and so common) that it helps to make daily builds, to insure that no breakage goes unnoticed. YES! We use unit tests and difference tests each day to ensure that we don't break any critical functionality in GEOS-Chem.

35 Joel Spolsky's 12 Steps to Better Code 4. Do you have a bug database? Joel says: Lots of programmers think they can hold the bug list in their heads. Nonsense. I can't remember more than two or three bugs at a time, and the next morning, or in the rush of shipping, they are forgotten. You absolutely have to keep track of bugs formally. YES! The GEOS-Chem wiki is our bug database. Bugs, fixes, and unresolved issues are listed for each model version. These are also cross-referenced in our Bugs and Fixes and Currently unresolved issues in GEOS- Chem pages.

36 Joel Spolsky's 12 Steps to Better Code 5. Do you fix bugs before writing new code? Joel says: In general, the longer you wait before fixing a bug, the costlier (in time and money) it is to fix...and if you find a bug in code that has already shipped, you're going to incur incredible expense getting it fixed. YES! We assign the highest priority to bug fixes and work to resolve them before adding new science to GEOS-Chem.

37 Joel Spolsky's 12 Steps to Better Code 6. Do you have an up-to-date schedule? Joel says: The...crucial thing about having a schedule is that it forces you to decide what features you are going to do, and then it forces you to pick the least important features and cut them rather than slipping into featuritis (a.k.a. scope creep). YES! Each GEOS-Chem version is now released nine months after the previous version. In addition, each release now has a period of 1 month for public comment. Whatever doesn't ship by release time will go into the next version.

38 Joel Spolsky's 12 Steps to Better Code 7. Do you have a spec (i.e. a blueprint)? Joel says: At the design stage, when you discover problems, you can fix them easily by editing a few lines of text. Once the code is written, the cost of fixing problems is dramatically higher. YES! New source code additions to GEOS-Chem (e.g. HEMCO, GIGC updates) were implemented according to previously thought-out design standards. (Excluding 3rdparty code that did not originate with us.)

39 Joel Spolsky's 12 Steps to Better Code 8. Do programmers have quiet working conditions? Joel says: We all know that knowledge workers work best by getting into flow, also known as being in the zone...if a coworker asks you a question, causing a 1 minute interruption, but this knocks you out of the zone badly enough that it takes you half an hour to get productive again, your overall productivity is in serious trouble. YES! Melissa, Mike, and myself have offices to ourselves, as opposed to Dilbert-style open cubicles.

40 Joel Spolsky's 12 Steps to Better Code 9. Do you use the best tools money can buy? Joel says: Top notch development teams don't torture their programmers. Even minor frustrations caused by using underpowered tools add up, making programmers grumpy and unhappy. And a grumpy programmer is an unproductive programmer. YES! We have access to cutting edge software packages, some free/open-source and some proprietary: IFORT compiler, Emacs editor, Totalview Debugger, IDL, Git, Matlab, R, Dreamweaver (for web design), etc.

41 Joel Spolsky's 12 Steps to Better Code 10. Do you have testers? Joel says: Skimping on testers is such an outrageous false economy that I'm simply blown away that more people don't recognize it. YES! We don't have dedicated testers like a software company would, but the GCST now can perform automated unit tests and difference tests.

42 Joel Spolsky's 12 Steps to Better Code 11. Do new candidates write code during their interview? Joel says: Would you hire a magician without asking them to show you some magic tricks? Of course not. N/A. This is more applicable to a software company developing in languages with standardized components. Our primary criteria is that a GCST member should be well-versed in a physical science before anything else. Programming skills can be taught more easily than science.

43 Joel Spolsky's 12 Steps to Better Code 12. Do you do hallway usability testing? Joel says: A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code. SORT OF. Melissa and I often ask each other to test our codes and documentation. We sometimes also ask other group members to test things that we have been working on. But we do this on an as-needed basis.

44 So how did we do? I would give us a Joel Test score of 10.5 / 11, or 95% Excluding point #11 which is not applicable The results are an objective confirmation of what we knew already :-) That G-C is a well-managed software product That we pay great attention to our user interface and documentation

45 Why use Fortran? C For numerical programming, Fortran is MUCH faster than other languages! Which is why real programmers use Fortran. :-)

46 Extra slides

47 UT will catch floating-point math errors Infinity REAL*4: any x < or x > REAL*8: any x < or x > Can be caused by a division by-zero NaN: Result of any illegal computation, e.g. LOG(x), x <= 0 SQRT(x), x < 0 TAN(x), x = +p/2 Etc.

48 UT will catch array out-of-bounds errors What happens when we make an illegal array assignment?! Declare a REAL*4 array w/ 3 elements REAL*4 :: ARRAY(3)! Assign values ARRAY(1) = 1.0 ARRAY(2) = 2.0 ARRAY(3) = 3.0 ARRAY(4) = 4.0! This is outside of array bounds! ARRAY(4) gets placed into the system memory following ARRAY(3), but those bytes may already be occupied by another variable! ARRAY(1) ARRAY(2) ARRAY(3) ARRAY(4)? This clobbers whatever value was stored immediately following ARRAY!! A junk value then will propagate thru your code!

49 UT will catch inefficient subroutine calls The calling routine (or program): System Memory USE CMN_SIZE_MOD, IIPAR, JJPAR, LLPAR REAL*4, ARRAY(IIPAR,JJPAR,LLPAR) ARRAY =! Some computation ARRAY (IIPAR x JJPAR x LLPAR)! Pass a non-contiguous slice CALL MY_ROUTINE( ARRAY(:,1,:) ) The routine that is called:: SUBROUTINE MY_ROUTINE( ARRAY ) USE CMN_SIZE_MOD, IIPAR, LLPAR REAL*4, INTENT(INOUT) :: & ARRAY(IIPAR,LLPAR) ARRAY =! Some other computation END SUBROUTINE MY_ROUTINE Temporary copy of a sub-slice of ARRAY (IIPAR x LLPAR) If we try to pass a noncontiguous array slice to a subroutine, then the compiler has to create an ARRAY TEMPORARY (i.e. a duplicate copy). This can cause a performance bottleneck.

50 UT will catch parallelization errors Creating a parallel DO-loop:!$OMP PARALLEL DO!$OMP+DEFAULT( SHARED )!$OMP+PRIVATE( I, J, P ) DO J = 1, JJPAR DO I = 1, IIPAR P = A(I,J) * 2.0 B(I,J) = P ENDDO ENDDO!$OMP END PARALLEL DO To look for parallelization errors, the Unit Tester will do 2 simulations as part of each requested unit test: (1) With parallelization turned OFF (2) With parallelization turned ON If both simulations do not yield identical results, there is an error. Most of the time parallelization errors are caused by a programmer forgetting to add a variable to the!$omp PRIVATE statement.

Debugging with the new GEOS-Chem Unit Tester

Debugging with the new GEOS-Chem Unit Tester Bob Yantosca Senior Software Engineer GEOS-Chem Support Team Jacob Group Meeting 27 Nov 2013 with Melissa Payer Sulprizio Contents Overview G-C software development