Debugging with the new GEOS-Chem Unit Tester

Transcription

1 Debugging with the new GEOS-Chem Unit Tester Bob Yantosca Senior Software Engineer GEOS-Chem Support Team Jacob Group Meeting 27 Nov 2013 with Melissa Payer Sulprizio

2 Contents Overview G-C software development Why benchmarking doesn't catch some bugs What is Unit Testing? GEOS-Chem Unit Tester Interactive demo!!! What types of bugs will it find? Avoiding bugs and numerical issues GEOS-Chem v9-02 status

3 G-C software development G-C is a model w/ an international user base Users are also developers Most updates are submitted by users G-C Support Team (GCST) manages development for the user community Version Control Benchmarking

4 G-C software development We use the Git version control system to keep track of several parallel lines of software development. Code updates made in one branch don t affect the other branches. Branches can later be merged back into the main line of development. Bob Y. IGC6 presentation But sometimes Git is a little too good! It will all grab of the source code updates from a user, even their bugs... Benchmark simulations to validate G-C output

5 G-C benchmark output 1-month and 1-year benchmarks See Budgets Concentrations, differences, ratios Frequency distribution histogram Emission ratios Aerosol scatter plots J-value ratios Seasonal profiles (vertical and surface)

6 G-C benchmark limitations G-C benchmarks are designed to reveal shortcomings in model science algorithms Is chemistry too hot? Do we have too much deposition? Is there a bias w/r/t observations?, etc. Bugs or numerical issues in the code may not always manifest themselves in the benchmarks Effects may be too small to show up on the plots But the code may be still doing something wrong!!!

7 Motivation to be bug-free Bugs waste time and time is $$$ Every minute we spend fixing bugs is a minute that we are not working on something else Bugs can damage our reputation Worst case scenario: Having to issue a retraction because that neat result we found was caused by a bug in the code.

8 Motivation to be bug-free The IGC6 software development discussion group recommended that we implement an automatic debugging scan for G-C. Goal: To remove bugs and numerical issues from the code before we submit benchmark simulations. Thus the G-C Unit Tester was born...

9 What is Unit Testing? Unit Tests are short simulations designed to reveal flaws in software. Unit Tests can be automated and run daily. Many software libraries that we use (ESMF, netcdf, HDF5, etc.) employ Unit Testing as part of their development.

10 Example: ESMF Unit Tests ESMF web site:

11 Example: netcdf unit tests NetCDF unit test site:

12 GEOS-Chem Unit Tester G-C Unit Tester is an external package Downloadable via Git git clone git://git.as.harvard.edu/bmy/geos-chem-unittest UT The Unit Tester package has several subdirs: perl: Contains scripts & input files runs: Contains GEOS-Chem run directories logs: where log file output will get sent jobs: where job files will be created doc: where documentation gets built

13 GEOS-Chem Unit Tester You can perform one or more G-C unit tests Each unit test validates a combination of met field + horizontal grid + simulation type geos5_2x25_co2 merra_4x5_hg geosfp_4x5_fullchem geos5_4x5_soa Etc.

14 Interactive demo! We will start a G-C Unit Test job! While the job runs, we'll examine what types of bugs the Unit Tester scans for.

15 An individual unit test For each combination of met + grid + sim: G-C is run with strict debugging options, twice: Once without OpenMP parallelization (aka sp ) Once with OpenMP parallelization (aka mp ) The debug options will catch common errors. Output files from sp and mp runs are checked. If not identical, there's a problem.

16 Debugging options Each unit test is designed to reveal: Floating-point math exceptions Array out-of-bounds errors Inefficient subroutine calls Parallelization errors Code optimization is disabled (-O0) Optimization can sometimes mask bugs This also facilitates use of a debugger (Totalview)

17 Floating-point math = 53 4-byte (32-bit), aka REAL*4 8-byte (64-bit), aka REAL*8 The bit is the basic computational unit. It represents an on-off switch (0 or 1). A group of 8 bits is called a byte. Each bit in the byte represents a power of 2. With one byte, you can represent values from Negative numbers are obtained by using one of the bits to represent the +/- sign. You have one less power of 2. Available values: -128 to 127. To represent higher numeric values, you string several bytes together.

18 Floating-point math REAL*4 (32-bit) number, IEEE 754 specification: 1 bit for the +/- sign 8 bits for the exponent 23 bits for the mantissa (fractional part) Range: -3.4 x to 3.4 x REAL*8 (64-bit) number, IEEE 754 specification: 1 bit for the +/- sign 11 bits for the exponent 53 bits for the mantissa (fractional part) Range: -1.8 x to 1.8 x

19 Floating-point math Denormal values -Infinity, +Infinity A value that is (smaller,larger) than the (min,max) value that can be represented in REAL*4 or REAL*8. e.g. Result of a division by zero NaN Not a number : Result of an illegal computation. SQRT( -1 ), LOG( 0 ), LOG( -1 ), etc.

20 Floating-point math Denormal values propagate thru code, e.g. Any value + NaN = NaN Any value - NaN = NaN Any value * NaN = NaN Any value / NaN = NaN Any value ** NaN = NaN etc. similarly for -Infinity, +Infinity

21 Example #1. Pretend somebody slipped this into G-C:! Force a div-by-zero error A = 1.0 B = 0.0 PRINT*, 'Result:', A/B If you didn't check for floating-point errors, you would see this: > Result: Infinity But the G-C Unit Tester (using floating-point error checking option FPE=yes) would stop the run w/ an error message: > forrtl: error (65): floating invalid Image PC Routine Line Source geos B1 Unknown Unknown Unknown geos C MAIN 12 main.f geos C Unknown Unknown geos libc.so B2A21A05 Unknown Unknown Unknown geos Unknown Unknown Unknown

22 Avoiding floating-point errors Make sure divisions are possible! Instead of this... C = A / B! Try this... IF ( ABS( B ) > 0.0 ) THEN C = A / B ELSE C = 0.0! Or some other value ENDIF! Or this... C = SAFE_DIV( A, B, 0.0 )

23 Avoiding floating-point errors Make sure you can take the LOG or SQRT! Instead of this... B = LOG( A ) C = SQRT( A )! Try this... IF ( A > 0.0 ) THEN B = LOG( A ) C = SQRT( A ) ENDIF

24 Array storage in Fortran The following Fortran code declares a 1-dimensional array:! Declare a REAL*4 array w/ 3 elements REAL*4 :: ARRAY(3)! Assign values ARRAY(1) = 1.0 ARRAY(2) = 2.0 ARRAY(3) = 3.0

25 Array storage in Fortran The following Fortran code declares a 1-dimensional array:! Declare a REAL*4 array w/ 3 elements REAL*4 :: ARRAY(3)! Assign values ARRAY(1) = 1.0 ARRAY(2) = 2.0 ARRAY(3) = 3.0 The array is stored into successive bytes in system memory. Recall that each element of ARRAY takes 4 bytes to store. ARRAY(1) ARRAY(2) ARRAY(3) In adjacent memory locations, other arrays or variables are stored. We usually won't know what values get put into a given memory location.

26 Array out-of-bounds error But what happens when we make an illegal array assignment?! Declare a REAL*4 array w/ 3 elements REAL*4 :: ARRAY(3)! Assign values ARRAY(1) = 1.0 ARRAY(2) = 2.0 ARRAY(3) = 3.0 ARRAY(4) = 4.0! This is outside of array bounds!

27 Array out-of-bounds error But what happens when we make an illegal array assignment?! Declare a REAL*4 array w/ 3 elements REAL*4 :: ARRAY(3)! Assign values ARRAY(1) = 1.0 ARRAY(2) = 2.0 ARRAY(3) = 3.0 ARRAY(4) = 4.0! This is outside of array bounds! ARRAY(4) gets placed into the system memory following ARRAY(3), but those bytes may already be occupied by another variable! ARRAY(1) ARRAY(2) ARRAY(3) ARRAY(4)? This clobbers whatever value was stored immediately following ARRAY!! A junk value then will propagate thru your code!

28 Array out-of-bounds error Example #2: Assume somebody slipped this into G-C: REAL*4 :: ARRAY(3) ARRAY(1) = 1.0 ARRAY(2) = 2.0 ARRAY(3) = 3.0 ARRAY(4) = 4.0! This is outside of array bounds! PRINT*, 'Result: ', ARRAY(1:4) Without checking for array out-of-bounds errors, you'd see this: > Result: You may not notice anything wrong at first, but the run may die later on. But the G-C Unit Tester (using the error checking option BOUNDS=yes) would stop the run w/ an error message: > main.f(12): error #5561: Subscript #1 of the array ARRAY has value 4 which is greater than the upper bound of 3 ARRAY(4) = 4.0

29 Avoiding out-of-bounds errors Make sure loop bounds are consistent with the declared size of arrays! Declare variables INTEGER :: N REAL*4 :: ARRAY(3)! Do the loop DO N = 1, 3 ARRAY(N) = N * 1.0 ENDDO The upper loop bound must be the same as the declared dimenison of ARRAY (both = 3).

30 Avoiding out-of-bounds errors Or, even better, use declared parameters for array dimensions:! Declare variables INTEGER :: N INTEGER, PARAMETER :: DIM = 3 REAL*4 :: ARRAY(DIM)! Do the loop! DO N = 1, DIM ARRAY(N) = N * 1.0 ENDDO If you change the value of DIM, it gets updated in all places where it is needed.

31 Efficient subroutine calls The calling routine (or program): System Memory USE CMN_SIZE_MOD, IIPAR, JJPAR, LLPAR REAL*4 :: ARRAY(IIPAR,JJPAR,LLPAR) ARRAY =! Some computation ARRAY (IIPAR x JJPAR x LLPAR) CALL MY_ROUTINE( ARRAY ) The routine that is called: SUBROUTINE MY_ROUTINE( ARRAY ) USE CMN_SIZE_MOD, IIPAR, JJPAR REAL*4, INTENT(INOUT) :: & ARRAY(IIPAR,JJPAR,LLPAR) ARRAY =! Some other computation END SUBROUTINE MY_ROUTINE Under most circumstances, the compiler will NOT have to make a duplicate copy of ARRAY. It will just tell the subroutine where it can find ARRAY in the system memory. This is called passing a pointer reference, which is very computationally efficient.

32 Inefficient subroutine calls The calling routine (or program): System Memory USE CMN_SIZE_MOD, IIPAR, JJPAR, LLPAR REAL*4, ARRAY(IIPAR,JJPAR,LLPAR) ARRAY =! Some computation ARRAY (IIPAR x JJPAR x LLPAR)! Pass a non-contiguous slice CALL MY_ROUTINE( ARRAY(:,1,:) ) The routine that is called:: SUBROUTINE MY_ROUTINE( ARRAY ) USE CMN_SIZE_MOD, IIPAR, LLPAR REAL*4, INTENT(INOUT) :: & ARRAY(IIPAR,LLPAR) ARRAY =! Some other computation END SUBROUTINE MY_ROUTINE Temporary copy of a sub-slice of ARRAY (IIPAR x LLPAR) If we try to pass a noncontiguous array slice to a subroutine, then the compiler has to create an ARRAY TEMPORARY (i.e. a duplicate copy). This can cause a performance bottleneck.

33 Finding inefficient subroutine calls Example #3: Assume somebody slipped this into G-C:! In the calling routine USE CMN_SIZE_MOD, IIPAR, JJPAR, LLPAR REAL*4, ARRAY(IIPAR,JJPAR,LLPAR) ARRAY = CALL MY_ROUTINE( ARRAY(:,1,:) )! In the called subroutine SUBROUTINE MY_ROUTINE( ARRAY ) USE CMN_SIZE_MOD, IIPAR, LLPAR REAL*4, INTENT(INOUT) :: ARRAY(IIPAR,LLPAR) ARRAY = END SUBROUTINE MY_ROUTINE But the G-C Unit Tester (using the error checking option DEBUG=yes (which sets -check arg_temp_created) prints: > forrtl: warning (402): fort: (1): In call to MY_ROUTINE, an array temporary was created for argument #1

34 Avoiding inefficient subroutine calls Pass entire arrays from calling routine to subroutine (and not array slices). Be aware of the types of arrays that you have in the calling routine and called subroutine Passing POINTER arrays can sometimes create array temporaries For more info, see: Passing array arguments efficiently in GEOS-Chem page on the G-C wiki

35 Parallelization with OpenMP Creating a parallel DO-loop:!$OMP PARALLEL DO!$OMP+DEFAULT( SHARED )!$OMP+PRIVATE( I, J, P ) DO J = 1, JJPAR DO I = 1, IIPAR P = A(I,J) * 2.0 B(I,J) = P ENDDO ENDDO!$OMP END PARALLEL DO Each CPU will get its own personal copy of the I, J, and P variables. The A and B arrays have the same scope as the loop (I,J) and thus don't have to be held PRIVATE. I,J,P I,J,P I,J,P I,J,P

36 Parallelization with OpenMP But what if we forget to declare the P variable as PRIVATE?!$OMP PARALLEL DO!$OMP+DEFAULT( SHARED )!$OMP+PRIVATE( I, J ) DO J = 1, JJPAR DO I = 1, IIPAR P = A(I,J) * 2.0 B(I,J) = P ENDDO ENDDO!$OMP END PARALLEL DO I,J I,J I,J I,J Each CPU gets its own copy of I and J, but not P. P gets overwritten by all CPUs simultaneously. This introduces numerical errors into the result.

37 Detecting parallelization errors Example #4: Assume somebody slipped this into G-C:!$OMP PARALLEL DO!$OMP+DEFAULT( SHARED )!$OMP+PRIVATE( I, J ) DO J = 1, JJPAR DO I = 1, IIPAR P = A(I,J) * 2.0 B(I,J) = P ENDDO ENDDO!$OMP END PARALLEL DO But the G-C Unit Tester compares the output of the sp and mp runs. If they are not identical, there is an parallelization error: ### File 1 : trac_avg.geosfp_4x5_fullchem sp ### File 2 : trac_avg.geosfp_4x5_fullchem mp ### Sizes : IDENTICAL ( and ) ### Checksums : DIFFERENT ( and ) ### Diffs : DIFFERENT

38 Avoiding parallelization errors Know what you have to declare PRIVATE within an OpenMP parallel loop See the guidelines posted on the GEOS-Chem wiki page: Parallelizing GEOS-Chem

39 G-C Unit Tester: Wrapup We now have an automatic way to detect common G-C bugs! Since v9-02o, we have caught approximately 20 bugs w/ the G-C Unit Tester. G-C Unit Tester is now being used in conjunction w/ the 1-month and 1-year benchmark simulations.

40 GEOS-Chem v9-02 status v9-02 is in the benchmarking phase Major updates GEOS-FP met (we are examining benchmarks) Merging isoprene species into std mechanism Removal of NOx-Ox partitioning Updated SOA simulation Emissions updates Merging updates from GIGC development

41 GEOS-Chem v9-02 status v9-02 is more efficient than v

42 GEOS-Chem v9-02 status Public release soon Public comment period to allow user to report any further issues. Next version will probably be named v10-01 Updates for consideration (G-C Steering Cmte) UCX strat chem Harvard Emissions component (HEMCO) FlexChem Etc.

43 Thanks and Happy Thanksgiving!