Benchmarks With the optimization of code and structure Basin: 384 grids, 6 GRUs (10 km scale), 730 simulation days ( )

Transcription

1 Benchmarks SA_MESH_ Added binary sequential format for forcing input files (Sapriza) Faster access time Contains data only for cells active in the basin (not the entire X-Y grid) SA_MESH_ Removed CLASS gather/scatter routines (after Marsh) Replaced (Grid, GRU) variables inside the time-loop with vectors Moved gather/scatter outside the time-loop for output and save state operations

2 Benchmarks TRUNK (up to r976) Changed the allocation of variables to use the number of active land elements in the basin (instead of the maximum number possible) to reduce memory footprint Removed repeated variables to reduce memory footprint Split calls into within tile, within grid, and between grid sections Parallelized vertical processes (e.g., within tile )

3 Benchmarks With the optimization of code and structure Basin: 384 grids, 6 GRUs (10 km scale), 730 simulation days ( ) 1200 Runtime (seconds) SA_MESH_ SA_MESH_ TRUNK.r976

4 Benchmarks With parallel execution using Message Passing Interface (MPI) Basin: 384 grids, 6 GRUs (10 km scale), 730 simulation days ( ) 900 Runtime (seconds) Number of nodes in Parallel Execution

5 Benchmarks TRUNK (up to r976) Added flags to disable unnecessary output to console and file

6 Benchmarks With and without output to console and file disabled Basin: 384 grids, 6 GRUs (10 km scale), 730 simulation days ( ) Runtime (seconds) TRUNK.r976 (Standard output) TRUNK.r976 (Reduced output)

7 Process Overview The original calls in MESH_driver.f90 were categorized into one of three groups: Within tile process Land Surface Scheme (LSS) Parallelizable Within grid process Redistribution between tiles (e.g., Blowing Snow algorithm) Parallelizable Between grid process Channel routing (Routing) Cannot be parallelized without further design criteria

8 Process Overview Modules are used to group together similar processes to organize the code Modules contain or contain calls to functions and subroutines Multiple sub-process calls (e.g., to components of the LSS) can be grouped inside a single module, and then be referred to by a single call Multiple modules can be used for dependencies and to represent complex hierarchies Driver Container Process Module Subroutine Program Module Subroutine Subroutine

9 Process Overview Modules are used to group together similar processes Modules contain or contain calls to functions and subroutines (example) Program Module Subroutine run _init RUNMESH sa_mesh _run run run _finalize

10 Process Overview Modules are used to group together similar processes Multiple sub-process calls (e.g., to components of the LSS) can be grouped inside a single module, and then be referred to by a single call (example) Module Subroutine Call RUNCLASS36 sa_mesh _run run RUNSVS113 WF_ROUTE

11 Process Overview Modules are used to group together similar processes Multiple modules can be used for dependencies and to represent complex hierarchies (example) Model Module Process variables WF_ROUTE _config _init _finalize _module _between_grid

12 Process Overview Modules are used to group together similar processes Multiple modules can be used for dependencies and to represent complex hierarchies (example) Splitting a process into multiple modules allows parts of that process (e.g., its parameters) to become accessible in other parts of the code without potentially recursive dependencies Program Module Process variables RUNMESH WF_ROUTE _config Subroutine read _run_options read _parameters _hydrology

13 Process Overview Within a single time-step, the within tile, within grid, and between grid modules are called in order The order facilitates information exchange by the Message Passing Interface (MPI) for parallel execution START sa_mesh _run time sa_mesh _run _within_grid sa_mesh _run _between_grid END

14 Passing Information Type-set variables, which are like objects that contain multiple variables, pass information between modules Explicit variables, like vectors, arrays, and/or subsets of these variables, pass information to subroutines Requires no change to the interface of subroutines taken from standalone models Required for Message Passing Interface (MPI) for parallel execution

15 Passing Information Specific type-set variables are stored at the program level (e.g., MESH_driver.f90) These variables are passed to the within tile, within grid, and between grid modules Any process calls within those modules can modify the variables Program Module Subroutine program pass_data type(data) :: dat allocate(dat%array(i)) call run_module(dat) end program subroutine run_module(dat) type(data) :: dat call run_sub(dat%array(i:i), i) end subroutine subroutine run_sub(array, i) integer i real array(i) array(i:i) = i end subroutine

16 Passing Information Important type-set variables: (passed to the within_tile, within_grid, and between_grid modules): shd (type: ShedGridParams): basin dimensions, location, grid information, soil profile, tile information, tile fraction cover shd%na: number of grids shd%naa: number of grids in the basin (with drainage area, DA >= 0.0) shd%lc: land cover information shd%lc%ntype: number of GRUs or sub-grid elements shd%lc%aclass: fraction of the given GRU inside the given grid shd%lc%nml: number of active land elements (Grid and GRU) in the basin shd%lc%ignd: number of soil layers in the land profile ic (type: iter_counter): current time-step and counter, date information ic%ts_count: count of time-steps from the start of the run ic%now_year: year of the date in the current time-step ic%now_jday: day from the start of the year in the current time-step ic%now_month: month in the year in the current time-step ic%now_day: day in the month in the current time-step ic%now_hour: hour in the day in the current time-step cm (type: clim_info): climate forcing variable Climate Indices (indx): cm%dat(indx)%gat: NML-based climate forcing input for the current time-step cm%dat(indx)%grd: NA-based climate forcing input for the current time-step

17 Passing Information Current state variables: (passed to the within_tile, within_grid, and between_grid modules): wb (type: water_balance): state of precipitation, evaporation, runoff, soil storage [mm] eb (type: energy_balance): state of energy fluxes sv (type: soil_statevars): soil profile temperature, liquid and frozen water content stfl (type: streamflow_hydrograph): observed and measured streamflow [m 3 s -1 ] rrls (type: reservoir_release): reservoir release, storage, and abstraction Future state variables: All CLASS prognostic variables at the program level Channel storage values

18 Indices Indices are determined by the number of nodes running the program Two groups of indices: 1:shd%lc%NML (il1:il2): tile element, smallest land element available at the program level 1:shd%NA (ii1:ii2): grid or large-scale element Preferred index notation: k: tile element (il1:il2 or 1:shd%lc%NML) j: soil layer (1:shd%lc%IGND) i: grid or larger-scale element (ii1:ii2 or 1:shd%NA or 1:shd%NAA) m: GRU or sub-grid element (1:shd%lc%NTYPE)

19 Indices The current range of tile elements for a given node (il1:il2) is determined by a subroutine in the mpi_utilities module: call mpi_split_nml(inp, izero, ipid, & NML, ILMOS, & il1, il2, ilen) Inputs: inp, izero, ipid: common variables determined by the program and accessible by the mpi_shared_variables module NML, ILMOS: available from the shd type-set variable (shd%lc%nml, shd%lc%ilmos) Outputs: il1: start index for NML-based vectors and arrays il2: stop index for NML-based vectors and arrays ilen: the size of a given NML-based vector or array (il2-il1+1) Indices in Fortran are, by default, 1-based (they do not start at zero)! Code that aggregates (i.e., rejoins) NML-based vectors and arrays (e.g., after receiving data from other nodes) should use il1:il2 notation. For nodes, these indices can be used to extract data from NML-based arrays (e.g., climate forcing data). Variables that are local to nodes, and that later do not have to be aggregated or connect to NML-based arrays, can be allocated and index with 1:ilen notation.