ESMF. Earth System Modeling Framework. Carsten Lemmen. Schnakenbek, 17 Sep /23

Transcription

1 1/23 ESMF Earth System Modeling Framework Carsten Lemmen Schnakenbek, 17 Sep 2013

2 2/23 Why couple? GEOS5 vorticity We live in a coupled world combine, extend existing models (domains + processes) reuse infrastructure STRAT-CHEM ozone

3 3/23 Why modular coupling? Distributed detailed process knowledge Many individual physical, biological, and chemical processes; requires specialists and specialist models Computing technology Increase in hardware and software complexity in high-performance computing; requires specialists Communication exchange data and codes exchange models, and compare exchange data with model

4 Earth System Modeling Framework 4/23

5 Earth System Modeling Framework 5/23 ESMF is a focused community effort to tame the complexity of models and the computing environment. It leverages, unifies and extends existing software frameworks, creating new opportunities for scientific contribution and collaboration. Definition (Framework) An environment where components are assembled to form an application which can be executed. Definition (Components) exchange information according to a specified set of rules (interfaces) and are therefore easily interchanged.

6 6/23 Gridded Component and transfer and regrid the data before passing it to another Gridded Component. atic shows a flow of data between two Coupler Components that connect three Gridded Components: an e model Coupling with a land model, topology and the same atmosphere model with a data assimilation system. Atmosphere Hierarchicalor star or... D A T A D A T A Figure 3: ESMF supports configurations with a single central Coupler Component. In this case inputs from a Components are transferred and regridded through the central coupler. AtmLandCoupler AtmAssimCoupler Atmosphere D A T A D A T A D A T A Land Land DATA Coupler DataAssim DATA Ocean D A T A 54 SeaIce

7 7/23 Fully coupled AGCM Figure 2: ESMF enables applications such as the atmospheric general circulation model GEOS-5 to be structured hierarchically, and reconfigured and extended easily. Each box in this diagram is an ESMF Gridded Component. GEOS-5 agcm history domain models process models dynamics physics gravity_wave_drag fvcore surface chemistry moist_processes radiation turbulence lake land_ice data_ocean land infrared solar vegetation catchment

8 Adoption 8/23

9 Code invasion 9/23 H.R.A. Jagers / Linking Data, Models and Tools: An Overview Comparison to other frameworks CCA topic defines framework X X X X X X X defines interfaces X X X X X X X provides (reference) implementation rx X X X X X rx X defines object model X X X X code invasiveness [Lloyd et al., 2009] +???? ++? plug & play (and graphical coupling) X (X) X X X support for HPC environment X X X X X C/FORTRAN support X X (X) W X X W W W Java support X (X) X X X.NET support X X ESMF HLA Kepler MCT Table 1. Comparison of coupling technologies OASIS OMS OpenMI TIME

10 10/23 Adoption guide P A R S E P repare A dapt R egister S chedule E xecute

11 11/23 Preparing structure Figure 2: A typical building block for an ESMF application consists of a parent Gridded Component, two or more child Gridded Components, and a Coupler Component. The parent Gridded Component is called by an application driver. All ESMF Components have initialize, run, and finalize methods. The diagram shows that when the application driver calls initialize on a parent Gridded Component, the call cascades down to all of its children, so that the result is that the entire tree of Components is initialized. The run and finalize methods work the same way. In this examplea hurricane simulation is built from ocean and atmosphere Gridded Components. The data exchange between the ocean and atmosphere is handled by an ocean-atmosphere Coupler Component. Since the whole hurricane simulation is a Gridded Component, it could be easily be treated as a child and coupledtoanothergriddedcomponent,ratherthan being drivendirectly by the applicationdriver. A similar diagram could be drawn for an atmosphericmodelcontaining physics and dynamics components, as described in Section Initialize, Run, Finalize AppDriver ( Main ) Call Initialize Call Run Call Finalize Initialize Run Finalize Parent GridComp Hurricane Model Call Initialize Call Run Call Finalize Initialize Run Child GridComp Atmosphere Finalize Initialize Run Child GridComp Ocean Finalize Initialize Run Child CplComp Atm-Ocean Coupler Finalize

12 12/23 More PARSEing Prepare user code Initialize, run, and finalize methods, decide on components, coupling fields and control flow Adapt data structures wrap data structures in Array- or Field-objects, wrap time information in Clock-object Register user methods attach user code methods to the framework through registration calls Schedule, synchronize, and send data write couplers preferably using internal regridding Execute Run components using an ESMF driver

13 ... but the user can decide how much to adopt. 13/23

14 Infrastructure 14/23

15 Infrastructure 15/23 The motivation for using the infrastructure is code reuse. Time Manager Config LogErr Communications: Redistribution, Halo Update, Gather, Scatter, etc. Regridding Figure 1: Schematic of the ESMF sandwich architecture. In this design the framework consists upper level superstructure layer and a lower-level infrastructure layer. User code is sandwiched be layers. ESMF Superstructure AppDriver Time Component Classes: GridComp, CplComp, State U ser Time Code ESMF Infrastructure Data Classes: Bundle, Time Field, Grid, Array Utility Classes: Clock, LogErr, DELayout, VM, Config operation, such as a description of the directory structure and how to build and run the ESMF exa Section 11 is an architectural overview that describes the framework s basic goals and features. Secti steps required to adapt a component for use with ESMF. Finally, to help you become familiar with ESM the last section in the User s Guide is a glossary.

16 Parallelization 16/23

17 Parallel task allocation 17/23 Figure 4: Schematic of the run method of a coupled application, with an Atmosphere and an Ocean Gridded Component running sequentially with an Atm-Ocean Coupler. The top-level Hurricane Model Gridded Component contains the sequencing information and time advancement loop. The application driver, Coupler, and all Gridded Components are distributed over nine PETs. Consecutive T i m e PETs AppDriver ( Main ) Concurrent Figure 5: Schematic of the run method of a coupled application, with an Atmosphere and an Ocean G Component running concurrently with an Atm-Ocean Coupler. The top-level Hurricane Model Gridded C nent contains the sequencing information and time advancement loop. The application driver, Coupler, and to Hurricane Model Gridded Component are distributed over nine PETs. The Atmosphere Gridded Compo distributed over three PETs and the Ocean Gridded Component is distributed over six PETs. PETs Call Run Run GridComp Hurricane Model T i m e AppDriver ( Main ) Call Run LOOP Call Run Run Run GridComp Atmosphere GridComp Hurricane Model LOOP Call Run Run Run Run GridComp Ocean GridComp Atmosphere GridComp Ocean Run Run CplComp Atm-Ocean Coupler CplComp Atm-Ocean Coupler

18 Parallelization abstraction ESMF_Field / ESMF_Array Decomposition #1 #2 #3 #4 #6 #7 #8 #9 #5 #10 #11 #12 #13 ESMF_Grid ESMF_DistGrid ESMF_Field ESMF_Array ESMF_LocalArray ESMF_DELayout ESMF_VM DE connection weights PET ~ OS Instance 0 p0(0)t0 1 p1(1)t0 2 p2(2)t0 MPI 1, MPI 2, armci SHMEM, InterCon lib 3 p4(3)t0 4 p4(3)t1 5 p4(3)t2 6 p7(4)t0 PETs instantiate user code MPI 1, MPI 2, armci SHMEM, OS IPC Pthreads PE Core CPU User TETs OpenMP or Pthreads SSI Definition of terms used in the diagram 18/23

19 19/23 Object Model Communication llowing is a simplified UML diagram showing the structure of the State class. States can contain FieldBu, Arrays, or nested States. See Appendix A, ABriefIntroductiontoUML,foratranslationtablethatli ls in the diagram and their meaning. Information exchange between states 0..n Time State 0..n Bundle 0..n Field 0..n Array 0..n Metainformation with attributes Class API Common Information Model (CIM) ESMF_StateAssignment(=) - State assignment NetCDF attributes

20 Discussion 20/23

21 ESMF Features 21/23 Since 1998 by Open source (esmf.sf.net) Library (-lesmf), built from 500 k code lines Code 60% Fortran90, 40%C++ APIs: use esmf, #include <ESMC> (partial), import ESMP (emerging) Performance optimized extends existing FMS, MCT Flexible processor layout Adoption guide (PARSE) >170,000 lines of example, unit and system testing code. well documented, tutorials, support portable across Linux/Unix/Mac/MinGW abstracts parallelization layer usually single exec, but also web services Nightly tested on >40 different OS/Compiler/MPI combinations

22 22/23 Summary and outlook Why ESMF? flexible integration of existing communities and models allow conservative approaches Try it yourself this afternoon Connect to Wifi Modular Coupling Workshop ssh -Y lastname # pw lastname.mc

23 ESMF Features 23/23 Since 1998 by Open source (esmf.sf.net) Library (-lesmf), built from 500 k code lines Code 60% Fortran90, 40%C++ APIs: use esmf, #include <ESMC> (partial), import ESMP (emerging) Performance optimized extends existing FMS, MCT Flexible processor layout Adoption guide (PARSE) >170,000 lines of example, unit and system testing code. well documented, tutorials, support portable across Linux/Unix/Mac/MinGW abstracts parallelization layer usually single exec, but also web services Nightly tested on >40 different OS/Compiler/MPI combinations