Environmental Modelling: Crossing Scales and Domains. Bert Jagers

Transcription

1 Environmental Modelling: Crossing Scales and Domains Bert Jagers 3 rd Workshop on Coupling Technologies for Earth System Models Manchester, April 20-22,

2 Vision: scales and process Gold mining has left mercury in the bed of the Sacramento River, so let s see how these deposits resurface and spread through the delta it shares with the San Joachim River WAVES FLOW MOR 3D 2D 1D How many model / components would you use? WAQ

3 From vision towards reality Delft3D Flexible Mesh

4 Domains & Scales

5 From local Urban Scale <1m resolution surface and sewer flows 3Di

6 From local Channel Scale 1D 2D/3D Delft3D Flexible Mesh

7 From local Estuary Scale mixing triangles and quadrangles (as well as pentagons and hexagons were useful) Delft3D Flexible Mesh

8 From local Coastal Scale Delft3D Flexible Mesh

9 From local Sea Scale Delft3D Flexible Mesh

10 To global South-East Asia Delft3D Flexible Mesh

11 To global North America Delft3D Flexible Mesh

12 To global Northern Europe Delft3D Flexible Mesh

13 To global Arctic Delft3D Flexible Mesh

14 Water, Sediment, Water Quality, Ecology, Need to dynamically couple components depending on the research question of the day Geospatial domain different Time scales different Components different Hardware different RAINFALL RUNOFF CONTROL WAVES FLOW GROUND WATER WIND TRANSPORT MOR WAQ/ECO OBSERV. DATA

15 all in a user friendly, GIS based environment

16 Delft3D From Engineering (short time scales)

17 to Geology (long time scales) Geleynse et al., 2011 Delft3D

18 Geleynse et al., 2011 Simulated stratigraphy Delft3D

19 Scalability Delft3D 30 km x 1200 m size 2 m x 2 m resolution = 9 million cells

20 Storage Intermezzo

21 File storage Prefer to build on netcdf Climate & Forecasting (CF) conventions Rapidly growing adoption UPDATE: Now also SGRID for staggered structured grids. See However, it lacks concepts for unstructured meshes New standard: UGRID 0.9

22 Model Restructuring and Coupling

23 Focus on FLOW, MOR, WAVE, and CONTROL Starting with FLOW, MOR, and WAVE FLOW and MOR run sequentially per time step (parallel model; MPI) WAVE may contain multiple nested domains (sequential) WAVE runs in parallel (OpenMP or MPI) Gridded (2D/3D) quantities exchanged Extend with CONTROL of hydraulic structures in FLOW Scalar quantities exchanged CONTROL WAVES FLOW FLOW MOR MOR

24 Removing MOR from FLOW in theory Delft3D-FLOW (including MOR) Delft3D-FLOW Delft3D-FLOW (including MOR) Flexible Mesh 1D-2D-3D MOSSCO MOR MOR MOR We are modifying the Delft3D morphology library to deal with memory management of other host codes. MOR contains all Delft3D morphology features such as different sediment transport formulae and bed stratigraphy.

25 Removing MOR from FLOW in reality Delft3D-FLOW Flexible Mesh 1D-2D-3D MOSSCO Delft3D-FLOW (including MOR) MOR MOR MOR We are modifying the Delft3D morphology library to deal with memory management of other host codes. MOR contains all Delft3D morphology features such as different sediment transport formulae and bed stratigraphy.

26 Removing MOR from FLOW current solution Delft3D-FLOW Flexible Mesh 1D-2D-3D MOSSCO Delft3D-FLOW (including MOR) glue glue glue MOR MOR MOR Glue layer not drawn to scale Glue layer should be standardized as much as possible, like a coupling layer

27 Interaction of FLOW (MOR), WAVE & CONTROL (1/2) Starting with FLOW (including MOR) and WAVE FLOW (MOR) runs in parallel (MPI) WAVE may contain multiple nested domains (sequential) WAVE runs in parallel (OpenMP or MPI) Gridded quantities exchanged Extend with CONTROL of hydraulic structures in FLOW Scalar quantities exchanged CONTROL WAVES FLOW MOR

28 Interaction of FLOW (MOR), WAVE & CONTROL (2/2) Vision: all memory based data exchange a single main exe (coded in C) multiple configurable components (coded in Fortran) MAIN (exe) FLOW FLOW MOR MOR (shared lib) (shared lib) WAVE MANAGER (shared lib) SWAN SWAN 1 1 (shared (exe) lib) SWAN 2 SWAN 2 (shared lib) (exe) SWAN SWAN 33 (shared lib) (exe) CONTROL (shared lib)

29 Step 1: Interaction of FLOW (MOR) & WAVE (1/2) Typical exchange fields FLOW to WAVE water depth, flow velocities WAVE to FLOW wave heights, wave periods, wave forces Historically implemented as three programs with file based exchange. FLOW MOR FLOW (exe) MOR WAVE MANAGER (exe) SWAN SWAN 11 (exe) (exe) SWAN SWAN 2 2 (exe) (exe) SWAN SWAN 3 3 (exe) (exe)

30 Step 1: Interaction of FLOW (MOR) & WAVE (2/2) Now for Delft3D Flexible Mesh implemented as A main program with two libaries FLOW (MOR) component WAVE manager component and still a separate WAVE executable. Data transfer between shared libs initially still via file. Regrid via ESMF. MAIN (exe) FLOW FLOW MOR MOR (shared lib) (shared lib) WAVE MANAGER (shared lib) SWAN SWAN 11 (exe) (exe) SWAN SWAN 2 2 (exe) (exe) SWAN SWAN 3 3 (exe) (exe)

31 Step 2: Adding CONTROL Extend with CONTROL as third library component under main. Data exchange with CONTROL via memory MAIN does not know difference between FLOW, WAVE & CONTROL Calling order and data exchange user configured MAIN (exe) FLOW FLOW MOR MOR (shared lib) (shared lib) WAVE MANAGER (shared lib) SWAN SWAN 1 1 (exe) (exe) SWAN 2 SWAN 2 (exe) (exe) SWAN 3 SWAN 3 (exe) (exe) CONTROL (shared lib)

32 Configuration of MAIN (1/3) <referencetime> t00:00:00</referencetime> <simulationstarttime> t00:00:00</simulationstarttime> <simulationendtime> t00:00:00</simulationendtime> <parallel> <start name="myflowcomponent"/> <startgroup> <exchangestarttime> t00:00:00</exchangestarttime> <exchangetimestep>pt1h<exchangetimestep> <exchangeendtime> t00:00:00</exchangeendtime> <start name="mywavecomponent"/> </startgroup> <startgroup> <exchangestarttime> t00:00:00</exchangestarttime> <exchangetimestep>pt1m<exchangetimestep> <exchangeendtime> t00:00:00</exchangeendtime> <coupler name="flow2control"/> <start name= mycontrolcomponent"/> <coupler name= control2flow"/> </startgroup> </parallel> <!-- sequential execution of waq after completion of flow-wave-rtc --> <start name="mywaqcomponent"/>

33 Configuration of MAIN (2/3) <component name="myflowcomponent"> <library>dflowfm</library> <workingdir>flowwork</workingdir> <process>0 1</process> <!-- Additional(component specific) keywords. --> <inputfile>input/flow/r17.mdu</inputfile> <partitionfile>partitions.nc</partitionfile> </component> <component name="mywavecomponent"> <library>wave</library> <workingdir>wavework</workingdir> <process> </process> <!-- Additional(component specific) keywords. --> <inputfile>input/wave/r17.mdw</inputfile> </component>

34 Configuration of MAIN (3/3) <coupler name= flow2control"> <sourcecomponent>myflowcomponent</sourcecomponent> <targetcomponent>mycontrolcomponent</targetcomponent> <item> <sourcename>observations/obs_a/water_level</sourcename> <targetname>pid_controller_a/h</targetname> <!-- Possibility for adapter arguments here in the future --> </item> <! - more exchange items --> </coupler>

35 API requirements (aiming for BMI update) IRF + get/set + self-describing (introspection) see Scott s talk integer(c_int) function initialize(c_config_file) result(c_iresult) bind(c, name="initialize") integer function update(dt) bind(c, name="update") integer function finalize() bind(c, name="finalize") subroutine get_start_time(t) bind(c, name="get_start_time") subroutine get_end_time(t) bind(c, name="get_end_time") subroutine get_time_step(dt) bind(c, name="get_time_step") subroutine get_current_time(t) bind(c, name="get_current_time") subroutine get_var_type(c_var_name, c_type) bind(c, name="get_var_type") subroutine get_var_rank(c_var_name, rank) bind(c, name="get_var_rank") subroutine get_var_shape(c_var_name, shape) bind(c, name="get_var_shape") subroutine get_var_count(c_var_count) bind(c, name="get_var_count") subroutine get_var_name(var_index, c_var_name) bind(c, name="get_var_name") subroutine get_var(c_var_name, x) bind(c, name="get_var") subroutine set_var(c_var_name, xptr) bind(c, name="set_var") grid information for spatial interpolation

36 API requirements (aiming for BMI update) IRF + get/set + self-describing (introspection) see Scott s talk integer(c_int) function initialize(c_config_file) result(c_iresult) bind(c, name="initialize") integer function update(dt) bind(c, name="update") integer function finalize() bind(c, name="finalize") simple: water_level multi subroutine level: observations get_start_time(t) bind(c, -> Obs_A name="get_start_time") -> water_level Risk subroutine time consuming get_end_time(t) parsing of bind(c, strings name="get_end_time") to get a single value. Use something like an id as alternative? subroutine Unfortunately get_time_step(dt) this would bind(c, imply name="get_time_step") buffering of references at the component side, so maybe subroutine this should get_current_time(t) be optional. bind(c, name="get_current_time") subroutine get_var_type(c_var_name, c_type) bind(c, name="get_var_type") subroutine get_var_rank(c_var_name, rank) bind(c, name="get_var_rank") subroutine get_var_shape(c_var_name, shape) bind(c, name="get_var_shape") subroutine get_var_count(c_var_count) bind(c, name="get_var_count") subroutine get_var_name(var_index, c_var_name) bind(c, name="get_var_name") subroutine get_var(c_var_name, x) bind(c, name="get_var") subroutine set_var(c_var_name, xptr) bind(c, name="set_var") grid information for spatial interpolation Multi level initialize and finalize needed Parallel version (no gather) Need a bit more flexibility than BMI currently offers

37 Coupling Variations on a Theme

38 Multiple instances of the same model Different groups responsible for different parts of the national model. Coupled via Rather than one model, maintain separate models that can be coupled. 41 branches 746 comp. points (to be split further) 6580 branches 5563 connections comp. points 67 branches 782 comp. points all together coupled to a national ground water model (almost via OpenMI)

39 Interactive modelling Interact with the model while it s running. You are one of the coupled components. watch?v=kzfbz8kawf4

40 Web based modelling Stateless Stateful Pull driven Increasing interactiveness query answer query a running process event driven Publish & subscribe pattern

41 Publish & Subscribe approach Model Message Interface (MMI) Built on top of Basic Model Interface JSON message ({ name : waterlevel, shape =[300,200]}) + Binary Message (Array) Built on top of message passing code/protocol ØMQ:

42 MMI wrapper Synchronous pull driven model interface Model implements Initialize, Timestep ( Run ), Finalize Get / Set functions Introspection The model does not have to call any other component: the wrapper will call you (Hollywood principle). Similar wrappers can be implemented for ESMF, OpenMI, CCA, and other frameworks. Model BMI interface

43 MMI wrapper Asynchronous remote control Visualizer Visualizer Controller query event trigger Visualizer Visualizer Controller Visualizer Controller Model BMI interface