Building Ensemble-Based Data Assimilation Systems for Coupled Models

Transcription

1 Building Ensemble-Based Data Assimilation Systems for Coupled s Lars Nerger Alfred Wegener Institute for Polar and Marine Research Bremerhaven, Germany

2 Overview How to simplify to apply data assimilation? 1. Extend model to integrate the ensemble 2. Add analysis step to the model 3. Then focus on applying data assimilation Lars Nerger et al. Building EnsDA Systems for Coupled s

3 PDAF: A tool for data assimilation PDAF - Parallel Data Assimilation Framework a program library for ensemble data assimilation provide support for parallel ensemble forecasts provide fully-implemented & parallelized filters and smoothers (EnKF, LETKF, NETF, EWPF easy to add more) easily useable with (probably) any numerical model (applied with NEMO, MITgcm, FESOM, HBM, TerrSysMP, ) run from laptops to supercomputers (Fortran, MPI & OpenMP) first public release in 2004; continued development ~200 registered users; community contributions Open source: Code, documentation & tutorials at L. Nerger, Lars Nerger W. Hiller, et al. Computers Building EnsDA & Geosciences Systems for 55 Coupled (2013) s

4 Application examples run with PDAF FESOM: Global ocean state estimation (Janjic et al., 2011, 2012) NASA Ocean Biogeochemical : Chlorophyll assimilation (Nerger & Gregg, 2007, 2008) HBM-ERGOM: Coastal assimilation of SST & ocean color (S. Losa et al. 2013, 2014) Sea surface elevation RMS error in surface temperature Sea ice thickness MITgcm: sea-ice assimilation (Q. Yang et al., , NMEFC Beijing) + external applications & users, e.g. Geodynamo (IPGP Paris, A. Fournier) MPI-ESM (coupled ESM, IFM Hamburg, S. Brune) -> talk tomorrow CMEMS BAL-MFC (Copernicus Marine Service Baltic Sea) TerrSysMP-PDAF (hydrology, FZJ) Lars Nerger et al. Building EnsDA Systems for Coupled s

5 Ensemble filter analysis step case-specific call-back routines Analysis operates on state vectors (all fields in one vector) Ensemble of state vectors X Filter analysis 1. update mean state 2. ensemble transformation For localization: Local ensemble Local observations Vector of observations y Observation operator H(...) Observation error covariance matrix R Lars Nerger et al. Building EnsDA Systems for Coupled s

6 Logical separation of assimilation system single program time state Ensemble Filter Initialization analysis ensemble transformation Core of PDAF state observations initialization time integration post processing modify parallelization mesh data Observations quality control obs. vector obs. operator obs. error Explicit interface Indirect exchange (module/common) Nerger, L., Hiller, W. Software for Ensemble-based DA Systems Implementation Lars Nerger and Scalability. et al. Building Computers EnsDA and Systems Geosciences for Coupled 55 (2013) s

7 Extending a for Data Assimilation single or multiple executables coupler might be separate program Start Initialize parallel. Initialize Initialize coupler Initialize grid & fields Do i=1, nsteps Time stepper in-compartment step coupling Post-processing Stop revised parallelization enables ensemble forecast Start Aaaaaaaa Initialize parallel. Init_parallel_PDAF Aaaaaaaa Initialize Initialize coupler aaaaaaaaa Initialize grid & fields Init_PDAF Do i=1, nsteps Time stepper in-compartment step coupling Assimilate_PDAF Post-processing Stop Extension for data assimilation plus: Possible model-specific adaption Possible adaption of coupler (e.g. OASIS3-MCT) Lars Nerger et al. Building EnsDA Systems for Coupled s

8 Framework solution with generic filter implementation Start Aaaaaaaa init_parallel_da Aaaaaaaa Initialize aaaaaaaaa Init_DA Do i=1, nsteps Time stepper Assimilate Post-processing Stop Subroutine calls or parallel communication Generic No files needed! PDAF_Init Set parameters Initialize ensemble PDAF_Analysis Check time step Perform analysis Write results Dependent on model and observations Read ensemble from files Initialize state vector from model fields Initialize vector of observations Apply observation operator to a state vector multiply R-matrix with a matrix with assimilation extension Core-routines of assimilation framework Case specific callback routines Lars Nerger et al. Building EnsDA Systems for Coupled s

9 2-level Parallelism Forecast Analysis Forecast Task 1 Filter Task 1 Task 2 Task 2 Task 3 Task 3 1. Multiple concurrent model tasks 2. Each model task can be parallelized Analysis step is also parallelized Configured by MPI Communicators Lars Nerger et al. Building EnsDA Systems for Coupled s

10 Building the Assimilation System Problem reduces to: 1. Configuration of parallelization (MPI communicators) 2. Implementation of compartment-specific user routines and linking with model codes at compile time Lars Nerger et al. Building EnsDA Systems for Coupled s

11 2 compartment system strongly coupled DA Forecast Analysis Forecast might be separate programs Comp. 1 Task 1 Cpl. 1 Comp. 2 Task 1 Filter Comp. 1 Task 1 Cpl. 1 Comp. 2 Task 1 Comp. 1 Task 1 Comp. 1 Task 1 Cpl. 1 Cpl. 1 Comp. 1 Task 1 Comp. 1 Task 1 Lars Nerger et al. Building EnsDA Systems for Coupled s

12 Configure Parallelization weakly coupled DA Comp. 1 Task 1 Cpl. 1 Comp. 2 Task 1 Comp. 1 Task 2 Cpl. 2 Comp. 2 Task 2 Forecast Filter Comp. 1 Analysis Filter Comp. 1 Logical decomposition: Communicator for each Coupled model task Compartment in each task (init by coupler) (Coupler might want to split MPI_COMM_WORLD) Filter for each compartment Connection for collecting ensembles for filtering Different compartments Initialize distinct assimilation parameters Use distinct user routines Lars Nerger et al. Building EnsDA Systems for Coupled s

13 Example: TerrSysMP-PDAF (Kurtz et al. 2016) l.: TerrSysM P PDAF 134 rce code TerrSysMP (and buildingmodel procedure), it was posne the model libraries for CLM and ParFlow SIS-MCT) Atmosphere: with the data assimilation COSMO libraries DAF in one Land mainsurface: program. Figure CLM 3 sketches omponents of the TerrSysMP PDAF framersysmp PDAF Subsurface: driver (i.e. the main ParFlow program) hole framework. This includes the initialisasation of MPI, coupled TerrSysMP with and PDAF PDAF using as well ping control wrapper for the model forward integration ssimilation. The TerrSysMP wrapper is used e driver program single executable with the individual model m and libparflow coupled via OASIS-MCT). driver controls program r(-defined) functions are specifically adapted and the desired assimilation scheme (EnKF in include, for example, the definition of the state ervation vector and the observation error cox. These data are either provided by the model tate vector) or are read from files or command.g. observations and observation errors). The ctions provide the algorithms for different fils. This part of PDAF is not modified for the n of TerrSysMP PDAF because the input for functions (e.g. state vector, observation vec- Tested using processor cores Figure 3. Components of TerrSysMP PDAF. W. Kurtz et al., Geosci. Dev. 9 (2016) 1341 processor is assigned either to CLM or ParFlow dependin on the processor rank within the model communicator. A example of this model assignment is given in Fig. 4. Th first processors within a model communicator are always as signed to CLM and the rest to ParFlow summing up to th total number of processors for each model realisation. Af terwards, each of the component models is initialised wit the initialisation function of the corresponding model librar (see above). In this step also the model communicators o Lars Nerger et al. Building EnsDA Systems for Coupled s

14 Example: ECHAM6-FESOM Atmosphere ECHAM6 JSBACH land Ocean FESOM includes sea ice Coupler library OASIS3-MCT Atmosphere Ocean Separate executables for atmosphere and ocean Data assimilation (FESOM completed, ECHAM6 in progress) Add 3 subroutine calls per compartment model Replace MPI_COMM_WORLD in OASIS coupler Implement call-back routines : D. Sidorenko et al., Clim Dyn 44 (2015) 757 Lars Nerger et al. Building EnsDA Systems for Coupled s

15 Summary Software framework simplifies building data assimilation systems Efficient online DA coupling with minimal changes to model code Setup of data assimilation with coupled model 1. Configuration of communicators 2. Implementation of user-routines for interfacing with model code and observation handling Lars - Building Nerger et EnsDA al. Building Systems EnsDA for Systems Coupled for s Coupled s

16 References Nerger, L., Hiller, W. Software for Ensemble-based DA Systems Implementation and Scalability. Computers and Geosciences 55 (2013) Nerger, L., Hiller, W., Schröter, J.(2005). PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering, Proceedings of the Eleventh ECMWF Workshop on the Use of High Performance Computing in Meteorology, Reading, UK, October 2004, pp Thank you! Lars.Nerger@awi.de Lars - Building Nerger et EnsDA al. Building Systems EnsDA for Systems Coupled for s Coupled s

17 Changes to FESOM Add to par_init (gen_partitioning.f90) after MPI_init #ifdef USE_PDAF CALL init_parallel_pdaf(0, 1, MPI_COMM_FESOM) #endif Add to main (fesom_main.f90) just before stepping loop CALL init_pdaf() Add to main (fesom_main.f90) just before END DO CALL assimilate_pdaf() OASIS3-MCT Assumes to split MPI_COMM_WORLD in oasis_init_comp (mod_oasis_method.f90) Needs to split COMM_FESOM Start Aaaaaaaa init_parallel_pdaf Aaaaaaaa Initialize generate mesh aaaaaaaaa Initialize fields init_pdaf Do i=1, nsteps Time stepper consider BC Consider forcing assimilate_pdaf Post-processing Stop Lars Nerger et al. Building EnsDA Systems for Coupled s

18 Changes to ECHAM6 Add to p_start (mo_mpi.f90) after MPI_init #ifdef USE_PDAF CALL init_parallel_pdaf(0, 1, p_global_comm) #endif Add to control (control.f90) before call to stepon CALL init_pdaf() Add to stepon (step.f90) before END DO CALL assimilate_pdaf() OASIS3-MCT Assumes to split MPI_COMM_WORLD in oasis_init_comp (mod_oasis_method.f90) Needs to split p_global_comm Start Aaaaaaaa init_parallel_pdaf Aaaaaaaa Initialize generate mesh aaaaaaaaa Initialize fields init_pdaf Do i=1, nsteps Time stepper consider BC Consider forcing assimilate_pdaf Post-processing Stop Lars Nerger et al. Building EnsDA Systems for Coupled s

19 Minimal changes to NEMO Add to mynode (lin_mpp.f90) just before init of myrank #ifdef key_use_pdaf CALL init_parallel_pdaf(0, 1, mpi_comm_opa) #endif Add to nemo_init (nemogcm.f90) at end of routine CALL init_pdaf() Add to stp (step.f90) at end of routine CALL assimilate_pdaf() For Euler time step after analysis step: Modify dyn_nxt (dynnxt.f90) Start Aaaaaaaa init_parallel_pdaf Aaaaaaaa Initialize generate mesh aaaaaaaaa Initialize fields init_pdaf Do i=1, nsteps Time stepper consider BC Consider forcing assimilate_pdaf Post-processing #ifdef key_use_pdaf IF((neuler==0.AND. kt==nit000).or. assimilate) #else Lars Nerger et al. Building EnsDA Systems for Coupled s Stop

20 Current algorithms in PDAF PDAF originated from comparison studies of different filters Filters EnKF (Evensen, perturbed obs.) ETKF (Bishop et al., 2001) SEIK filter (Pham et al., 1998) SEEK filter (Pham et al., 1998) ESTKF (Nerger et al., 2012) LETKF (Hunt et al., 2007) LSEIK filter (Nerger et al., 2006) LESTKF (Nerger et al., 2012) Not yet released: serial EnSRF particle filter Global EWPF filters NETF Localized filters Smoothers for ETKF/LETKF ESTKF/LESTKF EnKF Global and local smoothers Lars Nerger et al. Building EnsDA Systems for Coupled s

21 Parallel Performance Lars Nerger et al. Building EnsDA Systems for Coupled s

22 Time increase factor Speedup Parallel Performance Use between 64 and 4096 processor cores of SGI Altix ICE cluster (HLRN-II) 94-99% of computing time in model integrations Speedup: Increase number of processes for each model task, fixed ensemble size factor 6 for 8x processes/model task one reason: time stepping solver needs more iterations 64/512 proc. 512 proc proc proc. Scalability: Increase ensemble size, fixed number of processes per model task increase by ~7% from 512 to 4096 processes (8x ensemble size) one reason: more communication on the network Lars Nerger et al. Building EnsDA Systems for 64/512 Coupled proc. s 512 proc.

23 time for analysis step [s] Very big test case Simulate a model Choose an ensemble state vector per processor: 10 7 observations per processor: Ensemble size: 25 2GB memory per processor Apply analysis step for different processor numbers Very small increase in analysis time (~1%) Timing of global SEIK analysis step N=50 N= processor cores State dimension: 1.2e11 Observation dimension: 2.4e9 Didn t try to run a real ensemble of largest state size (no model yet) Lars Nerger et al. Building EnsDA Systems for Coupled s

24 Application examples run with PDAF FESOM: Global ocean state estimation (Janjic et al., 2011, 2012) NASA Ocean Biogeochemical : Chlorophyll assimilation (Nerger & Gregg, 2007, 2008) HBM: Coastal assimilation of SST and in situ data (S. Losa et al. 2013, 2014) Sea surface elevation RMS error in surface temperature STD of sea ice concentration MITgcm: sea-ice assimilation (Q. Yang et al., , NMEFC Beijing) + external applications & users, e.g. Geodynamo (IPGP Paris, A. Fournier) MPI-ESM (coupled ESM, IFM Hamburg, S. Brune) -> talk tomorrow CMEMS BAL-MFC (Copernicus Marine Service Baltic Sea) TerrSysMP-PDAF (hydrology, FZJ) Lars Nerger et al. Building EnsDA Systems for Coupled s