Software Infrastructure for Data Assimilation: Object Oriented Prediction System

Transcription

1 Software Infrastructure for Data Assimilation: Object Oriented Prediction System Yannick Trémolet ECMWF Blueprints for Next-Generation Data Assimilation Systems, Boulder, March 2016

2 Why OOPS? Y. Trémolet OOPS

3 Evolution of Forecasting: Earth System Modelling The expectations of society for better weather (and related) forecasts are pushing us to account for more of the Earth system Atmosphere, Land surface, Ocean, Sea ice, Atmospheric composition... Lines of code (x 1,000,000) IF statements (x 1,000) Each model is becoming more and more complex as science progresses. The models are becoming more and more coupled to account for interactions between all these aspects. Y. Trémolet OOPS 1 / 16

4 Earth System Data Assimilation Data assimilation systems have been developped for each model. Coupled data assimilation requires some common framework. Y. Trémolet OOPS 2 / 16

5 Evolution of Data Assimilation We are asking more from DA: Number and types of observations, Observation bias correction, Sophisticated observation operators, Correlated observation errors, Account for model errors, Estimate model erros. Variational methods: Minimisation and preconditioning, Sophisticated TL/AD models, Wavelet J b, Weak constraint, Saddle point optimisation, Long (overlaping) windows. Ensemble methods: Ensemble of 4D-Vars EnKF (many variants) 4D-En-Var EVIL... Today s best data assimilation systems are hybrid... which increases complexity: Each task is complex, Scheduling of jobs, Data flow. Y. Trémolet OOPS 3 / 16

6 Computing Environment Future forecast configuration with the current code structure on the next generation HPC will: not complete within operational schedule; require unaffordable levels of electricity. To reduce energy needs, new generations of HPC will include new technologies such as accelerators, GPUs, many core units... We need to prepare for expected future hardware technologies and their implications on code structure ensuring efficiency and code readability The success metrics are efficiency gains in Watts (not FLOPS) Y. Trémolet OOPS 4 / 16

7 The Scalability Question Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth (wikipedia) Y. Trémolet OOPS 5 / 16

8 The Scalability Question Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth (wikipedia) Computers are growing in size and complexity Y. Trémolet OOPS 5 / 16

9 The Scalability Question Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth (wikipedia) Computers are growing in size and complexity Data Assimilation systems are growing in size and complexity Models are growing in size and complexity The number of people (and organisations) working with the same codes is growing Y. Trémolet OOPS 5 / 16

10 The Scalability Question Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth (wikipedia) Computers are growing in size and complexity Data Assimilation systems are growing in size and complexity Models are growing in size and complexity The number of people (and organisations) working with the same codes is growing Scalability is not only about processor counts. Y. Trémolet OOPS 5 / 16

11 How can we do better? Y. Trémolet OOPS

12 What is a good software system for DA? Reliable The code must run without crashing. Additional aspects of reliablity are application dependent, for example the code must do what the user thinks it does. Efficient For operational systems this is a very strong requirement Readable Readability improves staff efficiency: it is as important as computational efficiency (it s just more difficult to measure). Flexible It should be easy to modify (new science, new functionality, scalability...) Modular? Yes, if it means independent units of code (not Fortran modules!). Modularity improves staff scalability: it is as important as computational scalability (it s just more difficult to measure). Y. Trémolet OOPS 6 / 16

13 Object-Oriented Programming These needs are not specific to DA: the techniques that have emerged in the software industry to answer these needs are called generic programming and object-oriented programming. The software industry moved to OOP 15 years ago. Why haven t we? Structured programming (subroutines) eliminated the GOTO, object-oriented programming eliminates the IF (the two most common sources of bugs). OOPS uses both generic and object-oriented programming IF ( L95 ) THEN... ELSEIF ( LQG ) THEN... ELSEIF ( LIFS ) THEN... ELSEIF ( LNEMO ) THEN... ENDIF Not OO, not sustainable Adding a model to OOPS does not require any modfication to the code. PS: The next most common source of bugs is memory handling: the next step would be functional programming which eliminates the need for variables... Y. Trémolet OOPS 7 / 16

14 Data Assimilation Objects J(x) = 1 2 (x 0 x b ) T B 1 (x 0 x b ) n [H(x i ) y i ] T R 1 i [H(x i ) y i ] i=0 The 4D-Var problem, and the algorithm to solve it, can be described with a very limited number of entities: Vectors: x, y, g and δx. Covariances matrices: B, R (and eventually Q). Two operators and their linearised counterparts: M, M, M T, H, H, H T. All data assimilation schemes manipulate the same limited number of entities. We do not need to mention details about how any of the operations are performed, how data is stored or what the model represents. These entities are our objects. Y. Trémolet OOPS 8 / 16

15 OOPS Abstract Design Applications Building Blocks Models Forecast 4D-Var EDA EPS EnKF... States Observations Covariances Increments... Lorenz 95 QG IFS NEMO Surface... OOPS The high levels Applications use abstract building blocks. The Models implement the building blocks. OOPS is independent of the Model being driven. Y. Trémolet OOPS 9 / 16

16 Separation of concerns The key is separation of concerns: All aspects exist but scientists focus on one aspect at a time. Different concepts should be treated in different parts of the code. Classes have well defined responsibilities: The State knows its values at any location, The Model advances the State in time, The ObservationOperator knows how to compute an observation equivalent given the values of the State at the appropriate locations,... OOPS doesn t know any data structures: it passes adresses or pointers between appropriate methods. There is another project at ECMWF to develop another software package to handles grids and interpolations (separation of concerns at project level!) Interfaces are the most important aspect of the design! The opposite to most software developments in Fortran. Y. Trémolet OOPS 10 / 16

17 Design Example: State-Observations Interactions Two classes make the link between the model and observation spaces: Locations ModelAtLocations The computation of observations equivalents is done in a PostProcessor: 1. Ask the Observations for a list of locations where there are observations (at the current time) 2. Ask the State for the model values at these locations 3. Ask the ObsOperator to compute the observations equivalents given the model values at observations locations. Y. Trémolet OOPS 11 / 16

18 Design Example: State-Observations Interactions Two classes make the link between the model and observation spaces: Locations ModelAtLocations The computation of observations equivalents is done in a PostProcessor: 1. Ask the Observations for a list of locations where there are observations (at the current time) 2. Ask the State for the model values at these locations 3. Ask the ObsOperator to compute the observations equivalents given the model values at observations locations. Last step can be performed on the fly or in the finalize method (memory vs. load balancing). There is no magic interpolation from any grid to any location in OOPS. Preserves encapsulation (model grid not visible in observation operator). Y. Trémolet OOPS 11 / 16

19 Design Example: State-Observations Interactions Two classes make the link between the model and observation spaces: Locations ModelAtLocations The computation of observations equivalents is done in a PostProcessor: 1. Ask the Observations for a list of locations where there are observations (at the current time) 2. Ask the State for the model values at these locations 3. Ask the ObsOperator to compute the observations equivalents given the model values at observations locations. Last step can be performed on the fly or in the finalize method (memory vs. load balancing). There is no magic interpolation from any grid to any location in OOPS. Preserves encapsulation (model grid not visible in observation operator). But it s up to each model implementation: OOPS does not prevent copying the full State in the GOMs... Y. Trémolet OOPS 11 / 16

20 Software engineering aspects OOPS is written in C++ OOPS is small (10k lines) and light-weight (CPU-wise) OOPS works with MPI and OpenMP Can handle several models in the same executable The system include testing mechanism: Internal consistency and correctness of results (not meteorological evaluation), Tests run on demand and automatically on push to source code repository. Not there yet: documentation, support, training Y. Trémolet OOPS 12 / 16

21 OOPS Suites and Scripts Like the Fortran code, the suite definitions and scripts have become more and more difficult to maintain and develop. Complexity will keep increasing in the future: Long overlapping 4D-Var windows, Hybrid data assimilation (EDA and DA coupled two-ways), Coupled ocean-atmosphere models... The suite definitions and scripts define the application at the highest level. We should think of them as part of the algorithm. Three levels are mixed together in the suite definitions and scripts: The model (IFS, NEMO...), although the top level of OOPS is generic, The scientifc description of the cycling, hybrid aspects... The workflow technical specificity (SMS, ecflow, qsub...). The three levels could be, and should be, isolated from each other. Y. Trémolet OOPS 13 / 16

22 Prototype with QG toy-model and ecflow class Analysis ( CompositeTask ): def compose ( self ): window = self. input ( window ) bg = self. bgfc ( window = window ) obs = self. fetchobs ( window = window ) (an,fb) = self. an4dvar (bg=bg, obs =obs, window = window ) self. archive_bg ( data =bg) self. archive_fb ( data =fb) self. set_output ( an, an) Note that GetBackground is a composite task as well. The workflow (ecflow) is abtracted from the suite definition. Y. Trémolet OOPS 14 / 16

23 Abstracting the workflow Scientists should think as if writing any algorithm. Executing the (python) code generates the suite (and scripts). A simple workflow can run the tasks on the fly (toy system on a laptop). For complex systems, each component can generate a single task or a family. The workflow is chosen when running the python program. The workflow can be specialized for specific operational needs (start at predetermined times, trigger product generation...). Everything else (algorithms and code) is the same: more can be shared between research and operations. Y. Trémolet OOPS 15 / 16

24 Status and final comments Initial focus is on variational methods: 4D-Var is written as a single executable (less I/O improves scalability) Most variational algorithms are available Work to adapt the IFS and NEMOVAR to OOPS is under way The OO layer developed for the simple models is not only a proof of concept: the same code is re-used to drive the IFS (generic). OOPS does not solve scientific problems in itself: it provides a more powerful way to tell the computer what to do. Object oriented technology is well known and proven: Difficulties are human. OOPS will be available under an open source licence (Apache 2). Y. Trémolet OOPS 16 / 16