End-to-end optimization potentials in HPC applications for NWP and Climate Research

Transcription

1 End-to-end optimization potentials in HPC applications for NWP and Climate Research Luis Kornblueh and Many Colleagues and DKRZ MAX-PLANCK-GESELLSCHAFT

2 ... or a guided tour through the jungle... MAX-PLANCK-GESELLSCHAFT

3 Joint DKRZ/MPIM initial brainstorming Courtesy of Joachim Biercamp, DKRZ Optimization potentials 3 / 25

4 Rationals Why do end-to-end management? Scientific responsible experimentation support Optimization potentials 4 / 25

5 Rationals Why do end-to-end management? Scientific responsible experimentation support Reduce workload of all members of the numerical experimentation community... Optimization potentials 4 / 25

6 An experimentation howto Title, statement of problem, and hypothesis Optimization potentials 5 / 25

7 An experimentation howto Title, statement of problem, and hypothesis Site, hardware, software stack, used programs (versioned), sources, and documented input and boundary data Optimization potentials 5 / 25

8 An experimentation howto Title, statement of problem, and hypothesis Site, hardware, software stack, used programs (versioned), sources, and documented input and boundary data Step-by-step description in such a way that the experiment can be repeated Optimization potentials 5 / 25

9 An experimentation howto Title, statement of problem, and hypothesis Site, hardware, software stack, used programs (versioned), sources, and documented input and boundary data Step-by-step description in such a way that the experiment can be repeated Run the experiment, NO, not three times only one time! Optimization potentials 5 / 25

10 An experimentation howto Title, statement of problem, and hypothesis Site, hardware, software stack, used programs (versioned), sources, and documented input and boundary data Step-by-step description in such a way that the experiment can be repeated Run the experiment, NO, not three times only one time! Observations made during and analysis of the results Optimization potentials 5 / 25

11 An experimentation howto Title, statement of problem, and hypothesis Site, hardware, software stack, used programs (versioned), sources, and documented input and boundary data Step-by-step description in such a way that the experiment can be repeated Run the experiment, NO, not three times only one time! Observations made during and analysis of the results Discuss possible errors that could have occurred in the collection of the data (experimental errors) Optimization potentials 5 / 25

12 An experimentation howto Title, statement of problem, and hypothesis Site, hardware, software stack, used programs (versioned), sources, and documented input and boundary data Step-by-step description in such a way that the experiment can be repeated Run the experiment, NO, not three times only one time! Observations made during and analysis of the results Discuss possible errors that could have occurred in the collection of the data (experimental errors) Publication of the results Optimization potentials 5 / 25

13 The context What is our context? complex, non-standardized workflows and toolchains various processing steps by various actors in climate research as well very often changing workflows Important to note - we are not doing mainstream computing! Optimization potentials 6 / 25

14 Target: provenance of the whole data life cycle Focus on adding: primary data generation primary data processing Already available to a large extent: data publishing secondary data processing data distribution further not controlable data processing Optimization potentials 7 / 25

15 Scientists define experiment easily should organize their individual experiments workflow easily should be enabled to program their individual tasks should not care on collecting provenance data Optimization potentials 8 / 25

16 Scientific programmers define experiment easily should organize experiment workflows easily should be enabled to program individual tasks should not care on collecting provenance data should be enabled to query provenance data for bug tracking, performance improvements,... Optimization potentials 9 / 25

17 Computing center staff should not care on collecting provenance data should be enabled to query provenance data for failure analysis, performance improvements,... Optimization potentials 10 / 25

18 Components of the basic systems Frameworks Setup GUI or CLI Domain Specific Language for Task Scripting (Provenance Collecting Enabled) Cylc Meta-scheduling Preparation Post-processing Model run Archiving Optimization potentials 11 / 25

19 Tools Packages in use: python as scripting language postgres - provenance data collection subversion/git (migration to git for parts or all later, if model developers get convinced) cmake (migration from autotools and self-maintained makefile generator) Web interface for site and compiler dependencies; dependencies versioned in line with model code namelist migration to xml as model source code, user interfaces: a kind of namelist and a GUI. Optimization potentials 12 / 25

20 Single task component Scripting user friendly modeling language restartability exception and error handling in scientist understandable form full support of modern programming concepts use the python hype to change from shell to python Optimization potentials 13 / 25

21 Model build requirements Standardized, convenient, and fast tools out-of-source build Optimization potentials 14 / 25

22 Model build requirements Standardized, convenient, and fast tools out-of-source build dependency resolution Optimization potentials 14 / 25

23 Model build requirements Standardized, convenient, and fast tools out-of-source build dependency resolution make, with hooks for actions at certain steps Optimization potentials 14 / 25

24 Model build requirements Standardized, convenient, and fast tools out-of-source build dependency resolution make, with hooks for actions at certain steps make test Optimization potentials 14 / 25

25 Model build requirements Standardized, convenient, and fast tools out-of-source build dependency resolution make, with hooks for actions at certain steps make test make install Optimization potentials 14 / 25

26 Use of optimized codes: mpiesm and icon In use vectorization (we never gave up on this!), hand gather, scatter, merge instead of standard conditionals, and exposing transcendentials nproma blocking for different architectures OpenMP orphaning - whole physics including radiation run in an single OpenMP directive MPI implementation constantly revisited including building up static load-balancing strategies real asynchronous parallel I/O invest in optimization of libraries Maybe give up on bit-reproducability for production, but not for development! Optimization potentials 15 / 25

27 Data compression On top compression for grib2: AEC (CCSDS algorithm). a standardization exercise got NASA US patent released reimplement from scratch to go around commercial copyright define grib2 template for WMO validate with independent software stacks (Q4 2014) Average reduction in data size over 4-byte float: factor 5, and for grib Encoding and decoding are really, really fast. Get this ported to netcdf4 (hdf5), started but process is slow. Joint work of Mathis Rosenhauer, DKRZ, Shahram Najm, ECMWF, Uwe Schulzweida, and Luis Kornblueh Optimization potentials 16 / 25

28 ICON ocean scaling improvements Wallclock time (optimized) Wallclock time (original) Wallclock time [s] MPI tasks Ocean scaling improvements: no land points, gmres restart, hybrid MPI/OpenMP, code rewrite. Courtesy of Leonidas Linardakis Optimization potentials 17 / 25

29 ECHAM6 scaling improvements Wallclock time [s] Wallclock (cdi) 10 Output (cdi) Wallclock (cdi-pio) 5 Output (cdi-pio) Output time [s] MPI tasks Change from serial (cdi) to parallel output (cdi-pio, using RDMA, I/O time on compute nodes is essentially zero). Courtesy of Irina Fast, Thomas Jahns, DKRZ, and Deike Kleberg Optimization potentials 18 / 25

30 Use of optimized post-processing tools Extend and improve our toolchain (cdi and cdo) Available basic optimized code compute intensive operators are OpenMP parallelized processing of data can be handled by an threaded pipelining method Future data streaming instead of file storage transfer DAG based processing for highest possible parallel efficiency database information system for online data Optimization potentials 19 / 25

31 Experiment organization cylc - the Meta-Scheduler design your own distributed suites of inter-dependent cycling tasks efficient, modular, and reusable validate and visualize workflows on the fly control your running suites diagnose failures (easily!?) simplify failure recovery benefit from expert experience with a specialized tool for meteorological forecasting systems Courtesy of Hilary Oliver, NIWA and contributors Optimization potentials 20 / 25

32 A task modeling framework (provenance data collection) Cylc controled tasks and provenance collection high level programming language python embedded abstraction layer for file operations abstract task description embedded provenance data collection (database stored) tightly connected to cylc connect provenance enabled workflow to ESGF data distribution Remark: introduces complexity reduction methods to all users Courtesy of Deike Kleberg, MPIM and Pavan Siligam, DKRZ Optimization potentials 21 / 25

33 Ensemble overview setup prepare sync_start models archiving sync postproc models ens_post postproc archiving sync ens_post ensemble Optimization potentials 22 / 25

34 Ensemble detail setup prep_1 sync_start prep_0 model_1 model_0 post_oce1 arch_atm1 arch_oce1 arch_lnd1 post_atm1 post_lnd1 sync post_atm0 post_oce0 post_lnd0 arch_lnd0 arch_atm0 arch_oce0 model_1 ens_post model_0 post_atm1 post_lnd1 arch_oce1 arch_atm1 arch_lnd1 post_oce1 sync post_oce0 post_atm0 arch_lnd0 arch_oce0 arch_atm0 post_lnd0 ens_post ensemble Optimization potentials 23 / 25

35 A cylc optimization step connect cylc from inside application (in C) use curl for submitting http/post request to a WebServer POST triggers CGI as interface to cylc Later: server part integrated into cylc Do not poll! It is expensive! Optimization potentials 24 / 25

36 What is coming next? Primary target: Get an optimized worflow system working AT OUR SITE! Keep being conservative in model source code adaptation: concentrate on what we have But: Outsource exploration of GPU handling (CSCS) and take over necessary changes Constantly observe development directions Explore CS concepts in the easier context of post-processing Do: Tutorials and training, training and tutorials, tutorials and training,... Optimization potentials 25 / 25