Approaches to I/O Scalability Challenges in the ECMWF Forecasting System

Transcription

1 Approaches to I/O Scalability Challenges in the ECMWF Forecasting System PASC 16, June Florian Rathgeber, Simon Smart, Tiago Quintino, Baudouin Raoult, Stephan Siemen, Peter Bauer Development Section, Forecast Department, European Centre for Medium-Range Weather Forecasts (ECMWF) florian.rathgeber@ecmwf.int ECMWF June 8, 2016

2 ECMWF s Production Workflow 2016: 9km resolution, 137 levels IFS NWP Model Product Generation ECPDS Member States & Customers Raw Output 7M Fields Parallel Filesystem Storage (Lustre) 20TB FDB 7TB PDB per forecast run Products 40M Time critical path 100TB / day 400 research experiments 400,000 jobs / day MARS Perpetual Archive by 2020: 5x increase in data volume! 2

3 Hardware constraints Daily IO profile includes peaks due to time critical runs Operational runs 2 hours from satellite cut-off to deliver forecast products 10 day forecast twice per day, 00Z and 12Z Boundary Conditions 06Z and 18Z, monthly, seasonal, etc. Requires an otherwise oversized parallel storage system 00Z Run 12Z Run Can we avoid hitting disk so often? Can we stream data directly from the Model to Dissemination? 3

4 Redirecting the Data Flows: MultIO Library Structure Each IO server node has a MultIO system implements IO multiplexing Operates independently No synchronisation implied Data Sink FDB IFS IO server Fields MultIO Data Sink Time critical path Hermes (Product Generation) Failed operations written to journal on Data Sink error. May be replayed at leisure Journal Data Sink Choice of data sinks is configurable May be asynchronous Errors caught and reported to MultIO DHS/MARS (?) 4

5 Hermes Computation Service for ECWMF Product Generation tasks supercomputer t1 User Requests generate f1 t2 t3 dispatch f2 t1 t2 Linux cluster workstation Frontend (definition) Backend (execution) 5

6 Hermes Computation Service: Broker Architecture < MARS request data > Clients / DataServers Client a (de)serialise Monitoring Logging (de)serialise Workers Host 1 Monitor Log Client b Client c Client X (de)serialise request > < response comm thread Request queue + Response queues (per client) Hermes Broker comm thread < request work push work > < done (de)serialise Host 2 Host 3 Host N 6

7 Hermes Computation Service: Work Selection Queue Picker 1 { repres : rg, grid : [1.0, 1.0],...} { repres : rg, grid : [0.5, 0.5],...} { repres : sh, grid : [0.25, 0.25],...} Clients request work matching to specification {} Picker 2 { repres : rg, grid : [1.0, 1.0]} Picker 3 { repres : sh, grid : * } Generic worker Optimise caching of interpolation matrix Optimise for any SH transforms { repres : sh, grid : [1.0, 1.0],...} Picker 4 { repres : sh, grid : [1.0, 1.0]} Optimise loading of SH coefficients 7

8 Possible Infrastructure for ECMWF Production Workflow Time critical path Hermes (Broker) Hermes (Workers) Product generation requests data from object store. IFS IO Fields MultIO FDB (mem) Domain-specific object store. Hot buffering improves performance. Object server (FDB and mem FDB) lifetime independent of other components. FDB MARS 8

9 Summary Filesystem IO is a bottleneck to time critical computing Stream Model output to Product Generation Minimize filesystem IO in the critical path Upcoming NVRAM can be used to temporarily store model output Flexible hardware and architecture requirements for product generation Thank you! Questions? ECMWF: NextGenIO: NEXTGenIO has received funding from the European Union s Horizon 2020 Research and Innovation programme under Grant Agreement no

10 What is NextGenIO? Integrated into ECMWF s Scalability Programme Exploring new NVRAM technologies to minimise Exascale I/O bottlenecks Partners EPCC (Proj. Leader) Intel Fujitsu T.U. Dresden Barcelona S.C. Allinea Software ARCTUR ECMWF Project Aims Build an HPC prototype system with Intel 3D XPoint technology Develop tools and systemware to support application development Design scheduler startegies that take NVRAM into account Explore how to best use this technology in I/O servers ECMWF Tasks Provide requirements and use cases Develop a I/O Workload Simulator Explore interation with I/O server layer in IFS Test and assess the system scalability - EU funded H2020 project 10