Data Issues for next generation HPC

Transcription

1 Data Issues for next generation HPC Bryan Lawrence National Centre for Atmospheric Science National Centre for Earth Observation Rutherford Appleton Laboratory Caveats: Due to time, discussion is limited to expected use of HPC for climate simulation, but I expect the arguments apply to pretty much the entire gamut of direct numerical smulation of the environment using HPC. Outline: Context & Futures

2 Context

3 FAR:1990 SAR:1995 TAR:2001 AR4:2007 AR5:2013

4 Using Compute Cycles They all end up inputing and producing more data!

5 Simulation Data Deluge 8 PB 4 PB 2 PB BADC Figure courtesy of Gary Strand (NCAR)

6 and let's not leave out EO! Source: ESA GSCB Workshop June 2009

7 and let's not leave out EO! Source: ESA GSCB Workshop June 2009 Crucial component of model evaluation!

8 State of the art: CMIP5: Fifth Coupled Model Intercomparison Project Projections/simulations Perturbation examples Centennial & Longer e.g. Pre-industrial control, historical, RCPs, Paleo Natural forcings only, GHG forcings only, abrupt 4xCO2 runs. Longer integrations Decadal 10 and 30 year hindcasts and predictions Hindcasts without volcanoes Atmosphere only AMIP aqua-planet ~90,000 years simulated in ~60 experiments ~20-30 modelling centres (from around the world) using ~60? major model configurations ~1-2 million output atomic datasets ~10's of petabytes of output, 1 PB at BADC! Why 1 PB at BADC? Example: Just the ocean 3d fields for the decadal experiments = 45 TB We can't have folks downloading that across the Atlantic

9 The pieces of CMIP5 support HARDWARE SOFTWARE COLLABORATION METADATA DEVELOPMENTS USAGE TOOLKITS Data storage International effort Approx. 1,000TB International effort Sub-setting Replication system Faster network Batch processing QC & Versioning systems Describing models, experiments and datasets Servers to deliver and process data MOHC DATA SUPPORT Data handling of MOHC models Checking and QC Connection to tools Interfaces to data Standard format and description for all NERC DATA SUPPORT INTERNATIONAL DATA SUPPORT Harmonisation Data handling of HIGEM and Paleo models Data handling of models Format conversion Connection to tools Checking and QC Checking and QC Connection to tools Re-gridding Format conversion Visualisation Analysis Platform UK Community Engagement with Impacts Community Public Sector, general public and Private Sector access Development of Derived Products (From a BADC perspective) Not just about storage!

10 Futures

11 A range of scales Thus far it has taken roughly five years for evey doubling of climate model resolution, so this would be a pretty aggressive speed of improvement, but recent emphasis on complexity is likely to replaced by an emphasis on resolution for a while...

12 Computing Futures Best available UK climate sys Bottom climate sys (~ tier 2) ca pa ci ty ca pa bi lit y Top 500 #1 Less traditional capacity

13 The Capacity Data Future CMIP CMIP A CMIP B CMIP Atmos Resolution 1 degree 0.5 degree 25 km 10km Atmos Gridpoints 360x x x x , , ,000 5,062, Atmos % Extra Variables Storage (GB/model-mon) Storage Ratio to Comp Ratio to 2012 (data alone) Comp Ratio to 2012 (inc CFL timestep) Atmos # Gridpoints Atmos # Vert Levels Archive Size 1 PB Still a HARD HPC problem 500 x? PB From a data point of view: increased resolution may be less of a problem than more science & bigger ensembles plus wide availability of petascale computing.? likely to be significantly greater 2 Exascale Data

14 The Capability Data Future Not planning for this as a data sharing activity yet! John Shalf, LBNL: May Accessed 24/01/2011

15 We can produce it, and we can store it but can we move it? Moore's Law: CPU power doubles every 18 months Gilder's Law: the total communication bandwidth triples every twelve months (doubling time about 8 months). Experience of RAL Wide Area Network (WAN) is consistent with Gilder's Law, with doubling time a little over 7 months over the last 15 years. Sounds good, but Nielsen's Law: internet bandwith doubles every months. High performance network cards have moved from 1 Gbit/s to 10 Gbit/s in a decade: Local Area Network (LAN) access performance doubling every 21 months. In 1998, the BADC moved to Token Ring (100 Mb/s) to cope with ECMWF data rates In 2010, BADC is seriously trying to exploit 10 Gbit/s cards to cope with CMIP5 data replication and delivery but fighting with the WAN headline speed is not easily achievable. Internationally (and for the Met Office) we still exploit sneakernet (sending personal RAID systems and USB disks via courier). Very staff time intensive, would prefer to use WAN! In 2022 we can expect WAN performance to cope with massive distribution, but LAN limiting massive I/O at any particular location unless we take action!

16 Practicalities of Data Volumes CMIP CMIP6 2017A CMIP6 2017B CMIP degree 0.5 degree 25 km 10km 1 Gbit/s ,289 17, Gbit/s , Gbit/s Atmos Resolution Download times (hours) for 10 years of (all) data from one ensemble member! Clearly users won't be downloading ALL data from ensembles, now or in the future! ExArch Data comparison: must avoid N x N data movements between N data production sites! Prefer N movements to M (<N) processing sites, exploit better algorithms! Data Movement might be practical MxN times, but only with lightpaths in place ( lightpath = dedicate network slice over dark fibre, provides guaranteed bandwidth unlike shared network). Want dynamic lightpaths: drop when not in use.

17 Modelling Post-Processing Infrastructure: UK interim Plan? &3?G Local Analysis Cloud (Users control the software environment but can mount high performance disk WITH DATA ALREADY THERE) Why light paths? For CMIP5, synchronising 1 PB archive at 1 % level implies 10 TB/day movement implies 1 Gbit/s requirement. International Light Paths will cost money!

18 Modelling Post-Processing Infrastructure: UK Long Term Plan? Move BADC to join LHC (successor) Tier1 and European Bioinformatics Institute and to a national big computing facility in Scotland (green power, easy cooling). (Latency will still matter, we can't go offshore. We can improve on lots of things with improved networks, but we can't beat latency. Data still has to be IN the UK.)

19 The Missing Link Importance of metadata: What simulations exist where? Who ran them? Why? Using what model? Configured how? With what output? Cf CMIP5: millions of atomic datasets What quality assessments have been made? Keeping data: evidence (you probably won't be able to rerun five years later!) Without metadata to drive automatic systems we have no show of managing data, and making it available to the research communities. We expect metadata costs (for metadata in a variety of guises) to be very significant!

20 Our Exascale Future SI prefix k kilo Name Power of 10 or 2 Status thousand Count on fingers M mega million Trivial G giga billion Small T tera trillion Real P peta quadrillion Challenging 103 E exa quintillion Aspirational Z zetta sextillion Wacko Y yotta septillion Science fiction Stuart Feldman, Google, 2010 Expect Exabytes of HPC data to be challenging in ten years time! Too easy to say! (bnl)