Modeling groups and Data Center Requirements. Session s Keynote. Sébastien Denvil, CNRS, Institut Pierre Simon Laplace (IPSL)

Transcription

1 Modeling groups and Data Center Requirements. Session s Keynote. Sébastien Denvil, CNRS, Institut Pierre Simon Laplace (IPSL)

2 Outline Major constraints (requirements' DNA) Modeling center requirements/constraints Data center requirements/constraints Take home slide

3 A one slide guide to CMIP5 from a data perspective Fifth Climate Model Intercomparison Project (CMIP5) 101 experiments 61 model variants 59,000 datasets! 4.5 million files 2 PB in global archive. Unknown PB locally! World Climate Research Programme WCRP- WGCM Involves all the major climate modelling centres. CMIP3 : 35 To Original Timing: o(2) PB of requested output from 20+ modelling centres finished early 2010! Actual Timing? Years late. PCMDI-led, Community developed (GO-ESSP) s/w infrastructure for data delivery: Earth System Grid Federation

4 Coupled Model Intercomparison Project CMIP6 International community under strong pressure CMIP6/AR6 cycle concept 2013 & definition Model version set-up CMIP6 Model runs & data archiving Need very strong argumentation to change things here! 03/17 IPCC AR6 Simulations Data archive ESGF Data 31/07/18 15/03/19 09/19 Paper subm Paper accept AR6

5 Outline Major constraints (requirements' DNA) Modeling center requirements/constraints Data center requirements/constraints Take home slide

6 Courtesy Thomas Dubos LMD/IPSL and Yann Meurdesoif LSCE/IPSL cores Year/day Mh/century , , ,22 ½ ,2 ½ ,5 ½ ,8 ¼ ¼ Extrapolated degree Measured Next generation model performance

7 To keep in mind the potential to interpret, compare and reuse climate information results is strongly related to the quality of their description But metadata alone won't get us there! Computation useless if results cannot be stored/distributed/read 7

8 Job_EXP00 Job_EXP00 Job_EXP00 Job_EXP00 Job_EXP00 Job_EXP00 PeriodLength PeriodLength $SCRATCHDIR/IGCM_OUT/.../REBUILD Post RebuildFrequency rebuild rebuild $SCRATCHDIR/IGCM_OUT/XXX/Output $SCRATCHDIR/IGCM_OUT/XXX/Restart Debug Post PackFrequency pack_output pack_output PackFrequency pack_restart pack_restart pack_debug pack_debug ncrcat $CCCSTOREDIR/IGCM_OUT/XXX/Output tar $CCCSTOREDIR/IGCM_OUT/.../RESTART DEBUG SeasonalFrequency TimeSerieFrequency Post TGCC Compute create_ts create_ts create_se create_se atlas atlas monitoring monitoring TS et SE : $CCCSTOREDIR/IGCM_OUT/ dods/store MONITORING et ATLAS : $CCCWORKDIR dods/work DodsCopy=TRUE/FALSE

9 TGCC computers and file system in a nutshell Computers login login front-end front-end compute compute thin thin nodes nodes -q -q standard standard large large nodes nodes -q -q xlarge xlarge hybrid hybrid nodes nodes -q -q hybrid hybrid airain airain nodes nodes airain airain front-end front-end File system Small precious files Saved space $HOME sources small results IGCM_OUT : MONITORING/ATLAS $CCCWORKDIR dods/work dods/work quotas quotas cp dods_cp temporary REBUILD IGCM_OUT : files to be packed outputs of post-proc jobs $SCRATCHDIR quotas quotas cp IGCM_OUT : Packed results Output, Analyse SE and TS Temporary space Saved space $CCCSTOREDIR dods/store dods/store Non saved space ccc_hsm get dods_cp Space on tapes Visible from www quotas quotas HPSS : Robotic tapes

10 Hype.vs. Reality Semantic Web OpenSearch Web Map Service OPeNDAP Web Coverage Services NetCDF CF-1 GCMD DIF HDF-EOS ECHO Catalog Services for the Web Technology Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity

11 Why is it good to log «all around»?

12 Synthesis is so important here also

13 Metrics Garden Metrics Garden User Web Interface Test Glecker like metrics on CMIP5 version of IPSL models d

14 Outline Major constraints (requirements' DNA) Modeling center requirements/constraints Data center requirements/constraints Take home slide

15 Hype.vs. Reality Semantic Web OpenSearch Web Map Service OPeNDAP Web Coverage Services NetCDF CF-1 GCMD DIF HDF-EOS ECHO Catalog Services for the Web Technology Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity

16 Earth Observations Complexity Resolution Ehanced computing resources produce MORE DATA Dur atio n and ens emb le s ize

17 My/Your Data Environment At your home institution, you: We all like control! Can buy/arrange more storage / compute on varying time-scales can optimise... We all like the (illusion?) that we can scale our resources as necessary. Are responsible for deleting / preserving your own data We all lose/destroy/duplicate data. Have (some) control over your software environment Favourite packages, e.g. IDL Familiar Linux Are likely to be duplicating data others have already downloaded in your own institution let alone within a larger collaboration. Most of us do our HPC remotely. Some of us do our analysis remotely. Why not more of us?

18 High Performance Data (HPD) Analysis Environment Mutualized Analysis capabilities Jointly delivered by IPSL laboratories. Joint users (initially): IPSL community Joint users (target): French Academic community Environmental Data Compute Service Web Service Provision for : Climate Science Earth Observation Environmental studies Access services to ESGF System Big DATA Platform Synchro-data find, download, and keep up to date the data users need. CMIP5, CORDEX Obs4MIPs And *MIP... Collaboration Environment Access to Curated Archive. Large shared Group Work Spaces climate analysis enabled system + 1 PB of high performance disk coupled to hundreds of cores configured for analysis

19 Outline Major constraints (requirements' DNA) Modeling center requirements/constraints Data center requirements/constraints Take home slide

20 Take home slide Be ready for CMIP6 to streamline production phase! Be in a position to make good decision from torrent of data. Turn data into information. We need operational intelligence. Roadmap, timeline, minute, milestone, deliverable, responsabilities... publicly available. The High Performance Data (HPD) Analysis Environment approach We will need not only to move computation to data, but aggregate our data collections

21 Thank you for your attention