The NCI High Performance Computing (HPC) and High Performance Data (HPD) Platform to Support the Analysis of Petascale Environmental Data Collections

Transcription

1 ESSI The NCI High Performance Computing (HPC) and High Performance Data (HPD) Platform to Support the Analysis of Petascale Environmental Data Collections Ben Evans 1, Lesley Wyborn 1, Tim Pugh 2, Chris Allen 1, Joseph Antony 1, Kashif Gohar 1, David Porter 1, Jon Smillie 1, Claire Trenham 1, Jingbo Wang 1, Irina Bastrakova 3, Alex Ip 3, Gavin Bell 4 1 ANU, 2 Bureau of Meteorology, 3 Geoscience Australia, 4 The 6 th Column Project (Second part of this talk is in next ESSI

2 1/25 High Performance Data (HPD) - data that is carefully prepared, standardised and structured so that it can be used in Data-Intensive Science on HPC (Evans, ISESS 2015, Springer) HPC turning compute into IO-bound problems HPD turning IO-bound into ontology + semantic problems What are the HPC and HPD drivers? How do you build environments for this infrastructure that is easy for users to do science?

3 Top 500 Super Computer list since /25 Current NCI Next NCI Fast-and-flexible data access to structured data is required The needs to be a balance between processing power and ability to access data (data scaling) The focus is for ondemand direct access to large data sources enabling High performance analytics and analysis tools directly on that content

4 Elephant Flows Place Great Demands on Networks 3/25 Physical pipe that leaks water at rate of.0046% by volume. Result % of water transferred. Network pipe that drops packets at rate of.0046%. Result 100% of data transferred, slowly, at <<5% optimal speed. essentially fixed determined by speed of light With proper engineering, we can minimize packet loss. Assumptions: 10Gbps TCP flow, 80ms RTT. See Eli Dart, Lauren Rotman, Brian Tierney, Mary Hester, and Jason Zurawski. The Science DMZ: A Network Design Pattern for Data-Intensive Science. In Proceedings of the IEEE/ACM Annual SuperComputing Conference (SC13), Denver CO, 2013.

5 4/25 Computational and Cloud Platforms Raijin: 57,472 cores (Intel Xeon Sandy Bridge technology, 2.6 GHz) in 3592 compute nodes; 160 TBytes (approx.) of main memory; Infiniband FDR interconnect; and 7 PBytes (approx.) of usable fast filesystem (for short-term scratch space). 1.5 MW power; 100 tonnes of water in cooling Partner Cloud Same generation of technology as raijin (Intel Xeon Sandy Bridge technology, 2.6 GHz) but only 1500 cores; Infiniband FDR interconnect; Collaborative platform for services and The platform for hosting non-batch services NCI Nectar Cloud Same generation as partner cloud Non-managed environment Weak integration

6 NCI Cloud 5/25 Lustre Per-Tenant public IP assignments (CIDR boundaries typically /29) NFS NFS FDR IB FDR IB FDR IB FDR IB FDR IB FDR IB SSD SSD SSD SSD SSD SSD OpenStack private IP (flat network*) - quota managed

7 NCI s integrated high-performance environment 6/25 Internet To second data centre NCI data movers 10 GigE /g/data 56Gb FDR IB Fabric Cloud Raijin Login + Data movers Raijin HPC Compute Raijin 56Gb FDR IB Fabric Massdata (tape) Persistent global parallel filesystem Raijin high-speed filesystem Cache 1.0PB, Tape 20PB /g/data1 7.4 PB /g/data PB /g/data3 9 PB /short 7.6PB /home, /system, /images, /apps

8 10+ PB of Data for Interdisciplinary Science 7/25 CMIP5 3PB Earth Observ. 2 PB Marine Videos 10 TB Astronomy (Optical) 200 TB Atmosphere 2.4 PB Water Ocean 1.5 PB Weather 340 TB Bathy, DEM 100 TB Geophysics 300 TB BOM GA CSIRO ANU Other National International

9 8/25 National Environment Research Data Collections (NERDC) 1. Climate/ESS Model Assets and Data Products 2. Earth and Marine Observations and Data Products 3. Geoscience Collections 4. Terrestrial Ecosystems Collections 5. Water Management and Hydrology Collections Data Collections CMIP5, CORDEX ACCESS products LANDSAT, MODIS, VIIRS, AVHRR, INSAR, MERIS Digital Elevation, Bathymetry, Onshore Geophysics Seasonal Climate Bureau of Meteorology Observations Bureau of Meteorology Ocean-Marine Terrestrial Ecosystem Reanalysis products Approx. Capacity ~3 Pbytes 2.4 Pbytes 1.5 Pbytes 700 Tbytes 700 Tbytes 350 Tbytes 350 Tbytes 290 Tbytes 100 Tbytes

10 9/25 Internationally sourced Satellite Data (USGS, NASA, JAXA, ESA, ) Reanalysis (ECMWF, NCEP, NCAR, ) Climate Data (CMIP5, AMIP, GeoMIP, CORDEX, ) Ocean Modelling (Earth Simulator, NOAA, GFDL, ) These will only increase as we depend on more data, and some will be replicated. How can we better keep this in sync, versioned, and back-referenced for the supplier? Organise long-tail data that calibrates and integrates with the big data. How should we manage this data, versioned, and easily attribute supplier (researcher? Collab? Uni? Agency?)

11 Some Data Challenges 10/25 Data Formats Standardize data formats time to convert legacy and proprietary ones Appropriately normalise the data data models and conventions Adopt HPC-enabled libraries that abstracts storage Expose all attributes for search not just collection-level search, not just datasets, all data attributes What are the handles we need to access the data? Provide more programmatic interfaces and link up data and compute resources More server side processing Add the semantic meaning to the data Create useful datasets (in the programming context) from data collections Is it scientifically appropriate for a data service to aggregate/interpolate? What unique/persistent identifiers do we need? DOI is only part of the story. Versioning is important. Born linked data and maintaining graph infrastructure

12 Regularising High Performance Data using HDF5 11/25 Compilers & Tools Fortran, C, C++ Python, R, MatLab, IDL Ferret, CDO,NCL, NCO, GDL,GDAL, GrADS,GRASS,QGIS Globe Caritas Open Nav Surface Metadata Layer netcdf-cf HDF-EOS5 ISO 19115, RIF-CS, DCAT etc Library Layer 1 NetCDF-4 Library libgdal [FITS] Airborne Geophysics Line data [SEG-Y] BAG Library Layer 2 HDF5 MPI-enabled HDF5 Serial Lustre Other Storage (options)

13 Regularising High Performance Data using HDF5 including Data Services 12/25 Compilers & Tools Fortran, C, C++ Python, R, MatLab, IDL Ferret, CDO,NCL, NCO, GDL,GDAL, GrADS,GRASS,QGIS Globe Caritas Open Nav Surface Services (expose data model+sema ntics) Metadata Layer OpenDAP OGC WMS OGC WCS OGC WFS netcdf-cf HDF-EOS5 ISO 19115, RIF-CS, DCAT etc OGC WPS OGC SOS Fast whole-oflibrary catalogue Library Layer 1 NetCDF-4 Library libgdal [FITS] Airborne Geophysics Line data [SEG-Y] BAG Library Layer 2 HDF5 MPI-enabled HDF5 Serial Lustre Other Storage (options)

14 Finding data and services 13/25 GeoNetwork catalogue Supercomputer access Virtual lab DAP, OGC, Services Lucene database /g/data1 /g/data2 Trialing Elastic Search

15 Prototype to Production - anti- Mine craft 14/25 Virtual Labs: Separating Researcher from Software builders Cloud is an enabler, but: don t make researchers become full system admins. save developers from being operational Project lifecycle and preparing success Perspiration Productivity Proj1:Start Proj1:End Proj2-4:Start Proj2-4:End

16 Prototype to Production - anti- Mine craft 15/25 VL Managers VL Managers Headspace hours Developers Developer VL Mgr. Developer? Development Phase in a project Poorly executed Reasonably executed Well executed

17 Prototype to Production - anti- Mine craft 16/25 VL Managers VL Managers Changed Scope adopted broadly Headspace hours Developers Developer VL Mgr Developer Development Phase in a project Poorly executed Reasonably executed Well executed

18 Virtual Laboratory driven software patterns 17/25 Basic OS functions Common Modules Bespoke Services Special config choices Super Software Stack NCI Stack 1 NCI Env Stack Analytics Stack Vis Stack Workflow X Gridftp P2P 2xStack1 Modify Stack1 Modify Stack 2 Take Stacks from Upstream And use as Bundles

19 Transition from developer, to prototype, to DevOps 18/25 Step 1: Development Get template for development What is special, separate out what is common Reuse other software stacks where possible Step 2: Prototype Deploy in an isolated tenant of a cloud Determine dependencies. Test cases to demonstrate correctly functioning. Step 3: Sustainability Pull repo into operational tenant Prepare bundle for integration with rest of framework Hand back cleaned bundle Establish DevOps process

20 19/25 DevOps approach to building and operating environments Virtual Laboratory Operational Bundle - Git controlled - pull model - continuous integration testing NCI Core Bundles Community1 repo Community2 repo

21 Advantages 20/25 Separates roles and responsibilities - from gatekeeper to DevOps management: Specialist on package VL managers system admin Architecture to Platform flexible with technology change makes handover/maintenance easier Both Test/Dev/Ops and patches/rollback become BAU Sharable bundles Can tag release of software stacks Precondition for trusted software stacks Provenance - Scientific / gov policy scrutiny

22 A snapshot of layered bundles to build complex VLs 21/25

23 Easy analysis environments 22/25 Increasing use of ipython Notebooks VDI - Easy In-situ environment using virtual analysis desktops.

24 VDI cont 23/25

25 NCI Petascale Data-Intensive Science Platform 24/25 Data Services THREDDS Server-side analysis and visualization VDI: Cloud scale user desktops on data 10PB+ Research Data Web-time analytics software

26 Summary: Progress toward Major Milestones 25/25 Interdisciplinary Science To publish, catalogue and access data and software for enhancing interdisciplinary, big data-intensive (HPD) science and with interoperable data services and protocols. Integrity of Science Managed services to capture a workflow s process as a comparable, traceable output. Ease-of-access to data and software for enhanced workflow development and repeatable science which can be conducted with less effort or an acceleration of outputs. Integrity of Data The data repository services to ensure data integrity, provenance records, universal identifiers, repeatable data discovery and access from workflows or interactive users.