Data Management A Scientific Tour

Transcription

1 Data Management A Scientific Tour Shaun de Witt 0, 1, Rob Akers 1, Hannes Thiemann 2, Margareta Hellström 3, Alberto Michelini 4, Massimo Farres 4, Peter Danecek Corresponding author shaun.de-witt@ukaea.uk 1 Culham Centre for Fusion Enenrgy 2 - Deutsche Klimarechenzentrum 3 University of Lund 4 - Istituto Nazionale di Geofisica e Vulcanologia CCFE is the fusion research arm of the United Kingdom Atomic Energy Authority. This work was part-funded by the RCUK Energy Programme [grant number EP/I501045] and the European Union s Horizon 2020 research and innovation programme.

2 Agenda What is Data Management Data Management in History Who cares about data management? What is this data management stuff anyway? What s wrong with the way we do it now? OK so how do other people do it? So how can we improve (aka the paradigm shift)

3 Data Management Definition and Reasons "Data Resource Management is the development and execution of architectures, policies, practices and procedures that properly manage the full data lifecycle needs of an enterprise. Data Management Association deliver the scientific discoveries and major scientific tools that transform our understanding of nature and advance the energy, economic, and national security of the United States. DoE Office of Science Good research data management is not a goal in itself, but rather the key conduit leading to knowledge discovery and innovation, and to subsequent data and knowledge integration and reuse. EC maximising the benefits from ARC-funded research, including by ensuring greater access to research data. Australian Research Council

4 A Pictorial History 1600 s the present PC s-1981-present Rise of Supercomputer 40,000-5, s-2000 s 2500BC BC 75-AD present BC 1950 s-1970 s Good archival properties under HDD Write Huge diskfor arrays (PB s) Slow Very High OK slow Write Data write and Density Rates Read rates Rates 2002 tape right conditions Floppy Disk -loss MB Even huger tape libraries OK Excellent High Subject long Write term to long Rate, preservation term and Slow Rise of Magnetic Media Slow write rates, low data Exabyte Tapes 20GB@3MB/s (500PB@>140MB/s) (needs archival read corruption rate right conditions) Winchester hard disk (60density Tape Libraries 7 tapes Subject Corruption Good Low data Archival to density deliberate unlikely Material and 140MB) Easily corrupted/destroyed accidental Mainly Not Many subject Uses usedsubjective to fordeliberate tax Reel Tapes Read is very corruption/overwrite recording corruption (160MB@57kbps) Lacks Standardisation Mainly Needs good used for archival recording hunting techniques parties

5 Why the Explosion? Research Infrastructure trends: Internationalisation Diversification Large Scale Projects: SKA ITER LHC Human Brain Project IPCC/CMIP ITER middle age 19th century 20th century 21st century

6 Mirrored in Fusion Volume of Data (GB/Day) Daily Data Volumes MAST JET 2007 JET 2017 ITER DEMO

7 Data Management and the Data Lifecycle Digital Curation Centre UK Data Archive

8 Data Management The FAIR Principle Promoted by EC and RDA Findable Descriptive Metadata, Human and Machine Readable, Defined Ontology Accessible Unique and persistent identifier, metadata, security, licensing DOI, PID, ARK, PURL, URN, Open Data Interoprable Well defined file format, machine (and human) readable, metadata Within a community as well as outside Reusable Bit preservation, format migration, metadata, provenance Within a community as well as outside 4-star competing principle

9 Fusion Data Management Fusion Experiments operate as Islands Each site does an excellent job of curation Getting remote access is a bit like making a phone call in the 1930 s Sometimes it s easier to go to the data but that can be difficult and expensive Local Standards (at worst) but two main ways of accessing data HDF5 (open standard, widely used, self describing) MDSPlus (open source, optimised for fusion, bridges for reading HDF5 files) UDA provides abstraction (another standard?) ITER Data may be federated geographically Islands won t work any more

10 Future of Fusion DM? Data still stored at islands Automated Data Replication Federated and Optimised Data Access (bridging the islands) Ability to prestage data to HPC Data Abstraction Access any data in a well defined format Without the need for code recompilation Using something like OpenDDL (bridging the formats) Why? Analyse any data, anywhere, anytime (AAA) Test UDA at scale What s Missing? Federated AAI, Metadata Standards, data placement rules

11 Federated Authentication is HARD??? BUT IT CAN AND HAS BEEN DONE Problem is one of politics and perception, not technical

12 Examples of Federated Data Mangement Michael Loughlin

13 European Plate Observing System (EPOS) 25 Countries 141 Institutions 128 Laboratories 7224 Sensors 256 Research Infrastructures >1PB of data

14 Metadata

15 The ESGF peer-to-peer enterprise system provides services and resources essential for global-scale Earth system science. This system is developed, deployed and maintained by an international multi-agency federation. ESGF s open source, operational code base disseminates petabytes of data including model simulations, observational, and reanalysis data for research assessments. 1 CEDA UK 2 DKRZ Germany 3 ANU NCI Australia 4 NOAA GFDL 5 NASA GSFC U.S. U.S. 6 IPSL France 7 NASA JPL U.S. 8 DOE LLNL U.S. 9 LiU Sweden

16 Data Management Distributed Search Federation Analysis and Visualization Provenance Capture Security Network Compute Facilities Dynamic Resources Data Transfer ESGF ensures equal access to large disparate datasets. It enables scientists to evaluate models, understand their differences, and explore the impacts of geophysical disturbances through a common interface, regardless of data location. Long-Tail Publication Data Citation Machine Learning Current Capabilities Future Capabilities

17 ESGF: Digital Footprint Supports >700,000 datasets from universities as well as national and international laboratories. ~4 million datasets downloaded. Manages >5 PB of data in the total ESGF federated archive, which is expected to expand to >40 PB of uncompressed data, distributed across >25 projects and ~70 model intercomparison projects (MIPs). Services 18 highly visible national and international geophysical data products, including CMIP3, CMIP5, and soon CMIP6.

18 External HPC & HTC services Metadata registry & catalogue services (User-initiated) On-demand computing ICOS repository (data & metadata) Data products & metadata Raw data (Near Real-Time) sensor data + metadata Compute request Data ingestion Metadata services Data discovery & access Usage tracking Ecosystem Thematic Centre Wide range of end user communities User 1 User 2 User 3 User 4 Results ICOS Carbon Portal Central Analytical Laboratory Measurement & station metadata Elaborated data products Near Real-Time data & metadata QA/QC:ed data & metadata Data management Repository administration Preservation planning User community support Atmospheric Thematic Centre Ecosystem, atmosphere and ocean measurement stations Oceanic Thematic Centre TB/yr (RAW) 2 GB/year (processed) 100GB/year (high level) Anyone can ACCESS (after registration) More formal procedures for upload SparQL end-point for machine searchability Custom AAI CPAuth ncluding local accounts and social ID

19

20 R Made possible through the use of xrootd ( Data accessed via GUID or algorithmic data placement X509 Security with Virtual Organisations and Roles

21 European Open Science Cloud Data Management Services Global searchable metadata service Long Term Archival Repository Services Data Transfer Services Federated AAI Federated Compute Services (HTC) Cloud Computing Infrastructure Application on Demand Data Transfer Services Federated AAI Execution Frameworks PaaS Data Analytics Services Federated AAI

22 Key Challenges to Federated Data Management Agreements are Key AAI and Metadata Make sure users are involved in decision making process Good governance Make a function centric architecture Communication Semantics and syntax Loss of control What s in it for me Lack of funding Data models don t replace the need for information architecture Lack of infrastructure Legacy systems Derived from Enterprise Architecture Shared Interest Group 2003

23 Conclusion Scaling up current solutions is not be an option Federation and ease of access are key It takes longer to set up am infrastructure than planned Start designing and limited scale testing now 5 years before first ignition is too late Start with the hard things at current scale Metadata, federated authentication, provenance, interoperability, efficient data movement TRUST is KEY Make sure ALL stakeholders are involved But drive has to come from the top

24 BACKUP

25 EUDAT Authentication (SAML/X509)

26 Integrating an existing identity provider

27 ESGF AAI

28 Accessible Attributes (EGI X509/openID/SAML) Attribute friendly name Attribute OID Example value edupersonuniqueid urn:oid: ef ffe53c39b75bdcef46689f 5d26ddfa cc4fb5ce97e9ca8 mail urn:oid: displayname urn:oid: John Doe givenname urn:oid: John sn urn:oid: Doe edupersonassurance urn:oid: distinguishedname urn:oid: /C=NL/O=Example.org/CN=John Doe edupersonscopedaffiliation urn:oid: edupersonentitlement urn:oid:

29 EGI Application on Demand