DataONE Cyberinfrastructure. Ma# Jones Dave Vieglais Bruce Wilson

Transcription

1 DataONE Cyberinfrastructure Ma# Jones Dave Vieglais Bruce Wilson

2 Foremost a Federa9on Member Nodes (MNs) Heart of the federa9on Harness the power of local cura9on Coordina9ng Nodes (CNs) Services to link Member Nodes Inves9gator Toolkit (ITK) Tools for the whole data lifecycle Interoperability 2

3 Requirements for DataONE Scalable Usable by people and agents Resilient to technical and ins9tu9onal change Adap9ve to evolving standards Inclusive of exis9ng communi9es and tools Cognizant of sociological drivers Informed by prior and current work

4 Why a Federa9on? Diverse Federa9on == Resilience Failover for temporary outages Insurance against project/ins9tu9onal failure Diverse Federa9on == Scalability Storage increases with Member Nodes Incremental costs to each MN to replicate Distributes sustainability costs 4

5 Member Nodes Authorita9ve members of the Federa9on Curate their own data holdings Provide unique iden,fiers for each object Ensure availability, quality, and reliability Replicate holdings for other MNs Provide access and access control Log and report accesses to objects Engage with DataONE community Deploy a DataONE- compa9ble sovware system 5

6 Implementa9on Tiers Tier 1 Supports publicly readable content without authen9ca9on or more specific access control rules. Tier 2 Tier 1 plus access control support Tier 3 Tier 2 plus ability to add content through the DataONE service interfaces and provides full support for interac9on with DataONE Inves9gator Toolkit applica9ons and plugins. Tier 4 Support the full set of DataONE APIs and can operate as replica9on targets, accep9ng content from compa9ble (technical and policy) Member Nodes and fully suppor9ng the DataONE content access control rules. 6

7 Characterizing Member Nodes Diverse Contributors Data Types Ecological Environmental Demographic Social/Legal/Economic 45 Individual inves9gators 30 Field sta9ons and networks Government agencies 15 Non- profit partnerships 0 Scien9fic Socie9es Synthesis centers MB 7 60 < Data Sizes % >200

8 Characterizing Member Nodes Diverse Contributors Data Types Ecological Environmental Demographic Social/Legal/Economic 45 Individual inves9gators 30 Field sta9ons and networks Government agencies 15 Non- profit partnerships 0 Scien9fic Socie9es Synthesis centers MB 7 60 < Data Sizes % >200

9 Coordina9ng Nodes Provide coordina9ng services Search and Discovery Preserva9on monitoring Object tracking and replica management User iden9ty management Logging and monitoring Op9mized High availability Performance Scalability

10 The Inves9gator Toolkit Inves9gator Toolkit Web Interface Analysis, Visualiza9on Data Management Client Libraries Java Python Command Line Discovery tools Data Management tools Analysis and modeling tools Cita9on and publica9on tools

11 Data Lifecycle Collect Analyze Assure Integrate Describe Discover Deposit Preserve

12 Data Lifecycle Collect Analyze Assure Integrate Describe Morpho Discover Deposit Preserve

13 Iden9fy objects Goal: Uniquely iden9fy data or metadata objects Support the several iden9fier types widely used Iden9fiers assigned by Member Nodes Uniqueness ensured by Coordina9ng Nodes Resolu9on through Coordina9ng Nodes GUID {3F2504E0-4 LSID PURL

14 Iden9fy people Iden9ty provider selected by the user Member nodes define access rules Rules propagated by Coordina9ng Nodes Iden9ty and access control consistent across en9re infrastructure

15 Deposit Data and Metadata Proxy KNB Native Generic <meta> Science metadata EML, FGDC, DC, ISO, DIF, System metadata Globally unique IDs for data & metadata (DOI, GUID, Hdl, ) Checksums of objects Object policies

16 Preserve Data and Metadata Metadata mirrored at Coordina9ng Nodes Data replicated between Member Nodes CNs manage copies Checksums recorded and verified Promote quality metadata Coordina9ng Nodes

17 Discover Content

18 Integrate and Analyze water temperature (bottom, 10m ADCP)!!!! Graphs and derived data can be archived in DataONE Temperature degrees C !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 01:00 05:00 09:00 13:00 17:00 Time!!!!!!!!!!!!! 16

19 Analysis and Visualiza9on Diverse bird observa9ons and environmental data from 300,000 loca9ons in the US integrated and analyzed using High Performance Compu9ng Resources Model results Occurrence of Swainson s Hawk Land Cover Meteorology MODIS Remote sensing data Spa9o- Temporal Exploratory Model iden9fies factors affec9ng pa#erns of migra9on Jan Apr Jun Sep Dec Examine pa#erns of migra9on Infer how climate change may affect bird migra9on Slide from S. Kelling

20 DataONE System Overview

21 DataONE System Overview

22 DataONE Ac9vi9es Through Year 2 Deploy core infrastructure suppor9ng four fundamental services: Persistent, unique iden9fiers Bit- level preserva9on Search and retrieval Federated iden9ty Along with: Build out and deployment of Member Nodes Add ITK func9onality Test, test, test Ramp up R&D on addi9onal features

23 SoVware Delivered at Public Release Inves9gator Toolkit SoUware SearchPortal R Client Morpho Zotero Fuse FS Excel Mendeley Client Libraries Java Python Command Line DataONE Service Programming Interface (SPI) Member Node SoUware Metacat Dryad GMN CUASHI MerriZ Coordina9ng Node SoUware Service Interfaces Resolu9on Replica9on Iden9fiers Preserva9on Object Store Registra9on Discovery Catalog Monitor Index

24 DataONE Ac9vi9es: Years 3-5 Data sub- selng, transforma9on Visualiza9on Workflow support Seman9c search Seman9c data integra9on Computa9onal, or specialized nodes Inves9gator Toolkit expansion DMP-Tool

25 Cyberinfrastructure Outline CI Architecture, Requirements, and Design Member Nodes Coordina9ng Nodes Inves9gator Toolkit Demonstra9ons

26 Demonstra9ons Collect Analyze Assure Integrate Describe Discover Deposit Preserve 23

27 Demonstra9ons Collect Analyze Assure Integrate Describe Morpho Discover Deposit Preserve 23

28 Demonstra9ons Collect Analyze Assure Integrate Describe Morpho Discover Deposit Preserve 23

29 Demonstra9ons Collect Analyze Assure Integrate Describe Morpho Discover Deposit Preserve 23

30 Demonstra9ons Collect Analyze Assure Integrate Describe Morpho Discover Deposit Preserve 23

31 Demonstra9ons Collect Analyze Assure Integrate Describe Morpho Discover Deposit Preserve 23

32 Describing and deposit with Morpho 24

33 Data discovery 25

34 File system access 26

35 R plugin demonstra9on 27

36 Value of DataONE Discovery and access: Enabling discovery and universal access to data about life on earth from around the world Data integra9on and synthesis: Providing transforma9onal tools that enable cross- culng research Educa9on and training: Providing essen9al skills (e.g., data management training, best prac9ces) for scien9fic enquiry Building community: Combining exper9se and resources across diverse communi9es to collec9vely educate, advocate, and support stewardship of scien9fic data Data Sharing: Providing incen9ves and infrastructure for sharing data from federally funded researchers 28