Metadata Zoo Dataset Metadata Rebecca Koskela Execu4ve Director, DataONE

Transcription

1 Metadata Zoo Dataset Metadata Rebecca Koskela Execu4ve Director, DataONE eurocris September 9, 2013

2 Outline Data Challenges Metadata Solu=on DataONE addressing the Data Challenge Enabling Scien=fic Discovery Example 2 2

3 Data Management Challenges Entropy Discovery Heterogeneity 3

4 Data Entropy Time of publica4on Specific details Informa4on Content Accident General details Re4rement or career change Death Time (Michener et al. 1997) 4

5 Data Discovery 5

6 Data Heterogeneity Syntax (format) Schema (model) Seman=cs (meaning) Jones et al

7 Research and data life cycle integra=on Plan Analyze Collect Integrate Assure Discover Describe Preserve Many itera=ons of data to final data product 7

8 Open Science Movement 8

9 Metadata as a Solu=on Descrip=ve Subject maver of content Structural Content types and avributes Administra=ve Who, when, how content created Defines who can access, how it can be used IDENTIFY keywords geographic location time period attributes ASSESS use constraints access constraints data quality availability/pricing ACCESS online access order process contacts 9

10 Dataset Metadata Has Value to ALL Data developers Data users Metadata helps Organiza4ons 10

11 Proper Dataset Cura=on Enables Data Reuse 11

12 DataONE addressing the Data Challenge Provide universal access to data about life on earth and the environment 1. Building community 2. Developing sustainable data discovery and interoperability solu=ons 3. Enabling science through tools and services Plan Analyze Collect Integrate Assure Discover Describe Preserve 12

13 Discovery via DataONE Three major components for a flexible, scalable, sustainable network Coordina4ng Nodes retain complete metadata catalog indexing for search network- wide services ensure content availability (preserva=on) replica=on services 13

14 Discovery via DataONE Three major components for a flexible, scalable, sustainable network Coordina4ng Nodes retain complete metadata Member Nodes catalog diverse ins=tu=ons indexing serve local for community search network- wide provide resources services for managing ensure content their data availability (preserva=on) retain copies of data replica=on services 14

15 Discovery via DataONE Three major components for a flexible, scalable, sustainable network Coordina4ng Nodes retain complete metadata Member Nodes catalog diverse ins=tu=ons indexing serve local for community search network- wide provide resources services for managing ensure content their data availability (preserva=on) retain copies of data replica=on services 15

16 Discovery via DataONE Three major components for a flexible, scalable, sustainable network Coordina4ng Nodes retain complete metadata Member Nodes catalog diverse ins=tu=ons Inves4gator indexing for Toolkit serve local community search network- wide provide resources services for managing ensure content their data availability (preserva=on) retain copies of data replica=on services 16

17 Enable Data Discovery ORNL DAAC FGDC, ISO, DIF, FGDC USGS CSAS KNB PISCO SANParks ESA ONEShare UC MerriV CLO/AKN FGDC, ISO, FGDC EML, ISO EML EML, FGDC EML EML EML EML Extract and Align Metadata Augment Metadata Internal Metadata Index Search API LTER EML 17

18 Community Listening Understanding Engaging 18

19 Listening stakeholder surveys persona and scenario development scien9sts library s & librarians data managers cyberinfrastructure development usability tes9ng external assessments / surveys 19

20 Scien=sts want to share data Use other researchers datasets if easily accessible Willing to share data across a broad group of researchers 84% 81% Appropriate to create new datasets from shared data 76% Currently share all of their data 6% 676 Metadata standards DIF DwC DC EML FGDC Open GIS ISO My Lab none 20

21 Understanding 2/3 rd report that organiza=onal help and support is lacking 21

22 The Long Tail Science Volume Specialized repositories (e.g. GenBank, PDB) Orphan data Most of the bytes are at the high end, but most of the datasets are at the low end Jim Gray Rank frequency of datatype (B. Heidorn) 22

23 Intercept researchers where they already work 23

24 Check for best prac=ces Create metadata Connect to ONEShare Data & Metadata (EML) 24

25 Enable Data Discovery Ease of good search key for users Metadata quality key enabler Scaling search is hard Access search API Promote quality metadata Enable search refinement Develop & curate metadata Promote quality metadata Index & replicate metadata Fast, robust search API Validate metadata Seman=cs research collabora=on Drive usability improvements 25

26 Preserve Data and Metadata Support common metadata standards Use checksums for invariance Replicate metadata across all CNs Replicate data across par=cipa=ng MNs Assemble data packages Curate data & metadata Establish replica=on policies Promote quality metadata Manage system metadata Replicate and preserve metadata Enforce checksum consistency Broker & maintain data replica=on 26

27 Enabling Scien=fic Discovery Example 27

28 Data Intensive Science in Biodiversity Research Data Collec9on Organiza9on Valida9on Preserva9on, & Access Data Explora9on, Visualiza9on, And Analysis Kelling et al The full data life cycle from data cura=on to scien=fic learning engage scien=sts, land- managers, policy makers, students, educators, and the public

29 Joining Disparate Data

30 High Performance Compu4ng and Climate Change: Environmental Cues of Migra=on Teragrid SuperCompu=ng

31 Climate Change: Environmental Cues of Migra=on Spring Arrival Dates Expected Spring Arrival Dates with leaf out two weeks early

32 Enabling Scien=fic Discovery Diverse bird observa=ons and environmental data from 300,00 loca=ons in the US integrated and analyzed using High Performance Compu=ng Resources Model results Occurrence of Indigo Bun4ng (2008) Land Cover Meteorology MODIS Remote sensing data Spa=o- Temporal Exploratory Model iden=fies factors affec=ng paverns of migra=on Jan Apr Jun Sep Dec Examine paverns of migra=on Infer how climate change may affect bird migra=on 32

33 Thank you Rebecca Koskela, Execu=ve Director 33