Dataset Interoperability Working Group

Transcription

1 Dataset Interoperability Working Group Co-Chairs: Charlie Zender and Peter Leonard Ed Armstrong, Mary Jo Brodzik, Joe Glassy, Aleksander Jelenak, Siri Jodha Khalsa, Wenli Yang; Steve Berrick, Chris Lynnes, Dan Marinelli (ESDIS POCs); S. Bailey, W. Baskin, S. Bennett, D. Blodgett, E. Criswell, F. Ding, S. Flynn D. Fowler, A. Herbert, J. Johnson, A. Khan, H.J. Lee, J. Lee, X. Li, E. Pauli, R. Ramapriyan, K. Yang (and of course Steve Olding!) ESDSWG Meeting Greenbelt, MD April 6-8, 2016 Seminar on Web

2 DIWG Mission Increase Earth Science dataset interoperability "Interoperable" considers compliance with and exploitability by HDF, netcdf, CF, ACDD, and other standards and conventions. netcdf ACDD ISO HDF CF

3 DIWG 2015 Accomplishments RFC, Public Draft of 12 recommendations from DIWG Progress on Swath Structures in Earth science products Three new (2015) interoperability recommendations Continued improvement of metadata compliance checking Leadership of CF2 extensions including group hierarchies

4 DIWG 2015 Accomplishments RFC, Public Draft of 12 recommendations from DIWG Progress on Swath Structures in Earth science products Three new (2015) interoperability recommendations Continued improvement of metadata compliance checking Leadership of CF2 extensions including group hierarchies

5 Twelve Recommendations Public Draft (here) advanced through new ESDS RFC Process. Revisions responded to dozens of comments. DIWG thanks Elli Pauli, Allan Doyle, Yonsook Enloe!

6 DIWG 2015 Accomplishments RFC, Public Draft of 12 recommendations from DIWG Progress on Swath Structures in Earth science products Three new (2015) interoperability recommendations Continued improvement of metadata compliance checking Leadership of CF2 extensions including group hierarchies

7 Progress on Swath Structures 1. Defined (here) types of swath data structures under discussion 2. Expanded DIWG GitHub library of CDL swath practices (here) 3. Identified key issues to address (georeferencing, multiple orbits...)

8

9 DIWG 2015 Accomplishments RFC, Public Draft of 12 recommendations from DIWG Progress on Swath Structures in Earth science products Three new (2015) interoperability recommendations Continued improvement of metadata compliance checking Leadership of CF2 extensions including group hierarchies

10 3 New DIWG Recommendations: 14. Embed complete coordinate reference system in file (details). 15. Use unique filenames with these minimal ingredients (details). 16. Avoid NaN and Inf in all data and metadata (details).

11 3 New DIWG Recommendations: 14. Embed complete coordinate reference system in file (details). 15. Use unique filenames with these minimal ingredients (details). 16. Avoid NaN and Inf in all data and metadata (details).

12 Embed Complete CRS in File 14. We recommend that Earth Science data products provide a complete set of grid coordinate reference system parameters and that the information be encoded in the data files in an interoperable way such as using CF-netCDF coordinate variables and HDF5 dimension scales, with additional parameters added if necessary (details).

13 3 New DIWG Recommendations: 14. Embed complete coordinate reference system in file (details). 15. Use unique filenames with these minimal ingredients (details). 16. Avoid NaN and Inf in all data and metadata (details).

14 Unique Filename Requirements 15. We recommend that each Earth Science product file in a public archive should be uniquely named to improve interoperability and avoid confusion. The minimum requirements for a unique filename are 1) a unique product name, 2) the date and time of the first observation in the file, and 3) a unique version number for each release of the product. While uniqueness and machine predictability are important considerations, these minimal elements of the name should be easy for humans to understand (details). > zender@aerosol:~/nasa_data$ ls MOD10CM.A hdf # Fine MOD10CM.A hdf # Fine MODIS_L2N_ T1120.nc # Version? MODIS_L2N.hdf # Date? Version? my_cat.hdf # Unique?

15 3 New DIWG Recommendations: 14. Embed complete coordinate reference system in file (details). 15. Use unique filenames with these minimal ingredients (details). 16. Avoid NaN and Inf in all data and metadata (details).

16 Au revoir Nan et Inf We recommend that Earth Science data products not store NaN or Inf in any field values, nor as an indicator of missing or invalid data (details).

17 DIWG 2015 Accomplishments RFC, Public Draft of 12 recommendations from DIWG Progress on Swath Structures in Earth science products Three new (2015) interoperability recommendations Continued improvement of metadata compliance checking Leadership of CF2 extensions including group hierarchies

18 2013: CF-compliance HDF/netCDF files with ncdismember First checker to work with hierarchical HDF/netCDF...

19 : Metadata Compliance Checker (MCC) by Ed Armstrong, Oliver Chang, Dave Foster

20 DIWG 2015 Accomplishments RFC, Public Draft of 12 recommendations from DIWG Progress on Swath Structures in Earth science products Three new (2015) interoperability recommendations Continued improvement of metadata compliance checking Leadership of CF2 extensions including group hierarchies

21 2015: Extending CF1 to CF2 1. NSF-funded EarthCube Integrative Activity (PI: E. Davis, UCAR) 2. CF2 Workshop (details) in Boulder in May, 2016 (and 2017!)

22 DIWG Follow-On Activities: 1. Best-Practices for Swath-like Structures 2. Improve interoperability and compliance tests (Simplify and infuse CF-checking into NASA) 3. Extend CF metadata for groups (CF2)

23

24 Supplementary Slides