Dataset Interoperability (formerly HDF5 Conventions) Working Group

Transcription

1 Dataset Interoperability (formerly HDF5 Conventions) Working Group Co-Chairs: Charlie Zender and Peter Leonard Ed Armstrong, Sean Bailey, Bill Emanual, Fan Feng, Doug Fowler, Ted Haberman, Beth Huffer, Aleksander Jeknah, James Johnson, SiriJodha Khalsa, H. Joe Lee, Jeff Lee, Dan Marinelli (ESDIS POC), Evelyn Nakamura, Lindsay Parker, Joel Plutchak, Pamela Rinsland, Jennifer Wei, Barry Weiss, Kent Yang (and of course Steve Olding!) ESDSWG Meeting Greenbelt, MD March 24-26, 2014 Seminar on Web

2 DIWG Mission Increase interoperability of NASA Earth Science datasets "Interoperable" considers compliance with and exploitability by HDF, netcdf, CF, ISO, and ACDD standards and conventions. netcdf ACDD ISO HDF CF

3 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

4 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

5 Design HDF5 files to maximize netcdf4 interoperability Format/Structure, Tests, Limitations, Incompatibilities

6 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

7 Disambiguate HDF4- from netcdf-packing algorithm/convention HDF4_CAL: unpacked = scale_factor*(packed add_offset) netcdf: unpacked = scale_factor*packed + add_offset

8 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

9 Include (about four to ten) basic CF attributes in datasets units, long_name, standard_name, _FillValue...

10 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

11 Check CF-compliance of NASA-distributed HDF/netCDF files with CFchecker and NCO s ncdismember

12 Check CF-compliance of NASA-distributed HDF/netCDF files reports CF issues with netcdf3/4 and HDF4/5 files...

13 Check CF-compliance of NASA-distributed HDF/netCDF files Produced HOWTO do CF-checking for DAACs...

14 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

15 Engage with CF community on guidelines to exploit CF conventions in group hierarchies Our proposal defines main tenets (attribute scope, inheritance) necessary to make CF group-aware...

16 Engage with CF community on guidelines to exploit CF conventions in group hierarchies CF community concerns must be addressed first...

17 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

18 Use NASA-endorsed profile of ISO and related standards, as defined by MENDS Illustrate, evolve best practices with structured metadata. Case studies SMAP, ICESat-2

19 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

20 Use existing mapping between ACDD and ISO developed by ESIP ACDD mapping to ISO global and variable-level metadata works well...

21 DIWG Recommendations 1. Design HDF5 files to maximize netcdf4 interoperability 2. Add two new attributes to disambiguate HDF4-packing from netcdf-packing (a.k.a. scaling or calibration) algorithms/conventions 3. Include (about four to ten) basic CF attributes in datasets 4. Check CF-compliance of NASA-distributed HDF/netCDF files (e.g., with cfchecker/ncdismember) 5. Engage with the CF community on guidelines to exploit CF conventions in group hierarchies 6. Use NASA-endorsed profile of ISO and related standards, as defined by MENDS 7. Use existing mapping between ACDD and ISO developed by ESIP 8. Participate with ESIP in the development of Version 2.0 of ACDD

22 Participate with ESIP in the development of Version 2.0 of ACDD DIWG members updating ACDD (e.g., groups...)

23 DIWG Follow-On Activities & Approaches: 1. Practical definition of interoperability 2. Working requirements and tests for interoperability and compliance Simplify and infuse CF-checking into NASA 3. Recommend metadata practices/extensions to harmonize conventions with HDF capabilities Iterate group-aware metadata conventions 4. Identify best practices to make NASA-distributed data accessible through netcdf API Survey Aura, SMAP, ICESat2, TROPOMI...

24 DIWG Follow-On Deliverables: 1. Improved group-aware metadata... a. Conventions As potential extension to CF b. Practices To deliver metatdata (e.g., ISO for SMAP) 2. More robust CF-compliance checking Handle corner cases, make HOWTO for DAACs 3. Constraints on group/variable names Length, character set

25 Supplementary Slides