High Performance Data Efficient Interoperability for Scientific Data

Transcription

1 High Performance Data Efficient Interoperability for Scientific Data Alex Ip 1, Andrew Turner 1, Dr. David Lescinsky 1 1 Geoscience Australia, Canberra, Australia

2 Problem: Legacy Data Formats holding us back GA s geoscientific data is currently held in several legacy formats, some of them proprietary Data is not readily interoperable each different format requires custom development.

3 Metadata Tedious but Vital Metadata is of varying quality and accessibility usually held separately to data. Discovery is hit-or-miss. Metadata is sometimes deduced from filename, context or actual data values Need Further Information!

4 Modern Container Formats (e.g. NetCDF/HDF) Transparency and interoperability of text-based file formats, combined with efficiency of binary formats Encapsulated metadata ensures that data and metadata never become separated Internal indexing for rapid subsetting to support large-scale parallel processing Compression and chunking for efficient storage and retrieval Robust, well-proven software libraries for most languages and environments Able to be consumed via web services (e.g. OPeNDAP) using identical code for direct file access

5 INTEROPERABILITY Container Format Options Which one? NetCDF-CF Highly Interoperable Extensive Toolsets Available Large User Base NetCDF4 Simple Standardised Proven (at large scale) HDF5 Self-Describing Flexible & Extensible Efficient & Scalable Robust Well Supported NetCDF Climate & Forecasting conventions (NetCDF-CF) provides enhanced interoperability by further constraining NetCDF4/HDF5, allowing us to leverage existing tools Network Common Data Form v4 (NetCDF4) facilitates the use of data in HDF5 by providing conventions to constrain structure and metadata. Hierarchical Data Format v5 (HDF5) Flexible, proven, domain-agnostic scientific data format

6 What GA has been doing with its geophysics data GA has been working to convert geophysics data (magnetics, radiometrics, & gravity) to netcdf GA already delivers much of its geophysics data (mostly gridded) from the NCI NERDIP infrastructure NCI has helped GA achieve compliance with relevant data & metadata standards and conventions, e.g: netcdf, CF (Climate & Forecasting), ACDD (Attribute Convention for Data Discovery), etc. GA now has systems to synchronise metadata between its catalogue and the data files Have implemented Jupyter Notebook demonstrators for geophysics data

7 Gridded Data vs. Un-gridded (Point & Line) Gridded data is relatively straightforward in HDF/NetCDF (e.g. Earth observation, climate model output, etc) Un-gridded point and flight-line (trajectory) geophysics data is new for netcdf. Approach based on existing NOAA approaches to oceanographic & atmospheric data. Structure of point/line netcdf is very simple each pointwise attribute is a separate variable. Able to be consumed via OPeNDAP web services Metadata requires careful attention. Need to standardise naming conventions for possible inclusion in CF convention

8 Representing Geoscientific Data in NetCDF4/HDF5 netcdf P452MAG { dimensions: variables: sample = ; line = 114 ; line_index = 115 ; int index_lines(line_index) ; int LINE(line) ; index_lines:my_standard_name = "index_lines" ; index_lines:long_name = "zero based index of the first sample in the line, and then one past the end of the last line" ; index_lines:units = "1" ; index_lines:_fillvalue = ; index_lines:_storage = "contiguous" ; index_lines:_endianness = "little" ; LINE:my_standard_name = "line_number" ; LINE:long_name = "line identification number" ; LINE:units = "1" ; LINE:_FillValue = ; LINE:_Storage = "contiguous" ; LINE:_Endianness = "little" ; short bearing(line) ; bearing:my_standard_name = "aircraft_bearing" ; bearing:units = "degrees" ; bearing:_fillvalue = s ; bearing:_storage = "contiguous" ; bearing:_endianness = "little" ; int datecode(line) ; datecode:my_standard_name = "date" ; datecode:long_name = "local date on which the line was flown" ; datecode:units = "yyyymmdd" ; datecode:_fillvalue = ; datecode:_storage = "contiguous" ; datecode:_endianness = "little" ; short flight(line) ; flight:my_standard_name = "flight_number" ; flight:long_name = "flight identification number" ; flight:units = "1" ; flight:_fillvalue = s ; flight:_storage = "contiguous" ; flight:_endianness = "little" ;

9 Making Geoscientific Data Available with Web Services Using OPeNDAP: Data is served remotely in full or as subset Data can be delivered in several formats including ASCII or binary

10 Accessing geoscientific Data in NetCDF4/HDF5 Data can now be rapidly accessed not only in-situ, but also via OPeNDAP from the NCI directly into environments including Python, R or MATLAB

11 Case Study Airborne Electromagnetic (AEM) Data Subsurface conductivity section visualisation in Jupyter Notebook by Neil Symington Originally tab-delimited text, now netcdf. Provides sub-second data reads c.f. minutes. Raw AEM data or derived inversions both handled.

12 Case Study Airborne Magnetic & Radiometric Survey Line Datasets Trajectory (line) data encoded as netcdf, with line groupings and full metadata. Fully-automated discovery of survey line datasets via CSW, data subset retrieval for area of interest via OPeNDAP web service.

13 Case Study Ground Gravity Survey Point Datasets Currently only available as CSV dumps from GA database ground gravity survey point datasets processed into netcdf with full metadata Both raw data and analysis-ready adjusted values contained in the same file NetCDF permits encoding of point-based metadata as enumerated types (e.g. data quality string, instrument, etc) No sexy visualisations (yet) translation work was only completed last week.

14 FAIR Data Principles and GA s netcdf Findable Datasets are catalogued in GA s ISO19115 metadata catalogue. Metadata is harvested by the NCI, ANDS and data.gov.au, amongst others. Accessible Data is published from the NCI via web services (or via HTTP download) Interoperable netcdf is self-describing, open format, with libraries implemented in many different programming environments. Reusable Data has embedded metadata, with links to full metadata.

15 Summary NetCDF-CF is a highly suitable container format for multiple types of geoscientific data GA is in the process of translating several geoscientific data types as part of a pilot program with the NCI NERDIP program Proposed controlled vocabularies will be presented to communities of practice for approval, before being submitted for inclusion in the CF convention Layered, standards-based architecture will speed the development of future-proof systems Improving data interoperability will help break down silos and drive exciting new, transdisciplinary science.

16 Thank you! Acknowledgements: Ross C Brodie, Yvette Poudjomdjomani, Neil Symington Geoscience Australia Kelsey Druken, Lesley Wyborn National Computational Infrastructure Edward King, Matt Paget - CSIRO Phone: Web: alex.ip@ga.gov.au Address: Cnr Jerrabomberra Avenue and Hindmarsh Drive, Symonston ACT 2609 Postal Address: GPO Box 378, Canberra ACT 2601