Ocean glider data management: UK and European progress, and the global context

Transcription

1 Ocean glider data management: UK and European progress, and the global context Justin Buck + many collaborators British Oceanographic Data Centre

2 The numerous collaborators Sylvie Pouliquen Thierry Carval Jean-Philippe Rannou Marine Autonomous Robotic Systems (MARS) facility Task Data management Cost action ES0904 Data STSM partners Mark Hebden Lise Quesnel

3 Outline Introduction EU FP7 GROOM Project task 3.2 Common data tools On-going and future work

4 Introduction

5

6 GOOS status (JCOMMOPS)

7 User expectations Data easily accessible from a unique point/portal Data coherent in terms of : Data format Data Quality Processing chain ( clearly documented) Additional requirements for Monitoring and forecasting users Data are available in near real time (within 24 hours) Data are available in delayed mode after calibration and /or validation (typically within 12 months)

8 Stakeholder expectations Opportunity to use more observations than they could afford alone Operate jointly part of the network Benefit from the other partners' experience from design to implementation to data management and user uptake A key phrase for funding bodies: Acquire once, use multiple times

9 However... Broadly speaking, two distinct types of deployment: 1) Process type studies short duration, spatially restricted, typically associated with a cruise. Designed to answer a specific question and can have data restrictions. 2) Sustained observation studies longer duration, regional scale missions. Typically repeated sections. Of interest to the ocean modelling and forecasting community. So there is potential for conflict between project interests and operational interests!

10 EU FP7 GROOM Project

11 EU FP7 GROOM project Gliders for Research, Ocean Observation and Management (GROOM) Design study to evaluate the requirements to set up a sustainable European glider infrastructure to safely operate individual and fleets of gliders in order to create a continuum of observations. Task 3.2 aims to deliver a template data management

12 GROOM task 3.2 partners HZG : Lucas Merckelbach, Rolf Riethmueller OGS : Elena Lauri CSIS : Tomeu Garau, Joaquin Tintore NURC : Reiner Onken PLOCAN : Carlos Barrera CNRS LOCEAN: Pierre Testor, Laurent Mortier CNRS DT-INSU : Laurent Beguery, Karim Bernardet, Elodie GONINHO LOV: Vincent Taillandier, Fanrizio D Ortenzio, Lars Stemman HCMR : Dimitris Kassis Kostas Knittis NERC : David Smeed UIB : Peter Haugan IFM-Geomar: Gerd Krahmann OC-UCY: Dan Hayes AWI: Agnieszka Beszczynska- Möller SAMS: Estelle Dumont UEA: Jan Kaiser Ifremer: S Pouliquen T Carval L Petit de la Villéon

13 Task 3.2 data system goals

14 Task 3.2 data system goals

15 GROOM data assembly centres Country DAC Glider Operator Delayed Mode PI UK BODC UEA, SAMS, NERC (NOC, POL) CEFAS, BAS France CORIOLIS LOCEAN, LOV, MIO, LEGOS, LPO, DT-INSU Italy CMRE CMRE UEA, SAMS, NERC (NOC, POL), CEFAS, BAS LOCEAN, LOV,MIO, LEGOS, LPO OGS/CORIOLIS OGS OGS Germany HZG HZG HZG CORIOLIS GEOMAR,AWI GEOMAR, AWI SPAIN CORIOLIS PLOCAN PLOCAN SOCIB SOCIB, CSIC SOCIB, CSIC Norway UiB/IMR UiB UiB Cyprus OC-UCY OC-UCY OC-UCY Greece CORIOLIS HCMR To Be Defined Finland To Be Defined To Be Defined To Be Defined Ireland BODC? MI MI Poland CORIOLIS IOPAS IOPAS

16 A common data exchange format EGO glider data format established by the GROOM community (October 2012): Climate and Forecast (CF) and SeaDataNet compliant NetCDF. Interoperable with data standards being developed internationally (e.g. IMOS in Australia and IOOS in the U.S). Standard quality control protocols for both near real-time and delayed-mode glider datasets (utilising Argo). Ensures that glider data, metadata and technical information are stored and distributed in a consistent manner.

17 Common data tools

18 System requirements Modular [for easy modifications and maintenance] Consistent input/output interfaces between modules Simple! Flexible enough to handle varying use cases Robust and handles any error with known rollback points Reports every failure Can be mixed and matched with other systems, especially external to BODC Similar tasks such as reading from the database are done using similar approaches

19 Automated (Provider) upload Data Provider External Database Unauthorised user Authorised user Source data Source meta Delayed mode meta/data Discovery Automated (BODC) download Prepare request Checkout BODC workflow Register Arrival Archive Convert to standards [meta/data] Error Handling & Reporting Data Scientist Authorise Authenticate QC/Calibration Automated download/upload Merge as relevant Manual Processing Store as relevant Database meta File System meta/data Database data

20 BODC workflow & common tools Automated (Provider) upload Data Data Provider Source data Delayed collection mode Source meta meta/data and secure Automated (BODC) download archive Database meta Register Arrival Archive Convert to standards [meta/data] QC/Calibration (EGO code) Merge as relevant Store as relevant Database data File System meta/data External Database Error Handling & Reporting Reformatting & processing Internal storage Data Scientist Manual Processing Prepare request Unauthorised user Authorised user Data Authorise Authenticate Automated download/upload Discovery Checkout delivery

21 STSM to develop common tools Brest, December 2012 Goal to develop first version of common tools Tomeu Garau, Daniele Cecchi (NURC) Thierry Carval, Jean-Philippe Rannou (Ifremer) Justin Buck, Mark Hebden, Lise Quesnel (BODC)

22 Common tools workflow Source data Seaglider Slocum Other gliders Collection of source data readers, multiple readers per platform type. Data from single transmission converted to a.mat Conversion of.mat output to EGO format NetCDF for transmission Processing modules Single transmission quality control routines Merge single transmission files to produce EGO NetCDF containing trajectory for deployment Time series quality control routines Corrections/calibration routines Data delivery Originator GDAC GTS

23 JSON files EGO NETCDF writer controlled by JSON files. Example on right describes CTD sensor metadata.

24 JSON files drive a generic system Raw data files JSON files describing variables to transfer JSON files describing deployment, glider and sensor configurations JSON files describing EGO format EGO NetCDF writer EGO NetCDF file *The flexibility means system is usable in other projects e.g. SMRU animal tags

25 Real time quality control Test are adaptations of established Argo QC RTQC configured via JASN files Presently implemented tests Valid range (e.g. TEMP, PRES, speed etc) Regional range Gradient Spike Stationary Position on land Density inversion

26 On-going sensor corrections/qc Within GROOM there are pertinent on-going actions linked to sensor corrections and RTQC Pressure Correction for Slocum Gerd Krahmann Oxygen : Thierry Carval Chlorophyl-a: E Mauri/ F D ortenzio

27 Sharing of code Code is available on the Ifremer SVN repository with Mantis used for bug tracking Pooling of common code from multiple centres Code has a reciprocal public license A SeaDataNet login is required Thierry Carval is administrator and can grant access

28 Data availability/time scales During Summer DACs to test and start to generate their glider files RTQC BODC, CORIOLIS, CMRE, UCY ( HZG, SOCIB,OGS,UIB- IMR) Verification with PI of the files produced September Start to populate the Groom GDAC Advertise within groom users November Open Groom GDAC to outside users

29 The last bits of work Additions to code to fully comply with the EGO/GROOM format (version 2) _QC and _UNCERTAINTY channels Addition of generic glider phases channel Integrate code from other STSM members into workflow e.g. SLOCUM reader Quality control code Documentation is on-going

30 Common tools workflow Source data Seaglider Slocum Other gliders Collection of source data readers, multiple readers per platform type. Data from single transmission converted to a.mat Conversion of.mat output to EGO format NetCDF for transmission Processing modules Single transmission quality control routines Merge single transmission files to produce EGO NetCDF containing trajectory for deployment Time series quality control routines Corrections/calibration routines Data delivery Originator GDAC GTS

31 On-going and future work

32 Delayed mode working groups WG1: QC for T, S Goal correct offset or drift greater than 0.01 in Salinity ; in Temperature Lead : Sunke, Daniele Members : Gerd, Justin,Carlos, Simon, Tomeo, Agnieszka WG2: QC for fluorescence. Lead: Vincent, Members : Simon, Dan (UCY)? WG3: QC for oxygen surface? lag correction? Lead : Gerd/Johannes Members :Bastien, Estelle, Carlos, Rolf( HZG)? WG4: QC for average currents Lead: Lucas Members :Tomeo, Daniele, Alvaro, Svein, Agnieszka

33 International Harmonisation Glider data formats between GROOM, IOOS (US) and IMOS (Australia) are based on the OceanSITES NetCDF format (based on the Argo format) Subtle differences between formats Meet with IOOS and IMOS to harmonise formats making interoperability simple

34 Questions?

35 Persistent identifiers? Thorny topic for real-time/evolving data Important in terms of data citation and publication Building links with other EU projects Potential solution: PID in real time [issuing centre/identifier] i.e. epic:1234/56789 Evolves to DOI when data are final i.e. doi: /56789