Open Access to the GTS: A JCOMM Observations Coordination Group Pilot Project Kevin O Brien University of Washington, USA NOAA/PMEL, USA David Legler NOAA/OOMD, USA Nadia Pinardi University of Bologna, Italy Bill Smith, Michelle Little, Kevin Kern NOAA/NDBC, USA Simona Simoncelli, Alessandro Grandi INGV, Italy Vanessa Cardin, Alessandro Crise, Pierre Poulain, Elena Mauri OGS, Italy Begona Perez, Fernando Manzano, Enrique Alvarez Fanjul Puertos del Estado, Spain James Potemra University of Hawaii, USA Eugene Burger, Heather Tabisola, Jessica Cross, Meghan Cronin, Noah Lawrence-Slavas NOAA/PMEL, USA
Introduction Goal is to provide data producers a simpler method of submitting real time data to GTS infrastructure and data consumers a simpler method of access A pilot project to take well understood physical ocean data from known platform types and inject in near real-time onto the GTS for distribution globally. Retrieve these and other data from the GTS and make available through interoperable web services. * in situ marine observations and upper air observations are regarded as essential by WMO Resolution 40 and they can be exchanged without charge and with no conditions on use
Partners NOAA Observing System Monitoring Center (OSMC), USA Data management, encoding/decoding GTS data, providing GTS data access Puertos del Estado, Spain Moorings, tide gauges NOAA/PMEL, USA 5 SAILDRONE platforms (3 in Arctic, 2 in Tropical Pacific) National Institute of Oceanography and Experimental Geophysics (OGS), Italy E2M3A mooring US Integrated Ocean Observing System/PacIOOS, USA Bottom Pressure Recorder National Institute of Geophysics and Volcanology (INGV), Italy GTS Access and GTS data tests JCOMMOPS, France WMO IDs for GTS platforms
Why? Real time data (via GTS) has very high societal value Local and global forecasts Tsunami detection and warning Resource allocation and management Rescue missions at sea Many more Carse, F., et al. (2015) Roquet, F., et al. (2013) discuss improvements using tagged seal data Carse, F. et al 1-6% improvement in root-mean-square error statistics using data provided by tagged seals Roquet,F. et al Inclusion of seal-derived data substantially modifies the estimated surface mixed-layer properties and circulation patterns within and south of the Antarctic Circumpolar Current. GTS is a convenient way to SHARE data GLOBALLY Barriers exist to distributing and accessing real time data on the GTS
GTS diagram circa 1994 Courtesy of Mark Bushnell
Barriers Complications for GTS data insertion WMO ID required Unique ID for every platform reporting on the GTS For platforms such as Tide Gauges, can be unclear who should generate a WMO ID Connection to National Data Center required Complexity of data format required BUFR is replacing TAC Currently not within scope of short term or small projects
Barriers Complications for GTS data access How to get to GTS data? BUFR decoders are required Even decoded BUFR messages are difficult to fully understand Data missing from GTS Feeds Identifying new GTS headers difficult Decoding issues can cause gaps Transition to BUFR highlights this in near term Area all GTS nodes created equal? How to ensure machine to machine access of data?
Data Ingestion Platform observations and metadata to be available in known format (netcdf, csv, xml) and from well understood platforms Metadata requires WMO ID to be assigned to platform WMO IDs generated by JCOMMOPS Data and metadata will then be ingested into ERDDAP server/framework Participating GTS institution (NDBC) will use the defined ERDDAP API to access the platform observation data and metadata Participating institution (NDBC) will encode the data and metadata using the appropriate BUFR template and submit data using custom GTS header ( Participating institution will inject encoded BUFR message onto GTS This process will be automated for daily or more frequent data ingestion, as required
Data Ingestion GTS Add WMO ID ERDDAP data platform Harvest from ERDDAP Transmit to GTS ENCODE into BUFR
Partners WMO ID Platform Name Platform Type Institution Location (approx) 6101404 Algeciras Coastal Mooring PdE Med Sea 6201070 Langosteira Coastal Mooring PdE Atlantic (Spanish coast) 6101403 E2M3A E2M3A OGS Adriatic Sea 4802011 1001 Saildrone PMEL Arctic 4802012 1002 Saildrone PMEL Arctic 4802013 1003 Saildrone PMEL Arctic 5801950 1005 Saildrone PMEL TPOS 5801951 1006 Saildrone PMEL TPOS 5100275 Aloha Cabled Observatory Bottom Pressure Recorder PacIOOS Hawaii Gijon Tide Guage PdE Ferrol Tide Guage PdE Hielva Tide Guage PdE Fuerteventura Tide Guage PdE Palma de Malloca Tide Guage PdE
Partners Volvo Ocean Race 2017-2018 7 teams
Data Access The OSMC project already pulls data hourly from the GTS at the National Data Buoy Center and makes this data available through an ERDDAP framework The OSMC project will retrieve the newly inserted platform data from the GTS, decode the BUFR data with the appropriate decoder and then add the data and metadata to the OSMC database OSMC project will develop webaccessible tools and visualizations to illustrate the platform data in context with the other real time observations available from the GTS
Data Access Google Earth Live Access Server DECODE from BUFR ERDDAP data platform ipython notebook GTS Metadata available through international standards IOOS coastal modeling Matlab ESRI Geoportal Data flows directly into users applications
Saildrone TPOS atmp Saildrone 2012 atmp Saildrone 2011 atmp Saildrone 2013 atmp Algeciras sst E2M3A atmp Langosteira atmp ACO sea water pressure
Data Access Data Access test and comparison with Copernicus Marine Environment Monitoring service (CMEMS) and the Observing System Monitoring Center (OSMC) GTS feed for Mediterranean sea in month of September, 2017 Simona Simoncelli and Alessandro Grandi, INGV Ship, VOS, VOSclim data in GTS, not in CMEMS Fixed buoys and moorings available in CMEMS not in GTS Drifting buoys in CMEMS not in GTS (BUFR issue)? Some CMEMS floats missing from GTS No TSG or river flow data data in GTS
Data Access Data Access test and comparison with Copernicus Marine Environment Monitoring service (CMEMS) and the Observing System Monitoring Center (OSMC) GTS feed for Mediterranean sea in month of September, 2017 Simona Simoncelli and Alessandro Grandi, INGV Conclusions Exercise illustrated importance of GTS data for forecasting activities Most GTS data available within 12 hours 43% of CMEMS NRT data only available after 24 hours GTS availability is fundamental to increase predicative capabilities Thanks to 2x daily observations available within 12 hours of collection Both GTS and CMEMS were missing some data types
Next Steps (ingestion) Lots of enthusiasm for the project Support from JCOMM (Open Access GTS Task Team?) Need support from National Data Centers (GISC, DCPC,NC) who would agree to harvest and encode data Maybe the future would allow a GTS encoder in a box - not ready now Need to entrain individual data collectors or global GTS Data Assembly Centers in use of ERDDAP
Next Steps (ingestion) Participating National Data Center Entrain/Enable GTS Data Assembly Centers GTS Data Assembly Center
Next Steps (ingestion) NetCDF/HDF as potential GTS transport format WMO has formed the Inter-Programme Expert Team on Operational Weather Radars (IPET OWR).This team is evaluating CfRadial2 (based upon HDF/netCDF) as a probable candidate to replace the BUFR. GTS WIS Task Team on Evolution of WIS (TT ewis) Development of WIS 2.0
Next Steps (access) Need support from National Data Centers (NCs, DCPCs) who would agree to decode and provide access to data US NOAA Observing System Monitoring Center (OSMC) already does this through ERDDAP server
Next Steps (access)
Next Steps (access) Need support from National Data Centers (NCs, DCPCs) who would agree to decode and provide access to data US NOAA Observing System Monitoring Center (OSMC) already does this through ERDDAP server Further investigation of GTS as a distribution mechanism Develop GTS data flow metrics Recommend ERDDAP as the data platform for uniformity of both ingest and access of GTS data Would integrate with current delayed mode DM activities and strategy supported by JCOMM OCG
Open Access to the GTS: A JCOMM Observations Coordination Group Pilot Project Thank You! Kevin.M.O Brien@noaa.gov kob@uw.edu