Data discovery and access via the SeaDataNet CDI system Central dataproducts and data services on distributed data. Peter Thijsse MARIS CLIPC IS-ENES workshop, KNMI, November 2014
Outline 1. Introduction to Common Data Index (CDI) system 2. Options of metadata and data access: a) WMS / WFS b) GI-CAT CSW c) OpenSearch d) Using a dedicated buffer and services 3. Discussion
1. Introduction to CDI
Common Data Index (CDI) Data Discovery and Access Service is a fine-grained index (ISO 19115 / ISO 19139) to individual data measurements (such as a CTD cast or moored instrument record). The CDI Data Discovery and Access Service provides users a highly detailed insight and unified access to the large volumes of marine and oceanographic data sets managed by the distributed data centres An intelligent middle tier connection is configured between the SeaDataNet portal and the local data management systems at each of the data centres. A shopping basket allows users to submit a shopping request for multiple data providers in one go and to follow its processing by each of the providers via an online transaction register.
Delivery Services CDI V3 architecture
All directories support the steps from data discovery to unified data access and delivery
Common Data Index (CDI) data discovery and access service of archived data quality checked data.
> 1,6 million CDI entries from 34 countries and >100 data centres and 487 originators for physics, chemistry, geology, geophysics, bathymetry and biology; years 1800 2014; 85% unrestricted or under SeaDataNet licence
2. Interoperability options
OGC WMS / WFS CDI supports OGC WMS v1.3 and WFS v1.1 services Horizontal exchange of geographical coverage of metadata Applicable in any map viewer WFS points back for more metadata and request for data acces to the central CDI system http://geoservice.maris2.nl/wms /seadatanet/cdi_v2/seadatanet
GICAT / CSW Horizontal exchange to overarching portals CDI database metadata divided in parts: per discipline per datacenter Each part described with ISO19139 metadata and Supplied via CSW or OAI-PMH protocol A.o. available in Ocean Data Portal and GEOSS
OpenSearch Now available for horizontal exchange of metadata Simple search options: Free text Geographical search (radius, box) Temporal search (date from/to) Response in JSON/Atom RSS. Available on virtual subsets (entry points similar to CSW) of CDI database Link per subset leads exactly to search result in CDI interface http://seadatanet.maris2.nl/opensea rch/os_description
Use of dedicated data buffers External portals could make use of a data buffer. According to a specific profile (Specify: datatype, region, datacenters, etc) in the SeaDataNet buffer system a subset is created. The SDN robot builds up a (consistently updated) specific subset of data relevant for external portal. To create this buffer an agreement of involved data centers is required. => Download only for creating dataproducts and after agreement of data owner. Not allowed to redistribute the data, other than via CDI system and download directly from partner
Example use of data buffer
Services on top of buffer Dedicated user interface for quick retrieval of data (no need to wait for order system and/or confirmation of datacenter). User are registered SDN users and activity is measured. WMS/WFS possible on top of subset for integration in external portal. OpenSearch option for integration in external portal. WMS/WFS and OpenSearch point back to CDI portal for data order. Machine to machine services API to be used to create dataproduct: Create_order: enabling to configure a discovery profile for shopping in an authorised central buffer database; Get_order_info: enabling to retrieve info from the RSM about the progress of the order processing; Download_order: enabling to download the orders, when ready, as zip files with metadata and data sets. Services secured by SDN CAS (AAA) login
EMODNet and SeaDataNet action results: Visualisation of climatologies on top of validated buffer http://www.seadatanet.org/products
Regional products (Internal SDN product group): Temperature Baltic Sea 1975-2008
Regional products (SDN product group): Temperature 1975-2008 North Atlantic Ocean
Regional products EMODNet: Example catalogue EMODNet Chemistry http://sextant.ifremer.fr/en/web/emodnet_chemistry/catalogue
Regional products EMODNet: Visualisation Seasonal nitrate (1970-2009) from catalogue EMODNet Chemistry
Specifications Previous products are developed using a.o.: Thematic and regional buffers (data collections) Harmonised raw datasets in ODV format (some NetCDF) using ODV software for aggregation: http://www.seadatanet.org/standards-software/software/odv DIVA reanalysis software (Univ of Liege): http://www.seadatanet.org/standards-software/software/diva Oceanbrowser visualisation software (Univ. of Liege), using OGC standards, NetCDF files Sextant data product catalogue (IFREMER) based on GeoNetwork (able to be accessed via CSW, OAI-PMH), products described using SeaDataNet vocabularies where possible.
THANK YOU!