Kick off meeting - REPORT Deliverable 4.2 Gap Analysis Report Grant Agreement number: 312118 Project acronym: COOPEUS Project title: Strengthening the cooperation between the US and the EU in the field of environmental research infrastructures Funding Scheme: FP7.INFRASTRUCTURES-2012-1 Deliverable No: 4.2 Prepared by: INGV COOPEUS project coordinator s office: MARUM Bremen University Loebnerstrasse 28359 Bremen Tel: +49 421 218 65629 Fax: +49 421 218 65607 E mail: coopeus@marum.de Project website address: http://www.coopeus.eu/
Table of content Executive Summary... 2 1. Status and main features of OOI and EMSO data infrastructures... 2 2. Gap Analysis: goal and objectives... 4 3. General principles for a shared data policy... 5 4. Initial plan for interconnecting data and metadata, and sensor metadata... 5 4.1 Generic Science Case... 7 4.2 User Scenario: Tsunami wave generation and dynamics... 7 4.3 Compatibility of selected sensors metadata... 8 4.4 Sensor registry concept... 8 5. Implementation strategy... 8 6. Conclusions... 9 1
Executive Summary In the recent decades valuable understandings and predicting skill about the environment and climate changes have been developing by scientists in favour of a sustainable management of the environment and for minimising the human ecological footprint. These long-lasting efforts however confront with some limitations related to the still inadequate Earth coverage with sustained in situ observations over the long-term and to the difficulty to access the bulk of the data that various institutions and research groups spread all over the world collect and hold. These limitations are particularly crucial for the oceans, extending over most of the planetary surface, which are the engine driving weather-climate systems across the ocean basins and the continents, directly affecting food production, drought, and flooding on land. The oceans long- and short-term temporal variations and the complexities of ocean dynamics are poorly understood and the seafloor remains not well mapped because of lastingly under-sampling. Innovative and complex marine research infrastructures such as OOI and EMSO have been developing to cover the continental scale with in situ ocean measurements on a continuous and long-term basis and to provide data infrastructures fed with valuable observations from key sites. They represent a breakthrough in the observation capabilities to increase the fundamental scientific understand, to tackle the relevant questions at the scales necessary to understand climate change and its impacts, model ocean circulation and exchanges with atmosphere and even to improve geo-hazard early warning. The potential of a joint use and integration of oceanic data from OOI and EMSO large-scale infrastructures is expected to be great to forming and revise experimental and model frameworks by improving their representativeness and extension, to verify the accuracy of predictions, assess proximity to disruptive change and monitor the effectiveness of adaptation and mitigation strategies [4]. Key aspects of moving forward are global data and long time series. Global context will be provided by the interoperability; partnering and interoperability will increase efficiencies and make a better case for sustained operation. OOI and EMSO are expected to contribute to the global strategy and coordination efforts to achieve real interoperability of oceanic data getting advantage of COOPEUS project framework thus laying the ground for a real use of global data to test recent findings, models and theories and to provide scientific community with new source of global data. The present documents gives the status of the implementation of EMSO and OOI data infrastructures, presents the results of the gap analysis and drafts the actions to start removing the obstacles to the interoperability, sharing and joint use of the data produced by RIs. Specific Actions on which the collaboration is being developed are detailed as well as the selected User Scenario which is considered to strengthen the collaboration on a real challenging scientific question. 1. Status and main features of OOI and EMSO data infrastructures EMSO and OOI have different implementation approaches and time lines: EMSO is founded on a coordination effort among different research institutions, countries and funding agencies while OOI relies upon a single organizational body and a single funding agency although many different institutions are actively involved. Many EMSO nodes have been operating from early 2000 and data are currently flowing into a distributed data infrastructure which is under further coordination. At the time of this report writing, OOI is in the process of completing design and construction and just beginning the installations of its subsea infrastructure and as a result does not yet have an operational data infrastructure. The following information have thus to be considered valid for the collaboration on COOPEUS although they could undergo to modifications according to the evolution of the respective design. The data infrastructure for EMSO is designed as a distributed system: EMSO data collected either continuously or during experiments at each EMSO site, are locally stored and organized in catalogues or relational databases run by the responsible of the regional EMSO nodes. The data management architecture 2
presently includes subsystems which are at a different development stage and which relate with the following functions: - Data Acquisition, - Data Curation, - Data Access, - Data Processing. The EMSO data acquisition sub-system collects raw data streams from EMSO observatories as provided by sensor arrays of varying geometry and various instruments. The currently practiced data management strategies address real time as well as archived and delayed-time data within the EMSO distributed data information system. Real time data are intended as in situ measurements continuously acquired by the sensor equipments and immediately transmitted to land via cables or via acoustics and satellite connections for feeding the data information system. These measurements can have a wide range of sampling rates from very high (e.g., bioacoustics, seismology) to low rates (e.g., physical oceanography, geomagnetism). Whereas real time is immediately available, the so called delayed-time data are those stored locally at the nodes which become available for the data management system only after visits during dedicated ship expeditions when the instruments are recovered or maintained. An additional data typology is constituted by data acquired through laboratory studies performed on material or samples collected at marine observatory sites such as multidisciplinary analyses of water samples, sediment cores, tow or trap catches. The EMSO data access sub-system enables discovery and retrieval of data housed in data resources managed by a data curation sub-system. EMSO offers data discovery via a common metadata catalogue and web portal which can be visited at http://dataportals.pangaea.de/emso. The portal is based on the brokerage system panfmp (http://www.panfmp.org) and uses Open Archives Initiative Protocol for Metadata Harvesting (OAI- PMH) or simple file transfer via FTP/HTTP to harvest metadata from EMSO distributed regional node data archives and their archival systems PANGAEA, MOIST and EUROSITES. The EMSO data portal offers machine-human as well as machine-machine search facilities and discovery services based on the collected metadata. This includes a simple web-based user interface, a data search engine, which is offered at the EMSO data portal in a Google like style. In addition the data portal offers a common discovery service following the OpenSearch specification including the OpenSearch-Geo extension. A Open Geospatial Consortium (OGC) Catalogue Service for Web (CS-W) interface is currently under development. A centralized data export service for these archived data is not implemented or planned, therefore, unless each EMSO data archive offers its own NetCDF data transformation service (see above) data requests are not yet processed by the EMSO data portal but are redirected to the hosting data archives which provide their own data access services for data retrieval. Access to real time data is also offered via the EMSO data portal. EMSO has chosen to implement core standards of the OGC Sensor Web Enablement (SWE) suite of standards, such as Sensor Observation Service (SOS) and Observations and Measurements (O&M) to deliver real time data. These interfaces and formats are used to offer a common, web based SOS client which provides interactive visualizations of real time data. Centralized data processing sub-system which aggregate the data from various resources and provides computational capabilities and capacities for conducting data analysis and scientific experiments are not yet implemented for EMSO. Once more regional EMSO node and their data archives support NetCDF data export we envisage to introduce data visualization and plotting services at the EMSO data portal.. However presently, data processing services such as visualization, mining as well as statistical services are exclusively provided by each regional node and its responsible data center. The OOI-CyberInfrastructure (OOI-CI) supports observational data ingestion into repositories (managed by the infrastructure) alongside with their metadata. It also supports syntactical data and information format transformations as well as ontology-supported semantic mediation, providing access to various information products based on meta-data and other search criteria. Standard models applied include the VSTO Ontology model for data access and conventions from the Marine Metadata Interoperability (MMI) project. These services enable data and meta-data access spanning time and space tailored to the individual needs (e.g., data format, subsets/regions of interest, etc) of the stakeholders. The following is a summary of Data Management capabilities of the OOI-CI: 3
- Online Data Repository - Data and Metadata - Persistent Archive - Cataloguing, Preservation & Curation - Asset Validation - Integrity and Authenticity - Aggregation - Classification, Categorization, Grouping - Attribution - Associate Attributes, Semantic Ontology - Metadata Search&Navigation - Query/Browse by context - Dynamic Data Distribution Services - Publish, Subscribe and Query for dynamic data resources - Data Access Services - Suite of External Interfaces To address interoperability issues between data represented in various formats, OOI-CI employs a semantic framework derived from the Earth System Grid technology, a canonical representation for scientific data based on DAP / OPeNDAP, and related data publishers such as ERDDAP. The semantic framework enables the data transformation services to process data and meta-data in multiple formats both for ingestion and later presentation. 2. Gap Analysis: goal and objectives The Gap Analysis (GA) analysis applied to EMSO and OOI is to help in the achievement of the general goals of the COOPEUS project through - providing scientists worldwide with really global scale data, with homogeneous metadata and comparable data products across the ocean observation infrastructures. - tuning the investigation analysis across the Atlantic on science challenges in order to leverage the present capacities of the scientific communities on both side of the Atlantic and increase them - really sharing a continuous experience on the use of sensors and devices, on the performance assessment to update best practices and converge on new ones. GA is thus aimed at identifying lacking and not optimized issues/elements for the sake of interoperability/integration and even enhancement of the overall capacities of OOI and EMSO research infrastructures in order to face effectively the global challenges. These GA issues/elements can even be expected improvements of a research infrastructure from the other (US-OOI from EU-EMSO and vice versa) and necessary enhancements for both research infrastructures. Actions to overcome identified gaps are driven by real Science questions/needs as well as by recommendations from strategic initiatives underpinning the collaboration across the Atlantic, such as the recent Symposium EU-US-CA held in Rome 2013 at the Canadian Embassy in Italy, which has underlined the importance of a collaboration effort in specific subjects: data quality control, preservation and curation and data products. At this report time, both EMSO and OOI are engaged in the deployment of sensor components, even with different extent: OOI plans most of the installation of the Global Nodes in 2013-15 while EMSO, with 8 sites operational out of 13, is considering an expansion of the monitoring capabilities in few nodes with addition of sensors and equipment s and at the same time the development of a generic sensor package to be deployed in all operational nodes in order to reach a good coverage and homogeneity for a defined set of measurements. Basically, the main objectives of the actions in COOPEUS for OOI and EMSO can be grouped in two categories: General principles for a shared data policy facilitation the crossing use of data Interconnection of data, data metadata and sensor metadata including sensor registry (to compare, integrate, interoperate data provided by RI and exchange experience and best practice on RI sensor use) 4
3. General principles for a shared data policy In order to facilitate the mutual and from-outside access to research data provided by the infrastructures OOI and EMSO will follow some general requirements about data and metadata access and publication. Data access - Free an open access to data produced by Research Infrastructures facilities is strongly supported and OOI and EMSO feel committed to work towards the fulfilment of this principle; the infrastructures shall identify unnecessary or obsolete barriers towards open access to their data and will continuously work towards the stepwise elimination of these obstacles. Potential reasons to restrict public access to data resulting from national and international legal and ethical obligations and data policy have to be mutually acknowledged. Undisclosed research data shall be made available on demand to researchers representing OOI and EMSO infrastructures - as far as possible within the legal, ethical and policy framework of the data holding infrastructure. Each infrastructure shall nominate data stewards to help to accommodate such internal data requests. Metadata access and attributes - Free an open access without any restrictions shall be granted to the metadata of the data holdings of the infrastructures in order to enable and ease the discovery. Metadata shall contain information to attribute identification, authorship, geographical and temporal coverage, classification of measurements and observations, access constraints as well as the responsible organisation or principal investigator (PI). A sensor registry is under implementation in both the research infrastructures to ensure the maintenance, updating and curation of the use history (e.g., installation configuration, operational conditions, calibration) of the sensors. EMSO and OOI have started working to harmonise the respective sensor registry for an easy comparison and performances of similar sensors of the two observation infrastructures (see next sections). Publication - If data or information from OOI and EMSO infrastructures is used in published or unpublished work, attribution for the used resources is required. Data citations shall exclusively use the information provided within the metadata of each data set. Data which is not publicly accessible but has been disclosed to individual researchers through COOPEUS requires explicit allowance of the responsible PI prior to the publication of results derived from this data. Although OOI and EMSO data infrastructures development follows different architectures and time lines, they have similar main capabilities and are mature to outline, plan and even initiate actions for underpinning a long-term transatlantic cooperation by sharing: - Core data standards, to homogenise formats, protocols, metadata, harvesting versus distributed approach, - Data quality, to set up principle for a shared basic quality assessment key indicators and to expand the details on quality management procedures and on calibration with regard to instruments used within research infrastructures. - Practices/protocols for observations planning field work and sharing field resources. 4. Initial plan for interconnecting data and metadata, and sensor metadata Specific Actions have been identified as intermediate steps toward the GA main objectives achievements. For the sake of efficiency, the Specific Actions are to be developed within a scientific framework constituted by Science Cases and User Scenarios. The following table summarises the logical connection between GA main objectives, Specific Actions and the scientific framework. The expected results are also indicated. 5
Objectives Specific Action(s) Application to Science Case(s) and Use Scenario(s) Select a subset of physical parameters monitored by the RI Data metadata interconnection: compare, integrate, interoperate data provided by RI harmonise the existing metadata of the selected physical parameters Harmonise data quality assessment criteria and procedures select physical parameters needed to constitute the basic scientific information Identify, compare and agree on metadata description for the selected physical parameters Identify, compare and agree on basic criteria for data control and quality assessment Expected outcomes/results Initial common vocabulary for the selected physical parameters with unambiguous, univocal definitions Initial common catalogue of metadata related to selected physical parameters in term of attributes definition Identification of common basic control checks and set-up of corresponding quality levels Sensors metadata interconnection: exchange experience and best practice on RI sensor use Compare RI common sensors features, installations, performances identify sensor types/models used within RIs and providing the measurements of the previously selected physical parameters Sharing a common sensor catalogue for a basic suite of sensors including (settings parameters, calibration procedures) compare and harmonise sensor metadata Identify, compare and agree on sensor metadata description for the selected sensors Setting up principles, practical and strategic, for a shared sensor registry. adoption of similar sensor metadata descriptions, attributes, attribute content and formats. 6
4.1 Generic Science Case A generic set of parameters selected to develop the Specific Actions are basic physical oceanography parameters: Temperature, Turbidity, Conductivity, Dissolved Oxygen, Density Currents (single-point and layers) Sea bottom water Pressure, According to the Table above these parameters will be subjects of the Specific Actions development. The selected parameters can been initially described according to NERC Vocabulary Server version 2.0 (NVS2.0) (http://vocab.nerc.ac.uk/). 4.2 User Scenario: Tsunami wave generation and dynamics It is accepted that exposition to tsunami threat is common to Europe in the Mediterranean basin and North-East Atlantic regions, and to North America in the North-East pacific coasts because of near-shore tsunamigenic areas. The common need is to deepen the experience in testing methodology and developing real-time processing of tsunami field data and modelling. According to the Specific Actions the following initial assumption are adopted: parameters describing the tsunami generation and propagation are initially identified: - Sea-bottom ground velocity - Sea-bottom ground acceleration - Seismic wave velocity in water - Sea-bottom water pressure - Sea-level - Water density - Water temperature needed instruments for tsunami data analysis and modelling are identified for revision: - broad-band three-component seismometers - Accelerometers - hydrophones, - bottom pressure sensor, - CTDs The User Scenario will consider: 1. Available tsunami data: Tsunami field data are already available at RIs for past events from ocean observatories. Initial selected type of parameters for the purpose are: - Ground motion - Bottom water pressure - Sea-level The work will include: - Comparison and harmonisation of measurement metadata (attributes) - Specification of sensor types/models: BPR, seismometers, Hydrophones, etc - Comparison and harmonisation of sensor metadata and sensor registry - Identify, compare and agree on basic criteria for data control and quality assessment: specify basic key indicators for users (gaps, out-of-range, corruption, etc ) 7
2. Real time tsunami data: a few EMSO observatories can presently provide these data; for the future a review of sensors is needed to align OOI and EMSO to provide comparable real-time data Review sensors needed for the purpose and agree settings: - High sampling rate BPRs (10-100 Hz) - 3 comp. Broadband Seismometers - 3 comp. Accelerometers + Gyro - High sensitivity low frequency or new Broadband Hydrophones ( 0.05-50000 Hz) As further improvement, OOI installations can be equipped with an already available algorithm for tsunami detection tested at EMSO nodes: 4.3 Compatibility of selected sensors metadata The following attributes constitute a starting list to be considered for harmonising the sensor metadata: Sensor identification: long name short name sensor type id id-sensor model numner part number serial number manufacturer Use information calibrations documents log Exterior weight length width height material Operation features storage type memory capacity battery current 4.4 Sensor registry concept The Sensor Registry enables the registration of sensors and instruments in SensorML. By registering sensors, the provided information becomes discoverable from data portals and related catalogue services. Encoding standard is the OpenGIS Sensor Model Language (SensorML 1.0.0), a GEOSS standard for sensor metadata. The sensor registry prototype developed within ESONET NoE EC can be adopted as initial reference scheme for the purpose within COOPEUS project (see ESONET Sensor Registry Entry Form). To develop a really comprehensive sensor registry additional metadata could be necessary (eg., name of the Observatory and name of the site) and an integration of the starting list of metadata can be requested. 5. Implementation strategy In order to provide an official frame the crosscutting use of the data, EMSO and OOI will facilitate the implementation of the reciprocal/integrated use of respective data through the drafting of a MoU including the shared principles for Data Policy. The MoU shall also give directions for 8
- the mutual and 3 rd party data and metadata access to OOI cyber-infrastructure and to EMSO data portal - standard for data and metadata format, and for sensors metadata - the further development of the sensor registry. Specific Actions for the development of Generic Science Case will be developed through: - direct contact with PIs involved in the OOI-CI - participation of COOPEUS- WP4 leader. to workshops of and organisation and planning of specific workshops sessions about EMSO-OOI collaboration in COOPEUS OOI in 2014-15. (the Irminger Sea Global Scale Node Workshop already held in Dec. 2013, in Boston). User Scenarios will be developed through specific meetings with OOI involved in the Endurance Array Regional Scale Node in 2014-15. 6. Conclusions While OOI and EMSO development have different time-line, schedule and prioritisation, specific subjects about data and metadata sharing and interoperability have been identified, detailed and discussed during the COOPEUS annual meeting and during the Irminger Sea Workshop (Dec. 2013). Other workshops are foreseen in 2014 and 2015. The actions to be developed during 2014 and 2015 on these subjects have been also agreed: - draft of MoU framing the data and metadata sharing and mutual use - harmonisation of Research Infrastructures data formats and metadata of data and sensors to make them discoverable from respective data portals and steps toward a common sensor registry. 9