Requirements for Observational Metadata: CAS

Transcription

1 Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Requirements for Observational Metadata: CAS Measurement and Data Department, MeteoSwiss TT-WMD-1, Geneva, March 2013

2 Overview GAW in a nutshell Metadata what are we talking about? The data value chain why do we care about metadata? WDCs and other data centres and WIS/WIGOS GAWSIS and metadata Profiling ISO19115 CAS/GAW Metadata Requirements Conclusions 2

3 GAW in a nutshell Global Atmosphere Watch (GAW) is an observing system coordinated by WMO guided by Scientific Advisory Groups (SAGs) and Expert Teams (ET) Scope of GAW is air quality as well as climate, on urban, regional and global scales. Mainly network(s) of ground-based stations, but also uses ship-, air-, and satellite-borne sensors. GAW is the atmospheric chemistry component of the Global Climate Observing System (GCOS). 3

4 GAW Services and Products Scientific stewardship for atmospheric composition monitoring Framework for consistent quality of observations Long-term observations for trend analyses Long-term archiving of data ( GAW WDCs) Ground-based data for satellite validation Early warning system for atmospheric change Contribution to quadrennial Ozone Assessment Contribution to IPCC Process Ozone, Greenhouse gas bulletins 4

5 Metadata What data? Where to find them? How to get them? Any strings attached? Discovery metadata Method of observation? Quality control applied? Data processing applied? Uncertainty of data? Interpretation metadata 5

6 The data value chain why do we care about metadata? ISO/WMO GAW Guidance documents ISO standards on uncertainty, aggregation Discovery Retrieval Generation of Raw Data Generation of Finalized Data Archiving Dissemination Data Use Revisions Data provider Data user Data archive 6

7 Generation of raw data Purpose DQO Sampling Measurement Analysis Data Storage Level 0 Data Site description Sampling Analytical + + = procedure procedure Level 0 metadata Level 0 Product 7

8 Generation of finalized data Scale Operator judgement Level 0 Data Calibration Level 1 Data Quality control Aggregation Uncertainty assessment Level 2 Data Level 0 metadata + Traceability chain + Calibration QC procedure + Aggregation + procedure procedure = Level 2 metadata Level 0 Product Raw data finalized data Amount of data Level 2 Product Amount of metadata 8

9 Data submission Level 2 Data World Data Centre Guidelines Level 2 metadata Submission Archiving Dissemination Level 2 Product Different data formats Different metadata requirements Different client handling Specialized data archives 9

10 GAW World Data Centres WOUDC Ozone/UV WDCPC Precip Chem WDCA Aerosols WRDC Radiation WDC-RSAT Satellites WDCGG Gases Data Policy For scientific purposes, access is unlimited and free. offer of co-authorship whenever substantial use is made of data. acknowledgment of data providers or owners and the data centre when used within a publication. [GSP, 2007] Purpose collect and archive processed GAW data, make data publicly available, provide support for quality, assurance, analysis and interpretation Commitment align operations to the needs of data submitters and data users alike. [GSP, 2007] 10

11 Other Data Centres hosting GAW Data AERONET AGAGE BSRN CapMon CDIAC EANET EBAS (NILU) GALION (Earlinet, ) NADP NOAA/ESRL/GMD RAMCES SHADOZ SKYNET TCCON (CalTech) [One for each satellite] Reasons for co-existence History Specific mandate Data policy Visibility Limited flexibility of WDCs? Ad-hoc permanent? Lack of coordination Result of co-existence + Expert knowledge - Exact duplication - Content duplication - Different metadata - Different versions - Loss of traceability Need for ET-WDC GEO AQ CoP WIGOS Standards Interoperability 11

12 The WISard of WIGOSW WMO Information System (WIS) succeeding Global Telecommunication System (GTS) Former GTS Exchange of n.r.t. data Catalogue of data Inter-operability of data centres Discovery metadata Very different creatures overcome big obstacles and achieve great things by supporting each other. WMO Integrated Global Observing System (WIGOS) Framework for the integration of GOS, GAW, GCW, WHYCOS, etc. Coordinated governance structure Coordinated quality management Coordinated data management and delivery TT on WIGOS Metadata Interpretation metadata 12

13 Task Team on the WIGOS Metadata (TT-WMD) Terms of Reference To identify the information that is needed to allow the majority of users to use WIGOS observations in appropriate contexts and in a defensible way To create the WIGOS Core Metadata Standard that allows the essential information to be exchanged unambiguously, regardless of the format used for the transfer To define the mechanism for maintaining the WIGOS Core Metadata Standard, including how metadata might be provided that is additional to the Core. To implement within the WIGOS Core Metadata Standard, and the WMO Core Metadata Profile, a standard method of providing users with an indication of the quality of the data, and to do so in a way that distinguishes with the quality management of the data ( quality of the observation ) and ensuring that the user is able to identify which applications the data are suitable for ( classification of the observation ). To complete its tasks and hand over additional requirements to its successor (if required) by 31 December Interpretation metadata 13

14 GAWSIS in a nutshell Some WIS functionality for GAW Official catalogue of GAW ground-based networks(s) 800+ stations/sites data series 900+ contacts 600+ bibliographic references GAWSIS Regional stations Global stations Associated and partner networks Contributing stations Interactive, web-based metadata archive Link to data across WDCs, other data centers Clearing house for identification of stations (colocation) 1-stop shop for the ground-based GAW network(s) 14

15 GAWSIS Metadata Integration for GAW WMO Global Atmosphere Watch World Data Centre for Aerosols GAWSIS Global GAW stations WDCPC Regional GAW stations CAPMon Associated and partner networks Contributing GAW stations AERONET Operational & comprehensive Operational, incomplete No direct link at present 15

16 GAWSIS Main Features Lists (and maps) of stations Lists of contacts Station reports Site characterization Measurement program Meta data for each series Method of observation Period, frequency, etc. Instrument history... Hyperlinks to data (archive) Contacts Bibliographic references Clearinghouse for 3-letter station codes (incl. GAW IDs) 16

17 GAWSIS XML Metadata Profile based on ISO19115 mandatory elements 1 metadata set for each data series WMO Core Metadata Profile compliant (presumably!!) Reviewed and slightly extended during ET-WDC meeting in May 2012 (WMO Geneva) XML representation constructed from GAWSIS relational database using R Mixture of discovery and interpretation metadata Still experimental and subject to further improvement Available at

18 Profiling ISO19115 Type-1 profile Type-1 profile Type-2 profile The WMO Core Metadata Profile is a type-1 profile. This implies it only RESTRICTS the original standard and adds / amends Code Lists. The GAW Profile is also a type-1 profile, but an extension of the WMO Core profile! 20

19 WMO Core Profile v1.2 vs v1.3 Core metadata record including all optional elements of WMO Core Data Types in WMO core metadata profile Metadata profile v1.2, including all optional elements v1.2, minimum profile v1.3, mandatory elements v1.3, full profile 21

20 CodeLists: Recommended terms for GAW 22

21 CodeLists: Recommended terms for GAW DRAFT not approved For observation regardless of platform For upper-air observations regardless of platform For upper-air observations at marine sites For observations regardless of platform For satellite observations For airplane observations For description of platform only (metadata record does not describe an observation) 23

22 CodeLists: Recommended terms for GAW DRAFT not approved Proposed additions to Table 16 WMO_CategoryCode atmosphericcomposition uvradiation solarradiation thermalradiation or: irradiation? For all observations of atmospheric composition, i.e. ParamType = {Aerosol Precipitation Chemistry Ozone Greenhouse Gas Reactive Gas Radio Nuclide Archived Sample POPs} except optical properties of aerosols} For all observations of solar radiation, i.e. ParamType = {Radiation} & ParamGroup = {UV} For all observations of solar radiation, i.e. ParamType = {Radiation} & ParamGroup = {Solar} For all observations of thermal radiation, i.e. ParamType = {Radiation} & ParamGroup = {IR} Recommendation : ET-WDC recommends to extend the WMO_CategoryCode code list with the terms atmospheric Composition, uvradiation, solarradiation, thermalradiation Action item: jkl to contact IPET-MDRD (Jeremy Tandy, Steve Foreman 24

23 CAS/GAW Metadata Requirements (1/2) Elements in CIMO Guide (WMO No. 8) as listed in Doc-6.5 paragraphs 23 and 24 of this session Further elements that are essential to GAW specification of characteristics of the vicinity of an observing platform that potentially influence the measurement result (soil cover, horizon, local sources, ) Specification of measurand (observed variable in context) specification of sampling procedures, sample treatment and analytical procedures (to be covered with MI_Acquisition?) specification of central laboratories if applicable 25

24 CAS/GAW Metadata Requirements (2/2) traceability/uncertainty information (cf. calibration information) constraints on meaningful application of the data (cf. network purpose ) formal metadata describing data quality / scope of use Data format, version of data being described history of data/metadata long-term value of data data policy, use constraints, acknowledgement (partly covered already) Requirement for extensibility of WIGOS metadata standard 26

25 Open Issues Who is the authority to define the metadata profile? WIGOS not the only player on the field!! Can we agree on one profile? Who is the authority to define and maintain vocabularies? Can CF accomodate all the needs of WIGOS? Role of IUPAC? ISO? WMO? EPA? EEA? 27

26 Conclusions (Meta)data management starts with the definition of the purpose of the observation and the DQOs (and not the data!). Data value chain, data amount, but the metadata amount. Value of data = data quality + discoverability + documentation. Feed-back of data users to archives and originators critical! Metadata are of the same importance as data For CAS, ET-WDC is the focal point for metadata issues related to atmospheric composition. We re still on a learning curve 28

27 Thank you for your attention! Acknowledment: Data center managers and individual users for their support of GAWSIS 29