Meta4+CMIP5. Bryan Lawrence, Gerry Devine and many many others

Transcription

1 Meta4+CMIP5 Bryan Lawrence, Gerry Devine and many many others,

2 Outline A metadata taxonomy: a place for everyone (and everyone in their place :-) Introduction to Key Metafor Concepts CIM Metafor and IS-ENES services (an acronym feast ) On portals and services (why portals should be a thin veneer over substantial service infrastructures!) CMIP5 information flow Data Node Distribution Quality control for CMIP5 Citing data (relationship to Metafor),

3 Discovery, Documentation, Definition

4 Discovery, Documentation, Definition (& examples) CF-NETCDF META4-CIM IS-ENES Metrics GEOSS Journal Paper

5 Metafor: Facts and Figures Common Metadata for Climate 11 partners Modelling Digital Repositories Total budget of 2.2M SEVENTH FRAMEWORK PROGRAMME Started March 2008, duration 3 years NCAS Climate, NCAS CMS. Reading Uni, UK (Coordinator) NCAS BADC (STFC CEDA), UK CERFACS, France Models and Data, Max Planck Institute for Meteorology, Germany Research Infrastructures INFRA Scientific Digital Repositories Heritage: ENES: PRISM project (FP ) Followed by the PRISM Sustained Initiative (PSI) Institute Pierre-Simon Laplace, CNRS, France University of Manchester, UK Met Office, UK Administratia Nationala de Meterologie, Romania Météo France, CNRM, France CLIMPACT, France CICS, Princeton University, USA

6 Discrimination and Documentation The METAFOR Problem Simulations have a key role in climate science in constructing understanding, and in producing predictions. Discriminating between two simulations is not easy, not even when you were responsible for them! Documentation currently revolves around (at best) the runtime, but not the scientific detail and relevance of the model components. Little or no documentation of the simulation context (the whys and wherefores and issues associated with any particular simulation).

7 Discrimination and Documentation The METAFOR Problem Simulations have a key role in climate science in constructing understanding, and in producing predictions. Discriminating between two simulations is not easy, not even when you were responsible for them! Documentation currently revolves around (at best) the runtime, but not the scientific detail and relevance of the model components. Rowan Sutton: There are no end-users of climate predictions Gavin Schmidt: the public will get used to dealing with climate model outputs however, an increased amount of handholding will be necessary But engagement requires effort and time!... and it's not always the public who require engagement above and beyond normal academic behaviour, misunderstanding abound and even appear in the peer reviewed literature! Little or no documentation of the simulation context (the whys and wherefores and issues associated with any particular simulation). Of course IS-ENES is addressing this too!

8 METAFOR Work plan... build a Common information Model: CIM! Initial Infrastructure is a prototype, but condition of funding is that services must continue after the end of METAFOR

9 Original Metafor PERT chart WP2 develops a COMMON INFORMATION MODEL WP4 provides the integrating architecture and delivers services,

10 CIM development strategy The purpose of the Common Information Model (CIM) is to describe climate data, the models, and the activities that produce data in a standard way Premise : Domain analysis is captured in a formal model procedure, which drives as much as possible, implementations. Separation of concerns between (and within) STRUCTURE and VOCABULARY. Argument : Analysis can be done at the right level of abstraction Applications derived from a conceptual view are more robust than software developed directly on a specific technology Goal : One normative artifact the UML model. Derived schema (initially XML:XSD, possibly OWL) generated automatically. CIM governed by a yet to be established international committee. CIM complemented by individually managed and governed Controlled Vocabularies (many to be managed by a new committee of CF under the auspices of WGCM).

11 CIM UML Structure Six Packages An activity uses software to produce data to be archived in a repository. Multiple classes within each package. Varying levels of interpackage dependency Relationships designed to describe scientific process and structure (s).

12 The CIM has been broken down into several packages: Shared contains those elements that are used in many different packages. Quality contains elements used to express diverse quality metrics for CIM metadata or the artefacts that metadata describes. Grids provides a complete description of the horizontal and vertical discretisation of modelling elements: this may refer to grids that data is mapped onto, software adheres to and/or activities constrain. Activity specifies the experimental design including the experimental requirements and descriptions of how simulations conform to these requirements. Software specifies all the modelling software components used within the experiment process. Data describes the data output from the climate modelling process as well as that for any input data.

13 Defining Experiments in Metafor Lighter boxes introduced as a consequence of describing CMIP5 experiments

14 Controlled Vocabularies Gathering a way of describing (in a community consistent way) the scientific properties of model subcomponents: Hundreds of controlled terms!

15 Questionnaire Deployment 1,

16 Security Deployment ,

17 CMIP5 metadata collection

18 On portals and services An example: the NERC data discovery service: Metadata exposed via service interface at multiple data centres: (BADC, BODC, EIDC, NGDC etc; i.e. all over the UK) Metadata harvested to a discovery database (@ BADC) Database exposes search interface (@BADC) Multiple portals (NERC@BADC, MEDIN@BODC) use customised human interface. Existing portals can compose a simple interface into their existing design (e.g. into search box a la Google).

19 Discovery Architecture Red arrows = Web Calls

20 NERC & MEDIN Discovery NERC portal hosted at BADC (Didcot, near Oxford) MEDIN portal hosted at BODC (Liverpool) Both use a web service interface to the data harvested from the community and held at BADC

21 NDG search inside BADC portal

22 CIM Creation (internal Django db) XML Meta4 Questionnaire (Internal Sqlalchemy db) NetCDF data CIM XML atom CIM XML Thredds ESG Data Node atom ESG2CIM? ncml CIM XML atom XML Centre CIM DB (e.g. IPSL) All creation tools provide atom feeds of their internal contents. XML documents are initially persisted locally and may be duplicated remotely (and persisted in different formats).,

23 Atom Feed Essentially just some documents available at a specific URL <?xml version="1.0" encoding="utf-8"?> <feed xmlns=" FEED DESCRIPTION <id> <title>cmip5 model experiment metadata</title> <subtitle>metafor questionnaire - completed experiment documents</subtitle> <updated> t00:00:00z</updated> <link href=" rel="self"></link> <author><name>the metafor team</name></author> <generator version="r33" uri=" <entry> <id>urn:uuid:1fb380d df-924b-00163e9152a5</id> ENTRY DESCRIPTION <title>5.5-1 esmfdbk1( ESM feedback 1)</title> points to XML payload <updated> t00:00:00z</updated> <published> t00:00:00z</published> <summary>impose conditions identical to 3.1::Control but radiation code sees CO2 concentration increase. </summary> <content src="/cmip5/experiment/1fb380d df-924b-00163e9152a5/1/" type="application/xml"></content> </entry> <entry> <id>urn:uuid:1fd2019c df-924b-00163e9152a5</id>...,

24 CIM Portals atom XML (exist) RDF Query Engine http atom Metafor coding assistance Java Hibernate Metafor Portal RDF Query Engine ESG Portal atom Metafor and IS-ENES query engines have http service interfaces so they can be composed into third party portals., XML (exist) RDF Query Engine http IS-ENES Portal

25 Metafor Software Stack Persistence is decoupled from documents, which may be logically arranged in a variety of ways with associated query interfaces. Tools manipulate the query interfaces, and are exposed by web services. Applications exploit the web services to provide user services (generally as HTML pages exploiting Ajax to interact with RESTful web services). Document Differencing IS-ENES! Faceted Browse Portal Service These two examples depend on metafor documents being persisted in different formats, with different syntax (XML versus RDF triples) but a common semantic data/document model (CIM),

26 Services CIM Viewer given an ID, display a document CIM Query tool given a query, return a result set (just like the NDG example we've just seen) CIM Differencing tool CIM Document tracking (IS-ENES) Faceted Browse Aim: compose these services into portals which navigate through these options in user-community-friendly ways.

27 CIM Viewer

28 Query Tool

29 Differencing Tool CIM Differencing differs from other types of feature comparison tools because... There will be several variants of comparisons depending on the type of CIM instances being compared and the type of information being requested. The set of features being compared is potentially orders of magnitude larger than those typically found in online catalogs (hundreds vs. tens). CIM instances have a very rich structure to draw on. Sometimes this helps; Other times it is a hindrance. So... Only small focused comparisons across the same document type will be supported And users should be able to constrain how the results are presented to them in real-time

30 Differencing Tool

31 Document Tracking Use case: User has downloaded data. What simulation was it from? Data file has internal tracking id urn. Use tracking portal to go from tracking-id back to metadata. Service interface means tracking-id-to-metadata can be composed into any other portal. atom, List in memory Tracking ID Service http Tracking Portal

32 Customizing Metafor & IS-ENES,

33 & So to CMIP5 (With thanks to Stephen Pascoe) A quick introduction to the information flow within the ESG Federation (ESGF). ESGF: a federation of the Earth System Grid (US) and IS-ENES (Europe) and other activities (e.g. Australia and Japan).

34 ESGF Architecture

35 Data Provider Perspective METAFOR Gateway Questionnaire CIM Gateway Model Data Node CMOR CMIP5 Tables ESG Publisher NetCDF RDBMS DN Manager st IS-ENES 1 General Assembly THREDDS XML TDS

36 Quality Level One Access? Data owner should use ESG access control to limit access to friends and family, i.e. folks who can critically evaluate data sanity. Early versions can be deleted without any worries.

37 Quality Level Two Shared refereeing of data quality by PCMDI/DKRZ/BADC. Access? Data access available to anyone with ESGF credentials! Versions can still be deleted without any worries. (Some details of ordering yet tbd)

38 Quality Level Three Access? Public One a final DOI has been assigned, version persisted indefinitely. Timescale: probably about three months from first publication on a data node to Publication with DOI. NB: only replicated data Published

39 Quality Control Dataset Publish Dataset Replicate Dataset Citable Dataset DOI QC Level 1: CMOR2, ESG, CIM Conformance QC Level 2: WDCC Conformance and subjective controls QC Level3: STD-DOI Publication preliminary; no user notification about changes; performed for all data preliminary; no user notification about changes; performed for core data published and persistent data with version and unique STD-DOI citation; performed for core data Access constrained to data author (modeling center) constrained to CMIP5 members and IPCC WG I community open for IPCC process (WG I III) and research community Access Control PCMDI on the behalf of WMO/WGCM PCMDI on the behalf of WMO/WGCM IPCC-DDC on the behalf of TGICA Citatio n no citation reference preliminary citation reference final citation reference Quality Flag automated conformance checks passed subjective quality control passed Data approved by author (in case of newer DOI available: approved by st IS-ENES 1author, General butassembly Barcelona, May 2010 suspended ) Analysis and post processing of data: Post modelling group data can also be deposited. Independent quality control mechanism. Working with ASL on mechanism to publish data analysis.

40 Owning the processes PCMDI governs ESG data node publication into catalog systems. PCMDI governs the qc level 2 process DKRZ governs the qc level 3 process (and with TIB, assigns the DOIs). DOI's will point at a page held at BADC (eventually under auspices of IPCC, using metafor technology). Metafor page will point at PCMDI version of catalog (and data). PCMDI catalog points to the ESGF view of (multiple) copies of the data. IS-ENES infrastructure adds support for European component of ESGF and for local, and supplementary services (and extra data). IS-ENES portal additionally supplies views of category C interfaces, as well as exposing these underlying category B (Metafor CIM) and A (CF-NetCDF) interfaces, and composing them in different ways.

41 User Perspective Gateway User NetCDF Wget script Data Node New Version Notification