Atmosphere The C3S Climate Data Store and its upcoming use by CAMS Miha Razinger, ECMWF thanks to Angel Alos, Baudouin Raoult, Cedric Bergeron and the CDS contractors
Atmosphere What are C3S and CDS? The European Commission has entrusted ECMWF with the implementation of two Copernicus services: the Copernicus Atmosphere Service (CAMS) and the Copernicus Climate Change Service (C3S) The mission of Copernicus Climate Change Service is to provide authoritative, quality-assured information about the past, current and future states of the climate in Europe and worldwide. The service will give access to information for monitoring and predicting climate change and will, therefore, help to support adaptation and mitigation. The Climate Data Store (CDS) will be at the heart of the C3S
Climate Data Store Requirements Atmosphere The Climate Data Store will provide access to diverse set of data: observations, reanalyses, seasonal forecasts and climate projections The CDS will be designed as a distributed system, providing access to existing datasets through a single web interface The CDS will also provide a set of reusable software components (tools) that will allow the users to process data in the system by developing workflows and applications which will make use of content of the CDS This service needs to support the highly diverse set of users that will include policy makers, experts, developers and general public The CDS will be progressively extended to serve users of the Copernicus Atmosphere Service
CDS Architecture Atmosphere Architecture Service-oriented Simple and intuitive web interface and robust back-end Modular and scalable by design API for every CDS function
Adaptors Transparent Data Access Atmosphere Adaptors are data access abstraction layer Their role is to translate generic data request into dataset-specific requests Flexible to allow integration of new data sources Transparent (for the users) access to multiple data sources Current implementations: OPeNDAP ECMWF Web API WCS Others will be developed as required
Data Providers Cheklists Atmosphere 1. Check conformity with Common Data Model and data format specifications 2. Create dataset catalogue records: ISO19115+INSPIRE, keywords, license 3. Specify data retrieval form 4. Provision of Download and Preview Service: Preferably by reusing existing adaptors, e.g. OPeNDAP Provide service to preview/sample data: e.g. WMS
Atmosphere Climate Data Store Toolbox
CDS Toolbox Atmosphere The Toolbox will be composed of: Tools perform basic operations on data, such as arithmetic operations, computation of statistics, sub-setting, extracting value at points, plotting, etc Workflows combine tools by chaining them so that the output of one tool is used as input to another Applications make use of workflows and selected data and products of the CDS, to build interactive web-pages allowing end-users to interact with the CDS A Toolbox Compute layer: When possible tools will be executed next to the data (at the data suppliers) Otherwise, computations will be performed in a dedicated compute layer Use of cloud technologies
What is a typical workflow? Atmosphere RETRIEVE RETRIEVE RETRIEVE Common Data model Mapping Import Convert Rename Variable Variable Variable Pressure Realization N-dimensional Arrays VISUALIZE Get Data and Provenance Longitude Lead Time WORKFLOW SCHEMA PROCESS Parallelism PROCESS Parallelism Parallelism PROCESS
Atmosphere Applications
Atmosphere CDS Operations Reporting Capacity planning Usage statistics Service level agreement On-call support Help desk High-availability Backup
CDS Development Status in May 2017 Atmosphere Product listing Product details Data request form Beta version of CDS due in Summer 2017: a few datasets published (ERA5, seasonal forecasts, climate projections) data discovery, search and data download no tools yet, first due in ~Q4 2017 Data request history
CDS and upcoming use in CAMS Atmosphere Opportunity for CAMS to piggyback on the sizeable C3S investment and acquired expertise due to modular and flexible system architecture CAMS CDS (ADS?) system could either be an independent system or share some elements with C3S but keeping catalogue and front-end separate CAMS in healthy position: product catalogue populated, some adaptors already available, smaller data diversity We will soon be talking to data provides to create new CAMS adaptors
Atmosphere Want to know more? http://www.ecmwf.int/en/newsletter/151/meteorology/climate-service-develops-user-friendly-data-store
Atmosphere Spare slides
CDS Architecture Atmosphere CDS Administrator End User Data Supplier External Information Systems CDS EQC / Metrics Authentication *Creation Modification Deletion Data/Products Catalogue Browse Bowse API Catalogue Export Users Management* User Settings User Requests BROKER / Scheduler Data/Products Catalogue Discover/View/ Download Data/Products Catalogue Iso19115 Export WEB PORTAL BACK END Web Pages Content Management System Data/Product Metadata Management* Tools/Workflow Inventory Tools/Workflow Management* Application Management* Compute Applications Use TOOLBOX INFRASTRUCTURE Applications inventory / Metrics CLIMATE DATA STORE Queues Status Tools/Workflows Repository Computation Exec. Computation Exec. Staging Results/Cache Adaptor Adaptor Adaptor Adaptor Adaptor DATA Site 1 Data Elaborated Computation Tools Repository DATA Site 2 Data Elaborated Computation Tools Repository DATA Site 3 DATA Site 4 DATA DATA Site n DATA SUPPLIERS DATA
CDS Vision Atmosphere More an open framework than a classical data portal Distributed system with a unique interface Based on state-of-the-art technologies Data and services quality control Continuous interaction with stakeholders Support to users and data suppliers. Workbench to allow analysis close to the data (tools) Flexible and scalable to support the growing and evolving needs Focus on interoperability and open standards
Atmosphere CDS infrastructure and toolbox
What are tools? Atmosphere Basic Operations arithmetic operations numerical operations logical comparisons Reduction and Statistical Operations minimum, maximum, sum operations mean, standard deviation operations probabilities, point extraction operations... Field-set Manipulation Operations re-gridding, sub-setting, interpolation Conversion Operations Charts and plots
Atmosphere CDS toolbox: Application workflows
Atmosphere COMMON DATA MODEL The purpose of the Common Data Model (CDM) is to provide a uniformed description (conventions, structures, formats etc.) of all data and products in the CDS, so that they can be combined and processed by the Toolbox in a consistent fashion N-dimensional Arrays Climate Data Store RETRIEVE RETRIEVE RETRIEVE Common Data model Mapping Import Convert Rename Variable Variable Pressure Realization Longitude Lead Time
Atmosphere CDS Toolbox Technical challenges: Diversity of users Diversity of data sets Very large data volumes Data residing at different locations Interoperability, efficiency User-defined workflows Variety of presentation methods Need for interactivity Access via API User management Performance monitoring Reanalysis Observations projections