Copernicus Climate Change Service

Size: px

Start display at page:

Download "Copernicus Climate Change Service"

Theodore Willis
6 years ago
Views:

1 Climate Data Store, Toolbox Geneva, 6-8 th of December 2016 Cedric Bergeron Angel Lopez Alos Baudouin Raoult

2 Budget of 4.3 Bn for COPERNICUS Copernicus, previously known as GMES (Global Monitoring for Environment and Security), is the European Programme for the establishment of a European capacity for Earth Observation Space Component Service Component In-situ Component USERS Full, free and open access to data Source: copernicus.eu, retrieved April 2014

3 COPERNICUS Space components Sentinel-1 (A/B) SAR imaging All weather, day/night applications, interferometry 2014 Sentinel-2 (A/B) Multi-spectral imaging Land

sea/land surface temperature, altimetry Sentinel-4 (A/B) Geostationary atmospheric Atmospheric composition monitoring, trans-boundary pollution Sentinel-5

3 3 COPERNICUS Space components Sentinel-1 (A/B) SAR imaging All weather, day/night applications, interferometry 2014 Sentinel-2 (A/B) Multi-spectral imaging Land applications: urban, forest, agriculture, Continuity of Landsat, SPOT Sentinel-3 (A/B) Ocean and global land monitoring Wide-swath ocean color, vegetation, sea/land surface temperature, altimetry Sentinel-4 (A/B) Geostationary atmospheric Atmospheric composition monitoring, trans-boundary pollution Sentinel-5 precursor/ Sentinel-5 (A/B) Low Earth-orbit Atmospheric composition monitoring Jason-CS (A/B) Low inclination Altimetry Sea-level, wave height and marine wind speed 2020 Source: ESA

4 4 COPERNICUS Service components

5 5 COPERNICUS Service components Copernicus Services * * * * JRC * Operational & Delegation Agreement

6 6 Copernicus Services Deployment Copernicus Services implementation Schedule

7 C3S Vision The European Commission has entrusted ECMWF with the implementation of the C3S To be an authoritative source of climate information for Europe Brings a unique pan-european dimension to

7 7 C3S Vision The European Commission has entrusted ECMWF with the implementation of the C3S To be an authoritative source of climate information for Europe Brings a unique pan-european dimension to Climate Services Build upon, complement and add value to current capabilities in Europe Provide a one-stop-shop access to quality assured climate information, tools and good practices Facilitate uptake and growth of the climate service market Cross-cutting role and exploit synergies with other Copernicus services

8 C3S Components CLIMATE DATA STORE Sectoral Information System Evaluation and Quality Control Outreach and

to support adaptation and mitigation at global and European level Monitors quality of C3S products and services

8 8 C3S Components CLIMATE DATA STORE Sectoral Information System Evaluation and Quality Control Outreach and Dissemination ECVs past, present and future Observed, reanalysed and simulated Derived climate indicators Tools to support adaptation and mitigation at global and European level Monitors quality of C3S products and services Ensures C3S delivers state-of-the-art climate information to end-users Identifies gaps in service provision Bridges Copernicus with the research agenda in Europe (e.g. H2020, national research projects) Web content Public outreach Coordination with national outreach Liaison with public authorities Conferences, seminars Training and education

9 9 The Climate Data Store, Toolbox Concepts

10 10 Context The Climate Data Store will be at the heart of the C3S infrastructure and will provide information about past, present and future climate in terms of Essential Climate Variables and derived climate indicators The CDS will be designed as a distributed system, providing improved access to existing datasets through a unified web interface The CDS will contain observations, global and regional climate reanalyses, global and regional climate projections and seasonal forecasts The CDS will also provide an authoritative set of software (toolbox) that will allow the users to develop applications that will make use of the content of the CDS This service will accommodate the needs of the highly diverse set of users that will include policy makers, experts as well as scientists

11 Content Scientific basis: Essential Climate Variables as defined by

reanalysis Seasonal forecasts and climate projections Global estimates

Multi-model seasonal forecast products Reprocessed CDRs, reference

Support for data rescue, climate data collections Coupled climate

11 11 Content Scientific basis: Essential Climate Variables as defined by GCOS GCOS Status Report (GCOS-195) IPCC, CMIP Observations Climate reanalysis Seasonal forecasts and climate projections Global estimates of ECVs from satellite and insitu data Global atmosphere, ocean, land Multi-model seasonal forecast products Reprocessed CDRs, reference observations Regional reanalysis Access to CMIP data and products Support for data rescue, climate data collections Coupled climate reanalysis for 100 years European multimodel climate projections Climate Indicators

12 12 Content Example of Climate datasets expected to be accessible via the Climate Data Store at an early stage

13 13 Content Example of Climate datasets expected to be accessible via the Climate Data Store at an early stage

14 14 Sectoral Information Systems Overview Proof Of Concept Development of sectoral applications Copernicus Climate Change Service

15 Sectoral Information Systems Overview Seven proof of concept SIS contracts have been awarded: SIS water management: SWICCA (Service for Water Indicators in Climate Change adaptation) lead SMHI (Sweden) EDgE (End-to-End demonstrator for improved decision making in the water sector in Europe) Lead CEH (UK) SIS energy: CLIM4ENERGY (Climate for Energy) Lead CEA (France) ECEM (European Climatic Energy Mixes) Lead UEA (UK) SIS others: AgriCLASS (Agriculture Climate Advisory Services) Lead Telespazio Vega (UK) WISC (Windstorm Information Service) Lead CGI (UK) URBAN-SIS (touching health, infrastructure, water) Lead SMHI (Sweden) 15

16 Toolbox Challenges Technical challenges: Diversity of users Diversity of data sets Very large data volumes Data residing at different locations Interoperability,

16 16 Toolbox Challenges Technical challenges: Diversity of users Diversity of data sets Very large data volumes Data residing at different locations Interoperability, efficiency User-defined workflows Variety of presentation methods Need for interactivity text controls Access via API graphs, maps User management Performance monitoring

17 17 Concept The Toolbox relies on the CDS Infrastructure Cloud Hosting

18 18 Web Portal Content Management System (articles, news, events, ) Catalogue of products ISO19115, for interoperability with INSPIRE, GEOSS, WMO Information System, Toolbox Catalogue of analytics tools that can be invoked on the data and products of the CDS Describes algorithms and methods, input and output data, possible parameterization Users settings Profiles, preferences, licenses, Users requests Current and past users requests: data retrievals and computations Request management: monitor, cancel, download results

19 19 Broker/Scheduler & Adaptors Broker Manages all users requests (data retrievals and computations) Knows where is what Dispatches requests to the right location Adaptors Map broker requests (data retrievals and computations) to data supplier specific protocols Quality of Services Needed to guarantee a good service when CDS will be used by many simultaneous users Uses queues to throttle processing of requests so that available resources (CPUs, disks, network) are not exhausted Implements limits for a fair use of the system (per user, per dataset, per type of computation, etc) and a for a fine grained control of resource usage Implements priorities to schedule important requests (e.g. VIP users, interactive works) before others (e.g. batch access, large computations)

20 20 Data Suppliers Data suppliers will be: Distributed Data and products will mostly remain at the data provider s location Offering Datasets Reanalysis (Petabytes, GRIB) Climate projections (Petabytes, NetCDF) Seasonal forecast (Terabytes, GRIB/NetCDF) Gridded observation (Gigabytes to Terabytes ) Station observation (Gigabytes )

21 21 Toolbox The Toolbox will be composed of: Tools that perform basic operations on data, such as the computation of statistics, subsetting, averaging, value at points, etc. Workflows that combine tools by chaining them so that the output of some tools is used as input to others Applications that make use of workflows and selected data and products of the CDS, to build interactive web-pages allowing end-users to interact with the CDS A Toolbox Compute layer: When possible tools will be executed next to the data (at the data suppliers) Otherwise, computations will be performed in a dedicated compute layer Use of cloud technologies will be considered Compute layer will also hold intermediate results

22 22 SOA and Orchestration Service Oriented Architecture and Orchestration Tools are available either at the data providers or in the compute layer Tools will be invoked remotely by the broker as services (SOA), using adaptors when necessary Tools are combined into workflows An orchestrator will manage the execution of workflows Parallelisation of execution whenever possible Minimisation of data transfers

23 23 Concept The CDS Toolbox, from data to applications

24 24 Application Workflows

25 25 Application Framework

26 26 The Climate Data Store, Toolbox Implementation

27 Monitoring / Metrics Architecture Overview CDS Administrator End User Data Supplier External Information Systems CDS EQC Authentication *Creation Modification Deletion Data/Products Catalogue Browse Browse API Catalogue Export Users Management* User Settings User Requests BROKER / Scheduler Data/Products Catalogue Discover/View/ Download Data/Products Catalogue Iso19115 Export WEB PORTAL BACK END Web Pages Content Management System Data/Product Metadata Management* Tools/Workflow Inventory Tools/Workflow Management* Application Management* Compute Applications Use TOOLBOX INFRASTRUCTURE Applications inventory Monitoring / Metrics CLIMATE DATA STORE Queues Status Tools/Workflows Repository Computation Exec. Computation Exec. Staging Results/Cache Adaptor Adaptor Adaptor Adaptor Adaptor DATA Site 1 Data Elaborated Computation Tools Repository DATA Site 2 Data Elaborated Computation Tools Repository DATA Site 3 27 DATA Site 4 DATA DATA Site n DATA SUPPLIERS DATA

28 28 Operations Monitoring Reporting Capacity planning Usage statistics Service level agreement On-call and support Help desk High-availability Backup

29 29 Climate Data Store Infrastructure Release Cycle Project Duration: 36 months Kick-Off: 06 Jul 2016 One release every 6 months

5 Sep 16 6 Sep 16 04 Oct 16 04 Mar 17 30 Sep 17 Climate Data Store Toolbox Development

0 4 weeks 6 x 4 weeks 6 x 4 weeks KO Sprint 0 Sprints 1-6 Sprints 7-13 SC1 End We Are

30 5 Sep 16 6 Sep Oct Mar Sep 17 Climate Data Store Toolbox Development Timeline to release weeks 6 x 4 weeks 6 x 4 weeks KO Sprint 0 Sprints 1-6 Sprints 7-13 SC1 End We Are here Initial Product Backlog Roadmap Release 0.1 Roadmap Release 1.0 Development Environment Review Meetings (at ECMWF) Q1 Review v0.1 Review v1.0 Review 30

31 31 Development and environment tools Climate Data Store (CDS) Toolbox Programming languages: Python and Javascript Virtualisation technology: Docker / OpenStack (From infrastructure) Scientific Python packages: NumPy, SciPy, Pandas Climate community software tools: Metview, Magics, CDO, NCL, NCO,GDAL,.. Web Framework: React, D3 Software quality: pycodestyle, pyflakes, mypy

32 32 Thank You climate.copernicus.eu

The C3S Climate Data Store and its upcoming use by CAMS

Atmosphere The C3S Climate Data Store and its upcoming use by CAMS Miha Razinger, ECMWF thanks to Angel Alos, Baudouin Raoult, Cedric Bergeron and the CDS contractors Atmosphere What are C3S and CDS? The