Python in the Copernicus Climate Change Service

Transcription

1 Python in the Copernicus Service Gionata Biavati

2 C D S : A n e w w a y o f s e r v i n g d a t a Copernicus Services (C3S) ( is providing the Data Store (CDS) ( The CDS provides climate relevant datasets and tools to visualize and preprocess the data Datasets may be downloaded using web forms or a convinient API Preprocessing occurs in the cloud and it is done using the Toolbox

3 C l i m a t e D a t a S t o r e Searching and Browsing Functions

4 C l i m a t e D a t a S t o r e Discovery and Retrieve Functions Gridded Satellite Observation Example

5 C l i m a t e D a t a S t o r e Discovery and Retrieve Functions Reanalysis Example

6 Monitoring / Metrics C D S O v e r a l l t e c h n i c a l A r c h i t e c t u r e CDS Administrator Authenticatio n *Creation Modification Deletion Data/Products Catalogue End User Users Management* User Settings User Request s BROKER / Scheduler Data Supplier Data/Products Catalogue Discover/View/ Download Data/Products Catalogue Iso19115 Export WEB PORTAL Web Pages Content Management System Data/Product Metadata Management* External Information Systems Browse Browse API Catalogue Export BACK END Tools/Workflow Inventory Tools/Workflow Management* Application Management* Compute Applications Use TOOLBOX INFRASTRUCTURE Applications inventory CDS EQC Monitoring / Metrics CLIMATE DATA STORE Queues Status Tools/Workflows Repository Computation Exec. Computation Exec. Staging Results/Cache Adaptor Adaptor Adaptor Adaptor Adaptor DATA Site 1 Data Elaborated Computation Tools Repository DATA Site 2 Data Elaborated Computation Tools Repository DATA Site 3 DATA Site 4 DATA DATA Site n DATA DATA SUPPLIERS

7 I d e a l i z e d v i e w o f t h e t o o l b o x

8 C D S O p e n - S o u r c e T E C H N O L O G I E S CDO Magics Dask

9 C D S T o o l b o x i s h e r e!

10 H o w i s t h e t o o l b o x w o r k i n g? The Toolbox may be seen as many things 1. A cloud infrastructure 2. A collection of services 3. A platform for developing python code 4. A platform for developing web applications A user submit a python script called workflow The user then can instantiate the execution of a workflow (providing inputs if defined) The workflow code is executed in almost perfect isolation within a docker container The workflow may be seen as a schedulere for a series of tasks (services or tools) After the execution outputs are returned as json files pointing to the urls of the results (plots or data files)

11 C D S T o o l b o x - t o o l s We have five main categories of tools in the toolbox Application Catalogue Computation Graphics Dedicated tools

12 C D S T o o l b o x - t o o l s Application Define the web interface Define input and output Catalogue Computation Graphics Dedicated tools

13 C D S T o o l b o x - t o o l s Application Catalogue Retrieve data from known sources Computation Graphics Dedicated tools

14 C D S T o o l b o x - t o o l s Application Catalogue Computation Basic mathematical operations Standard operators (, ==,!=, +, -, *, /, ** ) Data cube manipulation (average along axes, concatenation, sub-setting) Max-Planck-Institut's Data Operators CDO Filtering on specific domains (Eurostat's nuts) Statistical operations (curve fitting, detrend, extrapolation and interpolation Meta data manipulation Graphics Dedicated tools

15 C D S T o o l b o x - t o o l s Application Catalogue Computation Graphics cdsplot (based on matplotlib) map (based on magics) chart (based on plotly for interactive plots) Dedicated tools

16 C D S T o o l b o x f o r e x p e r t s Application Catalogue Computation Graphics Dedicated tools (for Sectoral Information System) Navigation

17 F e a t u r e s t h a t w i l l b e i m p l e m e n t e d Support for observational data in form of ODB and CSV Support for shapefiles and other GIS formats ODB is a data format developed at ECMWF and is used to store any type of observational data. A C++ API is already available on the pubblic ECMWF's repositories. A pure python interface is currently under testing

18 Thanks Questions?