ECMWF Copernicus Procurement

Transcription

1 ECMWF Copernicus Procurement Invitation to Tender Copernicus Climate Change Service Volume II Software Development for the Climate Data Store (CDS) Toolbox ITT Ref: C3S_25 ISSUED BY: ECMWF Administration Department Procurement Section Date: 7 October 2015 Version: Final

2 Table of Contents 1 Introduction to the project A vision for the toolbox Definitions CDS Infrastructure Project Goals The Toolbox principles A simple Toolbox use case Outcome Setup Flow of execution Summary Project Objectives Project Deliverables Project Scope Technical Specifications Specific requirements Common Data Model (CDM) Libraries of helper functions and classes Tools Workflows and the Orchestrator Applications General requirements Open Source Software Programming Languages Support Project timeline Project Management & Delivery Methodology Project Delivery Methodology Page 2 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

3 4.2 Tasks to be performed Project team Tenderer project team ECMWF project team Tender Format Page Limits Specific additional instructions for the Tender Track Record Quality of Resources Applied Technical Solution Proposed Management and Implementation Plan APPENDICES Use Cases Use Case example Use Case example Use Case example Use Case example Use Case example Page 3 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

4 Table of Figures Figure 1: Expected CDS Infrastructure from ITT C3S_23a... 8 Figure 2: A typical example of a Toolbox application that combine three graphs generated from data obtained from three different data suppliers. The vertical axis is the average monthly temperature in C Figure 3: this figure shows a possible distribution of tools with the CDS and the data suppliers Figure 4: Sequence diagram for the workflow use case - in the CDS infrastructure Figure 5: Mock-up of what an application can look like Page 4 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

5 1 Introduction to the project The European Commission (EU) has entrusted ECMWF with the implementation of the Copernicus Climate Change Service (C3S). The Copernicus Climate Change service shall provide information to increase the knowledge base to support adaptation and mitigation policies. It shall, in particular, contribute to the provision of Essential Climate Variables (ECVs), climate analyses, projections and indicators at temporal and spatial scales relevant to adaptation and mitigation strategies for various sectoral and societal benefit areas. The Climate Data Store (CDS) constitutes the first building block of the C3S and will be at the heart of the C3S infrastructure. It shall provide information about past, present and future climate in terms of Essential Climate Variables and derived climate indicators. The CDS will be designed as a distributed system, providing improved access to existing databases via a one-stop shop, generating and maintaining a documented European catalogue. The CDS will contain at least the following data/products: observation collection, processing and reprocessing; observation gridded products; regional climate reanalyses; seasonal forecasts; global climate projections; regional climate projections. An additional aim of the CDS is to provide a set of generic software (the Toolbox) that will allow the users to develop applications that will make use of the content of the CDS. This service will benefit from the state of the art data analysis tools to generate the reference applications required to analyse, monitor and predict the patterns of both the climate drivers and impacts. This service must accommodate the needs of the highly diverse set of users that will also include, experts and scientists. In order to realise this infrastructure, which will be based on a Service-Oriented Architecture, the procurement has been split into two ITTs, and these are: C3S_23a : Software Infrastructure for the Climate Data Store (CDS); C3S_25: Software Development for the Climate Data Store (CDS) Toolbox. The ITT C3S_23a will aim at developing and implementing the CDS infrastructure. The ITT C3S_25 (this ITT) will be focused on the development and the implementation of a set of generic tools, workflows and applications inside the CDS infrastructure itself. The tenderer is invited to read the ITT C3S_23a (attached as Annex 1 to this document) for a better understanding of the current ITT. These two ITTs are strongly linked and close coordination between the two successful Tenderers is required throughout the development and implementation process. Page 5 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

6 1.1 A vision for the toolbox The CDS Toolbox is a platform that will be used by developers to create web based applications that will use the datasets available in the CDS. These applications will subsequently be made available to end-users. Users will be given some control over the applications by interacting with web form elements. This could involve the selection of a range of dates or a geographical area of interest, which will then be used to parameterise the application. All computations will be executed within the CDS infrastructure, in a distributed, service-oriented architecture. The data used by the applications will not leave the CDS, and only the results will be made available to the users. These results will typically be in the form of tables, maps and graphs on the CDS data portal. Users may be also offered the ability to download these results. The CDS will contain a wide variety of datasets, such as from reanalysis activities, seasonal forecasts, climate projections, observations, etc. These datasets will be distributed and located at many data suppliers, and will be accessible via a range of various protocols such as bespoke REST APIs, OpenDAP or OGC WPS. These datasets are encoded using various formats, such as WMO GRIB or NetCDF-CF or are stored in relational SQL databases. The data types range from single point observations or time series at a given location, to multi-dimensional global fields. The variety of data types, formats and structures make their combined use highly challenging. The aim of the Toolbox is to provide a set of high-level utilities that allow developers to implement applications without the need to know about these differences. The Toolbox should hide the physical location of the datasets, their access methods, formats, units, etc. from those who are developing the applications. Developers will be presented with an abstract view of all the data available in the CDS. It is anticipated that all the datasets share some common attributes. Most of them represent variables which are either measured or forecast (e.g. temperature, amount of precipitation, etc.), and which are located in time and space. It should therefore be possible to define a common data model that will be able to represent all datasets in the CDS. The Toolbox will also provide a series of tools that will perform some basic operations on the datasets, such as averaging, calculating differences, sub-setting, etc. For performances reasons, these tools may be specialized for certain datasets (e.g. sub-setting GRIB data and sub-setting NetCDF-CF data), and may be executed next to the data. This aspect will also be hidden from the application developer, who will only be provided with a single view of a tool (e.g. only one sub-setting operation can be selected). All tools will be registered in a database and documented within the CDS. Application developers will implement workflows that will invoke the various tools available to them on the datasets from the CDS. These workflows will also present the output for some of the tools as the input to other tools, in order to implement more complex algorithms. As is the case for tools, workflows will also be registered and documented so that they can be reused. Finally, application developers will create web pages that will present users with a series of widgets (check boxes, drop down menus, etc.) as well as various tables, maps on graphs, and associate the page with a selected workflow. The tables, maps and graphs will represent different views on the result of executing the workflow parameterised with the values selected in the widgets. These views will be dynamically updated as end-users will interact with the widgets. Page 6 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

7 After a validation process, the applications which have been created will be published on the CDS data portal and made available to all users. 1.2 Definitions In this document the following definitions shall apply for the purpose of this ITT. Where there is a difference between the definitions below and the definitions in Volume I of the ITT documents (Instructions and Conditions), the definitions below shall take precedence. Name API Application Broker CDS CDM Data Data repository Data Supplier EQC ESGF Metadata Product Product Catalogue Proof of Concept QoS SOA Tool Toolbox URL Users Definitions An acronym for Application Programming Interface An interactive web page that displays maps, graphs and/or textual information that are the results of computations performed on the data and products of the Climate Data Store A middleware software component which will schedule and forward requests from the web portal to remote repositories An acronym for Climate Data Store An acronym for Common Data Model The raw data collected A generic name for a system that holds data and/or products. This can be a database, a collection of files, etc. An organisation that make its data and products available through the Climate Data Store An acronym for Evaluation & Quality Control An acronym for Earth System Grid Federation Ancillary information about the data A derived, valued added piece of information, usually generated from raw data A list of available products The realization of a certain method or idea to demonstrate its feasibility or a demonstration in principle, whose purpose is to verify that some concept or theory has the potential of being used An acronym for Quality of Service An acronym for Service-Oriented Architecture A software that performs computations on some input data or products and produces some output data or products The set of available tools, workflows and applications within the CDS An acronym for Uniform Resource Locator The internal and external Users of the C3S infrastructure Page 7 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

8 Web portal WMO Workflow The web page to the CDS portal that enables the users to view information, access/perform tasks on the Product Catalogue, Toolbox and CMS An acronym for the World Meteorological Organization A series of invocations of software tools whereby the output of the preceding tool becomes the input for the one which follows it until the required processing chain is completed 1.3 CDS Infrastructure As mentioned in the introduction, the CDS Toolbox will rely on the CDS infrastructure and will be implemented using the various CDS functions. Therefore, as background information, it is important to outline the proposed CDS infrastructure here. As per the ITT C3S_23a, the functional view of the CDS infrastructure will be as following: Figure 1: Expected CDS Infrastructure from ITT C3S_23a The aim of this infrastructure is to provide a single point of entry for users to discover and manipulate the data and products which are provided through the distributed data repositories in the CDS. It also provides the required environment to manage users, catalogues and the Toolbox. Page 8 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

9 As shown on the figure above, the CDS infrastructure will consist of: a distributed data store built upon the infrastructure available at the data suppliers; a web portal to allow the users to browse and search for data and products of the CDS, perform data retrievals and invoke tools from the toolbox; a brokering software that will schedule and forward data and compute requests, and guarantees quality of service; a centralised catalogue that will describe the data and products of the CDS; a back-end that will hold the databases to support the portal service: a content management system (CMS), a data and product catalogue, a toolbox catalogue as well as the users preferences and status of requests; a Toolbox infrastructure (highlighted in green in Figure 1) that will contain software components to enable computations on the content of the CDS - (part of this current ITT C3S_25): o o o a dedicated section of the web portal that invokes/manages tools, workflows and applications on the data and products; backend catalogues that will describe the tools, workflows and applications; a compute layer that will perform computations on a combination of data and products retrieved from possibly several remote data repositories; This compute layer is functionally part of the CDS and the computations are done in that layer when they cannot be performed at the data location. It is anticipated that the compute layer will be built using cloud technologies. A REST API that will allow batch access to the CDS function, such as data and product retrieval and invocation of the toolbox. The interoperability with external information systems, such as the World Meteorological Organization Information System (WIS) or the Global Earth Observation System of Systems (GEOSS) will be ensured by the product catalogue export function to ISO compliant metadata records. 2 Project Goals The climate data and products available in the CDS will vary greatly in volume (kilobytes to petabytes), in format (GRIB, NetCDF, SQL tables ) and in nature (observations, fields, time-series ). One of the primary goals of this ITT is to develop a software platform that will enable users to develop applications that make use of all data and products in the CDS and combine them together, and display their results graphically on the web portal, or make their results available for download when appropriate. To achieve this the Tenderer must: design and implement a Common Data Model that will ensure a consistent view of all data and products in the CDS; design and implement a set of interoperable tools that make use of this data model and provide users with the necessary building blocks to create high-level applications; consider the need to process very large data sets, preferably by bringing the computational processing closer to the data. Page 9 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

10 These goals drive the deliverables required in this ITT and the project objectives related to the Toolbox. 2.1 The Toolbox principles As show in Figure 1, the Toolbox shall be built inside the CDS infrastructure. The Toolbox shall be composed of: tools that perform basic operations on data, such as the computation of statistics, sub-setting, averaging, value at points, etc.; tools shall conform to a Common Data Model to manipulate the data consistently. workflows that combine tools by chaining them so that the output of some tools is used as input to others; applications that make use of workflows, to build interactive web-pages.please note that the tools are either located in the compute layer of the CDS or at the data supplier, in particular for very large datasets for which transferring the data to the data layer would be prohibitive. As a consequence, workflows will invoke tools that will be distributed throughout the CDS and the data suppliers. These tools and workflows, in accordance with the required Service-Oriented Architecture (SOA) of the CDS infrastructure, shall be called as remote services. Then, two major components will come into play for the proper execution of workflows: the Orchestrator will manage the execution of workflows; the Broker will dispatch computation requests to remote services. The web portal and the REST API shall be the only entry points into the Toolbox. A SIMPLE USE CASE IS PROVIDED IN THE NEXT CHAPTER THAT ILLUSTRATES THESE MAIN PRINCIPLES. 2.2 A simple Toolbox use case The following use case is provided to demonstrate how tools, workflows and applications fit together and interact with the Orchestrator and the Broker. The following section is provided for the sake of illustrating how tools workflows and applications could fit together and is by no means an indication of the final solution Outcome This use case describes a hypothetical application (i.e. an interactive web page on the data portal) in which users can select a city from a drop down menu and be presented with the following graph: Page 10 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

11 Figure 2: A typical example of a Toolbox application that combine three graphs generated from data obtained from three different data suppliers. The vertical axis is the average monthly temperature in C. Each of the bars represents a monthly-average of the surface temperature at the selected location, where: ERA5 is a global reanalysis located at ECMWF, covering 1979 to today. The dataset is composed of global fields, encoded in WMO-GRIB and the temperature is expressed in Kelvin; CMIP6 is a 200 years climate projection scenario located in an ESGF node accessible via the CDS. The dataset is composed of global fields, encoded in NetCDF-CF and the temperature is expressed in Kelvin; Observations is a time-series of observed temperatures, located in a SQL database at one of the C3S data supplier called ClimDBase 1. The temperature is expressed in Celsius Setup We shall assume that there is a service that can map a city name (e.g. Reading, UK) into a latitude/longitude (e.g. 51N, 1W). The following assumptions are made: The ECMWF data repository holds amongst other tools, an extract-point tool that can be run directly on the data and interpolates the value of the data at the requested location from neighbouring points. It is therefore possible to extract data values for a point at given coordinates, from the whole ERA5 dataset. The ESGF data repository holds amongst others, a subset tool than can be run directly on the data, and can return data over a small area (four points) around a requested point. It is therefore possible to subset the whole CMIP6 dataset and extract the minimum amount of information needed to interpolate at a selected location. 1 This is an imaginary dataset, which is used for the sake of the example. Page 11 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

12 The ClimDBase data repository holds long time series of observations at various measuring stations and can be queried by station name. The compute layer of the C3S is equipped with many more tools: o extract-point: Interpolate values from neighbouring points at a given location o monthly-average: Compute the monthly average from a time-series. Returns a timeseries containing only 12 values in this case (one per month) o bar-chart: Plot a bar chart from several time-series. The figure below illustrates this setup: Figure 3: this figure shows a possible distribution of tools with the CDS and the data suppliers. The workflow corresponding to the application can be written in a functional manner as: bar-chart( monthly-average( extract-point(era15, temperature, 51N, 1W, interpolate) ), monthly-average( extract-point( subset(cmip6, temperature, 52N, 2W, 50N, 1W), 51N, 1W, interpolate), monthly-average( retrieve(sql, Reading) ) ) Each of the functions represents a remote procedure call (RPC) to one of the services managed by the Broker. Page 12 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

13 This pseudo code above clearly illustrates the fact that some of the functions are independent of the others. The Tenderer will have to ensure that the orchestration of workflows will make use of this to parallelise the invocation of remote services. Workflows are expected to be described using a scripting language, such as Python. The example below show what such a script could look like: from c3s import bar_chart, monthly_average, extract_point, subset t_era5 = extract_point(dataset= ERA5, parameter= T, location=(51, -1), method= interpolate ) t_cmip6 = subset(dataset= CMIP6, parameter= T, area=(52, -2, 50, -1)) t_cmip6 = extract_point(source=t_cmip6, method= interpolate ) t_sql = retrieve(dataset= SQL, parameter= T, city= Reading ) plot = bar-chart(data=[monthly_average(t_era5), monthly_average(t_cmip6), monthly_average(t_sql)], colours=[ blue, red, green ]) return plot Each of the functions imported from the c3s Python module are high-level wrappers to remote procedure calls to tools/services located at the data suppliers themselves and/or in the compute layer. Workflow authors should not be aware of the location of the services: services should be selected based on their availability within the CDS, the location of the data they are acting on, etc. The script could make use of promises (or futures) to ensure that functions are executed asynchronously and that execution is only blocked when a result of a remote service is needed as input for another service. The actual data do not need to be transferred between the services and the Orchestrator: the results of functions only need to be a reference (i.e. URL) to the actual data. Please note that the example above is merely provided to illustrate the concept of workflow, and that the solution proposed by the Tenderer will almost certainly be different Flow of execution Assuming the end user selects Reading in the application, the workflow associated with the application is passed to the Orchestrator with city=reading as a parameterisation. The workflow is then executed. Each tool is invoked in order via the Broker that will schedule the execution of the tool at the given location, either at a data supplier s location or in the compute layer (which may be composed of a cluster of machines). Every tool will write its output to a file in a staging area and return a URL. The sequence diagram below illustrates the flow of execution described above. Page 13 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

14 Figure 4: Sequence diagram for the workflow use case - in the CDS infrastructure Page 14 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

15 2.2.4 Summary This use case illustrates the following points: data and products in the CDS are available in different formats, with different units, etc. For data and products to be processed and/or combined seamlessly by the tools, a Common Data Model must be designed and implemented; tools will produce intermediate results: there will be a need for staging areas to hold temporarily these results, and as a consequence, there will be a need for housekeeping jobs to clean-up the staging areas regularly; the orchestration of workflows relies on the knowledge of where is what, i.e. where are the datasets located in the CDS (data suppliers), where are the tools located (i.e. at the data supplier or in the compute layer), where are intermediate result located (i.e. in which staging area). The workflows must then make use of this knowledge to call the most appropriate service when invoking a tool. If a tool is available at several location, the workflow should select the one that will make the most of available resources (e.g. minimising data transfers); calls to services, as well as their results, should only consist of references to the actual data, and rely on the URLs to point to them; when the results of a tool are the same when invoked with the same arguments, caching of results should be considered. 2.3 Project Objectives Project Deliverables Based on the elements shown in the use case, the Tenderer is expected to complete the following work packages: Work package 1: design a Common Data Model (CDM) that can be used to combine the data and the products of different types, contained in the CDS (Observation collection and processing, Observation gridded products, Regional Climate reanalyses, Seasonal Forecasts, Global Climate Projections, Regional Climate Projections ). Work package 2: design, implement and document a set of software libraries that will contain helper functions and classes that will be used by persons developing workflows and/or new tools; these functions and classes must provide sufficient high level functionalities such that common patterns and idioms can be abstracted, allowing developers to focus on the algorithm and analytics they wish to implement. In particular, these functions and classes should abstract data types and data format and provide developers with a unified view of all data according to the Common Data Model defined in work package 1, and any high-level operations that be performed on these data. These functions and classes should also offer common functionalities such as logging, error reporting, metric gathering, management of temporary files, handling of parameters as well as input and output files, etc. Page 15 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

16 Work package 3: establish a list of tools required for the climate data and products to be manipulated (subsetting, re-gridding, format conversions, statistical functions, ); review the existing software that could be used to implement this set of tools; implement the set of tools making maximum use of existing software. Work package 4: design a method by which workflows using data and products that are held in various distributed data repositories can be orchestrated, allowing computation to be performed close to the data, data transfers to be minimised (specifically in case of very large datasets processing) and be carried in a sandboxed environment; define a set of relevant workflows to fully demonstrate the methods implemented; these workflows should be based on the use cases in the appendices of this document implement the orchestration and the set of relevant workflows (It includes the management/creation of staging areas to hold the temporary results ) Work package 5: develop and implement an application development environment, allowing users to build, test and publish applications on the CDS web portal; implement a series of reference applications (based on the use cases in appendices of this document) that will demonstrate all functions offered by the Toolbox, and will be used as examples to prove the end to end system. Throughout the development and implementation process, the Tenderer will be required to work closely with the successful Tenderer of ITT C3S_23a. This coordination will involve the holding of joint workshops, cross-attendance to project reviews, generation and implementation of interfacing requirements, support to integration test campaigns, etc. and these activities should be costed in the proposal. At the end of the contract resulting from this ITT, the tenderer will provide an environment where C3S registered users will have access to the following: the description of the Common Data Model that is used to combine CDS data and products of different types and also for future data suppliers to map their data into it; the possibility to invoke workflows that make use of the tools and can be parameterised through the web portal; the possibility to build, test and publish applications based on existing workflows; the possibility to interact directly with the applications. The Tenderer shall implement the CDS Toolbox as a software framework. The Tenderer must include in their response how the proposed CDS Toolbox is extensible; it should be possible to add new tools, new application widgets etc. Page 16 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

17 2.3.2 Project Scope The following lists details what will be considered in the scope for this ITT List of Datasets The following datasets will be initially used as a proof of concept (note that this list may not be the final one): Supplier: ECMWF Data Type: Global atmospheric reanalysis Dataset: ERA5 Expected Format: GRIB - 10-member ensemble Supplier: UERRA and EURO4M consortia Data Type: Regional atmospheric reanalysis Dataset: to be defined Expected Format: GRIB or NETCDF-CF- multi-analysis ensemble Supplier: ESGF Data Type: Global and Regional Climate Projections Dataset: CMIP6 datasets to be defined Expected Format: NetCDF-CF Supplier: To be defined (ECMWF at minima) Data Type: Seasonal forecast products Dataset: Forecasts and reforecasts Expected Format: GRIB multi-model runs Supplier: To be defined (ESA CCI at minima)- Data Type: Climate Observations and ECV datasets Dataset: Time series to be defined Expected Format: To be defined (JSON, NetCDF-CF, CSV ) As new data suppliers join the CDS, the list will grow in data types and formats. It is paramount that the solution proposed by the Tenderer can accommodate these additions with minimum additional development List of Tools The Tenderer will define a list of valid and relevant tools that can perform basic computations on the different datasets provided by the CDS. The contractor will review tools that may currently be in use and propose a list of tools to be used and/or developed within the scope of this ITT. This list will be agreed with ECMWF before implementation. A preliminary list of proposed tools is to be provided as part of the proposal. Please refer to the technical specifications in Section 3 for details. Page 17 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

18 List of Workflows The Tenderer will develop a list containing workflows, using tools from the list of tools in and the list of datasets limited to those described in that will support the reference application development This list of workflow will be developed to support the reference applications. This list will be agreed with ECMWF before implementation. A preliminary list of proposed workflows is to be provided as part of the proposal. Please refer to the technical specifications in Section 3 for details List of reference applications The Tenderer will develop a series of relevant reference applications among the use cases listed in the appendices. These reference applications should be comprehensive enough to prove that the feasibility of the Toolbox concept is fully demonstrated. This list will be agreed with ECMWF before implementation. A preliminary list of proposed applications is to be provided as part of the proposal. Please refer to the technical specifications in Section 3 for details Documentation The Tenderer will have to provide ALL the documentation, in electronic format and in English, related to the principles, overviews, descriptions and how to implement/use for the common data model, the tools, workflows, and reference applications and how to add new tools The Tenderer will have to provide such list of documents as part of the proposal. 3 Technical Specifications 3.1 Specific requirements Common Data Model (CDM) The purpose of the Common Data Model (CDM) is to provide a uniformed description (conventions, structures, formats etc.) of all data and products in the CDS, so that they can be combined and processed by the Toolbox in a consistent fashion. The Common Data Model must, amongst other aspects: propose a set of common spatio-temporal coordinate systems that caters for all data and products; propose a common naming convention for all climate variables, as well as an agreed definition for each of them; propose a common set of units, and possible mappings between them; account for the ensemble nature of many of the datasets in the CDS; propose a naming scheme for all datasets available in the CDS. Page 18 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

19 The definition of a CDM will provide ECMWF with a consistent way of categorising and describing data to facilitate data sharing, integration and ensure best practices are implemented. A study shall be carried out by the Tenderer of available industry standard common data models and the results of this study shall be reported to ECMWF. A recommendation for which one to implement will have to be agreed by ECMWF. The Tenderer is invited to consider the existing CF conventions and INSPIRE conventions ( as starting points for the CDM, and propose possible extensions to them. The Tenderer is required to create a mapping between all the CDS datasets onto the recommended CDM. The tenderer will write a reference document describing the CDM. This document will be published onto the C3S web portal Libraries of helper functions and classes The purpose of the libraries of helper functions and classes is to simplify the work of developers of workflows and/or tools by implementing common tasks, in particular: implementing functions and classes that provide a unified view of all data available in the CDS, irrespective of their format (e.g. GRIB, NetCDF, JSON, etc.) and nature (e.g. reanalyses, climate projections, observations, etc.), according to the Common Data Model; provide high level data manipulation functions; provide functions to perform data retrievals and invoke tools in the CDS; provide functions and classes that allow logging, error reporting, metric gathering, ; provide functions and classes that manage input parameters; provide functions and classes that abstract management of input and output files (e.g. by resolving URLs when input data are only remotely available) ;... Please note that the list above is provided as a guidance, and other functionalities must be identified by the Tenderer during the requirements gathering phase. The helper functions and classes must be provided in the same programming language as the one used for the workflows. Tenderers are invited to review the list in the next chapter for possible software solutions Tools The tools shall be invoked remotely through the Broker as remote procedure calls (RPCs) when located in the compute layer, or via the adaptors when located at the data suppliers location. The tools are Unix/Linux programs that transform some input data into some output data and can be parameterised. The tools shall have the ability to perform a wide range of functionality/computation over gridded data (point-wise operations) and/or time series data, such as: performing basic operations: o arithmetic operations (e.g. +, -, /, *, ) ; Page 19 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

20 o numeric operation (e.g. cos, sin, sqrt, floor, ); o date time operation (e.g. format, round, ) ; o logical comparison (e.g. <, >, =, ); o performing statistical operation: o aggregate functions (e.g. min, max, mean, sum, stddev, ); o performing field set manipulation : o sub-setting; o re-gridding (time and space); o interpolation; o performing data format/type conversion produce graphics: o 2D geographical maps, with various styles (contours, shading, streamlines, wind arrows, etc) o graphs: line plots, bar charts, error bars, o meteograms, cross-sections, Hovmöller diagrams, etc It should be noted that graphics are ultimately intended for displaying on a web page. The Tenderer can either consider plotting as a part of the set of tools, or consider rendering the plots in user s web browser, using existing JavaScript libraries. The Tenderer shall describe all tools in the Toolbox catalogue, which will be made available on the CDS web site. The description should provide information about input and output data types and formats, possible parameterisation and what algorithms are being used. A large amount of software already exists (see table below) which can already cover most of the tools requirements of the Toolbox. It is expected that the toolbox will merely wrap this software so that they can interoperate with the rest of the Toolbox (e.g. downloading of remote data, management of cache and temporary results, logging, error reporting, etc) The Tenderer shall provide a set of tools to be agreed by ECMWF. They shall be compatible with the CDS infrastructure and frameworks The following list gives some tools and libraries which could be considered: Package name URL Comments Metview GRIB & NetCDF UI & batch processing High-level macro language for field manipulation Magics GRIB & NetCDF visualisation Page 20 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

21 CDO GRIB & NetCDF Operators for Climatologies IRIS GRIB& NetCDF Python framework NCO NetCDF Operators GrADS GRIB & NetCDF desktop tool & batch NCL GRIB & NetCDF Interpreted language gdal NetCDF (limited GRIB) GIS tool set Ferret/PyFerret NetCDF Interactive UI & scripting ESMF/ESMPy NetCDF Earth System Modeling Framework software for building and coupling weather and climate applications UV-CDAT NetCDF (limited GRIB) Climate Data Analysis Tools Workflows and the Orchestrator A workflow can be defined as an orchestrated and repeatable pattern of processing steps that produce an outcome. In the context of the CDS, workflows can be viewed as scripts that invoke remote services, the tools, through RPCs, dispatched by the Broker to either the compute layer or the data suppliers. The Orchestrator is the process that will execute the workflows. If the solution chosen for workflows is a scripting language such as Python, the orchestrator will be merely the Python interpreter itself, supported by a series of Python modules that communicate with the Broker and other parts of the CDS. Workflow processing shall be carried out in isolation (sandboxing) and the use of systems like Docker ( LXC ( or OpenStack ( should be considered. In the event of a runtime error, the workflow shall propagate any meaningful message to the end user, and shall terminate gracefully. Workflows shall collect execution statistics, such as elapsed and CPU times, amount of data processed. Page 21 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

22 Like tools, workflows can be parameterised, can make use of input data, and produce output data. As for tools, the Tenderer shall be described all workflows in the Toolbox catalogue, which will be made available on the CDS web site. The description should provide information about input and output data types and formats, possible parameterisation and what algorithms are being used. Workflows shall be implemented using a scripting language with a preference for Python. The Tenderer shall develop a software framework to allow the development, testing and running of workflows. The Tenderer shall also identify and implement a set of workflows to be agreed by ECMWF Applications Applications are web pages that may display a series of widgets, charts, maps and text. By using the widgets, the users can interact with the information displayed on the page. The following sites implement similar functionalities: Maprooms: Plotly: Tableau: In the context of the CDS, applications can be seen as web pages that are associated with a workflow, some data sources, and some parameterisation. Each time the user interacts with the widgets on the web page, the workflow is executed, parameterised with the values of the widgets, and any resulting plots or graphs are updated. Figure 5: Mock-up of what an application can look like Page 22 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

23 The figure above is a mock-up of what an application can look like. When a user selects a new period or a new area using the drop-down menus, the map and graph are automatically updated accordingly. There are several actions associated with applications: Authoring: a user can create/update an application by selecting a workflow, providing some fixed parameterisation, laying out some text, maps and graphs and adding some widgets. Publishing: once authored, an application can be published and catalogued, so that other users can use it. Users can clone published applications and modify them for their own use or publish a modified version. Using: users are limited to the interaction provided by the widgets. The Tenderers are invited to consider existing software such as Jupyter, tmpnb (web-based interactive computational environments), with support for Python and R, and its associated interactive widgets for implementing applications, as well as Bokeh (a Python interactive visualization library). The Tenderer shall develop a framework to allow the management of web based applications. The Tenderer will also implement a set of reference applications, to be agreed by ECMWF, that will demonstrate the capabilities of the proposed solution. The reference applications will also be used for documentation and training purposes. 3.2 General requirements Open Source Software The Tenderer shall propose software solutions which it considers to be the most practical and cost effective for ECMWF s intended use. ECMWF expects the Tenderer to propose open source software, whenever it is available and can provide or contribute to a practical solution. A considerable amount of open source software exists, which the Tenderer shall consider as opportunities for use. As part of the proposal for each work package, the Tenderer must provide a list of: all Open Source software considered; all Open Source software proposed for use; all components being developed; all programming languages that will be used; any proprietary or non-open Source item that may be used. If a proposed solution would require the modification of open source software, the Tenderer will first inform and consult with ECMWF about the permissions or restrictions imposed by the relevant open source licence and the risk that the modified software will become obsolete and unsupported over time. Although ECMWF would prefer a solution based on Open Source software, it is possible to make use of commercial software. In this case, the cost of said software, and any associated support costs, must be taken into account when costing the overall solution. Page 23 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

24 3.2.2 Programming Languages The programming language used for the tools can be any programming language executable on Unix/Linux with a preference for Python, C/C++ and/or FORTRAN. As described previously in the document, the tools are intended to be called as services. Therefore, existing tools may have to be wrapped into a thin software layer: to ensure interoperability with the CDS broker to comply with the Common Data Model of the CDS The programming language used for the workflow is preferably a standard scripting language such as Python or R Support The Tenderer must propose their warranty agreement in respect of all software developed under this tender. As a minimum, a one year warranty is required to apply from acceptance. The Tenderer must provide information about the support services they would provide for software after it has been developed, tested and commissioned. ECMWF is expecting on-call support during ordinary working hours (Monday to Friday, 9am to 5 pm UK local time), for one year after acceptance, as a minimum Project timeline The Tenderer must provide as part of the proposal a detailed schedule of the activities to be carried out within the contract including proposed reviews, milestones, and validation and verification points. Dependencies will exist between this ITT and other C3S ITTs, namely: C3S_23a: Software Infrastructure for the Climate Data Store (CDS) (attached as Annex 1 to this document) C3S_51: Evaluation and Quality Control Function for the Climate Data Store (attached as Annex 2 to this document) C3S_52: Evaluation and Quality Control Function for the Sectoral Information System (attached as Annex 3 to this document) In order to minimise the dependencies between the development of the toolbox and the implementation of the CDS infrastructure, the following timeline is suggested: Phase Task Dependencies Phase 1 Work Package 1 Work Package 2 Work Package 3: development of tools Phase 2 Work Package 3: deployment of tools in the CDS Work Package 4 With CDS Infrastructure None Broker / Compute layer Page 24 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

25 Phase 3 Work Package 5 Data portal / CMS The Tenderer is required to describe in the proposal how these dependencies will be managed and controlled in the development process and is free to describe its own project timeline. 4 Project Management & Delivery Methodology 4.1 Project Delivery Methodology Given the developmental nature of this project, ECMWF anticipates that the adoption of an iterative and incremental approach (including prototyping) will be needed to ensure its successful delivery. This would typically involve a design framework comprising of: requirements gathering prototyping visual designs functionality Such project delivery framework will enable effective development for the project in a time-box approach. Notwithstanding the above, ECMWF welcomes suggestions from the Tenderer on what methodology they propose to apply for the software development based on their knowledge and experience with software projects of similar nature. They must provide examples of how they applied this approach in similar projects they have previously worked on. The development methodology proposed by the Tenderer must ensure that the final delivery is fit for purpose and remains within project cost and schedule. 4.2 Tasks to be performed The Tenderer is required to: implement an iterative/continuous improvement development approach; conduct requirements analysis with key stakeholders; produce a requirements specification in the form of user stories/use cases, to define structure and functionality of the proposed solution; create visual design briefs following the Copernicus graphical charter; review and report on possible synergies with existing projects, that can be used with CDS; develop technical specifications for the tools/workflows to be used/developed; produce a prototype to demonstrate structure and functionality (proof of concept); produce the components of the pre-operational phase; conduct component, unit, system, and integration testing; deliver the system and perform acceptance tests at ECMWF; deliver all required documentation, in English; carry out training of ECMWF personnel as required; ensure that interfaces and interactions with ITT C3S_23a, C3S_51 and C3S_52 are properly taken into account Page 25 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

26 The Tenderer shall provide as part of the proposal a detailed description of the design and development methodology that they propose to implement for phases 1 (proof of concept) and 2 (pre-operational) of the project (see section 3.2.4). In particular, the proposal shall describe: design and development approach requirements capture and analysis requirement priorities setting and control prototyping incremental system build, test, integration and delivery interactions with ECMWF project team: methodology, review, and validations steps Tenderers must provide references of previous similar projects and high level descriptions of how the proposed methodology has been previously implemented by the Tenderer. They should also provide a summary of lessons learned during the execution of these projects. 4.3 Project team Tenderer project team The Tenderer will appoint a Project Manager, responsible for the delivery of the project, to oversee the progress of this project. The Tenderer must outline the project team. The outline must contain the following: relevant experience of key staff and management personnel; names of project manager, and main technical contact and number of work hours dedicated to the project team for the duration of the project; the Tenderer must state if there are any sub-contractors, and define their roles; an indication of how many staff will be part of the project team and at what level. The Tenderer project team is expected to work very closely with ECMWF team for the duration of the project ECMWF project team ECMWF will appoint a Product Owner and a technical lead to oversee the development of this project. The Product Owner will be the point of contact for the Tenderer. The Product Owner will: monitor the Tenderer work execution; review Tenderer s specifications and architectures to ensure that they are fit for purpose ; be the focal point to provide the Tenderer with the ECMWF inputs required at each stage; validate and prioritise the requirements list; set customer priorities; be the focal point to support the incremental testing of each iterative phase; sign off key milestones and deliverables. Page 26 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

27 The ECMWF team will attend project meetings as deemed necessary for the monitoring of the Tenderer s activities, and will be granted unrestricted access by prior agreement to the Tenderer s facilities where the work is being carried out. Monthly progress teleconferences will be held with the ECMWF teams The following documents should be delivered as part of the project: an updated project management plan including milestones; progress reports; work package documentation; requirements specifications; design, development, test plan, test case tests, test reports and test scripts; detailed description of the test cases used for the internal validation of the software including test results; functional tests; performance tests; availability tests; interface descriptions of all modules; software quality assurance plan; risk register; sign offs; training plans; documentation (system, software, source code); release notes; user guide, data supplier guide, administration guide, installation guide;; 5 Tender Format General guidelines for the Tender are described in Volume III. Specific requirements for this particular ITT are described in the next few sub-sections. 5.1 Page Limits As a guideline, it is expected that individual sections of the Tenderer s response do not exceed the page limits listed below. These are advisory limits and should be followed wherever possible, to avoid excessive or wordy responses. Table 1 Page limits per section Section Track Record Quality of Resources Applied Technical Solution Proposed Page Limit 2 (for general) and 2 per entity 2 (excl. Table 1 in Volume IIIB and CV s with a maximum length of 2 pages each) 30 pages in total for the technical solution and Work Package (Table 2 in Volume IIIB, the section on reference, publications, patents and any preexisting IPR is excluded from the page limit and has no page limit) Page 27 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

28 Management and Implementation 10 (excl. Table 3 and Table 5 in Volume IIIB) + 2 per each Work package template (Table 4 in Volume IIIB) Pricing table No limitation 5.2 Specific additional instructions for the Tender The following is a guide to the minimum content expected to be included in each section, additional to the content described in the general guidelines of Volume III. This is not an exhaustive description and additional information may be necessary depending on the Tenderer response Track Record The Tenderer shall demonstrate for itself, and for any proposed subcontractors that they have participated in national or international research and private sector software development projects in the last 5 years for the activities for which this Tender is proposed. ECMWF may ask for evidence of performance in the form of certificates issued or countersigned by the competent authority Quality of Resources Applied The Tenderer shall propose a team that meets at least the following requirements: A Project Manager with more than 5 years experience in managing activities related to an ITT of this size, with experience in the appropriate project methodology proposed in Section 4. A technical project team with more than 5 years experience on performing activities related to the various aspects of this ITT CV s, proven track record and certification of key individuals is required Technical Solution Proposed The Tenderer is expected to provide a short background to the proposed solution to demonstrate understanding of that solution. In cases where alternate solutions are deemed suitable this should be detailed in this section. This section should include background on the Tenderer s understanding of the Copernicus Climate Change Service. An exhaustive and detailed description of the proposed technical solution for all work packages described in section 2.3.1, should be given. The Tenderer should include in this section any other third party suppliers that will be used as part of this technical solution. The Tenderer should include a statement of compliance for each requirement formulated throughout this document (including those from section 3 above), describing how the proposed solution maps to the requirement. This statement can be provided in the form of a summary compliance table Management and Implementation Plan The Tenderer shall provide a detailed implementation plan for the proposed solution broken down per work package using the table in Volume IIIB as a template. The Tenderer shall provide a table for each work package describing the main objectives, proposed activities and set of deliverables and milestones. Page 28 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

29 A detailed breakdown of the following is required: the design and development methodology proposed staff resources per work package budget breakdown for each staff member overhead costs to be incurred As part of the general project management description the Tenderer shall consider the following elements (this is not an exhaustive list): Quarterly, annual and final reports shall be provided in accordance with the Framework Agreement Article 2.3. Implementation plan for the year N+1 shall be provided in February of the year N for ECMWF approval. Monthly teleconferences with ECMWF and a proposal for involvement of ECMWF in major project reviews shall be provided as part of the management plan. A proposed payment plan shall be provided as part of the proposal. The payment plan shall be based on milestones completion and associated deliverables. A list of sub-contractors describing their contribution and key personnel, legal name and address shall be provided. The tenderer shall describe how the Framework Agreement, in particular Clause 2.9 has been flowed down to all their sub-contractors. Page 29 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

30 6 APPENDICES 6.1 Use Cases Page 30 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

31 6.1.1 Use Case example 2 Use Case Name Sector Expert consultant contracted by a hotel owner (in the Alps wants to improve their business website with seasonal outlooks of weather and snow conditions Tourism Actor Pre- Conditions Expert consultant contracted by hotel owner 1. Access available to data (seasonal forecasts; climatology; local weather observations). 2. Tools available to carry out the analysis via the CDS Toolbox (refer to Tender C3S_25 Software development of the Climate Data Store Toolbox). Post - conditions 1. Display a dedicated web page application of current weather/snow conditions and seasonal outlook on the CDS portal, with automatic updates based on CDS content. 2. Link from business website to application page. Normal flow of events 1. Select data and information products from CDS catalogue: a. ENSgram (time series plot for a given location) for the coming snow season, which shows a seasonal monthly and spatially aggregated forecast of snow probability and temperature for the region. b. Climatology for the region based on observations and/or reanalysis. c. Recent observations of snow/temperature for the region. 2. Develop mapping application showing recent and current weather/snow conditions on a map, with ability to scroll in time. 3. Develop a time series application showing seasonal outlook with uncertainties. 4. Generate user-friendly reports on verification of seasonal outlooks against insitu observations and reanalysis data. 5. Develop tools for automatic CDS access and data processing. 6. Combine all elements on the application. Alternate flow Steps 2-6 implemented remotely using elements of the CDS Toolbox. Page 31 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

32 6.1.2 Use Case example 3 Use Case Name Evaluation of fitness for purpose of the Sectoral Information System Sector Actor Pre- Conditions Expert consultant contracted by DG Climate 1. Access available to data (seasonal forecasts; climatology; local weather observations). 2. Tools available to carry out the analysis either locally or (alternatively) via the CDS Toolbox (refer to Tender C3S_25 Software development of the Climate Data Store Toolbox). Post - conditions 1. Display a dedicated web page application the quality of seasonal temperature forecasts. 2. Link from business website to application page. Normal flow of events 1. Select data and information products from CDS catalogue: a. Monthly means of individual ensembles of seasonal forecasts for the months May, Jun and July, lead time month 2, variable: 2m temperature, time period b. Gridded recent observations of temperature for the region. 2. Compute mean and standard deviation for each ensemble, for each issue date. 3. Compute the Continuous Rank Probability Score. 4. Combine all elements on the application. Alternate flow Step 4 implemented remotely using elements of the CDS Toolbox. Page 32 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

33 6.1.3 Use Case example 4 Use Case Name Sector Long term planning of design and maintenance of infrastructure Infrastructure Actor Pre- Conditions Expert consultant contracted by an entity in the infrastructure sector 1. Access available to data (seasonal forecasts; climatology; climate projections). 2. Tools available to carry out the analysis either locally or (alternatively) via the CDS Toolbox Post - conditions 1. Display a dedicated web page application the quality of seasonal temperature forecasts. 2. Link from business website to application page. Normal flow of events 1. Select data and information products from CDS catalogue: a. Climatological monthly means and standard deviations of monthly averaged precipitation from re-analysis. b. Climatological monthly means and standard deviations of monthly averaged precipitation from climate projections. c. Climatological monthly means and standard deviations of monthly averaged precipitation from seasonal forecast. 2. Compute 20, 100 and 1000 year (with option to choose others) return periods for all data. 3. Compare return periods from climate projections to climatology and feed into building design. 4. Compare return periods from seasonal forecasts and feed into maintenance schedule. Alternate flow Steps 2-5 implemented using elements of the CDS Toolbox. Page 33 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

34 6.1.4 Use Case example 5 Use Case Name Sector Hedging of grain prices Agriculture Actor Pre- Conditions Commodity fund trader 1. Access available to data (seasonal forecasts; climatology). 2. Tools available to carry out the analysis either locally or (alternatively) via the CDS Toolbox (refer to Tender C3S_25 Software development of the Climate Data Store Toolbox). 3. Access to in-situ data (land use, maps of soil properties, crop yield). Post - conditions 1. Display a dedicated web page application on the predicted crop yield. 2. Link from business website to application page. Normal flow of events 1. Select data and information products from CDS catalogue: a. Daily soil moisture (root zone depth), precipitation and temperature forecasts from seasonal forecasts. b. Climatology of daily soil moisture (root zone depth), precipitation and temperature forecasts from reanalysis. 2. Apply crop growth model using the input data. 3. Develop mapping application showing climatological crop yield on a map, with ability to scroll in time and aggregation to relevant administrative units. 4. Develop a time series application showing seasonal outlook with uncertainties. 5. Generate user-friendly reports on verification of seasonal outlooks against in-situ observations and reanalysis data. 6. Develop tools for automatic CDS access and data processing. 7. Combine all elements on the application. Alternate flow Steps 2-6 implemented using elements of the CDS Toolbox. Page 34 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

35 6.1.5 Use Case example 6 Use Case Name Sector Water level forecasts for major European Rivers Water Actor Pre- Conditions Waterway and Shipping Administrations and transport companies 1. Access available to data (real time and historic seasonal forecasts ensembles). 2. Tools available to carry out the analysis either locally or (alternatively) via the CDS Toolbox (refer to Tender C3S_25 Software development of the Climate Data Store Toolbox). 3. Access to in-situ data and information on hydrological modelling and ice formation 4. Hydrological model Post - conditions 1. Display a dedicated webpage application on the distribution of areas at risk of ice jams, flooding or low flows. 2. Link from business website to application page. Normal flow of events 1. Select data and information products from CDS catalogue: a. Daily temperature, evaporation and precipitation. 2. Apply hydrological model (including ice formation on rivers) using the input data. 3. Compute the Extreme Forecast Index of relevant outputs (water level, discharge, water temperature, ice formation) 4. Develop mapping application showing risk of ice jams, flooding or low flows aggregated on user defined river stretches and user defined temporal resolutions (e.g. weekly, monthly, daily aggregations of outputs) 5. Develop tools for automatic CDS access and data processing. 6. Combine all elements on the application. Alternate flow Steps 2-6 implemented using elements of the CDS Toolbox. Page 35 of 35 C3S_25 - Software Development for the Climate Data Store (CDS) Toolbox

36 ECMWF Copernicus Procurement Invitation to Tender Copernicus Climate Change Service Annex 1 to C3S_25 Volume II Software Infrastructure for the Climate Data Store (CDS) Note: This Annex is a partial reproduction of the Specifications volume of the ITT Ref C3S_23a (including Clarifications issued), serving as background information for Tenderers interested in ITT Ref C3S_25. This shall not be construed as an invitation to tender for ITT Ref C3S_23a.

37 Table of Contents 1 Specification of Requirements Definitions Project Objectives Project Vision Proposed Project Architecture Project Scope Technical Specification General requirements Project timeline Specific Requirements Architectural Requirements Interoperability Content Management System Open Source software Data Licensing and Data Policy Support Project Management & Delivery Methodology Project Delivery Methodology Tasks to be performed Project team Tenderer project team ECMWF project team Work Packages Work Package C3S_23a.1 Web Portal Work Package C3S_23a.2 Backend Work Package C3S_23a.3 Toolbox Work Package C3S_23a.4 Broker/Scheduler Work Package C3S_23a.5 Adaptors Page 2 of 67 C3S_23a: Software Infrastructure for the CDS

38 3.4.6 Work Package C3S_23a.6 Compute layer Work Package C3S_23a.7 Application Programming Interface (API) Work Package C3S_23a.8 Monitoring and Metrics Work Package C3S_23a.9 Content Management System APPENDICES Use Cases Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Use Case example Standards, protocols and APIs List of Copernicus Projects, ESA CCI Projects, Programmes and Projects C3S Precursor Projects ESA CCI Projects Programmes and projects Potential data repositories Existing portals Software Organisations Page 3 of 67 C3S_23a: Software Infrastructure for the CDS

39 4.8 A sample list of ECV s selected from List of climate variables Clarifications issued for C3S_23a Page 4 of 67 C3S_23a: Software Infrastructure for the CDS

40 1 Specification of Requirements 1.1 Definitions In this document the following definitions shall apply for the purpose of this ITT. Where there is a difference between the definitions below and the definitions in Volume I of the ITT documents (Instructions and Conditions), the definitions below apply for all aspects of this ITT C3S_23a. Name Agile API Application Broker C3S CDS Copernicus Data Data repository Data Supplier ECMWF EQC F4P ITT KPI Metadata Pre-Operational phase Product Product Catalogue Definitions A group of software development methods Application Programming Interface A program for the end user A middleware software component which will schedule and forward requests from the web portal to remote repositories Copernicus Climate Change service Climate Data Store Means the European Commission project for which Services under this ITT will be procured The raw data collected A generic name for a system that holds data and/or products. This can be a database, a collection of files, etc. An organisation that make its data and products available through the Climate Data Store European Centre for Medium-Range Weather Forecasts Evaluation & Quality Control Fitness for Purpose Means this invitation to tender, comprising an on-line portal containing a set of ITT questions and associated documentation Means Key Performance Indicators, a quantifiable measurement that reflects the critical success factors of an activity Ancillary information about the data (e.g. What was done, when it was done and conditions) Phase 2: Testing and building of operational procedures inside the components A derived, valued added piece of information, usually generated from raw data Holds a list of available products Page 5 of 67 C3S_23a: Software Infrastructure for the CDS

41 Proof of Concept phase QoS SBA SIS Tenderer Tool Toolbox URL Users Web portal Workflow Phase 1: Building and testing of the overall architecture Quality of Service Societal Benefit Areas Sectoral Information System A respondent to this ITT Means software that performs computations on some input data or products and produces some output data or products For the purpose of this ITT, the toolbox refers to the database referencing the list of available tools, workflows and applications Uniform Resource Locator Users of C3S infrastructure The web page to the CDS portal that enables the users to view information, access / perform tasks on the Product Catalogue, toolbox and CMS A series of invocations of software tools whereby the output of the preceding tool becomes the input for the one which follows it until the required processing chain is completed Page 6 of 67 C3S_23a: Software Infrastructure for the CDS

42 1.2 Project Objectives The purpose of this project is to overcome the current issues with climate data. At present, data is segregated, therefore locating it becomes difficult; there is no consolidated view; data quality is questionable, and its verification difficult, therefore this can lead to inaccurate conclusions; data input and output is available in various formats with no standardisation applied. In order to overcome the above issues, the objective of this ITT is to design and implement the Copernicus Climate Data Store (CDS) which will: provide a central and holistic view of all information available to the C3S; provide a consistent and seamless access to existing data repositories that are distributed over multiple data suppliers; provide a catalogue of all data and products available; provide quality information on all data and products; provide access and establish links between the software toolbox and its tools, allowing users to perform computations on the data and products. The toolbox content (i.e. tools and applications) will be part of a different ITT; provide means to continually improve its operational system by monitoring of performance in terms of usage, system availability of the CDS, response time(s). This service will benefit from state of the art tools and techniques to generate the relevant set of climate variables, indicators and indices required to analyse, monitor and predict the patterns of both the climate drivers and impacts. C3S can position itself as an authoritative source of climate information for Europe. As a result, users will be empowered to make informed business and policy decisions. Increased availability of data will, in turn, stimulate innovation among data and information providers. 1.3 Project Vision The project vision is to: deliver to EU stakeholders, an innovative and integrated web portal through which a distributed climate data store can be accessed; improve data analysis by providing access to state of the art tools through this operational service which will assist with business development; provide data that is traceable and quality controlled. Page 7 of 67 C3S_23a: Software Infrastructure for the CDS

43 1.4 Proposed Project Architecture The C3S is looking to implement a system that can effectively deliver a seamless workflow to future EU climate users. The key challenges faced today are: i. the increasing number of diverse users; ii. volume of data being extracted; iii. diversity of products being produced; iv. distributed nature of the data held around the world. At present, there is no infrastructure in place to deliver a service that meets the project objectives (see section 1.2), and deals with the current challenges with climate data. The proposed architecture (see Diagram 1, below) aims to put into place key components to deliver its objectives. ECMWF is looking towards building a CDS system structure (architecture) that would consist of the following components, and potential functionality: Web portal: will be single point of entry for the discovery and manipulation of data and products available in the CDS. The portal will also display the latest information on events, current news regarding CDS. It also provides help desk facility, FAQs, user forum; Backend: will hold the databases that will support the portal service: the Content Management System (CMS), the data and product catalogue and quality information, the toolbox and as well the users preferences and status of requests; Broker/scheduler: schedules and forwards requests from the web portal to the remote distributed repositories. These requests can be either data retrieval requests or computation requests. This component will also orchestrate more complex workflows and guarantee Quality of Service (QoS) by implementing queues; Compute layer: will perform computations on a combination of data retrieved from possibly several remote data repositories. Computations are limited to those provided by the tools from the toolbox. They can be performed at the data repositories, at a centralised location, or in a cloud environment; Adaptors: these components will guarantee interoperability between the broker and the various kinds of data repositories, possibly translating data retrieval, or computation requests; Results Cache: this will hold the results of computation ready to be downloaded by the user; Monitoring/Metrics: these elements will collect statistics on the whole system to monitor operations, and measure KPI s respectively; Web API: A web based Representational State Transfer (REST) application programming interface (API) that will allow users to automate their interactions with the CDS. In addition to these components, the CDS will also provide access to existing or future Data Repositories, which will vary greatly in size and nature. It should be noted that some of the repositories will contain many petabytes (10 15 bytes) of data and several billion products. The list of expected repositories is given in Appendix 4.4. Diagram 1 below, is a systems design for the CDS. Please note that: Tenderers should include in their response a commentary on the suitability of the above proposed architecture and indicate any alternative approaches that they have proposed which would meet the requirements of this ITT in a more effective manner. Page 8 of 67 C3S_23a: Software Infrastructure for the CDS

44 Diagram 1: Proposed CDS Infrastructure Page 9 of 67 C3S_23a: Software Infrastructure for the CDS

45 1.5 Project Scope Diagram 2 below, details what will be considered in and out of scope for this ITT. By the end of the contract resulting from this ITT the C3S infrastructure will have incorporate those elements In-Scope (see Diagram 2 below): Web portal; Backend: a centralised data and product catalogue (database); a centralised CMS database; a centralised database for the toolbox; auxiliary information (e.g. user settings); Compute; Broker / scheduler; Adaptors; Monitoring / Metrics; Web API. In addition to the components above, the Tenderer must: provide system training to ECMWF personnel enabling ECMWF to operate, maintain and upgrade the system; provide user documentation, to be published on the portal, to enable non-trained staff to efficiently use the system; provide system documentation, installation documentation, and tools for the administrators of the system. Within the scope of work of this ITT, the Tenderer will review existing technologies, platforms and tools that could be re-used. A preliminary assessment of these reusable elements is to be provided as part of the proposal. For this ITT, the following will be out of scope: tools, workflows and applications that make up the toolbox to enable further manipulation/extraction of data and products. This requirement will be detailed in Tender C3S_25: Software development of the Climate Data Store toolbox; definition of the business processes that will need to change as a result of this ITT; direct management of external data sources. Page 10 of 67 C3S_23a: Software Infrastructure for the CDS

46 Diagram 2: In & Out of Scope Page 11 of 67 C3S_23a: Software Infrastructure for the CDS