IS-ENES2 1 st General Assembly th June 2014 UPC Campus, Barcelona, Spain. ESGF Use, CORDEX, ongoing developments

Size: px

Start display at page:

Download "IS-ENES2 1 st General Assembly th June 2014 UPC Campus, Barcelona, Spain. ESGF Use, CORDEX, ongoing developments"

Bathsheba Norman
5 years ago
Views:

1 IS-ENES2 1 st General Assembly th June 2014 UPC Campus, Barcelona, Spain ESGF Use, CORDEX, ongoing developments! 1

2 ESGF: Status IS- ENES ESGF Portals BADC DKRZ IPSL SMHI CMCC DMI IS- ENES coordina4on: Same search facets (esp. for CORDEX) Consistent search results Consistent AuthenEcaEon / AuthorizaEon

side processing) Not announced to end- users (basic

3 ESGF: Status IS- ENES ESGF Portals BADC DKRZ IPSL SMHI CMCC DMI Some expose compute node: LAS server (server side processing) Not announced to end- users (basic scalability / resource usage problem) Visualize and Analyze

4 ESGF: Status IS- ENES ESGF Data Nodes DMI - 1 SMHI - 1 DKRZ - 6 BADC 2 CMCC 1 IPSL/CEA 4 E- inis.ie - 1 Norway 1 UniCan - 1 Nearly all IS- ENES Data Nodes expose Dashboard: Usage overview / staesecs To Do: StaEsEcs AggregaEon across nodes ESGF Data Nodes ( ): 40 worldwide 18 in Europe (coordinated in IS- ENES)

ESGF: Use Terabytes 200 150 100 50 Apr May Jun Jul Aug Sep Okt Nov Dec Jan Feb Mar Apr 2013 2014 Individual End Users

.) Organized User Groups Organize a local cache of required files Most of group don`t access ESGF but cache Trend Data

5 ESGF: Use Terabytes Apr May Jun Jul Aug Sep Okt Nov Dec Jan Feb Mar Apr Individual End Users Limited resources (bandwidth, storage,..) Organized User Groups Organize a local cache of required files Most of group don`t access ESGF but cache Trend Data Center Service Group Provides access to ESGF replica cache May also provide access to data near compute resources (BADC, DKRZ, IPSL, KNMI, UC) Needed: Replacement for Download and Process at Home Approach

6 CORDEX status ( ) ~ data- sets from 9 InsEtutes and from 12 models ESGF: CORDEX IS- ENES support: CORDEX specificaeon documents (CORDEX archive specificaeons, Variable Requirement Table,..) CoordinaEon of Data Node AdministraEon (Data Node Admin Document) Quality control data ingest and ESGF publicaeon CORDEX group and user list management CORDEX visibility checking tool Ongoing discussion at WCRP level with respect to IS- ENES maintaining/hoseng CORDEX CVs

7 ESGF: Recent Developments

8 Networking: ICNWG working group ESGF: Recent Developments Goal: Improvement of data bandwidth between core data centers Partners: DKRZ (DE) STFC (UK) KNMI (NL) ANU (Au) LLNL (US) IS- ENES partners (+ surfnet, es.net, DFN, janet,..)

9 IS- ENES (IPSL, SMHI) lead of ESGF data node installa4on team (esgf_iwt) ESGF: Recent Developments Mayor achievements so far: github based installer installaeon teseng procedure clear release management Next: Apache frontend: will allow for non java server side code ( transieoning to COG etc.) sys- admin friendlier installaeon process ( RPMizaEon )

10 ESGF: Recent Developments Support for server side processing: WPS Work also supported by Ex- Arch G8 project IniEal Deployments BADC: COWS WPS framework DKRZ: ClimDaPS WPS framework based on open source pywps implementaeon KNMI: pywps backend integrated in climate4impact portal (also external use planed) First User/Developer MeeEng in December in Hamburg organized by CSC/DKRZ Next steps: MeeEng Clipc / IS- ENES2 in November discussion on what technology can / should IS- ENES support User requirements gathering EUDAT GEF WPS prototype integraeon ParEcipaEon / ContribuEon to ESGF processing working team (esgf- pwt)

11 Long Term Archive Access through ESGF ESGF: Recent Developments DKRZ Long Term Archive Part (CMIP5 snapshot) exposed as ESGF data node WP5 in CLIP- C works on a generic solueon

ESGF: Recent Developments EUDAT ENES data replica4on EUDAT: safe replicaeon

Domain of registered data Security integraeon ENES replicaeon: DKRZ Jülich KIT,

replicaeon, metadata, pid infrastructure, portal.

partners have working irods installaeons (Sweden/NSC/LIU, Germany/DKRZ/Jülich,

12 ESGF: Recent Developments EUDAT ENES data replica4on EUDAT: safe replicaeon service based on irods and PIDs backup use case PIDs Policy rules EUDAT CDI Domain of registered data Security integraeon ENES replicaeon: DKRZ Jülich KIT, CINES CNRS/IPSL No EUDAT/ESGF infrastructure integraeon yet (security, replicaeon, metadata, pid infrastructure, portal..) irods / EUDAT data infrastructure something to watch for the future Many partners have working irods installaeons (Sweden/NSC/LIU, Germany/DKRZ/Jülich, France/CINES/IPSL, UK/STFC) NASA has an ESGF data node in produceon implemented based on irods Use cases: data replicaeon, data ingest,

13 ESGF: Recent Developments ESGF working teams ESGF InstallaEon Working Team (esgf_iwt) ESGF Support Working Team (esgf- swt) ESGF Compute Working Team (esgf- cwt) ESGF PublicaEon Working Team (esgf- pwt) ESGF Security Team ESGF Monitoring and StaEsEcs ESGF data idenefiers / data citaeon ICNWG Network Working Team IS- ENES partner real work commitments ½ FTE + ½ FTE (IPSL, LIU) ½ FTE (DKRZ) + ½ (BADC) (DKRZ, BADC, KNMI,..) ½ FTE (DKRZ), +.. > ½ BADC CMCC DKRZ? NERSC, DKRZ, KNMI Other ESGF Parts needing engagement: (P2P node manager) New ESGF Frontend: COG IS- ENES Portals (verc, climate4impact) AutomaEc ReplicaEon CMOR like data preparaeon tools Quality checking and data annotaeon.! SubstanEal ressources to evolve ESGF! PrioriEes..!?

14 ESGF: Summary Because of funding problems in US substaneal IS- ENES involvement in further development and evolueon of ESGF e- science infrastructure.. New installaeon team is a first real success story Also large part of ESGF support task on IS- ENES shoulders ESGF operaeons- teams very necessary and useful to manage operaeonal issues across ESGF sites A lot to do to meet future requirements and prepare for e.g. CMIP6

15 Thank You

16 Networking: ICNWG working group ESGF: Recent Developments

climate4impact.eu Christian Pagé, CERFACS

IS-ENES2 1 st General Assembly 11-13 th June 2014 UPC Campus, Barcelona, Spain Status of infrastructure climate4impact.eu Christian Pagé, CERFACS Working teams and institutions CERFACS: Christian Pagé