- C3Grid Stephan Kindermann, DKRZ. Martina Stockhause, MPI-M C3-Team

Transcription

1 A Collaborative Environment for Climate Data Handling - Stephan Kindermann, DKRZ Martina Stockhause, MPI-M C3-Team

2 Motivation Model Output Data + Observation Data + TeraByte Analysis Data Expected future rate: min. 10 PetaByte/year The mission for : Build up a collaborative environment to ease data discovery, data access and data processing Being part of a national Grid Infrastructure DFG Roundtable Tutzing 3

3 The Consortium Partners in information sciences Scientific Users MPI-M (Max Planck Institute for Meteorology) IFM-GEOMAR (Leibniz Institute of Marine Sciences) University of Cologne Freie Universität Berlin PIK (Potsdam Institute for Climate Impact Scientific Data provider University of Dortmund Zuse Institute Berlin (ZIB) DWD Research) DLR AWI World Data Centers WDC Climate WDC Mare WDC RSAT (German Meteorological Service) (German Aerospace Center) (Alfred Wegener Institute for Polar and Marine Research) GKSS DKRZ (German Climate Computing Center) PIK (Potsdam Institute for Climate Impact Associate Partners Research) AWI University of Hannover University of Bonn Forschungszentrum Karlsruhe (Alfred Wegener Institute for Polar and Marine Research) Sun Microsystems, Inc. NEC Corporation Brockmann Consult IFM-GEOMAR (Leibniz Institute of Marine Sciences) University of Cologne/ ZAIK/ FUB Further info:

4 in context Application Climate Community / Organizations GO-ESSP, CF, IPCC, Earth System Curator New FP7 projects Metafor, IS-ENES, Europs-Net.. Geosciences: INSPIRE ISO OGC Other Grids: EGEE, ESG, NDG D-Grid Security: Shibboleth, SAML Grid / Web Basics: OGF, Globus, WSRF, GSI, WSI,.. Infrastructure

5 Talk Overview ArchitectureKey Challenges / Problems Data discovery Automatic Metadata Generation Data access Bridging Heterogeneity Data processing Workflow Composition Access to resources Consistent Security Infrastructur

6 Discovery of data products: Open standards: Portal OAI-PMH harvesting Discovery Web Service DIS primary metadata ISO 19115/19139 Metadata Profile freely available discovery framework (see or sourceforge.net site) OAI-PMH Harvestin g ISO Profile: ISO Metadata Profile controlled vocabularies ( CF standard names, Coordinate Ref. Systems, ) hierarchical data organisation (parent child, aggregation info,..) data provenance (ISO lineage information) data access info ( specific)

7

8

9 Community Data Access: Bridging Heterogeneity geographical Generic Data Request Web Service Interface + vertical + temporal + content + file format selection Analysis Selection of preprocessing tools Metadata generation Provider specific data access interface Initial Implementation: WSDL Web Service next: WSRF Web Service base data preworkspace proc. metadata Compute Resource Grid based data management Implementation Examples: primary data - DB + Archive Wrapper (DKRZ, M&D) Data Warehouse (Pangaea) OGSA-DAI + DB (DWD)... netcdf, GRIB, HDF, XML,..

10 The SOA Portal Portal Distributed infrastructure grid DIS primary metadata DMS primary data local resources and interfaces Workflow Scheduler RIS base data preworkspace proc. metadata Compute Resource Data / Compute Providers World Data Centers Climat Mare RSAT e Research Institutes Universities IFMGeomarFU Berlin Uni Köln : Interface DWD DKRZ PIK GKSS AWI MPI-M

11 Problem: Automatic Metadata Life Cycle Management Problem A: Discovery of dynamically generated/inserted data products Portal Portal Distributed infrastructure Automatic (discovery) metadata generation as part of workflow, including grid DIS DMS automatic provenance info tracking security info aggregation info Automatic quality check Generic tool support primary metadata primary data local resources and interfaces base data?? preproc. Workspace / metadata C3 Repository Center Compute Resource

12 Metadata Handling Support Browser Interface: API: XML content visualisation workflow integration XML (template) generation e-science middleware integration Browser Browser Program interface XForms HTML + css Secure login API Property file interpreter OGSA-DAI Manipulation layer XSLT & XQuery + Checking Specification layer ISO XML template, Specification file, default XMLs Metadata basis ISO XMLs Technology: XML database (exist), XForms (Orbeon), python tools First reusable prototype, cooperation with EGEE

13 Discovery Metadata: Experiences, Problems For established data centers ISO discovery metadata generation was quickly done Substantial support necessary for generating or manipulating ISO discovery metadata for others Agreement on, maintainance of, controlled vocabularies very important USE / DISCOVERY metadata gap is an open issue (e.g. GRIB usage problem, )

14 Workflow Processing Portal Portal Distributed Problem infrastructure B: grid DIS DMS Flexibility of processing functionality?? RIS Workflow Scheduler predefined workflows (just parameterization) primary primary Compute base data workflow compositiondata based onpremodule metadata Resource library workspace proc. local resources and interfaces metadata user inserted programs

15 Workflow Processing: Status / Next steps Portal Initial set of fixed workflows integrated in portal JSDL based workflow description Scheduler controls execution (decision based e.g. on modules info + data avalability) modules info published to Grid Resource Information Service (MDS RIS Workflow Scheduler based) Compute Resource GT4 WS-GRAM Interfaces Preinstalled SW packages (use of modules system) workspace local resources and interfaces Open Issues: workflow composition support interdependency between processing and data user defined processing debugging, substantial user support needed; security!

16 Data access and security Portal AA single sign on support of users without grid certificates federated identity management Portal Distributed infrastructure grid DIS X509DMS grid certificates (EU-GridPMA CA) RIS Grid security infrastructure (GSI) Workflow Scheduler AA primary metadata primary data local resources and interfaces base data preworkspace proc. metadata Compute Resource Data / Compute Providers legacy AA infrastructure (LDAP, DB based,..) AA World Data Centers legacy data access infrastructure Research Institutes Climat Mare RSAT e Universities IFMGeomarFU Berlin Uni Köln : Interface DWD DKRZ PIK GKSS AWI MPI-M

17 Security Infrastructure Identity Provider Home Organisation SAML WAYF Identity Provider Virtual Organisation Home attributes + VO attributes SAML Portal Middleware Delegation Service GridShib SAML tools X509 Gridproxy wflow client <..SAML Assertions..> Grid Service Grid Resource SAML Grid Service SAML GridShib for GT policy GRAM / DataRAM SLCS (CA) MyProxy Personal / Group Account

18 Security Infrastructure Approach: 2. Shibboleth (WEB2.0) federated identity management, Home + VO attrib. 3. X509 Grid certificates + GSI in Grid Infrastructure 1 + 2: SAML assertions short lived certificate from SLCS CA Problem: Initial Idea was: Take and adapt existing technology Status is: ambitious, risky security infrastructure establishment new technology, often partly documented no working security infrastructure - no real added value services for users

19 Summary / Next steps Prototype running; first project phase ends Sept Next phase: transfer to production environment Establishment of a multi-petabyte distributed climate repository center (C3RC) for data storage Core Group has responsibility to keep this running contribution to IPCC AR5 data management Application for funds in context of next D-Grid Call

20 Thank you! further info:

21

22 Data provenance

23 Subject Issuer Public Key Validity Signature IdP entityid SAML Subject <Assertion> </Assertion >

24 Shibboleth & VO Steps: (3)IdP at Institutes (4)Initial SP s (5)GridShib tests