The EUDAT Collaborative Data Infrastructure

Transcription

1 The EUDAT Collaborative Data Infrastructure CSCS, Lugano, 8-9 September (Plan-E workshop) Damien Lecarpentier Project Director, Research Infrastructures, CSC EUDAT Project Director EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-infrastructures. Contract No

2 Research Infrastructures Where is it going? Research Infrastructure trends: Internationalisation Diversification Increasingly relying on on ICT Data deluge is a common challenge European Ris: Around billion investment middle age 19th century 20th century 21st century 2

3 A pan-european e-infrastructure solution for pan- European RI data Challenges All RIs are facing data challenges Where to store the growing amount of data? How to find it? How to make the most of it? Ø Solutions are needed at pan-european level We need to promote synergies Some services are common to many communities Costs and investments can be optimised Ø Better integration of e-infras and research infrastructures can be achieved 3

4 e-science Data Factory EUDAT Partners

5 Building solutions with the communities EUDAT services (B2 Service Suite) are designed, built and implemented together with user communites. Common Language Resources and Technology Infrastructure (CLARIN) European Network for Earth System Modelling (ENES) Distributed infrastructure for life-science information (ELIXIR) European Plate Observing System (EPOS) - Solid Earth sciences Research Infrastructure Integrated Carbon Observation System (ICOS) to quantify & understand greenhouse gas balance Long-Term Ecosystem Research (LTER) in Europe

6 Community-Driven Solutions EUDAT services are designed, built and implemented based on user community requirements. BIOMEDICAL & MEDICAL SCIENCES MATERIALS & ANALYTICAL FACILITIES MAPPER PHYSICAL SCIENCES & ENGINEERING

7 EUDAT Service Suite Covering both access and deposit, from informal data sharing to long-term archiving, and addressing identification, discoverability and computability of both long-tail and big data, EUDAT services seek to address the full lifecycle of research data

8 EUDAT DATA DOMAINS Linking igital Objects o Publications PUBLISHED DATA DOMAIN Discovery of Digital Objects Stage Digital Objects REGISTERED DATA DOMAIN WORKSPACE (TEMPORARY - TRANSIENT) Register Digital Objects

9 Who Citizens Scientists and small teams What Store and exchange data Synchronize multiple versions Ensure automatic desktop synchronization Why Ease of Use Trusted European Service EUDAT2020 Cloud Storage Federation, collaboration with GEANT in OpenCloudMesh B2DROP as a workspace area to computing facilities Integration with EUDAT CDI (e.g. B2SHARE) 9

10 Who Small to Medium Teams What Store data (incl. software) and add domain meta data Share registered research data worldwide Preserve (small-scale) research data for longterm Why Register Data for Publications Make known to wider community EUDAT2020 Focus on authorization Data versioning and annotation Extended HTTP Restful API interface Easy installable software package Integration with other services 10

11 Service Integration Bidirectional Integration

12 Who Community Data Managers Sophisticated Organisations What Provide an abstraction layer which virtualizes large-scale data resources Guard against data loss in long-term archiving and preservation Optimize access for users from different regions Bring data closer to powerful computers Why Performance Replication between trusted sites Data Preservation EUDAT2020 Support metadata Optimize and extend policies to support data curation and provenance Support authorization on basis of community access rules Integration with other EUDAT services 12

13 Who Users and Communities with Significant Computational Needs What Transfer large data collections from EUDAT storages to external HPC facilities for processing Copy large data sets, ingesting them onto EUDAT storage resources Why Integration/Collaboration with PRACE & EGI Simplify Data Transfer EUDAT2020 Further develop HTTP to a mature interface and extend functionality to metadata Extend EUDAT client API library to other B2 services (e.g. B2SHARE, B2FIND, PID) Further integration with B2ACCESS 13

14 Who Anyone What Find collections of scientific data quickly and easily, irrespective of their origin, discipline or community Get quick overviews of available data Browse through collections using standardized facets Why Unique collection Ease of Searching EUDAT2020 Harvesting of metadata stored in B2SAFE Community customizations Annotation of datasets Further assess RDF and Linked Data Further assess scalability and performance 14

15 Who Anyone wanting to use the B2 Services What Complies with community ownerships and access rights, basis of trust Credential conversion approach (e.g. SAML, OpenID, X.509, Username/password) Identity provider for citizen scientists Why Use your own ID in federated environment EUDAT2020 Integration with operational and B2 services B2SHARE B2DROP B2STAGE B2SAFE DPM CREG HTTP API GRIDFTP Integration with community IdP domains and portal environments Enabling access via edugain Social IdPs Focus on authorization 15

16 Service Portfolio Individual Researcher RI/ Community Manager Service Provider Service Function Status Data Discovery B2FIND Multi-disciplinary joint MD catalogue Active Metadata Catalogue MD extraction, MD store, index Under develop. Data Hosting, Registration & Management & Sharing B2DROP Cloud storage, sync & exchange Active B2SAFE Policy-driven data management Active B2SHARE Repository for sharable digital objects Active B2HANDLE Policy-based prefix & PID Active management Data Type Registry Under develop. Data Access, Interface & Movement B2ACCESS Federated multi-protocol IAM Active Generic API Common data interface service Under develop. B2STAGE Data staging service CDI ext. Active Subscription Data transfer subscription Under develop. Consultancy Training on services & data management Active Consultancy on licensing, certification, data Active privacy, data system design Helpdesk Support and enabling Active Operations Service Hosting PaaS, IaaS, SaaS Under develop. Monitoring Availability & reliability monitoring Active Accounting Storage & Data Usage Reporting Under develop. SLC Management Service Portfolio & Catalogue Active Coordination Project Implementation, Active ( ) Service & Resource Provisioning Site Registry Site, Service & Service Groups Active ( )

17 Research Data Management

18 Data management plans

19 Research Data Management. An introductory webinar from OpenAIRE and EUDAT Organised by: EUDAT and OpenAIRE Speakers: Sarah Jones (DCC), Tony Ross- Hellauer (Universität Göttingen) Date: 26 May 2016 Attendees: 233 (364 registered) Recording, slides, chatlog and write-up Q&A session available on the website: h-data-management-an-introductory-webinarfrom-openaire-and-eudat

20 How to write a Data Management Plan webinar Organised by: EUDAT and OpenAIRE Speakers: Sarah Jones (DCC), Marjan Grootveld (DANS) Dates: 7 and 14 July 2016 Attendees: 404 (32 countries) & 208 (31 countries) Recording, slides, chatlog and write-up Q&A session available on the website: %E2%80%9Chow-to-write-a-data-management-plan%E2%80%9D

21 The CDI A Service Infrastructure

22 Using the CDI Generic services supplied by EUDAT for public use. Interaction with one or several CDI service providers/nodes (possibility to sign specific SLAs with service providers) Users benefit from the common service management approach followed by the CDI service providers and the connection between sites.

23 Joining the CDI Joining the CDI backbone Possibility to provide EUDAT services as part of the CDI to a specific community. Requires commitment to some rules and behaviors to ensure CDI coherence à captured in the CDI Partnership Agreement.

24 Interoperable nodes a) Interoperable the CDI Web Service Providers are connected to the CDI using a CDI compliant approach for data access and discoverability. Interoperable nodes: must have a data repository in which they preserve or curate data from a single research community or host data from several research communities or experiments (storage and access services). must identify the data hosted in the repository via some form of persistent identifier (persistent identification), and it must be possible to harvest and discover the associated metadata through EUDAT s B2FIND service (metadata). Benefits: Better discoverability of data hosted at the site Interoperability with other service providers through CDI compliant repository Possibility to deploy and run EUDAT services and tools for local customers with support from central staff

25 Integrated nodes b) Integrated the CDI Cloud (includes a) Service providers are integrated; there is a single service desk, and this part of the CDI infrastructure is monitored and in part managed centrally.. Integrated nodes: Ø are interoperable (as in a) Ø provide a common data access layer Ø integrate with a common authentication & authorization infrastructure (AAI) Ø integrate their local data infrastructure with the CDI s data management services Ø connect their services to the common CDI service management infrastructure framework. Benefits: a) + Tighter integration with other service providers, offering seamless integration of resources to pan- European communities.

26 Organisational Roadmap Project Partnership Legal Foundation? ERIC? Council Transition Board Transition Appointment of Executive Roles Appointment of Secretariat Establishment of Legal Entity Transition of Executive Roles Transition of Secretariat Relationship with other e-infrastructures (single e-infrastructure ERIC?) Establish ERIC Transition of Council Current emphasis on moving towards a Partnership Agreement between service providers (establishment in September 2016) Movement beyond Partnership (e.g. ERIC) will depend on circumstances and will be decided in close discussion with the other e-infrastructures (e.g. PRACE, EGI, GEANT), in the context of the European Open Science Cloud

27 Digital Infrastructures for Research 2016 Designed with research communities in mind, DI4R aims to foster broader adoption of digital infrastructure services and promote user-driven innovation. DI4R is co-organised by EGI, EUDAT, GÉANT, OpenAIRE and RDA Europe, hosted by ACC Cyfronet AGH