EUDAT- Towards a Global Collaborative Data Infrastructure

Transcription

1 EUDAT- Towards a Global Collaborative Data Infrastructure FOT-Net Data Stakeholder Meeting Brussels, 8 March 2016 Yann Le Franc, PhD e-science Data Factory, France CEO and founder EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-infrastructures. Contract No

2 EUDAT European Data Infrastructure A consortium of high performance computing (HPC) / data centres, libraries, scientific communities, data scientists

3 A Changing Landscape Research Infrastructure trends: Internationalisation Diversification Increasingly relying on ICT Data deluge is a common challenge European Ris: Around billion investment middle age 19th century 20th century 21st century 3

4 A pan-european e-infrastructure solution for pan- European RI data Challenges All RIs are facing data challenges Where to store the growing amount of data? How to find it? How to make the most of it? Solutions are needed at pan-european level We need to promote synergies Some services are common to many communities Costs and investments can be optimised Better integration of e-infras and research infrastructures can be achieved 4

5 EUDAT Vision & Mission Our vision: Data is shared and preserved across borders and disciplines thereby enhancing the value and quality of research at large. Our mission: To enable data stewardship within and between European Research Communities through a Collaborative Data Infrastructure, a common model and service infrastructure for managing data spanning all European research data centres and community data repositories.

6 Community-Driven Solutions EUDAT services (the so called B2 Service Suite) are designed, built and implemented based on user community requirements. BIOMEDICAL & MEDICAL SCIENCES MATERIALS & ANALYTICAL FACILITIES MAPPER PHYSICAL SCIENCES & ENGINEERING

7 B2 Service Suite Covering both access and deposit, from informal data sharing to long-term archiving, and addressing identification, discoverability and computability of both long-tail and big data, EUDAT services seek to address the full lifecycle of research data

8 B2 Service Suite B2ACCESS B2Handle

9 Production Infrastructure Rep Operational Services Central Registry Sites&Services.. AAI PID providing PIDs Monitoring Data Project Coord. Resource&Service Prov. Thematic Service Provider Repository Provider Generic Service Provider (Archive, Large Storage System, HTC/HPC) Operational and Support services Helpdesk System PID Service Provider 9

10 Building solutions with the communities EUDAT services (B2 Service Suite) are designed, built and implemented together with user communites. Common Language Resources and Technology Infrastructure (CLARIN) European Network for Earth System Modelling (ENES) Distributed infrastructure for life-science information (ELIXIR) European Plate Observing System (EPOS) - Solid Earth sciences Research Infrastructure Integrated Carbon Observation System (ICOS) to quantify & understand greenhouse gas balance Long-Term Ecosystem Research (LTER) in Europe

11 Sharing Research Data (B2SHARE) Uses the repository for shareable digital objects (B2SHARE) and harvests the community metadata for inclusion into the Virtual Language Observatory. DKRZ is interested to run an EUDAT repository for shareable digital objects (B2SHARE) for ENES users LTER Europe will use the repository for shareable digital objects and evaluate the option for data repository for data objects / files. Fully integrate within DEIMS (LTER metadata editor and provider) & upload files to B2SHARE within DEIMS.

12 Data replication and long term preservation (B2SAFE) Data replication & long term preservation of the language data they are hosting & will extend the deployment throughout 33 centres Data replication & long term preservation of archived data to serve as basis of further developments & activities know location of all replica s for access optimization set up automated migration of daily sensor data streams of unique, non-reproducible observational data use for long-term storage of higher-level (processed) ICOS data products, including data sets from modelling activities know location of all replica s for access optimization For replication of virtual machines

13 Data staging (B2STAGE) Jointly developing a data subscription and distribution service Using B2STAGE to bring data to HPC centres Facilitate data movement and analysis of large volumes of data Set up pathways to allow fast and efficient data transfer to and from high-performance computing platforms such as EGI cloud

14 Multi-disciplinary joint metadata catalogue (B2FIND) allow data discovery followed by a subscription to ELIXIR/EGI/EUDAT destinations Find all EPOS data deposited in B2SAFE implement automated metadata upload for all ICOS data objects (not just those stored into the data replication and long term preservation service-b2safe). to deliver metadata to broader communities & evaluate options for MD integration & harmonisation (currently geonetwork in discussion)

15 Synchronising & Exchanging Research Data (B2DROP) use the secure and trusted cloud service to integrate their own workspaces into EUDAT. Thus allowing data retrieval and storage from the CLARIN community services. Policy-based prefix & PID management (B2HANDLE) Planned ENES data information tool with PID and versioning support using B2HANDLE library. Dependency on standardized PID types and may also display provenance information uses EUDAT brokered PIDs and is looking for special PID policies allowing explicit data verifiability

16 Authentication & Authorisation (B2ACCESS) Federated and multi protocol: Development of an AAI service to be used also outside of EUDAT data centers, to manage user authentication, authorization and identification needs Federated and multi protocol: Development of an AAI service to be used also outside of EUDAT data centers, to manage user authentication, authorization and identification needs

17 New Services in Development semantic annotation of data for the LTER community data Using a Data Type Registry

18 Investigating new services to support the Data Life Cycle Tracking data provenance across infrastructures Supporting Dynamic Data: working with incomplete and/or uncurated data Working with sensitive data: de-identification and anonymisation service Executing Workflows across infrastructures

19 Getting new communities on board and new requirements Data Pilot call (November 2015) 24 applications, of which 7 = Earth sciences, energy and environment, 6 = Biomedical and life sciences, 6 = Social Sciences and Humanities 5 = Physical Sciences and Engineering potential user audience = 40,000 users cumulative storage resource request of up to 4.3PB

20 Bridging with other e-infrastructures PRACE coupling EUDAT storage with PRACE HPC resources 1 st Pilot as part of PRACE 11 th Call (Feb 2015) 2 nd Pilot as part of PRACE DECI 13 Call (September 2015) PRACE applicants can now request resources to deposit computational results in EUDAT! EGI connecting data in the CDI to high throughput and computing resources provided through EGI Community pilots planned: EISCAT, ELIXIR, BBMRI, ICOS GÉANT - delivering EUDAT services to the end user through the NRENs First assessment shows that service providers comply with most NRENs requirements B2SHARE & B2DROP to be included in GEANT cloud catalogue OpenAIRE collaborating at policy level and connecting the two infrastructures through greater interoperability of the services and exchange of data. EUDAT has adopted OpenAIRE guidelines for data archives Discussion on collaboration between Zenodo & B2SHARE

21 European Open Science Cloud Through the promotion and adoption of common standards and protocols for all resources and digital services, the Open Science Cloud will connect networks, data, computing systems, software, tools and services for research as seamlessly as the Web connects information.»

22 Thank You!