I data set della ricerca ed il progetto EUDAT Casalecchio di Reno (BO) Via Magnanelli 6/3, 40033 Casalecchio di Reno 051 6171411 www.cineca.it 1
Digital as a Global Priority 2
Focus on research data Square Kilometre Array Large Hadron Collider Human Brain Project ESRF & ILL, Grenoble Daresbury Laboratory 3
G8+O5 working group and GSO Report Recommendations 9 & 10 Recommendation 9 - e-infrastructure Global research infrastructure initiatives should recognize the utility of the integrated use of advanced e-infrastructures, services for accessing and processing, and curating data, as well as remote participation (interaction) and access to scientific experiments. Recommendation 10 - exchange Global scientific data infrastructure providers and users should recognise the utility of data exchange and interoperability of data across disciplines and national boundaries as a means to broadening the scientific reach of individual data sets. 4
G8 + O5 White Paper Draft v1.0 5 Principles for an Open Infrastructure 1. Discoverability Implementation of appropriate persistent identifier frameworks, adoption of descriptive metadata standards, and the use of appropriate data formats and taxonomies. 2. Accessibility Publicly funded scientific research data should be made openly available with as few restrictions as possible. Ethical, legal or commercial constraints may be imposed on the use of research data to ensure that the research process is not damaged by inappropriate release of data. 3. Understandability Scientific data sets must be understandable in order to be effectively used. A set of numbers, texts, pictures or even videos alone cannot be understandable without additional context, semantics, data analysis tools, and algorithms. 4. Manageability management policies and plans must make it clear who is responsible for maintaining the availability of data and how the associated costs are to be met including issues associated with curation, storage and services. 5. People A global approach to research data infrastructure requires a highly skilled and adaptable workforce and culture that is able to capture the available data and make it available to those that are able to use it appropriately. 5
Sharing Vision Single Infrastructure à Single User Experience Raw Analysed Publica@on Different Infrastructures Analysis à Different User Experiences Catalogue Catalogue Catalogue Publica@ons Catalogue Raw Analysis Analysed Publica@on Publica@ons Experiment Raw Analysis Analysed Publica@on Publica@ons Observa@on Raw Analysis Analysed Publica@on Publica@ons Simula@on Capacity SoAware Publica@ons Storage Repositories Repositories Repositories 6
EUDAT project EUDAT is a three-year project that will deliver a Collaborative Infrastructure (CDI) Co-funded within the 7th Framework Programme Launched on 1 October 2011 7
EUDAT Collaborative Infrastructure Analysed Catalogue Publica@on Catalogue Generators Analysed Users Publica@on User functionalities, data capture & transfer, virtual research environments Curation Trust Analysed Community Support Services Publica@on discovery & navigation, workflow generation, annotation, interpretability Analysed Common Services Repositories Publica@on Persistent storage, identification, authenticity, workflow execution, mining 8
EUDAT Principles 1. deposited with the EUDAT CDI will be preserved in perpetuity 2. are best curated in their own communities 3. Access to data in the EUDAT CDI is free at the point of use 4. EUDAT will operate as a federation of community-facing repositories and back office hosting providers 5. EUDAT services and infrastructure must be a suitable target for TDR outsourcing (cf. datasealofapproval.org) 6. EUDAT will not assert ownership of any data it holds 9
Persistent Identifiers (PID) EUDAT relies on the EPIC service to associate persistent identifier to digital objects EPIC is an identifier system using the Handle infrastructure. Its focus is the registration of data in an early state of the scientific process, where lots of data is generated and has to become referable to collaborate with other scientific groups or communities, but it is still unclear, which small part of the data should be available for a long time period. Handle resolution 10
First Services in Preparation Metadata Catalogue AAI PID Aggregated EUDAT metadata domain. inventory Safe Replication preservation and access optimization Staging Dynamic replication to HPC workspace for processing Simple Store Researcher data store (simple upload, share and access) Network of trust among authentication and authorization actors Anchor for identifica -tion and integrity PID Metadata 11
Community Centres 12
EUDAT-EPOS success story The whole archive of the INGV s data center in Rome is replicated to CINECA, the EUDAT s data center in Bologna ( years:1990-2013 25 TB ) Granularity is at single file level (the collection of sesimic waveforms related to 1 sensor for 1 day) Metadata are replicated too, as files (dataless). The replicas are stored, synchronized and accessed by a single user, the INGV community manager. 13
Safe Replication Goals Relevant data needs to be replicated from community centers to a number of data centers in a safe way with several purposes in mind: data bit-stream preservation more optimal data curation better accessibility of data identification of data through Persistent Identifiers (PIDs) Common functionality: Create M replicas (identified by a PID record) at different data centers for N years, exclude centers X, maintaining the given access permissions 14
EUDAT domain data enrichment processing reduction domain of globally referenceable data analysis temporary data global registration registration referenceable data citable publication data acquisition generation description EPIC PID handle preservation 15
SmartDATA project CINECA s project to deal with Big and HPC: a Big Analytics engine for OpenDATA catalogues Initial pre-conditions: are archived, catalogued, searchable and accessible: - Robust and reliable infrastructure - Interoperability and adoption of standards Smart satisfies these requirements and besides: make available an HPC environment with analysis tools which foster the re-use of such data. ease the information exchange among experts belonging to different disciplines, from both the point of view of the methods and of the data. à Hence the data sharing creates new knowledge 16
SmartDATA project SmartDATA Portal and apps SmartDATA users orchestrator HPC Front-End Job Scheduler workflows ingestion Registry (irods) for OpenDATA sources repository Scratch staging Supercomputer analytic engine 17
EUHIT project EuHIT is a consortium that aims at integrating cutting-edge European facilities for turbulence research across national boundaries (2013-2017). It includes 25 research institutes and 2 industrial partners from 10 European countries. A total of 14 cutting-edge turbulence research infrastructures CINECA will provide the technical and hardware infrastructure for the turbulence data digital library service (DLTD), offering: A service to manage data coming from numerical simulations and experiments in the field of fluid dynamics. A storage space of 150 TB online. A helpdesk and specialized support. 18
EUHIT project TurBase, a freely accessible living knowledgebase for high quality turbulence data, will rely on the CINECA s DLTD service, which will be implemented using the same technology adopted by EUDAT to foster interoperability and accessibility 19
Human Brain project The Human Brain Project (HBP) is a large scale European research project which aims to simulate the world's first and most exact human brain in a supercomputer. Working together with 86 different European institutions the project is planned to last ten years (2013-2023). To image an entire brain many parallel stacks of images are acquired. They are afterwards merged with a custom-made algorithm suited to work with very large data sets (~ 1 TB) 20
Human Brain project WP7.5 High Performance Computing Platform: integration and operations CINECA is responsible for Task 7.5.4 The HBP supercomputer for massive data analytics: Implement and operate a data-centric HPC facility providing efficient storage, processing and management of large volumes of data generated by the HBP. The HBP Massive Analytics Supercomputer in CINECA will provide 2 Petaflop/ s of peak performance and 200 Terabytes of main memory, integrated with a mass storage facility of more than 5 Petabytes of working space with an aggregated file system bandwidth of about 70 GB/sec in a full production environment. The system will be integrated with a Facility of around 5 Petabytes for on-line disk storage repository and more 10 Petabytes for long term data archiving, providing efficient data life-cycle management of structured and un-structured data generated by the HBP. 21
CINECA Repository service The service is implemented through irods (integrated Rule-Oriented System - https://www.irods.org). It provides the following functionalities: Up/Download: the system supports high performance transfer protocols like GridFTP, or irods multi-threads transfer mechanism, and large interoperable ones like HTTP. Metadata management: each object can be associated to specific metadata represented as triplets (name,value,unit), or simply tagged and commented. Long Term archiving: the long-term accessibility is granted by means of a seamless archiving process, which is able to move the collections of data from the on-line storage space to a tape based off-line space and back, according to general or per-project policies. Stage-in/stage-out: the service is enabled to move data sets, requested as input for computations, towards the HPC machines' local storage space, commonly named scratch, and backwards as soon as the results are available Sharing: the capability is implemented via a unix-like ownership model. Moreover a ticket based approach is used to provide temporary access tokens with limited rights. Searching: the data are indexed and the searches can be based on the object locations or on the associated metadata. 22
Thanks for your attention! Questions? 23