NorStore. a national infrastructure for scientific data. Andreas O Jaunsen UNINETT Sigma as

Size: px

Start display at page:

Download "NorStore. a national infrastructure for scientific data. Andreas O Jaunsen UNINETT Sigma as"

Posy Alexander
6 years ago
Views:

1 NorStore a national infrastructure for scientific data Andreas O Jaunsen UNINETT Sigma as

2 About UNINETT Sigma UNINETT Sigma AS is a private company established by the Ministry of science and education (Kunnskapsdep.) and owned by UNINETT AS. Participation in national e-infrastructures is organized in a consortium with UiT, NTNU, UiB and UiO. Sigma has a coordinating role: Notur - national infrastructure for high perf. computing NorStore - national infrastructure for scientific data WLCG (Nordic Tier-1) - distributed (grid) services Sigma is the national representative in PRACE, EGI & EUDAT

3 NorStore and funding NorStore is on the national roadmap for infrastructures in science NorStore is funded by the RCN (67%) and the partners UiO, UiB, NTNU & UiT (33%) as part of the FORINFRA program. Currently funded period is RCN funding over 3.5 years is 37 MNOK Total project budget (including partner and user in-kind) is 58 MNOK Long-term funding is seen as a requirement due to the scope of most services provided and in securing commitments from the user communities

4 NorStore - human resources In total approx. 12 FTEs: project manager (1 FTE) administration (0.5 FTE) operations (1 FTE) user support (2-3 FTE) advanced user support (4-5 FTE) training (0.5 FTE) dissemination and outreach (0.5 FTE) development (1 FTE) technology coordinator / data manager (1 FTE)

5 Metacenter organisation a distributed e-infrastructure also needs to manage distributed man power the Metacenter (Notur, NorStore + PRACE, EGI, EUDAT) meet 1-2 times per year Metacenter activities are organised in task forces tasks forces meet F2F from once per month to few times per year) and when needed by video conference NorStore project group meet bi-weekly via video conference progress and results are disseminated/documented using wiki pages the Metacenter traditionally also meet during the annual Notur conference

UiB / Uni AS 29% Chemistry 4% Biosciences 25% Mathematics and

6 NorStore - usage statistics Primary disciplines Affiliation Computational Fluid Dynamics 11% Geosciences 39% Other 11% UiO 39% UiB / Uni AS 29% Chemistry 4% Biosciences 25% Mathematics and Informatics 7% Physics 4% IFE/Kjeller Nansensenteret 4% 4% NTNU 11% met.no 14%

7 NorStore - usage statistics Staffing Funding Perm staff 44% Other 29% EU 11% MSc students 13% guest 6% PostDoc 18% PhD students 19% Industry 2% Univ 25% SFF/SFI 12% NFR prog 22%

8 NorStore - usage statistics # people AccessOther DataManager DataPlan yes no # articles

9 NorStore - user requirements large storage capacities (scalable) with redundant copies of data computing-near-data secure storage and handling of (person-identifiable) sensitive data provide tools and user support for data management enable sharing of data (with colleagues) long-term archiving including curation and preservation of data tighter coupling between computing resources and data services easy access and authentication for non-traditional user groups (eg. webdav, cloud) authentication authorization infrastructure (AAI)

10 NorStore - e-infrastructure Research projects Research infrastructure Infrastructure A Commun. B Project C discipline services 10G UiT core services, user support NTNU UiA hardware, operations Org. B Inst. C UiB 10G 10G 10G UiO NorStore

11 User community services (bioinf)

12 User community services (bioinf)

13 Data life cycle

14 OAIS model Preservation Planning Descriptive Info Data Management Descriptive Info queries SIP Ingest Access result sets orders AIP AIP DIP Archival Storage Administration

15 NorStore - services application servers (ssh) WORK Generic Specific https API SSL Nationally federated identity (eg. Feide) Humanities Marine Climate & Environm. Language tech. Medical imaging Bioinfo

16 NorStore - services application servers (ssh) WORK irods PREPARE PRESERVE Generic Science archive Specific https API SSL Nationally federated identity (eg. Feide) Humanities Marine Climate & Environm. Language tech. Medical imaging Bioinfo

17 NorStore - services web-clients application servers (ssh) WORK irods PREPARE PRESERVE query access annotate ingest Feide Generic Science archive Specific https API SSL Nationally federated identity (eg. Feide) Humanities Marine Climate & Environm. Language tech. Medical imaging Bioinfo

18 NorStore - services web-clients HPC Notur application servers (ssh) stage data WORK irods PREPARE PRESERVE query access annotate ingest Feide Generic Science archive Specific https API SSL Nationally federated identity (eg. Feide) Humanities Marine Climate & Environm. Language tech. Medical imaging Bioinfo

19 NorStore - services web-clients HPC Notur application servers (ssh) stage data WORK irods PREPARE PRESERVE query access annotate ingest Feide Generic Science archive Specific THREDDS https AAI API SSL Nationally federated identity (eg. Feide) Humanities Marine Climate & Environm. Language tech. Medical imaging Bioinfo

20 NorStore - services web-clients HPC Notur application servers (ssh) user-funding stage data WORK irods PREPARE PRESERVE query access annotate ingest Feide Generic Science archive Specific THREDDS https AAI API SSL Nationally federated identity (eg. Feide) Humanities Marine Climate & Environm. Language tech. Medical imaging Bioinfo

21 Lessons learned (so far...) planning of services must be based on user input and requirements sustainability of services and funding is critical to ensure commitment from the community important to secure/build/recruit competence (data management, service & infrastructure design, storage technology, AAI, long-term archiving and preservation etc) secure commitment from user communities and partners (incl. funding) staffing > critical mass distributed project management is a challenge (need for frequent F2F meetings) use of task forces is one way to structure and secure ownership

22 Future infrastructure model AWS S3 9 Cloud Backed Storage 10 WebDAV Client 4 icat 1 irods IRODS Server 5 irods Clients Storage Devices 3 2 Swift Server 7 6 Cloud Clients Direct FileSystem Access On-premises storage devices irods Server On-premises irods server S3 Amazon storage service Swift Server On-premises OpenStack Storage

23 Future infrastructure model Zone A Zone B Swift Server A Swift Server B irods Server A 1 irods Server B Client irods A swift irods B Nova Nova 8 Nova Server Server 8 Server

Collaborative data infrastructure Data Generators Users

research environments Trust Data Curation Community

generation, annotation, interpretability Core Data

authenticity, workflow execution, mining Fig.

25 Collaborative data infrastructure Data Generators Users User functionalities, data capture & transfer, virtual research environments Trust Data Curation Community Support Services Data discovery & navigation, workflow generation, annotation, interpretability Core Data Services Persistent storage, identification, authenticity, workflow execution, mining Fig. from Riding the wave, EC High Level Expert Group on Scientific Data

26 NorStore WORK processing and analysis PUBLISH data generated science archive research community data-doc (annotate) publication PID doi:xxxx:yyyyyy ARCHIVE register and ingest quality control

Data Replication: Automated move and copy of data. PRACE Advanced Training Course on Data Staging and Data Movement Helsinki, September 10 th 2013

Data Replication: Automated move and copy of data PRACE Advanced Training Course on Data Staging and Data Movement Helsinki, September 10 th 2013 Claudio Cacciari c.cacciari@cineca.it Outline The issue