Bridging cloud computing innovation & Open Science

Transcription

1 Bridging cloud computing innovation & Open Science Updates & major achievements 1

2 Facts about HNSciCloud 2 Estimated Total Value excluding VAT: EUR Lead Procurer: CERN Start Date: January 2016 End Date: End of 2018 November 2016: Design Phase tender award February 2017: Assessment of the Design Phase deliverables April 2017: Prototype Phase award ceremony 6 February Launch of the Pilot Phase 29 November Results of the Pilot Phase The Helix Nebula Science Cloud (HNSciCloud) is a 5.3 million Pre-Commercial Procurement (PCP) tender for the establishment of a European hybrid cloud platform to support the deployment of high-performance computing and bigdata capabilities for scientific research. HNSciCloud is sponsored by 10 of Europe s leading public research organisations and co-funded by the European Commission.

3 THE HNSciCloud PCP The HNSciCloud PCP tender covers the procurement of R&D services for the design, prototype development and pilot use of innovative cloud services. The innovative cloud services provided through the platform are designed and implemented to address a set of challenges that require a combination of services at the Infrastructure as a Service (IaaS) level: Compute and Storage - support a range of virtual machines and container configurations working with datasets up to the petabyte range; Network Connectivity and Federated Identity Management - provide highend network capacity for the whole platform with common identity and access management; Service Payment Models - explore a range of purchasing options to determine the most appropriate ones for the scientific application workloads that will be deployed. Who can benefit from HNSciCloud? The procured cloud services integrated with the procurers in-house resources and publicly funded e-infrastructures provide a hybrid platform that serve scientists and engineers working in high-energy physics, astronomy, the life science including biomedical research, and the photon/neutron science in which the ten procurers operate. High Energy Physics Astronomy Life Sciences Photon/Neutron Sciences Long Tail of Science 3

4 January September 2016 The Tender Phase In March 2016, HNSciCloud organised the Open Market Consultation (OMC) event to analyse and discuss with over 70 cloud provider representatives the HNSciCloud requirements and use cases. The OMC outputs set the basis for the preparation of the tender material. The tender was launched in July 2016 and closed in September A total of 30 multinational companies, SMEs and public research organisations from 13 countries submitted bids during the summer. In November 2016, 4 Consortia were awarded contracts at the tender award ceremony at CNRS, in Lyon, France. The HNSciCloud Pre-Commercial Procurement tender in numbers 5.3 million Pre-Commercial Procurement (PCP) tender to establish a European hybrid cloud platform Tender launched on 21 July 2016 { Tender closed on 19 September 2016 { The tender generated a high-level of interest in the cloud services market A significant number of FP7 and H2020 projects were cited in the bids as sources of innovation contributing to the proposed solutions 4 consortia were awarded contracts for the Design Phase Phase 0: January October 2016: Solution Exploration { Over 50 press clippings promoting the tender Tender material downloaded more than 200 times More than 90 written questions received 1 Award Ceremony in Lyon, France 2 november tender Information Session, 7 September Days to bid More than 300 views of the animated video describing the tender 30 bidders from 13 countries including multinational companies, SMEs and public research organisations 4 winning innovative public-private consortia 11 winning economic operators Phase 1: November February 2017: Solution Design 4 Awarded Consortia Contractor 1: T-Systems, Huawei, Cyfronet, Divia Contractor 2: IBM Contractor 3: RHEA Group, T-Systems, Exoscale, SixSq Contractor 4: Indra, HPE, Advania, SixSq HNSciCloud Tender Award Ceremony

5 The Design Phase The Design Phase of HNSciCloud was kicked-off during the tender award ceremony with 15% of the tender budget allocated to this phase. In February 2017, the 4 consortia delivered their designs (including architecture and technical design of components together with unit costs). The eligible contractors submitted their prototype phase bids in early March and the HNSciCloud evaluation committee assessed the bids and selected the 3 most promising. The 3 winning Consortia moving to the HNSciCloud Prototype Phase were announced on the 3rd April 2017 during the awards ceremony held at CERN, in Geneva, Switzerland. Lessons learnt Prepare detailed information on each use-case Start the dialogue between the buyers group and the contractors early and continue discussions during the whole phase Set intermediate milestones for each contractor to present their progress Revise the buyers group effort planning when the Request for Tender is published»» Establish a dedicated technical team to collect input from all procurers, interact with the contractors and perform tests October March 2017 Regular meetings organized to facilitate the information flow between contractors and use case representatives and experts 4 design proposals presented and 3 prototype bids submitted Around 100 relevant stakeholders at the Prototype Phase award ceremony 3 consortia awarded contracts for the Prototype Phase Consortia moving from the Design to the Prototype Phase Contractor 1: T-Systems, Huawei, Cyfronet, Divia Contractor 2: IBM Contractor 3: RHEA Group, T-Systems, Exoscale, SixSq 5

6 April End of 2017 The Prototype Phase In the Prototype development Phase, which kicked-off in April 2017, the contractors built prototypes and made them accessible to experts from the buyers group for testing. In order to ensure a successful outcome of this phase, the buyers group closely monitored the contractors activities, performed extensive testing of the prototype solutions and organized meetings and progress review events to facilitate the information flow between contractors, researchers and IT experts. The prototype phase has produced results that address the major PCP challenges of Compute and Storage by supporting a range of virtual machines and container configurations working with datasets; Network Connectivity with high-end network capacity via GÉANT, and Federated Identity Management using the SAML2 protocol supporting edugain and ELIXIR. Consortia moving from the Prototype Phase to the Pilot Phase Contractor 1: T-Systems, Huawei, Cyfronet, Divia Contractor 3: RHEA Group, T-Systems, Exoscale, SixSq Recommendations for the pilot phase Precise definitions of key technical requirements including examples/ user stories should be provided during the earlier stages of the project. Feature freeze periods must be enforced more strictly, both in tender documentation and during the project execution. A ticketing system and shared collaboration tool between the buyers group and each contractor should be put in place at the very beginning of the phase. A single point of access to all procurers documentation is needed. Foreseen IaaS capacity and cloud resource utilisation should be planned carefully. Tender documentation should describe a solution in case not all of the resource were utilized due to delays. The involvement of the same partners, subcontractors or technologies in multiple bids should be avoided to ensure the solutions developed within the PCP are sufficiently distinct. 6 Consortia moving from the Design to the Prototype Phase

7 The Prototype Phase During the Prototype development Phase, kicked-off in April 2017, 3 prototype platforms were developed. In this phase, the contractors built prototypes including all components and made them accessible to experts from the buyers group for testing. Our prototype solution will benefit from the long-term experience with GÉANT, the European academic network: we believe that this will put us in a strong position to grant access to large amounts of data and be able to transfer large amounts of data across Europe. We also have good experience working in the science sector, and we are very anxious to learn how this will adapt to a large scale project like this and how we can show that it really performs up to the level that is required. Jurry de la Mar, Account Director in the Public Sector for T-Systems, representative of Contractor 1 Our objective is to build a hybrid solution based on existing IBM services products and technologies and develop new capabilities built on top of the IBM technologies to best fit the special requirements of Helix Nebula. Ezra Silvera, Researcher and Cloud Architect for IBM, representative of Contractor 2 The solution that we are going to prototype is a true multi cloud and hybrid cloud solution, composed by Exoscale and Open Telekom from T-Systems and managed by Nuvla, a service from SixSq. The data management layers will be provided by Cyfronet, while the open data solution and the entire project will be managed by RHEA. Marc-Elian Bégin, CEO and co-founder of SixSq, representative of Contractor 3 7

8 February Decenmer 2018 The Pilot Phase During the Pilot phase, which started in February 2018, contractors expanded prototypes that were firstly made available to IT experts who performed scalability tests and then to the end-users so theat they could deploy their applications. The assessment of each pilot provided feedback to the contractors. Phase objectives Assess each pilot providing feedback to contractors Open the pilot platforms to end users to deploy their applications Access to cloud service capacity 10k/1PB 3 Prototypes 2 Pilots 5k/500TB 3.5k/350TB 2k/200TB 100/10TB Cores/ Storage Functional Testing 10Gbps Scalability Testing End User Access 40Gbps JUN17 DEC17 FEB18 DEC18 8

9 HNSciCloud use cases The group of research organisations, the Buyers, tested the HNSciCloud pilot platform services in different use cases. These organisations used Computing, Storage and Identitfy Manager services across four scientific domains, comparing the performance and costs of diverse applications. In the following pages some of the use cases are presented. For more information visit / User area Using HNSciCloud, the PanCancer project will be able to determine genetic variation for more than 5000 tumor samples with more coming in on a monthly basis. Tony Wildish, EMBL-EBI Tested features Fields of Research Buyers AAI Federated Identity Storage blocks/objects Computing capacity (VMs) FDMNES Photon / Neutron Science High Energy Physic Astronomy Life sciences CrystFEL DODAS LOFAR PanCancer WeNMR/ HADDOCK useful, they can ease the way I do may job relevant, they can really improve the way I do my job strategic, they are fundamental to execute my job 9

10 LOFAR ASTRON Low Frequency Array Astronomy LOFAR (Low Frequency Array) is the first of a new kind of telescope that uses an array of simple omnidirectional antennas as opposed to a dish antenna for mechanical signal processing. The electronic signals obtained from the antennas are digitized and transported to a central digital processor. The antennas are simple enough, but there are a lot of them and indeed about 7,000 in the full LOFAR design. The main goal of the LOFAR use case is to test, and later put into production, separate location of data storage and computing resources distributed throughout Europe. The Helix Nebula Science Cloud would grant this opportunity due to the availability of the highspeed network connections of the Geant Cloud VRF infrastructure, and the pricing model based on compute and storage alone. The transparent data access functionality would even increase ease of use. CrystFEL Serial Femtosecond Crystallography Photon/Neutron Sciences The Crystfel framework is used for the technique of Serial Femtosecond Crystallography (SFX) and comprises programmes for data processing, simulation and visualization. It is a part of a complex, nonredistributable software stack, which is free for use by academia and non-profit organizations. The crystfel framework is increasingly used at various synchrotrons and FELs to analyze data from serial (femto-second) x-ray crystallography. The nature of these experiments make a cloud-based distributed pipeline particularly appealing, as the framework can fully exploit large computational resources with tunable demands. The objective of the CrystFEL use case is to run medium data intensive data analysis tasks (of one of the demanding photon science experiments). 10

11 3DIX Calculation of X-ray absorption spectra Photon/Neutron Sciences The 3DIX project produces 3D Imaging with X-rays. The objective is to make 3D images (volumes) of nanoscale objects to study their characteristics on a nanoscale. These studies can then be applied to a wide variety of objects and scientific fields such as chemical studies, life sciences, structure of materials etc. The calculations performed by the FDMNES programme require very little data transfer and storage. They are, however, often very compute intensive. Running such computations therefore uses considerable computing resources, but this load varies considerably over time depending on the experiments done and the research interests of the scientists. Instead of building upon the computing facilities in an institute for a rarely occurring maximum demand, the 3DIX project wants to offload such peak loads into the cloud. PanCancer: Supporting large scale genomics analyses in cancer studies Life Sciences The primary goal of the Pan Cancer initiative is to compare 12 tumor types profiled in the context of The Cancer Genome Atlas Research Network. Cancer can take hundreds of different forms, depending on external factors such as localization and cell type. PanCancer currently represents the most comprehensive computational study dealing with cancer genomics, with roughly 1 PB of data to be processed. This has forced researchers to implement new pipelines able to cope with the massive quantity of data, with a focus on leveraging cloud resources provided by public and commercial clouds, including the EMBL EBI Embassy Cloud. Using HNSciCloud, the PanCancer project will be able to determine genetic variation for more than 5000 tumor samples with more coming in on a monthly basis. 11

12 DODAS_CMS: Dynamic On Demand Analysis Services High Energy Physics CERN does not have the computing or financial resources to crunch all of the data on site. Recently cloud computing has come to public attention due to the promise of providing as much computing power as people require, simplifying management and reducing TCO. CMS is one of the LHC experiments in the Worldwide LHC Computing Grid (WLCG) collaboration, linking up 170 computing centres in 42 countries. Demonstrating that HNSciCloud can satisfy the requirements of LHC experiments has an enormous potential impact on cloud adoption in the particle physics environment. It could also be a driver for non-lhc experiments that do not have the human resources to move their computing to the cloud, by providing guidelines and best practices in order to simplify this process. WeNMR / HADDOCK - Information driven modelling of biomolecular complexes Life Sciences HADDOCK (High Ambiguity Driven protein-protein DOCKing) uses an informationdriven, flexible docking approach for the modelling of biomolecular complexes. Given the computational demands of the HADDOCK user group, the availability of easily accessible, affordable compute resources is crucial. The Helix Nebula science cloud would provide the opportunity to add these resources in a way that is seamless and invisible to the end user. To obtain this outcome, the resources in the Helix Nebula cloud will be made available to the current job scheduler of the HADDOCK project. 12

13 HNSciCloud Pre Commercial Procurement: lessons learned Project Timing & Structure Foresee a period of 2 months for each phase transition. The PCP execution process could be simplified as a single work package spanning de-sign, prototype and pilot phases. Select integrated set of tools with consistent access mechanisms to ensure sharing of material amongst the buyers group and with the contractors. Tender Preparation Phase Invest effort in the preparation phase (between 9-12 months) to agree and precisely define the R&D challenges, the objectives and expected outcome of the PCP. Prepare an in-depth needs assessment and an open market consultation activity during the tender preparation phase. Organize events where the procurers and potential tenderers can progressively refine the focus under the guidance of the expert Launch a survey among the known market players to allow the procurers to detect the capabilities and the willingness of the market to participate in the tender Tender Process Nominating a lead procurer that already had longstanding relationships with all members of the buyers group proved a successful approach in HNSciCloud. The close cooperation between the members of the buyers group was essential to the success of the project. Regular and intense interaction between the members of the buyers group as well as between the buyers group and the contractors has been an important element of the success of HNSciCloud. Include in the tender text a restriction forbidding a company from being a lead contractor and a participant in other bidding consortia. No more than one tender can be submitted by a natural or legal entity Include provisions in the selection criteria to ensure that proposed solutions by compet-ing consortia are sufficiently distinct and multiple solutions for each PCP challenge are developed. Strive to contract a higher number of consortia during each phase to ensure sufficient competition and increase the likelihood of a successful completion.»» Schedule intermediate reviews and payments as important checkpoints for the buyers group and contractors, to ensure all parties remained active and engaged. 13

14 Execution Phase Establish mechanisms for the buyers group to prioritise the challenges, communicate these priorities to the contractors and take them into account when allocating resources during the execution phase. The buyers group should consider how to deal with change requests and their impact on the execution of the project by the contractors. Consider the concept of an early adopter programme from the very beginning of the PCP, during the planning of funding projects. Commercialisation Plans A Total Cost of Ownership (TCO) study for selected use cases was introduced in the pilot phase to help the buyers group understand the impact of the commercialisation plans for their organisa-tions. Consider including a Total Cost of Ownership (TCO) study as an award criterion for each phase of the PCP execution (design, prototype and pilot).»» Consider engaging a specialist to review the tender specifications so that they are compatible with market offerings, to estimate the current market costs of the 14

15 IaaS capacity requested and to determine if the resulting services are competitive compared to the market prices In order for TCO studies to be a useful tool in service provisioning decisions, cost transparency needs to be increased for in-house provisioned service by public sector research procuring organisations Solution Testing Consider the development and execution of a testsuite as a separate task with its own milestones, deliverables and funded effort. Foresee a centralised repository of test-cases, testing material and test results to be shared across the buyers group and the contractors during the prototype and pilot phases. Include in the tender documents for the pilot phase progressive ramping-up of capacity of IaaS resources in order to contain the cost of IaaS provisioning over the lifetime of the project Monitor regularly the consumption of resources and make adjustments accordingly to control costs and ensure all the testing and deployment objectives can be achieved. Formalise the manner in which service outages are compensated in the tender document Network Connectivity The HNSciCloud tender required the commercial data centres to connect to the GÉANT network. Network connectivity is an essential element of a hybrid cloud model and HNSciCloud requested the contractors to implement a means of measuring network performance. The perfsonar software was jointly chosen by the buyers group and contractors as the means of performing and sharing end-to-end network measurements and installed at all the participating sites to monitor the bandwidth provided by the GÉANT connectivity» The network requirements for data intensive use cases should be established in detail during the tender preparation phase and provisions for network connectivity and data ingress/egress taken into account. 15

16 Training and Consultancy A series of training and consultancy events were included in the tender call off for the pilot phase of HNSciCloud. These events were hosted by members of the buyers group with the training and consultancy provided by the contractors according to the needs of the host. These events proved useful with feedback gathered from the participants and the contractors that served to improve the follow-on events. The importance of training and consultancy to facilitate the testing and uptake of the developed services should be not be underestimated and needs to be tailored to the audience, typically presenting different material for end-users, developers and site administrators. Consider having on-site consultancy to assist the buyers group members in deploying their selected use-cases at the beginning of the pilot phase. Access to innovative services beyond the project lifetime Include the option to purchase the resulting pilots through a license to use the final developed solution (for a period beyond the end of the project lifetime), including network access and use of future developments, after the project end, till the final product is commercialised. HNSCiCloud as a service provider for the EOSC The innovative services from the pilot platforms have been registered in the EOSC service catalog to promote their adoption by other user groups, reaching a wider European user base. 16

17 Early Adopter Group Over past months, the HNSciCloud consortium has promoted the common hybrid science cloud platform currently under development to groups of research communities, organisations and ESFRI research infrastructures across Europe resulting in a number of requests from organisations wishing to join the procurement group. HNSciCloud is now pleased to launch the Early adopter group with the purpose to expand the set of procurers The Voucher scheme HNSciCloud adopted a voucher scheme to lower the entry barriers to cloud services for new users. The scheme enabled community managers of the Early Adopter teams to distribute voucher codes to users, such that they could more easily access cloud resources. With a limited quota associated to a voucher, users could easily deploy cloud resources, associated to a community or customer account, and the procurer s had an oversight of the usage of the services they funded. Who joined the Early Adopter Group? The Early Adopter group includes research organisations wishing to become procurers for the resulting HNSciCloud services and wanting to take advantage of all the benefits related to joint procurement. The Early Adopter Group rules of engagement are available at www. hnscicloud.eu Benefits Opportunity to evaluate the use of commercially supported cloud services selected and tested by the HNSciCloud procurers»» Be able to fund the use of the cloud services via their regional, national or EC projects 17

18 What they say about us HNSciCloud listed among the EOSC in Practice Stories On 23 November 2018 the Final report and Recommendations on the European Open Science Cloud prepared by the 2nd High Level Expert Group on the European Open Science Cloud (EOSC), was officially launched. HNsciCloud is listed among the twelve EOSC in Practice Stories that could work as common service working models illustrating what the European scientific community is already achieving. The report was launched during the EOSC launch ceremony in Vienna as part of the programme of the Austrian Presidency of the Council of the European Union. The full report is available here ec.europa.eu/research/openscience/index.cfm?pg=open-science-cloud e-irg highlights Helix Nebula as a Cloud Service Provider for ESFRI communities In March 2017 e-irg released its support document Guide to e-infrastructure Requirements for European Research Infrastructures. Helix Nebula is listed among the cloud service providers for the ESFRI research community, with a dedicated section in the document which highlights the project s features and objectives. To read the latest e-irg Guide to e-infrastructure Requirements for European Research Infrastructures, visit goo.gl/scdxew 18

19 The HNSciCloud tender material speeds up the tender material preparation of the SELECT for Cities PCP project In February 2017, the SELECT for Cities initiative launched its call for tender for the development of an Internet-of-Everything (IoE) platform. HNSciCloud tender material has been used for the preparation of the tender documentation, as a source of good practices. In particular, the SELECT consortium took inspiration from the general structure of the documentation and used the same language and presentation model of the HNSciCloud tender material. To read the interview with Hugo Tamagnini Gonçalves, the SELECT for Cities Coordinator, visit goo.gl/likarr HNSciCloud is contributing to the establishment of the European Open Science Cloud On the 3rd of February 2017 renowned leading experts met the Commissioner for Research, Science and Innovation Carlos Moedas to discuss the next steps of the European Open Science Cloud. Bob Jones, CERN & HNSciCloud coordinator, brought to the table his experience and the major findings from the HNSciCloud Pre- Commercial Procurement and from the Helix Nebula Initiative. goo.gl/futzf4 19

20 Who s behind HNSciCloud? 20 Stay Tuned K A info@hnscicloud.eu This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit Helix Nebula - The Science Cloud with Grant Agreement is a Pre-Commercial Procurement Action funded by H2020 Framework Programme.