INDIGO-DataCloud Architectural Overview

Transcription

1 INDIGO-DataCloud Architectural Overview RIA Giacinto Donvito (INFN) INDIGO-DataCloud Technical Director 1 st INDIGO-DataCloud Periodic Review Bologna, 7-8 November 2016

2 Outline General approach of the INDIGO Platform The Platform in the proposal The overall view of the INDIGO Platform Standards, protocols, and implementations. 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 2

3 General approach of the INDIGO Platform Based on Open Source solutions widely supported by big communities whenever possible exploit general solutions instead of specific tools/services or put effort in increasing the generality of tools developed in a given community this will be important for sustainability of the architecture ensure that the framework offered to final users, as well as to developers, will have a low learning curve existing software suites like ROOT, OCTAVE/MATLAB, MATHEMATICA or R-STUDIO, will be supported and offered in a transparent way 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 3

4 INDIGO Global architecture INDIGO global architecture. Color codes: Yellow: implementation based on already available solution to be improved/changed; Orange: Completely new services to be implemented 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 4

5 Implementation approach Rely on standards µservice approach Modularity Pick the services you really need for your use-case And build your own platform based on your needs Each layer has clear interfaces and could be exploited directly by the end users The Authentication/Authorization is based on the concept of Delegation Each service could decide autonomously about the authorization Each service is requested exploiting the real end-user credential 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 5

6 Implementation approach Automation based on orchestrating resources This is done at different level (IaaS/PaaS/SaaS) Open: Not only Open Source, But the possibility to plug-in any supported services/protocols/resources in order to build the needed infrastructure Both private and public cloud resources could be part of the same INDIGO deployment You can build your own (private) infrastructure or provide a multi-tenant solution for your users. Depending on your goal 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 6

7 The high level view of the Architecture GUI Clients Admin Portlets Data Analitics WP6 Services User Portlets Workflow Portlets Future Gateway Portal Mobile Apps Open Mobile Toolkit Mobile clients Ophidpia plugin LONI plugin Taverna, Kepler plugin Workflows Other Science Gateways SG Mon Support services Future Gateway REST API Future Gateway Engine JSAGA/JSAGA Adaptors OIDC Kubernetes Cluster TOSCA REST/CDMI/Wedbav/posix/Gridftp Onedata Dynafed This is the INDIGO-DataCloud General Architecture* IAM Service TOSCA Infrastructure Manager PaaS Orchestrator CloudProvider Ranker QoS/SLA Monitoring Data Services Accounting FTS WP5 Services *: see details in or in Native IaaS API Non-INDIGO IaaS 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 7 TOSCA Aut. Scaling Service Heat/IM Smart Scheduling Mesos Mesos Cluster Cluster Spot Istances Native Docker Local Repository S3/CDMI/Posix/Webdav GridFTP Storage Service QoS Support Identity Armonization WP4 Services

8 IAM Service TOSCA Infrastructure Manager Monitoring CloudProvider Ranker counting WP5 Services Native IaaS API TOSCA Aut. Scaling Service Mesos Mesos Cluster Cluster S3/CDMI/Posix/Webdav GridFTP Storage Service Non-INDIGO IaaS Heat/IM Smart Scheduling Spot Istances QoS Support Identity Armonization Native Docker Local Repository WP4 Services 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 8

9 JSAGA/JSAGA Adaptors OIDC Kubernetes Cluster TOSCA REST/CDMI/Wedbav/posix/Gridftp Onedata Dynafed IAM Service TOSCA Infrastructure Manager PaaS Orchestrator CloudProvider Ranker QoS/SLA Monitoring Data Services Accounting FTS WP5 Services Native IaaS API TOSCA Aut. Scaling Service 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 9 Heat/IM Mesos Mesos Cluster Cluster S3/CDMI/Posix/Webdav GridFTP Storage Service QoS Support

10 WP6 Services GUI Clients Admin Portlets Data Analitics User Portlets Workflow Portlets Future Gateway Portal Mobile Apps Open Mobile Toolkit Mobile clients Ophidpia plugin LONI plugin Taverna, Kepler plugin Workflows Other Science Gateways SG Mon Support services Future Gateway REST API Future Gateway Engine JSAGA/JSAGA Adaptors OIDC Kubernetes Cluster TOSCA REST/CDMI/Wedbav/posix/Gridftp IAM Service PaaS Orchestrator QoS/SLA Onedata Data Services Dynafed 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 10 Accoun FTS

11 Some possible implementation scenario: 1. Enhanced Resource Virtualization -> Computing 2. Enhanced Resource Virtualization -> Storage 3. A complete federated storage access solution 4. Interactive usage of a Docker container with ssh 5. A web portal that uses a batch system to run applications 6. Virtual infrastructures for executing scientific applications and services exploiting Mesos 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 11

12 Enhanced Resource Virtualization -> Computing (OpenNebula) OCCI Support OneDock Orchestrator+TOSCA Support (IM) OpenNebula 1. IM: Provides a) Advanced IaaS Orchestrator capabilities b) TOSCA Support 2. OCCI: a) Enhanced Network capabilities b) Docker support 3. OneDock: a) Support for native Docker (on baremetal) 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 12

13 Enhanced Resource Virtualization -> Computing (OpenStack) OCCI Support NovaDocker Orchestrator+TOSCA Support (HEAT) OpenStack Synergy Spot Istances 1. TOSCA on HEAT 2. OCCI: a) Enhanced Network capabilities b) Docker support 3. Nova Docker: a) Support for native Docker (on baremetal) 4. Synergy: a) Fair-share on cloud resource usage 5. Spot-instances 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 13

14 Enhanced Resource Virtualization -> Storage (QoS) CDMI CEPH CDMI POSIX 1. CDMI service provides the capability to manage the QoS of storage 2. Independent from the technology used CDMI dcache 3. CDMI is not used to access files at the site level a) The files still could be accessed/stored using the original protocols a) b) c) d) 7-8/11/2016 1st INDIGO-DataCloud Periodic Review WebDAV Posix S3 GridFTP 14

15 Data Federation through INDIGO Onedata Amazon S3 NFS Server AWS USA INFN Italy VM nfs Docker Oneclient Docker DNS: p-aws-useast Docker Onezone VM onezone Docker VM oneprovider Docker Oneclient VM oneclient UPV Spain Docker Oneclient Docker Laptop OSX Docker Oneclient SAMBA Export POSIX Volume boot2docker 7-8/11/2016 VM: demo-onedata-upv-provider 1st INDIGO-DataCloud Periodic Review 20

16 Interactive usage of a Docker container with ssh - Overview User TOSCA Documents and Dockerfiles per Use Case 1.b) Automated Build 1.a.2) Dockerfile (commit) 1.a.1) INDIGO-DataCloud Docker Hub OrganizaLon build, push Champion + JRA WP6 3) Deploy TOSCA 4)Access Future Gateway API Server WP5 2) Stage Data Orchestrator Other PaaS Core Services Provider 5) Mount App SSHd Docker Public IP Container IM Cloud Site OneDock WP4 nova-docker 3 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 16

17 Interactive usage of a Docker container with ssh - INDIGO Services 1. TOSCA Template to describe the user service 2. Future Gayeway to configure and submit TOSCA Template in an easy way 3. Orchestrator + PaaS Core services + CloudProviderRanker + SLAM/QoS: a) To find the available IaaS b) That are correctly working c) That has SLA with the given user d) And supports the hw+sw requirements 4. Infrastructure Manager at the PaaS level in case the IaaS do not support native TOSCA-enabled orchestrator 5. IaaS Orchestrator (Heat/IM) supporting TOSCA 6. OneDock or Nova Docker to run Docker on bare metal at IaaS level 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 17

18 A web portal that uses a batch system to run applications - Overview 2) Deploy TOSCA with User Vanilla VM / Container TOSCA Documents and Dockerfiles per Use Case 1.b) Automated Build 1.a.2) Dockerfile (commit) 7-8/11/2016 WN 6)Access Web Portal 1) Stage Data Other PaaS Core Services Orchestrator Champion + JRA Virtual Elastic LRMS Cluster Future Gateway API Server WP5 1.a.1) INDIGO-DataCloud Docker Hub Organization build, push WN WP6 IM TOSCA Clues WN Galaxy Public IP 5) Mount IM OneZone Cloud Sites WP4 TOSCA Heat Front-End Provider 1st INDIGO-DataCloud Periodic Review OpenNebula OpenStack 18

19 Mesos PaaS solution exploiting INDIGO platform 2) Deploy TOSCA with User Vanilla VM / Container TOSCA Documents and Dockerfiles per Use Case 1.b) Automated Build Provider IM Public IP Workers Chronos/ Marathon Clues Mesos Masters 4) Install / Configure IM OpenNebula 7-8/11/ Other PaaS Core Services 5) Mount Virtual Elastic Mesos Cluster Workers WP5 Orchestrator Champion + JRA 1.a.1) INDIGO-DataCloud Docker Hub Organization build, push Future Gateway API Server 1) Stage Data 6) Access Mesos Services 1.a.2) Dockerfile (commit) WP6 1st INDIGO-DataCloud Periodic Review Cloud Site WP4 Heat OpenStack

20 A dynamic cluster to run applications INDIGO Services 1. TOSCA Template to describe the user service 2. Future Gateway to configure and submit TOSCA Template in an easy way 3. Orchestrator + PaaS Core services + CloudProviderRanker + SLAM/QoS: a) To find the available IaaS b) That are correctly working c) That has SLA with the given user d) And supports the hw+sw requirements e) That hosts the required data 4. Infrastructure Manager at the PaaS level in case the IaaS do not supports native TOSCA enabled orchestrator 5. IaaS Orchestrator (Heat/IM) supporting TOSCA 6. Onedata for shared and distributed data access 7. CLUES for driving the automatic resource provisioning based on the usage 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 20

21 Few examples of relevant tecnologies 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 21

22 Docker: Build-Ship-Run Docker is an open-source engine to easily create lightweight, portable, selfsufficient containers from any application. The same container that a developer builds and test on a laptop can run at scale, in production, on VMs, OpenStack/OpenNebula1 clusters, public clouds and more. Docker features: versioning (git-like) component re-use sharing (public repository) 1: thanks to ONEDock, an INDIGO-DataCloud development 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 22

23 TOSCA in a nutshell 7-8/11/2016 1st INDIGO-DataCloud Periodic Review Source: SCA/slideshelf 23

24 AAI Features INDIGO provides an advanced set of AAI features that includes: User authentication (supporting SAML, OIDC, X.509) Identity harmonization (link heterogeneous AuthN mechanisms to a single VO identity) Management of VO membership (i.e., groups and other attributes) Management of registration and enrolment flows Provisioning of VO structure and membership information to services Management, distribution and enforcement of authorization policies A Token Translation Service (TTS), creating credentials for services that do not natively support OpenID Connect. Services that do not support OpenID Connect are for example ssh, X509 based services S3 storage, OpenNebula. 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 24

25 Conclusions INDIGO has a modular approach in building its architecture This provides to the end users and to the resource/einfrastructure providers, the capability to build their platforms including the needed building blocks Or including new services developed by INDIGO in their own, already available, platforms INDIGO provide a high level of legacy compatibility: This allow also user community to easily plug their already available applications/services within the INDIGO Platform 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 35

26 Backup slides 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 26

27 INDIGO FAQ How do INDIGO achieve resource redundancy and high availability? This is achieved at multiple levels: at the data level, redundancy can be implemented exploiting the capability of INDIGO's Onedata of replicating data across different data centers. at the site level, it is possible to ask for copies of data to be for example on both disk and tape using the INDIGO QoS storage features. for services, the INDIGO architecture uses Mesos and Marathon to provide automatic service highavailability and load balancing. This automation is easily obtainable for stateless services; for stateful services this is application-dependent but it can normally be integrated into Mesos through, for example, a custom framework (examples of which are provided by INDIGO). How do INDIGO achieve resource scalability? First of all, we can distinguish between vertical (scale up) and horizontal (scale out) scalability. INDIGO provides both: Mesos and Marathon handle vertical scalability by deploying Docker containers with an increasing amount of resources. The INDIGO PaaS Orchestrator handles horizontal scalability through requests made at the IaaS level to add resources when needed. 7-8/11/ st INDIGO-DataCloud Periodic Review

28 INDIGO FAQ How do INDIGO achieve resource scalability? The INDIGO software does this in a smart way, i.e. for example it does not look at CPU load only: In the case of a dynamically instantiated LRMS, it checks the status of jobs and queues and accordingly adds or remove computing nodes. In the case of a Mesos cluster, in case there are applications to start and there no free resources, INDIGO starts up more nodes. This happens within the limits of the submitted TOSCA templates. In other words, any given user stays within the limits of the TOSCA template he has submitted; this is true also for what regards accounting purposes. How do you know when and where resources are available? We have extended the Information System available in the European Grid Infrastructure (EGI) to inform the INDIGO PaaS orchestrator about the available IaaS infrastructures and about the services they provide. It is therefore possible for the INDIGO orchestrator to optimally choose a certain IaaS infrastructure given, for example, the location of a certain dataset. 7-8/11/2016 1st INDIGO-DataCloud Periodic Review 35