Cloud interoperability and elasticity with COMPSs

Transcription

1 Cloud interoperability and elasticity with COMPSs Interoperability Demo Days Dec , London Daniele Lezzi Barcelona Supercomputing Center

2 Outline COMPSs programming model COMPSs tools COMPSs interoperability Demo Conclusions 2

3 BSC vision on programming models Applications Program logic independent of computing platform PM: High-level, clean, abstract interface Power to the runtime API General purpose Task based Single address space Reuse architectural ideas under new constraints 3

4 Programming objectives Reduce the development complexity of Grid/Cluster/Cloud applications to the minimum Writing an application for a computational distributed infrastructure may be as easy as writing a sequential application Target applications: composed of tasks, called several times Granularity of the tasks or programs Data: files, objects, arrays and primitive types Programming languages support Java (native) Python C/C++ 4

5 COMPSs Infrastructure User code + annotations Python C/C++ Language bindings Java Custom Loader Task interception TDG Tasks Grids Clusters Clouds Files, objects

6 Runtime System Task handling Task graph Scheduling Data transfers Resource management Cloud elasticity Support for federated heterogeneous clouds i.e. openstack + Amazon OpenStack 6

7 OCCI The demo: COMPSs and PMES Browser PMES allows deployment of COMPSs apps and services Contextualization of applications at deployment time Pre-allocation of static pool of VMs to save virtualization startup time Use of additional provider (Prisma) to execute additional tasks COMPSs and PMES are available in the appdb through a single Virtual Appliance PMES Dashboard Job Mgmt (BES) OCCI CESNET (OpenNebula) COMPSs Runtime COMPSs app package COMPSs Static VM OCCI User User Data Data BSC Storage COMPSs COMPSs app app package package Prisma INFN Bari(OpenStack) PMES Service Providers Configuration PMES Service VM COMPSs libraries COMPSs libraries COMPSs app COMPSs package app package Input files Input files COMPSs Worker VM COMPSs Worker VM COMPSs libraries COMPSs libraries COMPSs app COMPSs package app package Input files Input files COMPSs Worker VM COMPSs Worker VM

8 COMPSs development environment IDE graphical interface Runtime monitor Paraver traces COMPSs environment: IDE Graphical interface to help developers with COMPSs applications Annotation of main program and tasks Generation of project and resources files (xml) Deployment in the infrastructure Developed as a Eclipse plugin Available in the Eclipse marketplace 16 COMPSs enviroment: trace generation Automatic generation of Paraver tracefiles Paraver is the BSC tool for trace visualization Trace events are encoded in Paraver (.prv) format by Extrae Paraver enables different views and of a trace COMPSs environment: Runtime Monitoring The runtime of COMPSs provides some information at execution time so the user can follow the progress of the application: Real-time monitoring information ( ) # tasks Resources usage information Execution time per task Real-time execution graph

9 Runtime behavior: scheduling and resource management Task Scheduler Assigns tasks to VMs or physical resources Each VM or resource has its own task queue Scheduling Optimizer Checks status of workers Can decide To perform load balancing Create/destroy new VMs Resource Manager: elasticity Manages all cloud middleware related features Holds information about all workers and about cloud providers Scheduler Optimizer sends to the RM requirements about new VM characteristics Resource Manager, evaluates the cloud providers alternatives and chooses the best option More economic The decision can be to open a new private or public VM For each Cloud provider, a data structure stores the different available instances (with its features) and the connector that should be used 9

10 COMPSs Project page: Direct downloads page: Source code Sample applications & development virtual appliances Tutorials Red-Hat & Debian based installation packages EGI Cloud Marketplace VA: 10

11 EUBrazil Cloud Connect A federated environment Thanks to the collaboration between European and Brazilian research centres the project leverages a set of components for the use of supercomputing, private cloud and cloud opportunistic resources 11

12 The COMPSs team Rosa M Badia Pedro Benedicte (part time) Carlos Diaz Jorge Ejarque Fredy Juarez Daniele Lezzi Francesc Lordan Roger Rafanell Cristian Ramon (part time) Raul Sirvent Enric Tejedor 12

13 Thank you!