Cloud Computing mit mathematischen Anwendungen

Transcription

1 Cloud Computing mit mathematischen Anwendungen Dr. habil. Marcel Kunze Engineering Mathematics and Computing Lab (EMCL) Institut für Angewandte und Numerische Mathematik IV Karlsruhe Institute of Technology (KIT) 4.Cloud Applications KIT University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association

2 Examples of Cloud Applications 1. Rendering of movies 2. Management of digital data 3. Office work 4. Collaborative work 2 Cloud Computing SS2011 M.Kunze

3 Cloud Architectures using SQS, S3 and EC2 3 Cloud Computing SS2011 M.Kunze

4 Cloud Applications (1): Rendering of Movies! Architecture: Weakly coupled services! Use storage service for files! Use compute service for work! Organize workflow by use of a queuing service 1. Store files 2. Store messages with instructions 3. Start computation 4 Cloud Computing SS2011 M.Kunze

5 5 Cloud Computing SS2011 M.Kunze

6 6 Cloud Computing SS2011 M.Kunze

7 7 Cloud Computing SS2011 M.Kunze

8 8 Cloud Computing SS2011 M.Kunze

9 9 Cloud Computing SS2011 M.Kunze

10 10 Cloud Computing SS2011 M.Kunze

11 11 Cloud Computing SS2011 M.Kunze

12 Cloud Applications (2): SmugMug 12 Cloud Computing SS2011 M.Kunze

13 Cloud Applications (2): SmugMug Mashup S3, EC2, FPS, Mashup Management of Pictures and Movies Millions of users Billions of pictures Revenue: Several million $ per year 19 employees - IT Services: Amazon Web Services - User management: OpenID Application = Mashup of services of various providers 13 Cloud Computing SS2011 M.Kunze

14 Cloud Applications (3): Windows Live! Windows 7 ships with ready to go cloud applications! Windows Azure Software Development Kit 14 Cloud Computing SS2011 M.Kunze

15 Cloud Applications (4): Google Apps! Google Apps offers office applications! Collaboration within the company (e.g. KIT) 15 Cloud Computing SS2011 M.Kunze

16 16 Cloud Computing SS2011 M.Kunze

17 docs.google.com 17 Cloud Computing SS2011 M.Kunze

18 Optimize Linear Equations! An example of when and how to use Solve! Let's say you're a farmer and you want to decide what to grow. You know that each crop will bring in a certain amount of money and take a certain amount of land, capital, and fertilizer to grow! You can grow three things: wheat, corn, and broccoli! Each of them has its own costs and brings in a given amount of money. However, there are only limited resources available:! Money available: $170! Land available: 70 acres! Fertilizer available: 100 tons 18 Cloud Computing SS2011 M.Kunze

19 Optimize Linear Equations! Here's how you can express this information in the spreadsheet: 19 Cloud Computing SS2011 M.Kunze

20 Optimize Linear Equations! These are the formulas used in the spreadsheet:! Money (Cell B7) =9*A3+8*B3+7*C3! Land (Cell B8) =7*A3+8*B3+9*C3! Fertilizer (Cell B9) =2*A3+10*C3! Profit (Cell B16) =6*A3+8*B3+C3! When you run Solve, it can tell you what to grow: 20 Cloud Computing SS2011 M.Kunze

21 Hausaufgabe 4 Einem IaaS-Anbieter steht ein Rechenzentrum mit 100 Schränken und einer gesamten Anschlussleistung von kw zur Verfügung. In jedem Schrank sind Rechner mit insgesamt CPU-Cores und 1,5 TeraByte Memory installiert. Es sollen 3 verschiedene Klassen von virtuellen Maschinen angeboten werden: Micro, Small und Large. Jede Klasse hat spezifische Anforderungen an Memory, Zahl von CPU-Cores, Energieverbrauch und bringt einen gewissen Gewinn. Die Managementwerkzeuge können maximal Instanzen verwalten. Large Small Micro Nebenbedingung Memory (GB) Max. 150 TeraByte CPU-Cores Max CPU-Cores Energiebedarf (W) Max kw Gewinn (Cent/h) Zu maximieren Verwenden Sie zur Optimierung des Gewinns die Solve-Funktion der Tabellenkalkulation aus den Google Apps. Wie viele Instanzen sollten von jedem Typ gefahren werden, damit der Gewinn maximal ist? Wie groß ist der Gewinn pro Stunde? Wie groß ist in diesem Fall der Energiebedarf? 21 Cloud Computing SS2011 M.Kunze

22 Cloud Research and Development 22 Cloud Computing SS2011 M.Kunze

23 23 Cloud Computing SS2011 M.Kunze

24 Open Cirrus Cloud Computing Research Testbed An open, internet-scale global testbed for cloud computing research! A tool for collaborative research! Data center management & cloud services! Resources! Multi-continent, multi-datacenter, cloud computing system! Federated Centers of Excellence around the globe! each with nodes and up to ~PB storage! and running a suite of cloud services! Structure! Sponsors: HP Labs, Intel Research, Yahoo!! Founding partners: IDA Singapore, KIT, UIUC! New partners: CESGA, CMU, ETRI, MIMOS, RAS, China Mobile, China Telecom! Cover story in IEEE Computer, vol 43, no 4, April Cloud Computing SS2011 M.Kunze

25 Cloud Systems Research and Architecture! Perform research in the following areas:! Cloud services! Datacenter federation! Datacenter management and automation! Data-intensive applications and systems! Cloud management tools, e.g. KOALA Methodology! Perform experiments also on a low system level! Utilize flexible cloud computing frameworks! Compare different configurations and implementations! Simple, transparent, controllable cloud computing infrastructure 25 Cloud Computing SS2011 M.Kunze

26 Proprietary Cloud Computing Stacks 26 Cloud Computing SS2011 M.Kunze

27 Open-source Cloud Stack! OpenCirrus researchers have complete access to the hardware and software platform 27 Cloud Computing SS2011 M.Kunze

28 Open Cirrus Blueprint Cloud Application Services Virtual Resource Sets Cloud Infrastructure Services Eucalyptus OpenNebula IT-Infrastructure Layer (Physical Resource Sets) 28 Cloud Computing SS2011 M.Kunze

29 Physical Resource Sets (PRS)! PRS service goals! Provide mini-datacenters to researchers! Isolate experiments from each other! Stable base for other research! PRS service approach! Allocate sets of physical co-located nodes, isolated inside VLANs.! Leverage existing software (e.g. Utah Emulab, HP OpsWare)! Start simple, add features as we go! Base to implement virtual resource sets! Hardware as a Service (HaaS) 29 Cloud Computing SS2011 M.Kunze

30 Virtual Resource Sets (VRS)! Basic idea: Abstract from physical resource by introduction of a virtualization layer! Concept applies to all IT aspects: CPU, storage, networks and applications,! Main advantages! Implement IT services exactly fitting customer s varying need! Deploy IT services on demand! Automated resource management! Easily guarantee service levels! Live migration of services! Infrastructure as a Service (IaaS)! Implement compute and storage services! De-facto standard: Amazon Web Services interface 30 Cloud Computing SS2011 M.Kunze

31 31 Cloud Computing SS2011 M.Kunze

32 What is it?! Open-Source toolkit for building cloud infrastructures! Originates from the EC funded RESERVOIR project! Orchestrates storage, network and virtualization technologies to enable the dynamic placement of multi-tier services on distributed infrastructures, combining both data center resources and remote cloud resources, according to allocation policies! Provides internal and cloud administration and user interfaces for the full management of the IaaS cloud platform 32 Cloud Computing SS2011 M.Kunze

33 Architecture Cloud Plugins: Amazon EC2 and ElasticHosts connectors 33 Cloud Computing SS2011 M.Kunze

34 Features Feature Internal Interface Scheduler Virtualization Management Image Management Network Management Service Management and Contextualization Security Fault Tolerance Scalability Installation Flexibility and Extensibility Function Unix-like CLI for fully management of VM life-cycle and physical boxes XML-RPC API and libvirt virtualization API Requirement/rank matchmaker allowing the definition of workload and resourceaware allocation policies Support for advance reservation of capacity through Haizea Xen, KVM, and VMware Generic libvirt connector (VirtualBox planned for 1.4.2) General mechanisms to transfer and clone VM images Definition of isolated virtual networks to interconnect VMs Support for multi-tier services consisting of groups of inter-connected VMs, and their auto-configuration at boot time Management of users by the infrastructure administrator Persistent database backend to store host and VM information Tested in the management of medium scale infrastructures with hundreds of servers and VMs (no scalability issues has been reported) Installation on a UNIX cluster front-end without requiring new services Distributed in Ubuntu 9.04 (Jaunty Jackalope) Open, flexible and extensible architecture, interfaces and components, allowing its integration with any product or tool 34 Cloud Computing SS2011 M.Kunze

35 35 Cloud Computing SS2011 M.Kunze

36 What is it?! Open-Source software infrastructure for implementing Cloud computing on clusters from UC Santa Barbara! EUCALYPTUS - Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems! Implements Infrastructure as a Service (IaaS) gives the user the ability to run and control entire virtual machine instances (Xen, KVM) deployed across a variety of physical resources! Interface compatible with Amazon EC2! Includes Walrus, a basic implementation of Amazon S3 interface! Potential to interact with the same tools, known to work with Amazon EC2 and S3! Eucalyptus is an important step to establish an open Cloud computing infrastructure standard 36 Cloud Computing SS2011 M.Kunze

37 Architecture! Cloud Controller (CLC)! Collects resources information of the CCs! Manages the S3 / EBS Services! Cloud Meta-Scheduler Public Network Public Network! Cluster Controller (CC)! Select informa5on of free resources! Control the alloca5ons of the VMs on the NCs CLC CC Private Network CC Private Network! Node Controller (NC)! Controls the Xen / KVM Hypervisor! Sends resource informa5on to the Cluster Controller S3 / EBS DataBase NC NC NC NC NC NC 37 Cloud Computing SS2011 M.Kunze

38 AppScale Open-Source implementation of Google App Engine! Project driven by UC Santa Barbara! App Engine is a platform service that allows to run Web apps in Python (and JAVA) on the Google infrastructure! AppScale works transparently on Cloud infrastructures like Eucalyptus! AppScale is compatible with Google App Engine! Applications for Google App Engine can be developed and tested in a private cloud Quelle: Navraj Chohan 38 Cloud Computing SS2011 M.Kunze

39 Tashi Scheduler Virtualization Service Storage Service Cluster Manager Node Node Node Node Node Node! Co-Scheduling of CPU and data 39 Cloud Computing SS2011 M.Kunze

40 R&D Examples! Job flow to run Monte-Carlo simulation! HPC as a Service! Cloud management services 40 Cloud Computing SS2011 M.Kunze

41 A Grid Job Flow Scenario! Grid example: Run 1000 Jobs to generate 1 million Monte-Carlo events! The Grid is a complex system 41 Cloud Computing SS2011 M.Kunze

42 A Cloud Job Flow looks much smarter! Works the same like on a local computer! Develop and debug the program locally! Build a virtual machine with the following characteristics! Automated program start after machine launch (e.g. Amazon EC2)! Do the data processing (e.g. generate 1000 Monte-Carlo events)! Write output to persistent cloud storage (e.g. Amazon S3)! Automated shut down once finished to stop accounting! Just instantiate the suiting number of machines in the cloud! Simply launch 1000 machines to produce 1 million events! Cost is the same:! Run 1 machine 1000 hours! Run 1000 machines 1 hour! No need for batch queues and scheduling ( unlimited resources ) 42 Cloud Computing SS2011 M.Kunze

43 HPC as a Service Tradi9onal HPC Architecture! is characterized by very specific computer clusters designed for special applica5ons! offers pre- defined opera5on systems and user environments only! serves one single applica5on at a given 5me! provides restricted user access! provides management privileges exclusively to administrators Concept of HPCaaS! Capability of using clustered servers and storage as resource pools, fully automated management! Individual cluster configura5on on- demand! Flexibility to serve mul5ple user groups and applica5ons with varying requirements! Customers gain resource management privileges 43 Cloud Computing SS2011 M.Kunze

44 Cloud Management: KOALA IaaS PaaS KOALA! Mobile management of hybrid cloud resources as SaaS solution! Plan: Develop ipad/iphone/android application 44 Cloud Computing SS2011 M.Kunze

45 KOALA hsp://koalacloud.appspot.com - hsp://code.google.com/p/koalacloud! Advantages and Disadvantages of a Cloud- based Management of IaaS wrt. a local solu9on (e.g.: Elas9cfox, Hybridfox)! Advantages:! Flexibility concerning the used browser! Support of EC2/S3/EBS and Eucalyptus! No local installa5on necessary (except of the private key)! Cloud installa5on corresponds to the cloud compu5ng idea! Disadvantages:! Users have to trust the provider of the management applica5on concerning the data privacy and opera5on availability! Advantages and Disadvantages of KOALA wrt. the Amazon Management Tools (especially AWS Management Console)! Advantages:! Support of EC2/S3/EBS and Eucalyptus! KOALA can run in a private cloud (via AppScale)! Disadvantages:! Not all AWS- Features implemented (5ll now)! No support by Amazon 45 Cloud Computing SS2011 M.Kunze

46 KOALA hsp://koalacloud.appspot.com - hsp://code.google.com/p/koalacloud!!!! 46 Cloud Computing SS2011 M.Kunze AppScale runs in EC2 KOALA runs in AppScale The cloud can be managed out of itself It is not necessary to store credentials at Google

47 Good to know! All course info is on Cloud Computing SS2011 M.Kunze

48 48 Cloud Computing SS2011 M.Kunze