Building a Big IaaS Cloud. David /

Size: px

Start display at page:

Download "Building a Big IaaS Cloud. David /"

Ursula Ray
5 years ago
Views:

1 Building a Big IaaS Cloud David ke4qqq@apache.org / david@gnsa.us

2 #whoami Recovering Sysadmin F/LOSS contributor Committer on Apache CloudStack

3 Assumptions You have a need for an IaaS compute cloud platform You know what IaaS and cloud mean

4 Massively scalable Scalable - this is the easy part Massively - this part is much harder - getting to thousands of physical hosts is complex - getting to tens of thousands of physical hosts is a completely different magnitude of problem.

5 So I have some questions

6 Virtualization alone does not make a cloud Server Virtualiza0on Cloud Built for traditional enterprise apps & client-server compute Scale-up (pool-based resourcing) IT management-centric 1 administrator for 100 s of servers Proprietary vendor stack Designed around big data, massive scale & next-gen apps Scale-out (horizontal resourcing) Autonomic management 1 administrator for 1,000 s of servers Open, value-added stack

7 CloudStack Overview

8 What is Apache CloudStack? CloudStack is an open source Infrastructureas a-service (IaaS) orchestration platform that enables users to build, manage and deploy compute cloud environments. CloudStack was recently donated by Citrix to the Apache Software Foundation and is currently undergoing incubation.

9 Graphical User Interface CloudStack offers an administrator's Web interface, used for provisioning and managing the cloud, as well as an end-user's Web interface, used for running VMs and managing VM templates. The UI can be customized to reflect the desired service provider or enterprise look and feel.

10 CloudStack Web Services Query HTTP API is loosely based on the REST architecture and allows developers to create new management solutions or integrate existing systems with CloudStack. It supports output in both XML and JSON. EC2/S3 support (translation layer) is also present.

11 Benefits of CloudStack Capital Leverage Workforce Leverage Self Service Management Automation Workload Standardization Usage Metering Centralized Management Smarter Virtualization Remove IT as a service delivery cri1cal path Reduce IT opera1onal costs Consistent applica1on and service deployment Visibility into user and line of business usage Manage complete infrastructure, regardless of scale Drive reduced capital requirements

12 Create Custom Virtual Machines via Service Offerings

13 Dashboard Provides Overview of Consumed Resources Running, Stopped & Total VMs Public IPs Private networks Latest Events

14 Virtual Machine Management Users VM Operations Console Access VM Status Change Service Offering Start CPU U1lized 2 CPUs 4 CPUs Stop Network Read 1 GB RAM 4 GB RAM Restart Destroy Network Writes 20 GB 20 Mbps 200 GB 100 Mbps

15 Volume & Snapshot Management Add / Delete Volumes VM 1 Volume Create Templates from Volumes Volume Template Schedule Snapshots Now Hourly Daily Weekly Monthly View Snapshot History. 12/2/ am 2/2/ am

16 Network & Network Services Create Networks and attach VMs Acquire public IP address for NAT & load balancing Control traffic to VM using ingress and egress firewall rules Set up rules to load balance traffic between VMs

17 CloudStack Architecture

18 CloudStack Architecture Opera3onal Integra3on (OSS/BSS, Monitoring, Iden1ty Management, Etc) Integra1on API User Interface Developer API Administrator End User Console Amazon* Availability and Security Image Libraries snapshots LB HA Monitoring Applica1on Catalog Dynamic Workload Management Resource Management Servers Storage Network Custom Templates Opera1ng System ISOs Service Management (Metering, Accounts, etc.) Virtualization Layer Servers Network Storage

19 Cloud Infrastructure Overview - Summary One or more hosts grouped into a cluster One or more clusters grouped into a pod One or more pods grouped into a zone One or more zones controlled by one management server Management Server MySQL Cloud_db Secondary Storage

20 Components Hosts Servers onto which services will be provisioned Primary Storage VM disk storage Cluster A grouping of hosts and their associated storage Pod Collec1on of clusters in the same failure boundary Network Logical network associated with service offerings Secondary Storage Template, snapshot and ISO storage Zone Collec1on of pods, network offerings and secondary storage Management Server Farm Management and provisioning tasks Network Secondary Storage CloudStack Pod Zone Host Host Primary Storage Cluster Cluster VM VM CloudStack Pod

21 CloudStack Infrastructure - Overview CloudStack provides a number of infrastructure pieces, external to the management server that provide scalable services. Secondary Storage (SSVM) Console Proxy (CPVM) Virtual Router (VR or domr)

22 Secondary Storage Secondary Storage - provides storage for machine images and snapshots Secondary Storage VM - provides stateless and scalable management and interaction with Secondary Storage.

23 Console Proxy Hypervisors provide access to the console of a virtual machine generally via VNC. Accessing it requires direct access to the hypervisor - including credentials into the hypervisor directly. CPVM proxies access to the VNC session and provides access control so that others can t get access. Automatically scales to handle demand of console sessions. Provides an AJAX interface that is usable on virtually any device.

24 Virtual router Lowest common denominator (so far) is a virtual machine. Provides a number of services DHCP Routing DNS Loadbalancing Firewall NAT

25 CloudStack Scale Availability Zone 25 VMOps Pod VMOps Pod VMOps Pod CloudStack Pod CloudStack Pod CloudStack Pod CloudStack Pod

26 Availability Zones Deployed Globally CloudStack Cluster San Jose Austin Frankfurt Tokyo

27 Availability Zones Can be Private CloudStack Cluster San Jose Austin Frankfurt Private Delhi Tokyo Private Rio

28 Management Server Managing Multiple Zones Data Center 1 Zone1 Management Server Data Center 2 Data Center 2 Data Center 2 Zone 2 Zone 2 Zone 2Zone 3 Zone 3 Zone 3 Data Center 2 Data Center 2 Data Center 3 Zone 2 Zone 2 Zone 3 Zone Zone 4 3 Ø Single Management Server can manage mul1ple zones Ø Zones can be geographically distributed but low latency links are expected for beaer performance Ø Single MS node can manage up to 5K hosts. Ø Mul1ple MS nodes can be deployed as cluster for scale or redundancy

29 Multi-Site Deployment Availability Zone 1 Primary Management Server Data Center 1 Secondary Management Server Data Center 2 Availability Zone 4 Data Center 4 Availability Zone 2 Availability Zone 3 Data Center 3

30 Deployment Architectures

31 Deployment Architecture The architecture used in a deployment will vary depending on the size and purpose of the deployment. From a small-scale deployment useful for dev/test and PoC deployments To a fully-redundant large-scale setup for production deployments.

Replica3on Back Up DB Ø MS is stateless.

32 Management Server Deployment Architecture Single-node Deployment Multi-node Deployment User API User API Management Server Admin API Management Server MySQL DB Admin API Load Balancer Management Server MySQL DB Replica3on Back Up DB Ø MS is stateless. MS can be deployed as physical server or VM Infrastructure Resources Ø Single MS node can manage up to 8K hosts. Mul1ple nodes can be deployed for scale or redundancy Infrastructure Resources

33 Small-Scale Deployment Management Server Primary Storage Secondary Storage Public IP Internet Router & Firewall /24 Layer- 2 Switch NFS Server Compu1ng Nodes to

34 Large-Scale Redundant Deployment Internet Internet Layer-3 switches with firewall modules Layer-2 switches Computing Nodes Management Server Cluster Secondary Storage Servers NFS/Swii Secondary Storage NFS/Swii Secondary Storage Primary Storage Primary Storage Primary Storage Servers Primary Storage Primary Storage

35 The Three C s of Complexity Control Choice Compliance

36 Giving Control Brings Complexity End User Org A Admin Users Org B Admin Users Users ACL Limits Governance Admin Compute Network Storage VMware NetScaler Jun. SRX NFS Local Disk XenServer F5 iscsi SwiP KVM BareMetal FC HDFS

37 Guest Virtual Layer- 2 Network Guest 1 Virtual Network /24 Guest 1 VM 1 Guest Public IP Guest 1 Virtual Router Gateway Guest 1 VM 2 Guest Guest 1 VM 3 Guest Internet Guest 2 VM 1 Guest Public IP Guest 2 Virtual Router Gateway Guest 2 VM 2 Guest Guest 2 VM 3 Guest Guest 2 Virtual Network /24

38 Mul1-1er Network Virtual Network /24 VLAN 100 Virtual Network /24 VLAN 1001 Virtual Network /24 VLAN App VM Private IP Public IP Juniper SRX Firewall Public IP Private IP Netscaler Load Balancer Web VM Web VM Web VM App VM DB VM Web VM DHCP, DNS User- data Virtual Router DHCP, DNS, User- data Virtual Router DHCP, DNS User- data, Source- NAT, VPN Virtual Router Public IP

39 Unified Mul1-1er Network Internet Load Balancer Virtual Router IPSec or SSL site- to- site VPN Customer Premises Monitoring VLAN Virtual Router Services IPAM DNS LB [intra] S- 2- S VPN Sta1c Routes ACLs NAT, PF FW [ingress & egress] BGP Web VM 1 Web VM 2 Web VM App VM 1 App VM DB VM 1 Virtual Network /24 VLAN Web VM 4 Virtual Network /24 VLAN 1001 Virtual Network /24 VLAN 141

40 Other Topologies No services [Sta0c IPs] Dedicated VLAN with DHCP and DNS User can request specific IP[s] for NIC Guest Virtual Network /24 VLAN 100 Guest Virtual Network /24 VLAN 100 Gateway address Guest VM Guest VM Guest VM 2 Guest VM Gateway address Core switch Guest VM Guest VM Guest VM 4 Core switch Guest VM 4 Virtual Router DHCP, DNS User- data

41 Other Topologies MPLS Use Case Shared VLAN with DHCP and DNS Guest Virtual Network /24 VLAN 100 Guest Virtual Network /24 VLAN 100 MPLS VLAN 100 Gateway address Guest VM Guest VM Guest VM 2 Gateway address Guest VM 2 Core switch Guest VM Guest VM Guest VM 4 Core switch Guest VM 4 DHCP, DNS User- data CS Virtual Router CS Virtual DHCP, DNS Router User- data

42 Layer 3 Networking (Amazon Style) Web VM Web Security Group DB VM DB Security Group Web VM Web VM Web VM DB VM Web VM Web VM

43 User/API 8080 HTTP File Share CloudStack Mgmt. Server CloudStack Mgmt. Server 3306 MySQL (Master) 3306 MySQL (Slave) 80/ / / / Virtual Router Secondary Storage SSVM CPVM vcenter XenServer KVM Per Customer Per Availability Zone Per Pod / Cluster

44 Making it all scale

45 Thinking about cloud orchestration at scale Host management Capacity management What host to use to deploy a new VM Failure handling Security group propaga3on Set a goal

46 CPU utilization while deploying 30,000 VMs on 30,000 hosts CPU Utilization. 400% is maximum 20, Idl e Time

47 Deploy time from 25,000 to 30,000 VMs Seconds to deploy VM number:

48 Storage at scale Storage is cluster specific (typically 8-16 nodes) Scaling out with SAN typically doesn t do a good job - some new gen stuff helps, but still a daunting problem to keep up IO when you get to thousands of nodes. Distributed filesystems - they are better...but... Local storage - failure prone, but cheap, and scales easily with the number of nodes.

Xen and CloudStack. Ewan Mellor. Director, Engineering, Open-source Cloud Platforms Citrix Systems

Xen and CloudStack. Ewan Mellor. Director, Engineering, Open-source Cloud Platforms Citrix Systems Xen and CloudStack Ewan Mellor Director, Engineering, Open-source Cloud Platforms Citrix Systems Agenda What is CloudStack? Move to the Apache Foundation CloudStack architecture on Xen The future for CloudStack