Building a Big IaaS Cloud David Nalley @ke4qqq ke4qqq@apache.org / david@gnsa.us
#whoami Recovering Sysadmin F/LOSS contributor Committer on Apache CloudStack
Assumptions You have a need for an IaaS compute cloud platform You know what IaaS and cloud mean
Massively scalable Scalable - this is the easy part Massively - this part is much harder - getting to thousands of physical hosts is complex - getting to tens of thousands of physical hosts is a completely different magnitude of problem.
So I have some questions
Virtualization alone does not make a cloud Server Virtualiza0on Cloud Built for traditional enterprise apps & client-server compute Scale-up (pool-based resourcing) IT management-centric 1 administrator for 100 s of servers Proprietary vendor stack Designed around big data, massive scale & next-gen apps Scale-out (horizontal resourcing) Autonomic management 1 administrator for 1,000 s of servers Open, value-added stack
CloudStack Overview
What is Apache CloudStack? CloudStack is an open source Infrastructureas a-service (IaaS) orchestration platform that enables users to build, manage and deploy compute cloud environments. CloudStack was recently donated by Citrix to the Apache Software Foundation and is currently undergoing incubation.
Graphical User Interface CloudStack offers an administrator's Web interface, used for provisioning and managing the cloud, as well as an end-user's Web interface, used for running VMs and managing VM templates. The UI can be customized to reflect the desired service provider or enterprise look and feel.
CloudStack Web Services Query HTTP API is loosely based on the REST architecture and allows developers to create new management solutions or integrate existing systems with CloudStack. It supports output in both XML and JSON. EC2/S3 support (translation layer) is also present.
Benefits of CloudStack Capital Leverage Workforce Leverage Self Service Management Automation Workload Standardization Usage Metering Centralized Management Smarter Virtualization Remove IT as a service delivery cri1cal path Reduce IT opera1onal costs Consistent applica1on and service deployment Visibility into user and line of business usage Manage complete infrastructure, regardless of scale Drive reduced capital requirements
Create Custom Virtual Machines via Service Offerings
Dashboard Provides Overview of Consumed Resources Running, Stopped & Total VMs Public IPs Private networks Latest Events
Virtual Machine Management Users VM Operations Console Access VM Status Change Service Offering Start CPU U1lized 2 CPUs 4 CPUs Stop Network Read 1 GB RAM 4 GB RAM Restart Destroy Network Writes 20 GB 20 Mbps 200 GB 100 Mbps
Volume & Snapshot Management Add / Delete Volumes VM 1 Volume Create Templates from Volumes Volume Template Schedule Snapshots Now Hourly Daily Weekly Monthly View Snapshot History. 12/2/2012 7.30 am 2/2/2012 7.30 am
Network & Network Services Create Networks and attach VMs Acquire public IP address for NAT & load balancing Control traffic to VM using ingress and egress firewall rules Set up rules to load balance traffic between VMs
CloudStack Architecture
CloudStack Architecture Opera3onal Integra3on (OSS/BSS, Monitoring, Iden1ty Management, Etc) Integra1on API User Interface Developer API Administrator End User Console Amazon* Availability and Security Image Libraries snapshots LB HA Monitoring Applica1on Catalog Dynamic Workload Management Resource Management Servers Storage Network Custom Templates Opera1ng System ISOs Service Management (Metering, Accounts, etc.) Virtualization Layer Servers Network Storage
Cloud Infrastructure Overview - Summary One or more hosts grouped into a cluster One or more clusters grouped into a pod One or more pods grouped into a zone One or more zones controlled by one management server Management Server MySQL Cloud_db Secondary Storage
Components Hosts Servers onto which services will be provisioned Primary Storage VM disk storage Cluster A grouping of hosts and their associated storage Pod Collec1on of clusters in the same failure boundary Network Logical network associated with service offerings Secondary Storage Template, snapshot and ISO storage Zone Collec1on of pods, network offerings and secondary storage Management Server Farm Management and provisioning tasks Network Secondary Storage CloudStack Pod Zone Host Host Primary Storage Cluster Cluster VM VM CloudStack Pod
CloudStack Infrastructure - Overview CloudStack provides a number of infrastructure pieces, external to the management server that provide scalable services. Secondary Storage (SSVM) Console Proxy (CPVM) Virtual Router (VR or domr)
Secondary Storage Secondary Storage - provides storage for machine images and snapshots Secondary Storage VM - provides stateless and scalable management and interaction with Secondary Storage.
Console Proxy Hypervisors provide access to the console of a virtual machine generally via VNC. Accessing it requires direct access to the hypervisor - including credentials into the hypervisor directly. CPVM proxies access to the VNC session and provides access control so that others can t get access. Automatically scales to handle demand of console sessions. Provides an AJAX interface that is usable on virtually any device.
Virtual router Lowest common denominator (so far) is a virtual machine. Provides a number of services DHCP Routing DNS Loadbalancing Firewall NAT
CloudStack Scale Availability Zone 25 VMOps Pod VMOps Pod VMOps Pod CloudStack Pod CloudStack Pod CloudStack Pod CloudStack Pod
Availability Zones Deployed Globally CloudStack Cluster San Jose Austin Frankfurt Tokyo
Availability Zones Can be Private CloudStack Cluster San Jose Austin Frankfurt Private Delhi Tokyo Private Rio
Management Server Managing Multiple Zones Data Center 1 Zone1 Management Server Data Center 2 Data Center 2 Data Center 2 Zone 2 Zone 2 Zone 2Zone 3 Zone 3 Zone 3 Data Center 2 Data Center 2 Data Center 3 Zone 2 Zone 2 Zone 3 Zone Zone 4 3 Ø Single Management Server can manage mul1ple zones Ø Zones can be geographically distributed but low latency links are expected for beaer performance Ø Single MS node can manage up to 5K hosts. Ø Mul1ple MS nodes can be deployed as cluster for scale or redundancy
Multi-Site Deployment Availability Zone 1 Primary Management Server Data Center 1 Secondary Management Server Data Center 2 Availability Zone 4 Data Center 4 Availability Zone 2 Availability Zone 3 Data Center 3
Deployment Architectures
Deployment Architecture The architecture used in a deployment will vary depending on the size and purpose of the deployment. From a small-scale deployment useful for dev/test and PoC deployments To a fully-redundant large-scale setup for production deployments.
Management Server Deployment Architecture Single-node Deployment Multi-node Deployment User API User API Management Server Admin API Management Server MySQL DB Admin API Load Balancer Management Server MySQL DB Replica3on Back Up DB Ø MS is stateless. MS can be deployed as physical server or VM Infrastructure Resources Ø Single MS node can manage up to 8K hosts. Mul1ple nodes can be deployed for scale or redundancy Infrastructure Resources
Small-Scale Deployment 192.168.10.3 192.168.10.4 Management Server Primary Storage Secondary Storage Public IP 62.43.51.125 Internet Router & Firewall 192.168.10.0/24 Layer- 2 Switch NFS Server Compu1ng Nodes 192.168.10.10 to 192.168.10.13
Large-Scale Redundant Deployment Internet Internet Layer-3 switches with firewall modules Layer-2 switches Computing Nodes Management Server Cluster Secondary Storage Servers NFS/Swii Secondary Storage NFS/Swii Secondary Storage Primary Storage Primary Storage Primary Storage Servers Primary Storage Primary Storage
The Three C s of Complexity Control Choice Compliance
Giving Control Brings Complexity End User Org A Admin Users Org B Admin Users Users ACL Limits Governance Admin Compute Network Storage VMware NetScaler Jun. SRX NFS Local Disk XenServer F5 iscsi SwiP KVM BareMetal FC HDFS
Guest Virtual Layer- 2 Network Guest 1 Virtual Network 10.1.1.0/24 Guest 1 VM 1 Guest 10.1.1.2 Public IP 65.37.141.11 65.37.141.36 Guest 1 Virtual Router Gateway 10.1.1.1 Guest 1 VM 2 Guest 10.1.1.3 Guest 1 VM 3 Guest 10.1.1.4 Internet Guest 2 VM 1 Guest 10.1.1.2 Public IP 65.37.141.24 65.37.141.80 Guest 2 Virtual Router Gateway 10.1.1.1 Guest 2 VM 2 Guest 10.1.1.3 Guest 2 VM 3 Guest 10.1.1.4 Guest 2 Virtual Network 10.1.1.0/24
Mul1-1er Network Virtual Network 10.1.1.0/24 VLAN 100 Virtual Network 10.1.2.0/24 VLAN 1001 Virtual Network 10.1.3.0/24 VLAN 141 10.1.2.31 App VM 1 10.1.3.21 Private IP Public IP 10.1.1.111 65.37.141.111 Juniper SRX Firewall Public IP 65.37.141.11 2 Private IP 10.1.1.112 Netscaler Load Balancer 10.1.1.1 Web VM 1 10.1.1.3 Web VM 2 10.1.1.4 Web VM 3 10.1.2.21 10.1.2.18 10.1.2.38 10.1.2.24 App VM 2 10.1.3.45 10.1.3.24 DB VM 1 10.1.1.5 Web VM 4 10.1.2.39 DHCP, DNS User- data Virtual Router DHCP, DNS, User- data Virtual Router DHCP, DNS User- data, Source- NAT, VPN Virtual Router Public IP 65.37.141.115
Unified Mul1-1er Network Internet Load Balancer Virtual Router IPSec or SSL site- to- site VPN Customer Premises Monitoring VLAN Virtual Router Services IPAM DNS LB [intra] S- 2- S VPN Sta1c Routes ACLs NAT, PF FW [ingress & egress] BGP 10.1.1.1 10.1.1.3 10.1.1.4 Web VM 1 Web VM 2 Web VM 3 10.1.2.31 10.1.2.24 App VM 1 App VM 2 10.1.3.24 DB VM 1 Virtual Network 10.1.1.0/24 VLAN 100 10.1.1.5 Web VM 4 Virtual Network 10.1.2.0/24 VLAN 1001 Virtual Network 10.1.3.0/24 VLAN 141
Other Topologies No services [Sta0c IPs] Dedicated VLAN with DHCP and DNS User can request specific IP[s] for NIC Guest Virtual Network 10.1.1.0/24 VLAN 100 Guest Virtual Network 10.1.1.0/24 VLAN 100 Gateway address 10.1.1.1 10.1.1.1 Guest VM 1 10.1.1.1 Guest VM 1 10.1.1.3 Guest VM 2 Guest VM Gateway 10.1.1.3 address 2 10.1.1.1 Core switch 10.1.1.4 Guest VM 3 10.1.1.4 Guest VM 3 10.1.1.5 Guest VM 4 Core switch 10.1.1.5 Guest VM 4 Virtual Router DHCP, DNS User- data
Other Topologies MPLS Use Case Shared VLAN with DHCP and DNS Guest Virtual Network 10.1.1.0/24 VLAN 100 Guest Virtual Network 10.1.1.0/24 VLAN 100 MPLS VLAN 100 Gateway address 10.1.1.1 10.1.1.100 Guest VM 1 10.1.1.1 Guest VM 1 10.1.1.200 Guest VM 2 Gateway 10.1.1.3 address 10.1.1.1 Guest VM 2 Core switch 10.1.1.101 Guest VM 3 10.1.1.4 Guest VM 3 10.1.1.115 Guest VM 4 Core switch 10.1.1.5 Guest VM 4 DHCP, DNS User- data CS Virtual Router CS Virtual DHCP, DNS Router User- data
Layer 3 Networking (Amazon Style) Web VM Web Security Group DB VM DB Security Group Web VM Web VM Web VM DB VM Web VM Web VM
User/API 8080 HTTP File Share CloudStack Mgmt. Server 9090 8250 CloudStack Mgmt. Server 3306 MySQL (Master) 3306 MySQL (Slave) 80/443 8250 3922 111/2049 111/2049 443 22/443 22 Virtual Router Secondary Storage SSVM CPVM vcenter XenServer KVM Per Customer Per Availability Zone Per Pod / Cluster
Making it all scale
Thinking about cloud orchestration at scale Host management Capacity management What host to use to deploy a new VM Failure handling Security group propaga3on Set a goal
CPU utilization while deploying 30,000 VMs on 30,000 hosts CPU Utilization. 400% is maximum 20,0 00 500 0 500 0 Idl e Time
Deploy time from 25,000 to 30,000 VMs Seconds to deploy VM number:
Storage at scale Storage is cluster specific (typically 8-16 nodes) Scaling out with SAN typically doesn t do a good job - some new gen stuff helps, but still a daunting problem to keep up IO when you get to thousands of nodes. Distributed filesystems - they are better...but... Local storage - failure prone, but cheap, and scales easily with the number of nodes.