How to Tame your VM: an Automated Control System for Virtualized Services

Size: px

Start display at page:

Download "How to Tame your VM: an Automated Control System for Virtualized Services"

Cori Stewart
5 years ago
Views:

1 How to Tame your : an Automated Control System for Virtualized Services Akkarit Sangpetch Andrew Turner Hyong Kim asangpet@andrew.cmu.edu andrewtu@andrew.cmu.edu kim@ece.cmu.edu Department of Electrical and Computer Engineering Carnegie Mellon University LISA 21 - November 11, 21

2 Virtualized Services Web1 App1 DB1 App2 DB2 Web2 Db3 Web3 App3 Hypervisor Hypervisor Hypervisor Virtualized Infrastructure (vsphere,hyper-v,esx,k) Easy to deploy Easy to migrate Easy to re-use virtual machines Multi-tier services are configured and deployed as virtual machines 2

3 Problem: What about performance? Web1 App1 DB1 App2 DB2 Web2 Db3 Web3 App3 Hypervisor Hypervisor Hypervisor s share physical resource (CPU time / Disk / Network / Memory bandwidth) Need to maintain service quality Response time / Latency Existing infrastructure provides mismatch interface between the service s performance goal (latency) and configurable parameters (resource shares) Admins need to fully understand the services before able to tune the system 3

Existing Solutions Managing Service Performance Resource Provisioning Admins define resource shares / reservations / limits The correct values depends on hardware / applications / workloads

4 Existing Solutions Managing Service Performance Resource Provisioning Admins define resource shares / reservations / limits The correct values depends on hardware / applications / workloads Load Balancing Migrate s to balance host utilization Free up capacity for each host (low utilization ~ better performance) vm1 # Shares 5 vm1 vm3 vm2 vm4 vm2 1 Host 1 vm3 1 vm5 vm4 vm4 3 Host 2 4

Automated Resource Control vm1 vm2 vm3 vm4 vm1 vm2 vm3 1 1 25 25 25 25 Control system 5 vm1 response vm1 target vm2 target vm2 response vm3 target vm3 response vm4 target vm4 response vm1 target vm1

5 Automated Resource Control vm1 vm2 vm3 vm4 vm1 vm2 vm Control system 5 vm1 response vm1 target vm2 target vm2 response vm3 target vm3 response vm4 target vm4 response vm1 target vm1 response vm2 target vm2 response vm3 target vm3 response Dynamically adjust resource shares during run-time Input = required service response time I want to serve pages from vm1 in less than 4 seconds The system calculates number of shares required to maintain service level Adapt to changes in workload Admins do not have to guess the number of shares vm4 3 vm4 target vm4 response 5

6 Control System Calculate model parameters Record service response time Modeler Sensor Actuator K Host Sensor Actuator K Host Sensor Actuator K Host Adjust host CPU shares Controller Calculate # of shares required 6

Sensor unit Server Sensor analyzer tshark To Modeler {service:blog, tier:web, vm: www1, response time 25 ms}, {service:blog, tier:db, vm:db1, response time 2.

7 Sensor unit Server Sensor analyzer tshark To Modeler {service:blog, tier:web, vm: www1, response time 25 ms}, {service:blog, tier:db, vm:db1, response time 2.5 ms} Request Packets Host Bridge Request Packets xtables TEE Request Packets Record service response time Starts when the server sees request packet Until the server sends response packet Based on libpcap / tshark need to recognize the application s header 7

8 Actuator Unit Shares Allocation {server1:8, server2:2 } Server 1 Server Actuator Agent Hypervisor 8 CPU Cgroup Adjust number of shares based on controller s allocation We use Linux/K as our hypervisor Cgroup cpu.shares control CFS scheduling shares Similar mechanism exists on other platform (Xen weight / ESX shares) 8

9 Modeler Client T web Web server T DB Database S cpu The modeler calculates the expected service response time based on shares allocated to each We use an intuitive model for our 2-tier services in the experiment T web = f (T DB ) T DB = f (S cpu ) 9

10 Resource Shares Effect Test Clients Web server (Wordpress) SQL Server (mysql) Cruncher (1% cpu) Host 1 Host 2 NAS 1

11 Web Response Time (ms) # of DB CPU shares (out of 1) Daabase Response Time (ms) # of DB CPU shares (out of 1) SQL Server CPU Allocation effect Web Response Database Response Time step (5s) Time step (5s) thttp-msec cpu tdb-msec cpu 11

12 Database Response Time (ms) Database Response vs CPU # of CPU Shares allocated to SQL (out of 1) T DB = a (S cpu ) b 12

13 Database Response Time (ms) Database Response vs CPU (controlled) # of CPU Shares allocated to SQL server (out of 1) 13

14 Web Response (ms) HTTP vs Database response Database Response (ms) T web = a 1 (T DB ) + b 1 14

15 Controllers Model uses readings to estimate the service parameters <a,b,a 1,b 1 > T DB = a (S cpu ) b T web = a 1 (T DB ) + b 1 Controller finds the minimal S cpu such that T web < specified response time Long-term control: uses moving average to find model parameters & shares Short-term control: uses the last-period reading to find model parameters Avoid excessive response time violation while waiting for the long-term model to settle 15

16 System Evaluations Test Clients Web server 1 Web server 2 DB Server 1 DB server 2 Sensor Host 1 Sensor Host 2 Actuator Sensor Host 3 NAS 16

17 Service Response Time (ms) Service Response Time (ms) Static Workload web1 web1-target web2 web2-target web1 web1-target web2 web2-target 17

18 Service Response Time (ms) Service Response Time (ms) Dynamic Workload Web1 Load web1 web1-target web2 web2-target 5 per. Mov. Avg. (web1) 5 per. Mov. Avg. (web2) web1 web1-target web2 web2-target 5 per. Mov. Avg. (web1) 5 per. Mov. Avg. (web2) 18

19 Deviation from Target Static Load Without Control Mean Deviation (ms) Target Violation Period With Control Mean Deviation (ms) Target Violation Period Instance 1 54 % 19 (-8%) 9% Instance % 282 (-73%) 26% Dynamic Load Instance % 182 (-34%) 9% Instance % 336 (-5%) 12% Our control system helps track the target service response time 19

20 Further Improvements Enhance Service Model Service dependencies Cache / Proxy / Load balancer effects Non-cpu resource (disk / network / memory) Robust Control system Still susceptible to temporal system effects (garbage collections / cron jobs) Need to determine feasibility of the target performance 2

21 Conclusions Automated Resource Control system Maintain target system s response time Allows admin to express allocation in term of performance target Managing service performance could be automate with sufficient domain knowledge Need basic framework to describe performance model Leave the details to the machines (parameter fitting / optimization / resource monitoring) 21

22 Sample Response & Share values tweb p95 target shares share 22

Virtualization and the Metrics of Performance & Capacity Management

23 S September t b 2011 Virtualization and the Metrics of Performance & Capacity Management Has the world changed? Mark Preston Agenda Reality Check. General Observations Traditional metrics for a non-virtual