Power Control in Virtualized Data Centers

Transcription

1 Power Control in Virtualized Data Centers Jie Liu Microsoft Research Joint work with Aman Kansal and Suman Nath (MSR) Interns: Arka Bhattacharya, Harold Lim, Sriram Govindan, Alan Raytman December 2011, LCCC Workshop

2 Data Centers

3 Energy Expenditure of Computing The IT industry is on fire constitutes about 2% of total US energy consumption consumed 61 Billion kwh in 2006, enough to power 5.8 Million average US households is the fastest growing energy consuming industrial sector used twice the amount of energy in 2006 compared to 2001 expected to double again from 2006 to 2011

4 Provisioning Cost 50,000 server facility 19% 10% 18% 2% 9% 42% Land Core Electrical Mech Other Infrastructure Cost Breakdown Architectural Data courtesy James Hamilton and Mike Manos

5 Industry Responded Low power/efficient hardware Less AC Over-subscription Virtualization Renewable energy sources Supply and demand matching Play the price game

6 Power Consumption (Watts) Login rate (per second) Number of connections (millions) Data Center Dynamics Intel(R) 2CPU 2.4GHz Intel(R) 2CPU 3GHz Continuous Dynamics Workload changes reflected in power consumption fluctuation Weekly Messenger traffic on 60 servers Login Rate Connections Time in hours Sleep Idle 20% 40$ 60% 80% 100% CPU Utilization Typical Server Power Consumption Idle power takes a large percentage of total power consumption. Almost linear to its utilization once used. A MSGR server power consumption over a week Discrete Dynamics Service deployment and upgrades Machine crashes VM migrations

7 Data Center Power Over-subscription Savings from Power Capping Time reduce provisioning cost Rated peak (never reached) Possible peak (sum of server peaks) Allocated Capacity Actual power consumption (peak of the sum usually lower than allocated, but can exceed)

8 Power (Watt) Power Management Scenarios Power Capping Lower allocated capacity => lower provisioning cost (Slight perf hit) Possible because power can be capped if exceeds [Lefurgy et al. 2003, Femal et. al 2005, Urgaonkar et al. 2009, Wang et al. 2010] Power Tracking Utility power price changes Battery charging after outage Lead Acid Battery Charging Curve Time (hours)

9 Virtualization Improve server utilization Amortize idle power consumption However, hardware capping methods fall short Servers shared by VMs from different applications Not all apps/tiers are created equal Throttling a physical server affects performance of all apps on it VM VM VM VM VM VM Server-11 Server-12 Server-1j Rack

10 Power Management Challenges in Virtualized Data Centers Virtualization mixes physical and logical structures Power meters can only measure physical servers Control is better applied to applications Each VM affects the performance of a multi-tier application differently, Observability Controllability within an app and across apps Software power management may be too slow Monolithic modeling and control are not scalable Uncoordinated controllers may counter act Response Time Scalability Stability

11 Can Software Act Fast Enough? X-PDU REMOTE POWER PANEL RACK PDU SERVER UPS Magntic Breaker Thermal Breaker Circuit Break Architecture

12 Circuit Breaker (Slack) Tolerance of a Siemens 400A LCD circuit breaker against sustained current For 400A, the tolerance is infinite. For 2 x 400A, the tolerance is 20s

13 servers Worst-Case Power Rise in Servers Power rise in an Intel Xeon L5520 Server Fastest observed power rise : 100ms Power rise in an Intel Xeon L5640 Server Fastest observed power rise: 200ms Server spikes can be correlated. time Time

14 Time Line of Events Central controller gives actuation command Settings reflected by OS Command received by agent < 1ms ~20ms ms < 1ms <40-60ms in current implementation (using user-level code) time (not to scale) Command reaches destination server OS changes setting in hardware Power decreases E2E response latency: ~400ms Typical adjustable power/core: 10~20 W Distributed Decision Coordination & Control

15 Watts Watts Watts VM Joule Meter Estimate VM power consumption from performance counters Linear regression with whole machine (HW) power meters Component Dynamic Energy Performance counters CPU Memory Disk Power consumption Energy Model Error Measured Estimate Error Time(s) Time(s) 0 0 Component Dynamic Energy (Cumulative) 25 Application Dynamic Energy 2500

16 VM Performance Accounting CPU DISK

17 Benchmark Number Average Errors: SPEC CPU Error (%) Platform: HP DLG380 8-core Xeon, 16GB RAM Benchmark numbers 2 to 19 are SPEC CPU 2006 INT and FP benchmarks (ones that compile without Fortran). 8 copies of each benchmark ran to keep every core used. Benchmark #1 and 20 are a synthetic loads. Error is in line with errors reported in hardware meters

18 Training with Actual Workload Reduces error by more than 50% Implies continuous learning on line.

19 Performance (TPS) Actuator Two knobs: DVFS and CPU time cap Different marks are different DVFS levels, multiple marks correspond to different CPU time caps Perf gap at same power DVFS = 100 DVFS = 94 DVFS = 88 DVFS = 82 DVFS = 76 DVFS = 70 Power (Watt)

20 Hierarchical Control Framework PI + Weighted Fair Sharing Papp-1(t) PID to Tier 1 only Application Level Controller 1 P T (t) Data Center Controller Papp-n(t) Application Level Controller n Ptier-1(t) Ptier-n(t) Ptier-1(t) Ptier-n(t) Tier Level Controller 1 MPC Tier Level Controller n Tier Level Controller 1 Tier Level Controller n VM VM VM VM VM VM VM VM

21 Power (Watt) Workload (%) Experimental Results 40 VMs on 10 servers 3 Priorities (stock trader, web service, SPEC CPU) Take battery charging as complimentary power consumer MSN Messenger demand traces Time (s) Total Power Time (s) Uncapped MPC Controller Physical Hierarchy Controller Total Power Budget

22 The Complexity of VM Migration

23 VM Interference Co-located applications Processor Memory Bandwidth Memory App 1 App 2 Shared Cache DRAM Core-private cache Shared resource contention Static partitioning

24 Normalized Performance Degradation (%) Memory subsystem Interference Up to 125% degradation in Core 2 Duo, Intel Nehalem and AMD Opteron Quad Core processors Up to 40% degradation was observed among Google applications* lbm mcf bzip2 povray Vs lbm Vs mcf Vs bzip2 Vs povray Co-located Application on Intel Core 2 Duo *The impact of memory subsystem resource sharing on datacenter applications, Tang et al., ISCA 2011

25 lbm gcc mcf soplex omnetpp bzip2 gobmk povray perlbench libquant hmmer sjeng Application Perf. Degr. (%) scl Factor Perf. Degr. (Normalized to default) Interference Quantification and Prediction Cache Pressure modeling Tunable Cache Intensity lbm 8192 sets, 16 ways Cache Sets Performance: Bytes accessed per second Cache Ways 1 0 User equivalent cache load for prediction scl (sets, ways) LBM (8192, 16) Appln X Measured (Vs lbm) Predicted (Vs scl(8192, 16)) VM VM Core 1 Core 2 Shared Cache 10 0

26 Interference-Aware VM Assignment and Migration Given n jobs and m machines each with k cores Job degradation is specified over all job sets Given D, such that feasible degradation should be less than D Ever set of jobs has (energy) cost w(s) 1. Find a partition of jobs into b < m machines and minimize E = b i=1 w(s i NP-hard when k > 2 Polynomial time approximation ) E ALG H k E OPT H k ln (k) 2. Given an existing assignment and G allowable migrations, minimize the total cost of the new assignment after migration. NP-hard when k > 2 NP-hard to approximate

27 Conclusion Data centers are operating close to their designed capacities. Software-based power control is feasible Many practical concerns Need to be done carefully VM assignment and migration bring significant challenges at the discrete level.