Leveraging Energy Storage Towards Cost Effective Datacenters Di Wang

Transcription

1 Leveraging Energy Storage Towards Cost Effective Datacenters Di Wang Microsoft Research

2 Datacenters growing in numbers Microsoft has more than 100 DCs worldwide providing 200+ online services Datacenters have become as vital to the functioning of society as power stations. The Economist 2

3 size, 3

4 Billion kwh/year, and power consumption! Year If treated as a country, 5 th in the world for electricity use. US datacenter energy consumption increased from 28.2 billion kwh/year in 2000 to 91 billion kwh/year in By 2020, need over 50 power plants, imposing a peak load of over 20 GW on the grid. 4

5 Where do the $$$ go? Data centers can cost between $10M and $20M per Megawatt >80% of costs scale with power <20% of costs scale with space Server costs are trending town Power costs are trending up Total Cost of Ownership of a 1U Server 23% 12% 46% Total Equipment cost per server Energy Usage cost per server Datacenter Capital cost per server 19% Datacenter operating cost per server Server Lifetime 5 years Infrastructure Lifetime 15 years 5

6 Microsoft s Chicago Data Center (~200,000 servers) 700,000+ square feet 26,000 cubic yards of concrete 3400 tons of steel 2400 tons of copper 190 miles of conduit 7.5 miles of chilled water piping Capital investment over half a billion dollars! 100+ MW Power Capacity 60 MW Total Critical Power

7 What Is in a (Mega) Data Center? Internet gasoline power Power grid Generators PUE= Facility Energy Consumption Server Energy Consumption PDU Transformer UPS Racks air water CRAC Water chillers

8 Datacenter Raised Floor Cooling System 8

9 Microsoft s Data Center Evolution Generation 1 Generation 2 Generation 3 Generation 4 ~2 PUE PUE PUE PUE Colocation Density Containment Modular Server Capacity 20 year Technology Rack Density and Deployment Minimized Resource Impact Containers, PODs Scalability & Sustainability Air & Water Economization Differentiated SLAs ITPACs & Colos Reduced Carbon, Rightsized Faster Time to Market Outside Air Cooled Containment: tightly guide air flow Use outside air: locate in cooler region Operate servers hotter 9

10 Where do the $$$ go? Improve PUE 23% Cap-ex 12% Op-ex 19% Energy Cost 46% Equip. Cost 10

11 Power draw (W) Where do the $$$ go? Peak power draw 23% Cap-ex 12% Op-ex 19% Energy Cost 46% Equip. Cost Diesel Generator Power Distribution Unit (PDU) Utility Substation How to reduce peak draws? UPS Power Infrastructure Auto Transfer Switch (ATS) $10-20/watt Energy consumption (area under this curve) Server Racks Month 11

12 Power Demand Response to Lower Costs Savings in Peak Cost How to realize demand response in a datacenter? Re-shaped Power Time Cap-ex Saving Power Cap 12

13 Demand Response Knobs in a Datacenter Temporal Knobs (Load scheduling or deferral, e.g., DVFS) 1$/W 0.6$/W Utility Substation Diesel Generator UPS Auto Transfer Switch (ATS) 0.3$/W Power Distribution Unit (PDU) 0.2$/W Server Racks Power Cap 13

14 Demand Response Knobs in a Datacenter Spatial Knobs (request redirection, load migration) Power Cap Power Cap + 14

15 Power (W) Demand Response Knobs in a Datacenter New draw Power Cap How to provision and harness ESDs in datacenters? Time Energy Storage Device (ESD) (No Performance Impact) 15

16 ESDs in Current Datacenters Cost Saving Why restrict ESDs to any one level of the datacenter power hierarchy (e.g., central or server)? 1$/W Cost Saving 0.6$/W 0.3$/W Why restrict to single ESD technology (e.g., Lead acid battery)? 0.2$/W 16

17 Datacenter Power Demand: Temporal Statistics ACF with different time lags Zoomed-in view Key Insight Auto-correlation analysis confirms time-of-day behavior Di Wang, Chuangang Ren, Sriram Govindan, Anand Sivasubramaniam, Bhuvan Urgaonkar, Aman Kansal and Kushagra Vaid, "ACE: Abstracting, Characterizing and Exploiting Datacenter Power Demands", IISWC 13 17

18 Datacenter Power Demand: Spatial Statistics Cluster Server Key Insight Statistical multiplexing effect smoothen power fluctuations at higher levels Re-affirms prior observations [Fan et.al. ISCA 07] Di Wang, Chuangang Ren, Sriram Govindan, Anand Sivasubramaniam, Bhuvan Urgaonkar, Aman Kansal and Kushagra Vaid, "ACE: Abstracting, Characterizing and Exploiting Datacenter Power Demands", IISWC 13 18

19 Power Power Power Which ESDs to choose for peak shaving? Energy Time Energy Time Energy Time 19

20 power Which ESDs to choose for peak shaving? Time 20

21 power power Which ESDs to choose for peak shaving? Time Time 21

22 Specific Energy (Wh/kg) Ragone Plot 10,000 1, Compressed Air (CAES) Fuel Cell Batteries Lead acid Lithium ion Flywheels (FW) Ultracapacitors Supercapacitors (UC) Capacitors ,000 10, ,000 1,000,000 Specific Power (W/kg) 22

23 Power Hybrid ESD solution may be desirable Compressed Air Battery Ultracapacitor/flywheel Time 23

24 Multi-level Multi-technology ESDs Diesel Generator Utility Battery ATS ESD Flywheel Compressed Air PDU PDU PDU ESD ESD ESD Battery Capacitor Rack Rack Rack ESD Server H/W 24

25 Savings ($/day) Cost savings for MSN Workloads (Savings, ESD cost) 5,000 4,000 3,000 Rack: LA Server: Datacenter: LA LA Rack: UC + LA Datacenter: Server: FW+CAES UC + LA Server: UC (4.3k, 0.3k) (4.4k, 0.3k) (3.8k, 0.3k) (4.0k, 0.5k) (4.2k, 0.2k) (3.4k, 0.1k) 2,000 1,000 0 Single-tech, Single-level Multi-tech, Single-level Total cost without ESD is $15k/day Di Wang, Chuangang Ren, Anand Sivasubramaniam, Bhuvan Urgaonkar, and Hosam K. Fathy, "Energy Storage in Datacenters: What, Where and How Much?", SIGMETRICS 12. Multi-tech, Multi-level 25

26 Where do the $$$ go? Backup Total Cost of Ownership of a 1U Server Conversion 12% Op-ex Distribution Quality Reduce infrastructure Improve PUE 23% Cap-ex 19% Energy Cost 46% Equip. Cost Reduce Peak Power 26

27 Backup Power Infrastructure Problem 1.0$/W Backup Power Infrastructure 0.6$/W 0.2$/Wh 27

28 Backup Power Infrastructure Problem 1.0$/W Accounts for over 20% of power Cap-ex Backup Power Infrastructure 0.6$/W 0.2$/Wh 28

29 Backup Power Infrastructure Problem 1.0$/W Accounts for over 20% of power Cap-ex 0.6$/W 0.2$/Wh Backup Power Infrastructure Power Outage Frequency per Year 40% 30% 20% 10% 0% 40% 30% 20% 10% 0% 17% 40% 30% 13% None 1 to 2 3 to 6 7+ Power Outage Duration 31% < 1 minute 27% 14% 17% 1 to 5 5 to to 120 6% 5% 120 to 240 > 240 minutes Source: [1] U.S. business self-reports of annual outages, [2] National Survey of Datacenter Outages,

30 Backup Power Infrastructure Problem 1.0$/W Accounts for over 20% of power Cap-ex 0.6$/W 0.2$/Wh Backup Power Infrastructure Power Outage Frequency per Year 40% 40% 30% 30% 87% 20% experience 17% <= 6 outages 13% 10% 0% 40% 30% None 1 to 2 3 to % 27% 20% 17% Over 70% outages 14% <= 30 mins 10% 6% 5% 0% Power Outage Duration < 1 minute 1 to 5 5 to to to 240 > 240 minutes Source: [1] U.S. business self-reports of annual outages, [2] National Survey of Datacenter Outages,

31 Backup Power Infrastructure Problem 1.0$/W 0.6$/W 0.2$/Wh Power Outage Frequency per Year 40% 40% 30% 30% 87% 20% experience 17% <= 6 outages 13% 10% infrastructure? 0% None 1 to 2 3 to 6 7+ Can we under-provision backup power Accounts for over 20% of power Cap-ex Backup Power Infrastructure 40% 30% 31% 27% 20% 17% Over 70% outages 14% <= 30 mins 10% 6% 5% 0% Power Outage Duration < 1 minute 1 to 5 5 to to to 240 > 240 minutes Source: [1] U.S. business self-reports of annual outages, [2] National Survey of Datacenter Outages,

32 Power Backup Capacity: Power Capacity + Energy Capacity Normal Operation During Power Outage Utility Power Resumed Power capacity needed Energy capacity needed Outage Starts Time Outage Ends 32

33 Backup Infrastructure Design Space Power Capacity Energy Capacity 1.0$/W Power Capacity 0.6$/W Energy Capacity 0.2$/Wh 33

34 Backup Infrastructure Design Space MaxPerf DGpower =100% MinCost No DG No UPS UPSpower =100% UPSruntime=2min 34

35 Backup Infrastructure Design Space MaxPerf DGpower =100% UPSpower =100%, UPSruntime=2min MinCost No DG No UPS Varying DG power capacity Varying UPS power capacity Varying UPS energy capacity Combinations 35

36 Backup Infrastructure Design Space MaxPerf DGpower =100% MinCost No DG UPSpower =100%, UPSruntime=2min No UPS $100 $0 What do we lose when saving this cost? Varying UPS power capacity Varying DG power capacity DG Power=50% No DG UPS Runtime=2min $69 $38 Varying UPS energy capacity UPS Power = 100% Runtime = 30min No DG $55 Combinations UPS Power = 50% Runtime = 60min No DG $38 36

37 Power Power Under-provisioning Ramifications Reduced power capacity Normal Operation During Power Outage Utility Power Resumed Normal Operation During Power Outage Utility Power Resumed Degraded Performance Outage Starts Outage Ends Time Outage Starts Outage Ends Time 37

38 Power Power Under-provisioning Ramifications Reduced energy capacity Normal Operation During Power Outage Utility Power Resumed Normal Operation During Power Outage Utility Power Resumed Service Unavailability No Service Outage Starts Outage Ends Time Outage Starts Outage Ends Time 38

39 Power Under-provisioning Ramifications Loss of volatile app state Re-initialization of server components Consistency checks Reload OS & app stack Warm-up phase Re-computation Normal Operation During Power Outage Utility Power Resumed No Service No Service beyond outage Degraded Performance after power resumes Outage Starts Outage Ends Time for resuming to normal service Time 39

40 Cost Cost-Performance-Availability Trade-off MaxPerf How can we achieve this? MinCost Ideal Performance + Availability 40

41 Cost ($/KW/year) One Solution % of outages 20% of outages 10% of outages Get rid of Diesel Generators! (cost savings without performance loss) Cost of DG (Cap-ex) DGs from the cloud. Redirection to other datacenters Cost of UPS Use system techniques with performance degradation at lower cost. Sandy Outage Duration (mins) D. Wang, S. Govindan, et.al.,underprovisioning Backup Power Infrastructure for Datacenters, ASPLOS

42 Where do the $$$ go? Backup Total Cost of Ownership of a 1U Server Conversion 12% Op-ex Distribution Quality Reduce infrastructure Improve PUE 23% Cap-ex 19% Energy Cost 46% Equip. Cost Reduce Peak Power 42

43 An Efficient Cloud Reaches Beyond Datacenter -67% power plant Fossil fuel -10% transmission and distribution -33% cooling -4% lighting -15% UPS loss -10% air handling (fans) -35% power supply -85% underutilization -40% inefficient applications >99% of initial energy lost in conversion 100kW to the power plant 30kW to the data center 16kW to the server 9.5kW to the application 0.9kW to the customer 43

44 Fuel Cells are Getting Ready Fuel cell installation unit count Fuel Cell Electric Vehicles (Toyota) Fuel cell has surpassed conventional Micro-CHP in 2013 ebay s Utah data center (Bloom Energy) 44

45 USD Fuel Cells Comparing prices - Electricity vs. Natural Gas (Energy Equivalent) Electricity Natural gas Natural gas grid is 100 times more reliable than electrical grid. Gas grid failure is more graceful. Natural gas is easier to store than electrical or kinetic energy. 45

46 The next generation of energy technology 46

47 In Rack Fuel Cell Prototype 150kV - 15kV Medium voltage switch board 15kV V Up to 40% cheaper to build Up to 40% lower energy cost Batt ery Low voltage switch board Genera tor TVSS No electrical distribution No backup generators No central UPS Measured 53% efficiency 480 V /230 V Branch circuit distribution ATS SERVER SERVER SERVER SERVER SE RVE R SERVER SE RVE R Manifold distribution to each row SERVER SERVER SERVER Fuel Cell SERVER SERVER SERVER Connection to street header Pressure regulation Ana Carolina Riekstin, Sean James, Aman Kansal, Jie Liu, and Eric Peterson, No More Electrical Infrastructure: Towards Fuel Cell Powered Data Centers, HotPower

48 Technical Challenge: Load Following 48

49 Provisioning ESDs for load following Trace 1 Trace 2 Underprovisioning ESD Capacity Compensate with power capping techniques 49

50 Opportunities for Grid and Datacenters Grid Datacenters as energy buffer for the grid Demand Response Frequency Regulation 50

51 Acknowledgement Jie Liu Aman Kansal Sriram Govindan Sean James Badriddine Khessib Kushagra Vaid Eric Peterson John Siegler Bolun Xu Baosen Zhang Chuangang Ren Iyswarya Narayanan Anand Sivasubramaniam Bhuvan Urgaonkar Abdullah-Al Mamun Hosam Fathy Yang Li Onur Mutlu Li Zhao Jacob Brouwer 51