Next Generation Cooling

Transcription

1 Next Generation Cooling Server Manufacturer Perspective Practical and Cost Effective David Moss Data Center Strategist Dell 1

2 Why Chip, Row, or Rack? Density? Energy Don t be scared; I don t see 50kW/rack in mainstream products for a LONG time Your facility is essentially liquid cooled Whether it be DX or chilled water based, liquid is likely the transport mechanism that carries waste heat the farthest on its journey out of the building It is only the last, short distance that air is being used as the transport mechanism By shortening the distance the air must travel, or by eliminating the requirement for air completely, efficiencies can be improved 2

3 Typical Chilled Water Plant Air Liquid 330 Watt 5-15W ~55W ~123W ~44W 2-5% Overhead 17% Overhead 37% Overhead 13% Overhead CLF (cooling portion of PUE)=

4 Row, Rack, Chip Cooling Solutions Air Liquid 330 Watt 5-15W ~55W ~123W ~44W 2-5% Overhead 17% Overhead 37% Overhead 13% Overhead 4

5 Chip Level Cooling? Liquid 330 Watt 5-15W ~55W ~123W ~44W 2-5% Overhead 17% Overhead 37% Overhead 13% Overhead 5

6 Available Chip Liquid Cooling Available solutions do have challenges Ideal thermal solution is one that removes heat from all system components- Complete Spraycool Global solution Other vendor concepts Partial (compromise) solutions selectively harvest higher heat releases Liquid cold-plates rely on surface contact to select components 6

7 Challenges with Chip Liquid Cooling Complete solutions Service paradigms Business case Mechanical design and rack compromises Non-standard U? Segregated rack? Partial solutions Partial solutions tend not to harvest enough heat to be cost effective Compromise on access with tubes running everywhere Business case 7

8 Row/Rack a Good Compromise Don t see immediate mainstream use of chip level cooling; row/rack level better alternative at this point Bring the coils close to the load (at or in the rack) Containment if possible Maintains flexibility for IT hardware (air cooled in rack) Need to address redundancy Might see point product here or there that introduces chip level liquid Again, don t see 50 kw mainstream racks except for the following caveat If liquid enabled a dramatic shift higher in density, then it could become interesting for mainstream designs 8

9 Row/Rack Advantages Moves the efficient transport medium, liquid, closer to the heat load Reduces the length the less efficient medium, air, has to travel (reducing fan power) Offers different opportunities for containment (potentially less costly); containment: Eliminates mixing Raises return air temperatures (gaining back capacity) Normalizes temperatures to better afford the raising of data center temperatures 9

10 Data Center Lab Studies 1200 CFM per Tile 40 kw/rack Cooling; 36 kw Load Limit Airflow Effectiveness/ Energy Study 10

11 Next Generation Cooling Server Manufacturer Perspective - Rethinking Data Center Design Mark Monroe Director, Sustainable Computing Sun Microsystems, Inc 11

12 Rethinking Datacenters Moving Cooling Closer To Heat Sources 12

13 Conventional Datacenter Design Issues Designing and operating for the worst case is grossly inefficient Cool for hottest part of the room Design for highest redundancy requirements Raised floors and room level cooling can t handle high today s heat densities Designs are one-off and inflexible 13

14 Conventional Designs Can't Keep Up 14

15 Closer Cooling: The POD Concept A group of racks or benches with a common hot or cold aisle used as a building block to simplify datacenter design for power, cooling, & cabling. Vendor Independent, Slab or Raised Floor, Flexible, Scalable, High Density 15

16 Approach to Datacenter Design Scalable, repeatable, modular architecture Modular right-sized power and cooling Simplified, flexible cabling and plumbing Real-time energy monitoring Scale cost with use Easy to build and expand Entering a New Age of Engineered Datacenters 16

17 High Efficiency Water Cooling Modular Datacenter High Density Eco-Friendly Design 40% More Efficient Rapid Deployment Game Changing Economics 17

18 Why is Project Blackbox So Efficient? Innovation in action Unique, closed-loop water cooling system Water can carry as much as 3,500 times more heat than air in the same volume Self-contained, closed-loop air circulation system results in far less area to cool 18

19 This Changes the Datacenter Game 19

20 Corporate Data Center Data Center In A Box Traditional Data Center 700 systems M-value = 28 million 9000 sq ft 1MW power Compute Pod: 114 T5240 M-value = 51million 170KW power/cooling Storage Pod: 70 x4500 = 3.3 PB 144 KW power/cooling 314KW, 2x160 sq ft + exo-structure An M-value is a single number intended to represent the processing potential of a server running commercial workloads, taking into account the class of the server, the type and number of processors it features (including their clock speed and external cache size), and the release of Solaris being run. 20

21 Future On Chip Cooling? Gets heat into liquid even earlier High temperatures (80 o C-100 o C) provide more useful heat Used primarily in specialized applications today 21

22 Next Generation Cooling Cooling Manufacturer Perspective - Cooling Close to the Load Steve Madara VP and GM Liebert Precision Cooling Emerson Network Power 22

23 DCUG - Power Density (kw/rack) Percentage of Users 40% 35% 30% 25% 20% 15% 10% 5% 0% Average kw/rack of Primary Data Center 2 kw or less >2-4 kw >4-8 kw >8-12 kw >12-16 kw >16-20 kw >20-24 kw Greater than 24 kw Average kw/rack 2006 = = = = 12 Spring 2006 Spring 2007 Spring (Estimated) Maximum kw/rack of Primary Data Center Average Maximum kw/rack 2006 = = = = 16 Percentage of Users 35% 30% 25% 20% 15% 10% 5% 0% 2 kw or less >2-4 kw >4-8 kw >8-12 kw >12-16 kw >16-20 kw Spring 2006 Spring 2007 Spring (Estimated) >20-24 kw Greater than 24 kw If power & cooling were not an issue, 50% of the users would take the max kw/rack to >18 with 15% ready to go over 30kW/Rack 23

24 Challenges Facing The Data Center Max kw Per Rack Allowed Rack density guidelines set the limits on what can be loaded in a single rack 0 2 KW >2-4 KW >4-6 KW >6-8 KW >8-10 KW >10-12 KW >12-14 KW >14-16 KW >18-20 KW <24 KW >24 KW New focus on Energy consumed in the Data Center Cooling About 37%- 45% Air Movement 12% Electricity Transformer/ UPS 10% Cooling 25% Lighting, etc. 3% IT Equipment 50% 24

25 Types of Cooling Solutions User Requirements / System Design Need for IT hot swapability Redundancy and ride through capability Cooling reliability and cooling control Flexibility Reduced energy consumption per IT load Extended capability of solution Requires cooling fluid to/near the Rack Cooling Configuration Range (kw) Availability Cooling Plant Floor Mount Room Cooling Traditional <4 DX/CW Hot/Cold Aisle <5-6 Today DX/CW Containment < 8 Today DX/CW Supplemental Cooling Open Architecture Cooling Modules w Fans Today DX/CW Open Architecture Cooling Modules w/o Fans Limited DX/CW Closed Architecture Cooling Racks Today CW** Embedded Rack Cooling Cooling Coils inside rack up to 80 Limited DX/CW Component / Chip Server Cooling Spray * approx 30 Limited CW Contact up to 50 In Testing DX/CW * does not completely cool the server ** future pumped refrigerant 25

26 Supplemental Cooling Energy Efficiency Benefits Moving Cooling Close to the Load Traditional Cooling Only Fan Power >8.5kW per 100 kw of Cooling Average entering air temperature of F Cooling Unit XD & Base Cooling Fan Power- <3.5 kw per 100 kw of Cooling 2 kw per 100kW) Average entering air temperature of F >65% less fan power Greater cooling coil effectiveness 100% Sensible Cooling Supplemental Cooling Modules Lower fan power Higher coil effectiveness Add on capability Refrigerant kw Ch Water kw 26

27 Supplemental, Embedded and Component Cooling Configurations Cooling coils Enclosed Rack with Cooling Coil Cooling fans in rack High coil effectiveness Fixed Capacity Refrigerant * 30kw Ch Water kw Supplemental Rear Door Heat Exchanger No cooling fans Higher coil effectiveness Add on capability Embedded Coils No cooling fans, smaller server fans Higher coil effectiveness Refrigerant only 80 kw Redundancy available Hot swappable servers Component Server Cooling No cooling fans or server fans Max coil effectiveness Refrigerant only 50 kw Refrigerant 35 kw Redundancy available Ch Water kw Hot swappable servers 27

28 / / / Future System Opportunities for Improved Cooling Efficiencies % 0.2 Traditional Cooling Fluid Cooling Opportunities kw for cooling per kw of server heat load Enclosed Rack XD Modules (10 35 kw) Rack Cooling (10 20 kw) Embedded Cooling (35 60 kw) Component Cooling (>100 kw) Data Center Best Practices Energy Savings driven by reduction in Fan Power (cooling system and server) plus Heat Transfer efficiency 45% Cooling the Room Cooling the Rack Cooling the Load Optimal PUE-1.4 PUE

29 Comparison of Cooling Solutions Cooling Configuration Floor Mount Room Cooling Containment Supplemental Cooling Open Architecture Cooling Modules w Fans Open Architecture Cooling Modules w/o Fans Closed Architecture Cooling Racks Embedded Rack Cooling Cooling Coils inside rack Component / Chip Server Cooling Spray * Contact Efficiency Reliability Flexibility Density Redundancy Sound Floor Space Add-On Hot Swappable Servers Cost Commerical Availability Rating: 1- baseline 5 Optimal * Does not completely cool the server 29

30 Supplemental and Embedded Cooling Benefits Energy Reliability/Level of Control Space Cost Future Sound Footprint Challenges Power 30

31 Next Generation Cooling Consulting Engineer Perspective - Mechanical Systems Charles B. Kensky, PE, LEED AP Vice President Bala Consulting Engineers, Inc. 31

32 Increasing Increasing Heat Heat Loads Loads Increasing Increasing IT IT Loads Loads Data Center Loads 50kW 40kW 30kW 20kW 10kW kw/rack 32

33 Load Translation Today s IT loads: For every 1 square foot of data center space you can cool 300 square feet of office space Future IT Loads with Today s Cooling Options Cooling 1 Cabinet at 50kW Cooling for = 33

34 Load Translation Cooling 1 Cabinet at 50kW = 7 Pizza Ovens 17 2,400 s.f homes W Light Bulbs 34

35 50kW Per Cabinet Will Require Advances in Chip Technology Potential Reconfiguration of the Cabinet Potential Reconfiguration of the Data Center Room Perforated Doors vs. Solid Doors Paradigm Shift in Mechanical, Electrical, Plumbing Systems 35

36 MEP Systems: Cooling the Load At 50kW or Greater Air Must be the Secondary Cooling Medium. Air is a Poor Heat Transfer Medium Refrigerant Cooling is Better Water is Most Effective PUE Power Usage Effectiveness 36

37 Power Usage Effectiveness PUE = Total Facility Power IT Equip. Power Typical Data Center PUE Average Data Center Efficient Data Center Chilled Water Plant provides 10-15% power savings. Concentrated Cooling provides additional power savings. Air Velocity at cabinet face is 1180 fpm or 13 mph. 37

38 High Density Cooling Strategies High Density Cooling Options Cool the Room Zones / Containment Cool the Cabinets Cool the Chips/Servers Concentrated Cooling Mechanical Equip. Powered by UPS Dual Corded = Dual Piped? 38

39 High Density Energy Savings Energy Audits Free Cooling Water Side Air Side Reduced Air Flow Hot Aisle/Cold Aisle Heat Recovery Geothermal Room Temperature/Humidity Set Points Quicker Reaction Times 39

40 High Density Data Centers Energy Energy Energy - Processing Needs Increasing - Environmentally Sensitive / Cost Effective - Need Solutions Now! 40

41 Questions and Answers 41