Efficient Cooling Methods of Large HPC Systems

Transcription

1 Efficient Cooling Methods of Large HPC Systems Rudolf Lohner Karlsruhe Institute of Technology (KIT) Steinbuch Centre for Computing (SCC) STEINBUCH CENTRE FOR COMPUTING - SCC KIT Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft

2 Agenda The Struggle for Optimal Cooling in Data Centers Energy Consumption in Data Centers Cold Air Cold Water Warm Water Any Water (Green Cube) Heat Pipes Oil Immersion 2-Phase Immersion 2/30

3 Energy Consumption in Datacenters Power Consumption for IT and Cooling Server and network components Chillers and CRACs/CRAHs (cold air, cold water) Power supplies including UPS (power losses) Lots of fans Other Energy Consumption Building infrastructure Heating Air conditioning for offices Warm water Other building equipment (light, office equipment, elevators,...) 3/30

4 Energy Consumption in Datacenters Measure for efficiency of energy usage Power Usage Effectiveness PUE (The Green Grid, 2007) PUE = Total Power Datacenter (kwh) / IT-Power (kwh) Ideal value: SuperMUC at LRZ: Currently usual: PUE = 1.0 PUE < 1.1 PUE ~ 1.5 (good) 2.5 (bad) PUE is strongly time dependent - time of day / night - seasons: spring, summer, fall, winter 4/30

5 Special Situation in HPC-Centers Very high packing and energy density 30kW per rack already normal Soon 100 kw and more per rack Very high load on servers Homogeneous infrastructure Dedicated cooling circuit for one machine makes sense Makes also exotic cooling concepts reasonable Reuse of waste heat possible but not constantly available Unplanned (crashes) or planned (maintenance) shutdowns reduce availability times of waste heat 5/30

6 Cold Air Cooling Air Cooling Classical Method Chillers produce cold water CRAHs/CRACs make cold our of it Cold air is pressed in raised floor Exits in front of racks through open tiles Servers take in cold air, blow hot air in rear Evacuate hot air and cool down again Air Temperature and Humidity Avoid too cold and too dry air Too cold: humidity may condensate Too dry: danger of static electricity Intake temperature may be up to 30 C ASHRAE Thermal Guidelines for Data Processing Environments 6/30

7 Cold Air Cooling Airflow can be difficult to manage Use open floor plates only where necessary Avoid mixing of cold and warm air Use containment constructions for strict separation of cold and warm air Cold aisle Warm aisle Avoid high air velocity: Venturi effect may reverse air flow Close unneeded holes in tiles If cable outlet close with foam material Cost is very high Air has low heat capacity Mechanical chillers use much power Need for lots of air ducts and fans 7/30

8 Cold Air Cooling How to minimize overhead for cold air cooling? Choose suitable location for data center Iceland Finland Oregon Washington Ireland... Only possible for Big Players Google, Apple, Amazon, Microsoft,... Cloud Data Centers,... 8/30

9 Cold Water Cooling Cold water for cooling directly at rack level Need water pipes in computer room Still unpopular in many Datacenters Cold water cooling Variants: Heat exchangers in racks' rear doors Air tight sealed racks with internal heat exchangers and fans - Air flow horizontally (Rittal, HP MCS) - Air flow vertically (Emerson/Knürr) Direct chip cooling cold water flows directly through heat sink Details to consider Clean and accurate cabeling for free air flow Water temperature above dew point avoid condensate water 9/30

10 Cold Water Cooling Air tight sealed racks Horizontal air flow Control panel: - temperatures - flow rates Rear door heat exchanger Passive heat exchanger Air flow by fans in servers No active controls 10/30

11 Combined Approaches Combination of different concepts Summer / winter operation Dynamic adaption to ambient climate (HP EcoPOD) 11/30

12 Warm Water Cooling Direct chip cooling: warm water goes directly through heat sinks Outlet temperature: up to 40 C 50 C and even more Early Installations: IBM Aquasar ETH Zürich SuperMUC LRZ Munich Megware LRZ Munich Water quality can be critical 12/30

13 Warm Water Cooling Waste heat reuse easily possible Usage e.g. for building heating Backup heating might be necessary In case of longer downtimes (maintenance, upgrades, ) Driving adsorption chillers CoolMUC at LRZ: Temperature of ~ 60 C is high enough to produce cold air by use of adsorption chiller. 13/30

14 Basic Design Parameters ForHLR was built along these design parameters: ForHLR will be a Petaflop class system Electrical power up to 1 MW Very high energy efficiency wanted: - Warm water cooling for up to 750 kw - Free cooling and reuse of waste heat for heating office building: 40/45 C - Air cooling for up to 250 kw - Cold water from District Cooling with Combined Heat-Power-Refrigeration-Plant Building flexible enough for next generation systems Expandability planned from the beginning 14/30

15 Equipment in Server Room Sizes 14,17 m x 16,70 m = 236 m2 Height: 3,25 m Raised floor: 1 m gross, ca. 90 cm net Cooling Warm water piping in raised floor Power rails in raised floor Cold aisle containment Air cooling with CoolWall - Sucks warm air through heat exchanger wall - Blows cold air into raised floor Initial Installation Capacity about 500 kw for - HPC-System - Visualisation cluster - Filesystem 15/30

16 Warm Water Cooling Loops Warm Water Cooling Cools up to 750 kw of server heat Waste reuse for heating in winter (up to 100 kw) Free cooling all-year-round (with dry chillers) Three Cooling Loops coupled by Heat Exchangers Server room: Heat exchanger in server room, 42/47 C Stainless steel, pure water with biocides Building loop: Stainless steel, deionized water with 40/45 C for waste heat reuse and free cooling year-round Cooling loop to roof with glycol as antifreeze Water-Air Chillers on Roof 3x 375 kw Chillers 2+1 Redundancy Spare space for future extensions 1x 700 kw Chiller for free cooling in winter and refrigeration machine in summer 16/30

17 GreenCube ecube Concept GreenCube concept from V. Lindenstruth Water cooled rear doors at racks Adiabatic cooling no chillers Wet bulb vs. dry bulb temperature Solution implemented at GSI Darmstadt Universities of Frankfurt and Heidelberg 17/30

18 GreenCube ecube Concept Wet bulb vs. dry bulb temperature Mannheim /30

19 Heatpipes Extensive use of Heatpipes Direct chip cooling with heatpipes Avoid water inside Servers How Heatpipes work Working fluid evaporates at hot side and flows to cold side Condenses at cold side and flows back to warm side by capillary forces Heatpipes widely used Laptops, PCs/Servers, Satellites, Pipelines,... 19/30

20 Heatpipes HP Apollo 8000 System Direct chip cooling with heatpipes Water flow inside rack but outside of servers Additional heat exchangers and fans in racks Air tight closed rack 100% heat removal Controls outlet water temperature, e.g. 40 C Up to 80 kw power per rack 20/30

21 Heatpipes CALYTRONICS Two-Phase Cooling Solution From Belgian Company Calyos Uses heatpipes for cooling power electronics (trains, wind turbines, etc.) Technology miniaturized for HPC Special feature: Loop Heat Pipe Works as capillary pump Project with Bull Adapted solution to Bull B700 DLC Blade (prototype see above) No fans or pumps needed 21/30

22 Oil Immersion Cooling Green Revolution Cooling Servers completely immersed in mineral oil Density > 100 kw per rack possible Maintenance greasy Oil may creep through cables (capillary forces) 22/30 Cray 2

23 Oil Immersion Cooling Largest Productive Installation: Vienna Scientific Cluster VSC-3 More than 600 TFlops 2020 nodes using 540 kw 35 tons of special mineral oil Installed end of Q PUE of 1.02 as of end July /30

24 2-Phase Immersion Liquid Cooling Evaporative Cooling with 3M TM NovecTM Liquids Fully fluorinated carbons: C xfy O CmHn Environmentally friendly, inert, non-toxic and non-flammable Coolant Novec 649 boils at 49 C Cooling capacity 4 kw per liter Extremly dense packaging Internal 2-Phase Cooling Loop Runs fully autonomous Only driven by evaporation and condensation: Coolant boils at 49 C Vapor rises to top Condenses at cooling coil Drips back into liquid bath 24/30

25 2-Phase Immersion Liquid Cooling Videos demonstrating 3M Novec 2-Phase Immersion Liquid Cooling /30

26 2-Phase Immersion Liquid Cooling Allied Control, Hong Kong Bitcoin mining datacenter with extremely dense hardware Currently 80 kw per rack DataTank Container Unit 6 DataTank Immersion Cooling Racks of kW each 1.4 MW Planned upgrade: 3 MW Uses various 3M Novec fluids and boiling points (34 C, 49, 56 C, etc) 26/30

27 Courtesy of U. Brüning, Heidelberg Dense Node Architecture Dense Packaging of KNC and EXTOLL Folded into each other IN cabling on top No host for KNC required PCIe Backplane Hosts 8 nodes (16 boards) 18mm pitch for node 9mm distance board2board + PSU integration Lower level 8x 1200W 94% eff. 27 EXTOLL Measurement with an extreme LL IN on HP Server

28 Courtesy of U. Brüning, Heidelberg Immersion cooling 2-phase immersion cooling Using 3M Novec 649, boiling at 50 ºC Remove 10kW from chassis Efficient copper pipe Steep learning curve Test system in operation 28 EXTOLL Measurement with an extreme LL IN on HP Server

29 Courtesy of U. Brüning, Heidelberg Result: GreenICE GreenICE: immersion cooled electronics Hot water cooling chassis No fans, no moving parts just the external water pump Very energy efficient PUE = EXTOLL Measurement with an extreme LL IN on HP Server

30 Thank You 30/30