COOLING WHAT ARE THE OPTIONS AND THE PROS AND CONS FOR THE VARIOUS TYPES OF DATACENTRES SERVER AND COMMS ROOMS Dr Jon Summers Scientific Leader in Data Centers Part-time Academic in Thermofluids at the University of Leeds. RISE Research Institutes of Sweden ICT DIVISION SICS North AB Data Centres Ireland 22-23 November 2017
RISE SICS North. ice.sics.se 2
RISE SICS North Business model Big data Machine learning University IT Cloud delivering R&I projects Facility HW experiments and testing as-a-service Companies Utility
Cooling is a waste of energy and money! Luminous power Electrical power Thermal power <1% efficient! 4
Cooling is a waste of energy and money! Luminous power Electrical power Thermal power <1% efficient! 5
Getting cooling where it is needed is a problem of scales. Microprocessors are small Data centers are large 13,823,530 fold increase in size! 170mm 2 Intel Haswell EX (15 core) approx 160W 2,350,000,000mm 2 2MW facility with approx 400 cabinets. 6
Recirculation By pass air Cooling data centres Expansion valve Compressor GRILL HOT AISLE CONTAINMENT Ducting DATA CENTRE ROOM GRILL COLD AISLE CONTAINMENT Plenum Refrigerant Rack/ Servers DX CRAC Unit (inside) Air condensing Unit (outside) AIR LIQUID 7
Recirculation By pass air Cooling data centres (IT level) GRILL HOT AISLE CONTAINMENT Expansion valve DATA CENTRE ROOM Compressor Harvest the thermal energy from Ducting the microelectronics (no data centre cooling control) GRILL COLD AISLE CONTAINMENT Plenum Refrigerant ASHRAE 2015 recommended temperature and humidity Rack/ Servers AIR DX CRAC Unit (inside) LIQUID Air condensing Unit (outside) 8
Recirculation By pass air Cooling data centres (Data Hall level) Expansion valve Compressor GRILL GRILL HOT AISLE CONTAINMENT COLD AISLE CONTAINMENT Ducting DATA CENTRE ROOM Plenum Transport of the thermal energy for removal from the data centre. (layout and design critical) Refrigerant Rack/ Servers DX CRAC Unit (inside) Air condensing Unit (outside) AIR LIQUID 9
Recirculation By pass air Cooling data centres (Outside the data centre) HOT AISLE (a myriad of GRILL solutions depending Ducting CONTAINMENT Expansion valve Rejection of the thermal energy. on geographical location) DATA CENTRE ROOM Compressor GRILL COLD AISLE CONTAINMENT Plenum Refrigerant Rack/ Servers DX CRAC Unit (inside) Air condensing Unit (outside) 10 AIR LIQUID Evans, Tony (2004) The different technologies for cooling data centres, version 2. APC white paper #59
Recirculation By pass air Cooling data centres (Outside the data centre) HOT AISLE (a myriad of GRILL solutions depending Ducting CONTAINMENT Expansion valve Rejection of the thermal energy. on geographical location) DATA CENTRE ROOM Compressor GRILL COLD AISLE CONTAINMENT Plenum Refrigerant Rack/ Servers DX CRAC Unit (inside) Air condensing Unit (outside) 11 AIR LIQUID Evans, Tony (2004) The different technologies for cooling data centres, version 2. APC white paper #59
Cooling data centres (inside the data hall) Flooded Supply Targeted Supply Contained Supply 12 Flooded Return Targeted Return Contained Return No ducting or plenum. Air returns to CRAC or CRAH overhead. Hot and Cold Air mixes! No ducting or plenum to the rack. Air returns through overhead ducting. No ducting or plenum to the rack. Air returns through ducting from hot aisle containment. Use of underfloor plenum or overhead ducting. Air returns overhead. Example: underfloor plenum. Overhead ducting for return. Underfloor plenum with ducted hot aisle contained or ducted rack chimney. Fully ducted to rack with cold aisle contained. Air returns overhead. Fully ducted to racks with cold aisle containment and ducted overhead return. Combined hot and cold aisle containment. Very reduced hot and cold air mixing. Liquid cooling. Rasmussen, Neil (2003) Air distribution architecture options for mission critical facilities. APC white paper #55 Cho, J., Lim, T. and Kim, B.S., (2008). Cooling systems for IT environment heat removal in (internet) data centers. Journal of Asian Architecture and Building Engineering, 7(2), pp.387-394.
Cooling data centres (inside the data hall) Flooded Supply Targeted Supply Contained Supply 13 Flooded Return Targeted Return Contained Return No ducting or plenum. Air returns to CRAC or CRAH overhead. Hot and Cold Air mixes! No ducting or plenum to the rack. Air returns through overhead ducting. No ducting or plenum to the rack. Air returns through ducting from hot aisle containment. Use of underfloor plenum or overhead ducting. Air returns overhead. Example: underfloor plenum. Overhead ducting for return. Underfloor plenum with ducted hot aisle contained or ducted rack chimney. Fully ducted to rack with cold aisle contained. Air returns overhead. Fully ducted to racks with cold aisle containment and ducted overhead return. Combined hot and cold aisle containment. Very reduced hot and cold air mixing. Liquid cooling. Rasmussen, Neil (2003) Air distribution architecture options for mission critical facilities. APC white paper #55 Cho, J., Lim, T. and Kim, B.S., (2008). Cooling systems for IT environment heat removal in (internet) data centers. Journal of Asian Architecture and Building Engineering, 7(2), pp.387-394.
CFD Modelling of airflow (Targeted Supply, Flooded Return) [ o K] 14
Demonstration of realtime CFD 15
Cooling data centres (outside the data centre) 16
Cooling data centres (outside the data centre) We need to try and get rid of this additional plant. 17
Cooling data centres (outside the data centre) 18
Cooling data centres (outside the data centre) 19
Cooling data centres (outside the data centre) 20
Cooling data centres (outside the data centre) A2 allowable. 21
Cooling data centres (outside the data centre) HEAT PUMP TO DISTRICT HEATING 22
So what is the best approach? Aim for as close to contained supply and return as possible no mixing of hot and cold streams = highest temperature to outside. Look at the Coefficient of Performance (COP) of the outside heat rejection. Thermal Power Rejected/Electrical Power in Rejecting. Best in class is around 40.0 (direct/indirect air) or 4.5 for a Heat Pump to recover heat. Note that COP is dependent on external temperatures. If you want to talk PUE, then PUE = 1 + 1/COP 23
HOWEVER - History of the thermal problem Roger Schmidt, Liquid Cooling is Back, Electronic Cooling August 2005. M.J.Ellsworth. Interpack 11 Tutorial. 2017 - Nvidia V100 produces at peak 36.8 W/cm 2 Cross, T. "After Moore s Law: Double, double, toil and trouble." The Economist, Technology Quarterly, Quarter 1 (2016).
Liquid Cooling Solutions Moving towards total liquid cooling. Air ILC DLC TLC Mixes Facility and IT! ILC = Indirect Liquid Cooling 100% Air with liquid to the rack. DLC = Direct Liquid Cooling 0%<Air<100% with liquid to the rack. TLC = Total Liquid Cooling 0% Air with liquid to the rack.
Direct Liquid Cooling into the server.
Total Liquid Cooling by full immersion: rack to the lab heat transfer Using 20W central heating pump. Two 2.5kW domestic hot water radiators. Thermal images show surface temperatures of the rack and the radiators. With 2.2kW IT and 95W of pumps gives ppue = 1.05
Wed Sep 23 18:00:01 BST 2015 Sun Sep 27 02:00:01 BST 2015 Wed Sep 30 10:00:01 BST 2015 Sat Oct 3 18:00:01 BST 2015 Wed Oct 7 02:00:02 BST 2015 Sat Oct 10 10:00:02 BST 2015 Tue Oct 13 18:00:02 BST 2015 Sat Oct 17 02:00:02 BST 2015 Tue Oct 20 10:00:01 BST 2015 Fri Oct 23 18:00:02 BST 2015 Tue Oct 27 02:00:02 GMT 2015 Fri Oct 30 10:00:01 GMT 2015 Mon Nov 2 18:00:01 GMT 2015 Fri Nov 6 02:00:01 GMT 2015 Mon Nov 9 10:00:02 GMT 2015 Thu Nov 12 18:00:01 GMT 2015 Mon Nov 16 02:00:02 GMT 2015 Thu Nov 19 10:00:02 GMT 2015 Sun Nov 22 18:00:02 GMT 2015 Thu Nov 26 02:00:03 GMT 2015 Sun Nov 29 10:00:02 GMT 2015 Wed Dec 2 18:00:02 GMT 2015 Sun Dec 6 02:00:03 GMT 2015 Wed Dec 9 10:00:01 GMT 2015 Sat Dec 12 18:00:02 GMT 2015 Wed Dec 16 02:00:01 GMT 2015 Sat Dec 19 10:00:02 GMT 2015 Tue Dec 22 18:00:01 GMT 2015 Sat Dec 26 02:00:02 GMT 2015 Tue Dec 29 10:00:03 GMT 2015 Fri Jan 1 18:00:01 GMT 2016 Tue Jan 5 02:00:03 GMT 2016 Fri Jan 8 10:00:02 GMT 2016 Mon Jan 11 18:00:02 GMT 2016 Fri Jan 15 02:00:03 GMT 2016 Mon Jan 18 10:00:02 GMT 2016 Thu Jan 21 18:00:02 GMT 2016 Mon Jan 25 02:00:02 GMT 2016 Thu Jan 28 10:00:03 GMT 2016 Sun Jan 31 18:00:01 GMT 2016 Thu Feb 4 02:00:03 GMT 2016 Sun Feb 7 10:00:02 GMT 2016 Wed Feb 10 18:00:02 GMT 2016 Sun Feb 14 02:00:02 GMT 2016 Wed Feb 17 10:00:02 GMT 2016 Sat Feb 20 18:00:02 GMT 2016 Wed Feb 24 02:00:02 GMT 2016 Sat Feb 27 10:00:02 GMT 2016 Tue Mar 1 18:00:01 GMT 2016 Sat Mar 5 02:00:02 GMT 2016 Tue Mar 8 10:00:01 GMT 2016 Fri Mar 11 18:00:02 GMT 2016 Tue Mar 15 02:00:02 GMT 2016 Fri Mar 18 10:00:01 GMT 2016 Mon Mar 21 18:00:02 GMT 2016 Fri Mar 25 02:00:02 GMT 2016 Mon Mar 28 12:00:02 BST 2016 Thu Mar 31 20:00:01 BST 2016 Mon Apr 4 04:00:03 BST 2016 Thu Apr 7 12:00:02 BST 2016 Sun Apr 10 20:00:02 BST 2016 Thu Apr 14 04:00:02 BST 2016 Sun Apr 17 12:00:01 BST 2016 Wed Apr 20 20:00:02 BST 2016 Sun Apr 24 04:00:02 BST 2016 Wed Apr 27 12:00:02 BST 2016 Sat Apr 30 20:00:02 BST 2016 Wed May 4 04:00:01 BST 2016 Sat May 7 12:00:02 BST 2016 Tue May 10 20:00:01 BST 2016 Sat May 14 04:00:02 BST 2016 Tue May 17 12:00:03 BST 2016 60 Total Liquid Cooling by full immersion: rack 50 to the lab heat transfer Using 20W central heating pump. 40 Two 2.5kW domestic hot water radiators. 30 Thermal images show surface 20 temperatures of the rack and the radiators. 10 0 With 2.2kW IT and 95W of pumps gives ppue = 1.05
Total Liquid Cooling by full immersion Proof of Concept
Total Liquid Cooling by full immersion Proof of Concept
Total Liquid Cooling by full immersion 90 80 70 60 50 40 30 20 10 0 09:36 12:00 14:24 16:48 19:12 21:36 00:00 Proof of Concept
CONTACTS Dr Jon Summers Scientific Leader in Data Centers jon.summers@ri.se +46 70 450 31 53 RISE Research Institutes of Sweden ICT Division SICS North AB