Close Coupled Cooling for Datacentres Dr Fathi Tarada Mosen Ltd fathi.tarada@mosenltd.com www.mosenltd.com
Agenda Increasing Power Densities Raised Floor Limitations Hot and Cold Aisle Enclosures Close Coupled Cooling (CCC) Design Application Oliver s Yard
Agenda Increasing Power Densities Raised Floor Limitations Hot and Cold Aisle Enclosures Close Coupled Cooling (CCC) Design Application Oliver s Yard
Recent evolution of datacentre design 1970-1990 Monolithic custom designed rooms Datacentre information is processed in batches Raised floor cooling systems Unable to scale without disruption Low density (< 450 W/m 2 ) 1990 2010 Information is processed in real time Mainframes replaced by servers Datacentre demands high power (>1500 W/m 2 ) Room design power density is continuing to increase.
Increasing rack density loads Rack power density has increased 10x 2003 < 1.7 kw average per rack 2006 ~ 5.0 kw average per rack 2009 ~ 6 kw average per rack 2010 ~ 20 kw per rack peak loads Peak rack densities will continue to increase.
IT server air requirements Typical 1U Server 34 o C 42U rack 250 W per server is 10.5 kw requires 790 l/s of air 24 o C 75.5 l/s/kw 42U rack 3 blade chassis is ~12 kw requires 590 l/s of air Typical 7U Blade Server 24 o C 40-44 o C 49.6 l/s/kw Temperature rise and air volume varies with server type
Impact on IT Equipment Power Constant IT Server Fan Speed Variable IT Server Fan Speed 23.8 o C 21.5 o C 23 17 o C to 38 o C fan power is constant Component temp tracks inlet temp over range Maintains almost constant component temp < 23 o C fan power is constant, > 24ºC increases Component temperature tracks inlet temp Maintains constant component temp Fan power 11 W at 23 o C to >60 W at 35 o C Increased inlet temp does not effect reliability Total fan power (IT + facilities) may go up with warmer temperatures
Agenda Increasing Power Densities Raised Floor Limitations Hot and Cold Aisle Enclosures Close Coupled Cooling (CCC) Design Application Oliver s Yard
Cooling Designs - Perimeter Perimeter Units for 4-5 kw per rack Perimeter Units for 4-5 kw per rack 0.60m 2.70m 0.45m 0.60m 2.70m 0.45m Hot Air Return Plenum 32 o C Hot Air Return Plenum 32 o C Mixed Room Air Return 24 o C Mixed Room Air Return 24 o C Design Concepts s in room or mechanical corridor Ceiling void return air plenum Open room return air Supply air under floor Advantages Widely used for 40 years Large design base experience Low CapEx Large vendor base Disadvantages Limited density approx 4-5 kw/rack Less predictable performance Difficult to scale Room dimensions critical Floor depth, air leakage, room dimensions limit predictable cooling performance
Air distribution issues Uniform air distribution is difficult in large rooms
Agenda Increasing Power Densities Raised Floor Limitations Hot and Cold Aisle Enclosures Close Coupled Cooling (CCC) Design Application Oliver s Yard
Recirculation of hot air Two issues: Heat recirculation at row level and rack level Rack Level Row Level 90 F (32 C) 79 F (26 C) Side Side 80 F (27 C) 95 F (35 C) 35 C Hot Air Recirculation 73 F (23 C) 23 C 73 F (23 C) 23 C Blanking Panel Blanking Panel 35 C 83 F (28 C) 72 F 22 (22 C) 70 F (21 C) 23 C 73 F 22 (23 C) 72 F 22 (22 C) 70 F 21 (21 C) Server Inlet Temp Server Inlet Temp Blanking panels stop heat recirculation in the rack. Cold aisle enclosure prevents heat recirculation in the row. Enclosures stop recirculation of heat, allow full use of rack space
Raised floor cooling with Cold Aisle Enclosure Downflow units Fan Failure CAE with Supply Air Flooded cold aisle supplied from downflow s Eliminates hot air recirculation Actual cooling capacity limited by finite volume of air under floor Eliminates hot aisle air recirculation but can cause hot spots elsewhere
No Raised floor for cool air distribution Horizontal air distribution Piping protection on slab floor Solves raised floor air problems Reduced installation time Reduced cost Add power and cooling capacity as you add computing power
The infrastructure paradigm shift Room Level Rack Level UPS UPS Small Server Rooms 32kW IT load 6.4 m 4kW 4kW 4kW 4kW 4kW 4kW 4kW 4kW N+1 Increased Free Cooling 79% less fan power used 49% less floor space UPS 16kW 16kW 5.4 m 38% fewer racks No raised floor required 3.6 m 6 m No ceiling height restriction Less Space, Less cost, Lower CO 2 Emissions and much greater flexibility
The infrastructure paradigm shift 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Room Level Rack Level 38% fewer Room Level 49% less Rack Level Room Level Rack Level No. Racks Floor Space Fan power 79% less Less Space, Less cost, Lower Emissions and much greater flexibility
Cold aisle enclosure with CCC Cold Aisle Enclosure No raised floor required Ceiling height not an issue Redundancy at row level Eliminates warm air recirculation Cooling units can be fully or semi recessed BUT Mixed return air to CCC unit Predictable air distribution with close coupled cooling in the rack
Hot aisle enclosure with CCC Hot Aisle Enclosure No raised floor required Ceiling height not an issue Redundancy at row level Eliminates warm air recirculation Cooling units can be fully or semi recessed AND Maximises coil capacity Reduces chiller power Increases free cooling opportunity Heat removal at the source
Enclosed high density zones Zones can be replicated for consistent predictable performance
Improved cooling capacity with HAE Cooing capacity (kw) 60 30 50 25 40 20 kw30 15 20 10 Increases coil capacity with warmer return air Reduces chiller power Enables more free cooling hours +225% +213% 10 5 Return air 24ºc 43ºc temperature 24ºc 43ºc 0 75 T25 9 75 109 9T50 109 unit Increased cooling capacity with warmer return air
The high density advantage 160 kw total IT load 6 m Raised floor not required 10.8 m Increased server density 10.8 m 88% less fan power used 6 m 65% fewer racks 69% less space 10.8 m 10.8 m Perimeter Same cooling capacity In-row 1860 mm 50% lower fan power 52% smaller footprint 1860 mm Reduces CapEx and OpEx 600 mm
CCC scalability 10.8 m 10.8 m 10.8 m 10.8 m Same sever load, fewer racks Less first cost for cooling units Less floor space For Concurrent Maintenance: Downflow s N+1 = 8; CCC N+1 = 4
CCC scalability 10.8 m 10.8 m 10.8 m 10.8 m Twice the server load Half the floor space Same number of cooling units for twice the IT load
CCC scalability 10.8 m 10.8 m 10.8 m 10.8 m No scalability with s Difficult to exceed average density Zones can be replicated High peak density anywhere in row CCC is highly scalable
CCC scalability 10.8 m 10.8 m 10.8 m 10.8 m Limited to initial design load 4 times the server power Peak load in any rack Fast dynamic response CCC reduces first cost and allows future growh
Agenda Increasing Power Densities Raised Floor Limitations Hot and Cold Aisle Enclosures Close Coupled Cooling (CCC) Design Application Oliver s Yard
CCC Solutions Rittal LCP ENP (Knürr) Datacentience Delta-T Liebert XD CRV IBM Cool Blue APC InRow
Single row fully enclosed LCP cooling unit
CCC Solutions Rittal LCP ENP (Knürr) Datacentience Delta-T Liebert XD CRV IBM Cool Blue APC InRow
XD Pumped Refrigerant Systems
CRV
Close Coupled Cooling Solutions Rittal LCP ENP (Knürr) Datacentience Delta-T Liebert XD CRV IBM Cool Blue APC InRow
Individual sealed racks
Rear Door Heat Exchangers RDHx w/ fans RDHx w/o fans CDU
Delta-T rack mounted cooling T 25 T 50 T 25 T 50 Slides in like a server
Delta-T fully enclosed fan Flow paths out of the fully recessed fan. Front of rack Rear of rack Increased fan power needed in fully recessed position
Delta-T semi-recessed fan Removal of cabinet side panels allows natural radial air discharge and lower fan power. Front of rack Rear of rack All air flow becomes radial with fan wheel outside of rack
Agenda Increasing Power Densities Raised Floor Limitations Hot and Cold Aisle Enclosures Close Coupled Cooling (CCC) Design Application Oliver s Yard
Row Layout Room Configuration
Objects Modelled in CFD Objects modelled: Racks (standard 3 kw racks in Rows A/B, C/D & higher rated 3.9 kw racks in Rows E/F, G/H) Weiss Technik downflow units 8 number Power Distribution Units STS-PC-400- E454P (each producing 2 kw of waste heat) Underfloor cable racks Ventilated floor tiles
Normal Operation N+1 Units
Failed Controls & Unit for N Units Running Failed unit
Failed Controls & Unit for N Units Running
Case with One Unit Failed Failed unit
Failed Controls & Unit for N Units Running Failed unit
Failed Controls & Unit for N Units Running Failed unit Peak rack inlet temperature = 27.1 C
Summary Increasing Power Densities Raised Floor Limitations Hot and Cold Aisle Enclosures Close Coupled Cooling (CCC) Design Application Oliver s Yard