Data Centre Air Segregation Efficiency (ASE)

Transcription

1 Data Centre Air Segregation Efficiency (ASE) Ing Dr Robert Tozer MSc MBA PhD CEng MCIBSE MASHRAE Sophia Flucker CEng MIMechE Operational Intelligence Ltd. Abstract Data centres are large energy consumers. In recent years the industry has focussed on improving energy performance by identifying efficiency opportunities and implementing design and operational changes, thereby optimising operational costs. Typically the IT equipment and cooling systems consume the most energy and there is significant scope to target energy savings. Most data centres are air cooled therefore air management within the data hall is an important enabler to avoid energy wastage and allow savings. Different methods are employed to segregate hot and cold air streams; metrics describe how effectively data hall air is managed by quantifying bypass, recirculation and flow availability. This paper defines a new metric which quantifies the efficiency of air separation, helping users to quantify and define potential improvements, energy / operating cost savings whilst managing risk. Keywords data centre, energy efficiency, air management, containment 1 Introduction The total power requirement of the UK data centre industry in 2013 was estimated at 3.10 GW and is projected to reach 3.68 GW by 2016 [i]. Government policy in many countries is acting to limit energy consumption [ii]. Optimising cooling systems allows large reductions in data centre energy consumption. Cooling energy can be reduced by operating at higher temperatures. The industry has recognised the opportunity to save energy and IT equipment manufacturers have increased the environmental operating ranges of the air supplied to the hardware inlet [iii]. In order to operate at higher temperatures without having a detrimental impact on hardware reliability, data hall air management is required to ensure that hardware receives air within the correct temperature and humidity range, rather than recirculated warm air previously exhausted by IT equipment. The fans in the CRAH (Computer Room Air Handling) units and the fans in the servers are in series; each server draws in the volume of air it requires. Often, excess air is moved by the CRAH units and bypass air is present, where the cooling unit fans move air in the space which returns without passing through the IT equipment and doing useful cooling. Where fan speeds can be reduced, energy reductions can be realised. Metrics to quantify air performance have been defined in previous papers [iv, v]. By taking a suitable sample of temperature measurements within the data hall to define the average IT equipment and cooling unit inlet and outlet temperatures and applying mass flow balance equations, it is possible to quantify the amount of bypass and recirculation within the space and the ratio of cooling unit to IT equipment air flow volume. Page 1 of 11

2 Figure 1 Data hall air flows in section with key weighted average temperatures and typical values Figure 2 Data hall air flows in section showing components of mass flow balance!t max = T io "T co Page 2 of 11

3 !T c = T ci "T co!t i = T io "T ii m i = mass flow of air to IT equipment m c = mass flow of air from cooling units m f = mass flow of cold air to IT equipment BP = Bypass flow: cooled air that returns to CRAH (Computer Room Air Handling) unit without cooling IT equipment. Typically found at floor grilles in hot aisles and gaps in raised floor.! supply =1! BP = m f m c = T ci!t co T io!t co = "T c "T max! supply = supply performance (previously known as flow performance) R = Recirculation flow: discharge from IT equipment that is drawn back into IT equipment inlet. This can be addressed by installing rack blanking plates and ensuring sufficient air is supplied to IT equipment.! demand =1! R = m f m i = T io!t ii T io!t co = "T i "T max! demand = demand performance (previously known as thermal performance) NP = Negative Pressure flow: data hall air drawn into raised floor due to high under floor velocities (Venturi effect). This is not present in all data halls (whereas BP & R practically impossible to totally eliminate) and where it exists, is likely to comprise a relatively small proportion of total flow, thus making it difficult to measure. Hence the metrics model assumes NP = 0. Af = Flow availability: ratio of cooling equipment flow to IT equipment flow A f = m c m i = T io!t ii T ci!t co =! demand! supply = "T i "T c A free tool which calculates the metrics and generates the air flow diagram is available on the webpage [vi]. Users can input their measured / estimated data to quantify their air performance and help diagnose issues. Once the characteristic metrics have been calculated using the measured temperature data, these can be plotted on the following chart: Page 3 of 11

4 Figure 3 Metrics quadrant diagram An ideal data hall has results at the top right corner with minimum bypass and recirculation. In reality, an Af slightly above 1 is desirable to ensure positive pressure from cold to hot aisle streams; a small amount of bypass helps prevent recirculation (see section 4). The average result based on a sample of facilities (reported in 2008) is indicated at 50% bypass and 40% recirculation [vii]. In legacy environments, air was not managed but delivered on a flood cooling principle. As load densities have increased [viii] this method has become increasingly ineffective and awareness has grown around how to improve air delivery and benefit from higher reliability (address source of hot spots) and reduced energy consumption. This analysis can be used by data centre operators to better understand the air management problems within a particular space, which allows them to target improvements. For example, where recirculation is high and servers have been measured with high inlet temperatures it is important to implement remedial action to deal with this specific issue. Where bypass is high, this may result in a shortfall of cold air delivered to servers and compound recirculation; the servers will draw their required air volume whether this air is cold or hot. Sources of bypass within the room / rack should be identified and blocked. Where high flow availability is measured, this suggests an opportunity for reducing cooling unit fan speeds and is commonly found where the design has not been optimised for part load operation [ix]. By measuring performance before making changes, operators create a baseline from which to measure improvements and can identify estimated energy savings and payback. Page 4 of 11

5 2 Air containment As IT hardware load densities have increased [viii], so too has the temperature differential between the server inlet and outlet (more power consumed means more heat to reject). Average IT delta T measured from a sample of 30 data halls between 2005 and 2014 = 7.6K. Blade servers have a high design delta T of around 20-25K. Typically, operating servers do not reach their design delta T due to low processing utilisation and in legacy facilities it was common to find a large number of idle servers switched on but not doing any useful computation. One study reports a 6% average utilisation and up to 50% of idle servers [x]. In the drive for efficiency, practices such as consolidation and virtualisation are changing this. Combined with higher supply temperatures, the effect is an increasing hot aisle temperature which compounds the recirculation problem. Very high hot aisle temperatures also pose other problems, such as derating of devices in the hot aisle and concerns over health and safety of operators. Air containment has become a widely adopted best practice to separate hot and cold air streams and is found in most new build data halls. There are several types, summarised below: Type Image Pros Cons Cold aisle Easier to fit out in existing hot / cold aisle halls, as limited installation required above racks Most of the room exposed to hot aisle temperature May require modifications to lighting / fire protection / detection device locations Semi cold aisle Inexpensive Flexible good for retrofit Allows bypass air leakage Hot aisle Increased volume of cold air for thermal runaway Most of room is cold aisle temperature therefore easier to manage for operator working Coordination with overhead services Page 5 of 11

6 Chimney No hot air in the room Limits rack choice Certain solutions may be more suitable for different applications; all are effective at improving air performance. The quality of the installation is the most important factor in determining how close they are to reaching ideal (perfect) air separation. Results from one client s data hall using retrofitted semi cold aisle containment have better measured air performance than newer, purpose built installations using cold and hot aisle containment. Often best practices such as containment are undertaken by operators as a box ticking exercise, rather than with a full understanding or engagement of the reasons behind their implementation. For example, at a legacy facility blanking plates were found installed in front of a short depth server which had been installed back to front in the rack, thereby contributing to recirculation, rather than improving it. In one case, very high bypass was measured in a facility with a low energy design using hot aisle containment but operators were unaware of this, believing the design to be efficient. There is often a gap between the design intent and operational reality! 3 Air segregation efficiency (ASE) This defines how effectively hot and cold air streams are separated. The following diagrams show the boundary between hot and cold streams for a cold aisle containment system and leakage between the two. Figure 4 Segregation of hot and cold air streams (cold aisle containment) Page 6 of 11

7 Figure 5 Areas of leakage through cold aisle containment segregation Ideal air segregation efficiency = 1, where bypass and recirculation is zero, indicated on the quadrant diagram below. Figure 6 Air segregation efficiency Page 7 of 11

8 When improvements are made to air management and bypass and recirculation are reduced, the characteristic point on the quadrant diagram moves diagonally towards the top right corner. Af remains constant unless the flow rates of the IT equipment or cooling units changes, so the improved state will be further along the same Af line. Conversely, if no changes are made to air management but the air flow rate of the cooling units (or IT equipment) is changed, e.g. by changing the fan speed or changing the number of operating units, the characteristic point will move at 90 degrees to the Af line in an arc. Two cases are indicated in the diagram. In the top one, first air management is improved and then fan speed (and Af) is reduced to save energy. The lower one has a shortfall of air supply with Af around 0.3, therefore first air volume is increased (through increasing fan speed or adding CRAH units) to increase Af to approximately 1.2, followed by air management improvements. In this manner, the diagram can be used as a predictive tool to understand and manage the air performance in the data hall following changes to the supply / demand flows in the space. Air segregation efficiency (ASE) may be defined as follows: ASE = (1! BP)2 + (1! R) 2 2 In ideal conditions, where BP and R = 0, ASE max = ( ) 2 = 2 2 =1 Temperature surveys have been undertaken in a range of data halls and results from are shown on the diagram below. Blue points indicate results from open, legacy data halls without containment, points in red indicate results from data halls with containment and the average result is shown with a yellow point. The results show a range of performance with examples of poor and better air performance with both open and contained systems. The average result with BP = 0.53, R = 0.30, Af = 1.48 and ASE = 60% indicates that most data halls still have further scope for improvement to better manage and segregate their air. Although the sample size is small, there is a trend that contained systems generally have less recirculation, compared with the previous average reported (see Figure 3). With increasing temperatures and load densities, operators have focussed on reducing recirculation, to the detriment of BP and Af. The average ASE has shown a slight improvement (0.55 to 0.60). Most data halls are oversupplying air but in a few cases there is an undersupply. Page 8 of 11

9 Figure 7 Air performance results In some data hall environments there is a split in responsibilities between different stakeholders, for example in a colocation facility the client may be responsible for the IT racks and everything inside, whereas the host provides the room and manages the environmental conditions up to the rack (supplies air within the required range). Air management responsibilities inside and outside of the rack are controlled by different parties [xi]. Work is ongoing to apply these metrics to each of these distinct areas, i.e. analysis of air segregation performance within the rack and outside of the rack. Figure 8 Air flows inside and outside of rack Page 9 of 11

10 4 Control of CRAH fans in contained systems The recommended method of controlling CRAH fan speeds with contained systems is to use differential pressure control to maintain a fixed, slight positive pressure between the cold and hot aisles. If the pressure is too high, this results in excess bypass the excess air is pushed through the gaps and any servers which are off. If the pressure is negative, there is a risk of recirculation, with hot air passing in the wrong direction back into the cold aisle. 5 Conclusions Optimisation of data hall air performance enables operational energy savings [xii] and increases the opportunities for free cooling operation [xiii]. As energy efficiency best practice [xi] becomes more widely adopted, there is a desire within the industry to develop metrics which help better quantify environmental performance [xiv]. As demand grows, it is important that the industry continues innovating in order to address the increasing environmental impact of data centres, including its wider environmental impacts, e.g. embodied carbon in hardware [xv, xvi]. The air management metrics which have been developed can be used as a tool to quantify air performance, diagnose problems, predict the impact of changes and to continuously monitor improvements. The results recorded to date indicate that the installation of air containment systems alone does not guarantee good air separation. The principles behind the solutions need to be understood to ensure these are properly implemented and operated, e.g. controls are optimised and the operator derives the most benefit. There is still some way to go before best practice is fully understood and data centre energy management is optimised, however the case for investment in this area continues to strengthen. References i DataCenter Dynamics. DCD Industry Census 2013: UK Figures, DatacenterDynamics Focus, 3(32) (2013), pp ii The Green Grid. Energy policy research & implications for data centres in EMEA. White Paper #44 (2012) Available from: [Accessed: 19/11/13] iii ASHRAE, Thermal Guidelines for Data Processing Environments Expanded Data Center Classes and Usage Guidance, 2011 iv Robert Tozer, Munther Salim, Chris Kurkjian, Air Management Metrics in Data Centres, ASHRAE Transactions, vol. 115, part 1, Chicago meeting, CH , 2009 v Flucker S, Tozer R, Data Centre Cooling Air Performance Metrics, CIBSE Technical Symposium, Leicester, CIBSE, 2011 vi Operational Intelligence Air Performance Tool [Accessed: 19/11/13] vii Robert Tozer, Martin Wilson, Sophia Flucker, Cooling Challenges for Mission Critical Facilities, Institute of Refrigeration, Proc. Inst. R viii ASHRAE, Datacom Equipment Power Trends and Cooling Applications, 2nd Edition, 2012 Page 10 of 11

11 ix Flucker S, Tozer R, Scalable Data Centre Efficiency, CIBSE Technical Symposium, Liverpool, CIBSE, 2013 x Monroe, M., Stabinski, M., The power limit of servers, DatacenterDynamics Focus, 3(28) (2013), pp (originally from McKinsey s). xi European Commission, Best Practices for the EU Code of Conduct on Data Centres, version 4.0.5, 2013 xii Flucker S, Tozer R, Minimising Data Centre Total Cost of Ownership Through Energy Efficiency Analysis, CIBSE / ASHRAE Technical Symposium, London, 2012 Flucker S, Tozer R, Data Centre Energy Efficiency Analysis to Minimize Total Cost of Ownership, Building Services Engineering Research Technology 34(1) , DOI: / , 2013 xiii Tozer R, Flucker S., Zero Refrigeration for Data Centres in the US, ASHRAE Technical Paper, San Antonio, 2012 xiv The Green Grid. Proxy proposals for measuring data center productivity. White Paper #17 (2009) Available from: [Accessed: 19/11/13] xv The Green Grid. Data centre life cycle assessment guidelines. White Paper #45, v2 (2012) Available from: [Accessed: 19/11/13] xvi AVNIR LCA Conference, Whitehead, B., Andrews, A., Maidment, G., Dunn, A. (2012), The screening life cycle assessment of a data centre, Page 11 of 11