NUMERICAL MODELLING METHOD FOR THE DESIGN OF DATA CENTRE COOLING AMIN AZARMI Mechanical Engineer Aurecon amin.azarmi@aurecongroup.com MOSTAFA MAHDAVI Research Fellow Department of Mechanical and Aeronautical Engineering University of Pretoria AHMAD ERFANI MOGHADAM Research Assistant Institute for Frontier Materials Deakin University ABOUT THE AUTHORS Amin is Mechanical Engineer with 14 years of experience in Designing and Modelling of Building Services of major buildings & infrastructure in Middle East and Australia. He is specialist in HVAC system design, Computational Fluid Dynamic (CFD) modelling of the HVAC systems, Energy and Thermal modelling and analysis. Amin is Green Star Accredited Professional and NABERS Accredited Assessor. He is passionate about innovative HVAC system design & digital engineering to develop sustainable construction. Mostafa is a research fellow in the Department of Mechanical and Aeronautical Engineering at the University of Pretoria. He received his master's degree in Energy Conversion from Shiraz University in Iran. His research interests include heat and mass transfer, natural and force convection, turbulence, Nano fluids, particle migration and multiphase flow modelling. Ahmad is a research assistant in the Institute of Frontier Materials (IFM) at the Deakin University. He received his master's degree in Applied Mechanic of system design from Ferdowsi University of Mashhad in Iran. His research interests include Thermal & Fluid system design, Finite Element Analysis (FEA) and Computational Fluid Dynamic (CFD) modelling of the systems, and system optimisation. ABSTRACT Theoretical modelling analysis can couple with design decisions prior to the detailed design and construction of the ventilation systems for data centre construction. Computational fluid dynamic (CFD) tools effectively solve the governing flow equations for the data centre based on arrangements of equipment, the room and air conditioning standards. The flow was considered highly turbulent and two equation models were utilised for simulations due to high velocity and mixing. Due to many geometrical parameters involved in data centres, the main focus of this paper was concerned to less 1
studied areas such as failure of units and partitioning the aisles. Indeed, parameters such as supply to return heat index and return temperature index were introduced to understand functional performance of the cooling system within the simulation modelling. The application of the CFD in the data centre analysis would also be applied to study several failure scenarios of the CRAC and CRAH units enabling designers to optimise the control system for efficiency, as well as the least risk of equipment damage. In addition, temperature distribution was analysed to avoid maximum temperature in the case of failure. Keywords: Data Centre, CFD, CRAC/CRAH, Failure Scenario, Simulation Modelling INTRODUCTION The main centre of each information technology facility, such as computer servers and data storage systems, is undoubtedly a data centre (DC) housing a large number of electrical equipment. However, the efficiency of the servers highly depends on the environment temperature and humidity, and any damage or interruption in computer servers would cause huge costs for the company [1,6]. Hence, air conditioning and cooling the servers are a critical task in a DC. It is worth mentioning that some practical methods were suggested to improve the cooling system and optimise the energy demand in DC [7 9]. Computational fluid dynamic (CFD) is one of the strongest tools for predicting the efficiency and applicability of cooling systems in a DC. A large number of experimental and numerical studies on DC were performed in recent year [10,11]. Since simulation of the entire DC is somewhat expensive, a number of simply numerical and analytical approaches were proposed by researchers [12 14]. Schmidt et al. [13] applied a twodimensional depth-averaged model by only considering the raised floor and the resistance in upstream. Their obtained results were in good agreement with experiment considering the fact that the two-dimensional model can be recommended for 0.26m as the raised floor height. Karki et al. [14] modelled the same DC as Schmidt et al. [13], but in a three dimensional arrangement. They employed an empirical correlation for the resistance coefficient in tiles based on opening area. It should be mentioned that the research on thermal performance of DCs is still at the beginning stages in terms of numerical methods, and particularly three-dimensional models. Additionally, plenty number of geometrical and boundary parameters are involved, and further studies are required. In this paper, the results of the CFD simulations are initially compared with experimental measurements conducted by Schmidt et al. [13]. 1. GOVERNING EQUATIONS AND BOUNDARY CONDITIONS 1.1 Fluid flow equations Airflow is assumed to be turbulent in the entire DC; however, the flow patterns can considerably vary from the raised floor to the inner regions of aisles. The other assumptions are incompressible and single-phase flow being solved under the steady state condition. The flow is governed by three fundamental equations as conservation of mass, momentum and energy as follows [3-5]: u x i i 0 (1) 2
u u P u u u g S i i j i j i m x j xi x j x j T 1 T u k c T u q x j cp x j x j j L P j s where i and j are Einstein notation varying from 1 to 3 in 3D dimensions. For instance, ui represents u1 (velocity in x direction), u2 (velocity in y direction) and u3 (velocity in z direction). k L, Sm and q s are laminar thermal conductivity, source terms for momentum and energy equations, respectively. Due to transient nature of turbulent fluid, Reynolds-Average Navier stokes method is utilised to solve turbulent flow at steady state. Therefore, velocity and temperature are broken down into two parts, the time-averaged value ( u,t ) and the fluctuating part ( u,t ). One of the common approaches which can be applied to give an account for fluctuating velocity and temperature is to define turbulent viscosity and thermal conductivity. At this stage, turbulent k model is used to provide the ui general form for Reynolds stresses as uu i j t, known as Boussinesq hypothesis [4]. Two x additional transport equations for the turbulent kinetic energy (k), dissipation rate (ɛ) are solved to obtain the features of turbulence and turbulent viscosity. The transport equations and turbulent viscosity are as follows [5]: j (2) (3) k ( ku ) G G Y S x x x t i k b M k i i k i 2 t u C G C G C S x x x k k i 1 k 3 b 2 i i i (4) (5) 2 k (6) t C tc p k t Pr t (7) Where Pr t is Turbulent Prandtl number with fixed value of 0.9. Since the airflow is weak on top of the racks, the effects of buoyancy force need to be observed in equations. The coupling between momentum and energy equations is conducted by replacing the air density with Boussinesq s approximation as follows: (T T ) (8) ref ref Finite volume method along with staggered grid were utilised to iteratively solve the flow and transport equations. The SIMPLE algorithm developed by Patankar [3] was employed to couple momentum and energy equations based on accuracy of continuity equation in each iteration. 3
Flow rate (L/s) Flow rate (L/s) Flow rate (L/s) Flow rate (L/s) a) b) Figure 1. Two sets of experiments are considered for validation purposes. In each set, one of the CRAC units will be shut down. 100 Row 1 100 Row 2 50 50 0 0-50 Experiment Numeric -50 Experiment Numeric -100 90 5 10 15 5 10 15 Tile number Tile number a) b) Row 3 Row 4 100 60 50 30 0 0 Experiment Numeric -50 Experiment Numeric 5 10 15 Tile number c) d) Figure 2. Volume flow rate through each tile when CRAC A is off. Experiment by Schmidt et al. [13] 1.2 Turbulent wall treatment and boundary conditions 5 10 15 Tile number The boundary conditions for CRAC units consist of volume flow rate, the capacity of the unit and desirable cold temperature. However, Patankar [16] pointed out the desirable temperature could be less than this value and even could be close to 13 C. Two different servers are assumed in each rack, -100 4
one with volume flow rate of 0.14m 3 /s and heat generation of 1700W, and the other one with 0.1m 3 /s and heat generation of 1200W. 2. PHYSICAL MODEL FOR VALIDATION The first step in any CFD calculation is to ensure the capabilities of the method in modelling of this specific case. The experimental findings by Schmidt et al. [13] are borrowed to compare with the numerical results. Two CRAC units are considered in front of each other with four rows of perforated tiles and 15 tiles in each row. The height of the raised floor and the size of the tiles are 284mm and 600mm 600mm, respectively. The configuration of the DC is observed in Figure 1, including CRAC units, perforated tiles, the tube next to CRAC-A as an obstacle and stanchions at the corners of the tiles. The results of the simulations are compared with experimental measurements by Schmidt et al. [13] in Figure 2. There is good agreement between results with average deviation of 20% which could be accurate for this type of simulations with many geometrical parameters as well as further simulations with heat transfer presented in the next sections. 3. GEOMETRY PRESENTATION FOR MODELLING AND BOUNDARY CONDITIONS The entire size of the data centre for modelling is considered 15300mm 4770mm with 3500mm height. Fifty-two (52) racks are located in the domain, each twenty-six (26) racks in an isolated room enclosed by glass. Each rack cabinet contains two servers. Two of the CRAH units are always on duty and the other two starts running in the case of failure. The height of the raised floor and ceiling void is approximately 1100mm. The dimensions of the data centre are presented in Table 1, with schematic and geometry of the physical model in Figure 3. Base holders with diameter of 100mm are used at each corner of perforated tiles and also beneath the racks. 4. SIMULATION RESULTS AND DISCUSSION In the current study, several grid sizes were tested to ensure the independency of the simulation on number of cells. The resultant flow rates for two rows of tile in one cold aisle are shown in Figure 6 in terms of air flow (L/s). Finally, the grid size of 2.2 million cells were chosen due to the predictable slight change of accuracy in comparison to smaller grid. One of the most important tasks is to ensure that temperature at the server intake remains in the range of desirable values. Hence, Rack Cooling Index (RCI) [15] is introduced to compare the observed temperatures with those recommended in guidelines such as ASHRAE [2]. The recommended and allowable values are summarised in Table 2. RCI is calculated as follows: RCI 1 Total over temperature Max allowabletemperature (9) Supply air diffuser size Cold-aisle width Hot-aisle width 600mm 600mm 1200mm 600mm 5
CRAH unit power 142kW Server power density 2.33kW Number of servers 104 Supply temperature 24.5 o C Ceiling exhaust grilles size 600mm 1200mm Table 1. Details of the data centre sizes. (a) (b) Figure 3. (a) Schematic illustration of racks, raised floor and ceiling void. (b) Top view of the entire data centre. (c) Computational domain of the numerical model Total over temperature is the intake temperature of those servers above maximum recommended. The summation is calculated for the use of RCI. The other terms on the bottom is the multiplication of rack number and the difference between maximum allowable and recommended temperature. The graphical presentation of RCI is illustrated in Figure 4. The rectangle area between max allowable and recommended temperature presents the max allowable amount in RCI. The other important parameter is Return Temperature index (RTI). RTI is the measure of recirculation of hot air in hot aisle to cold aisle and also by-passing of cold air in cold aisle to the CRAH unit without passing through the servers. It is defined as the difference between supply and extract air temperature in air handler over the induced temperature difference in servers. RTI can be also regarded as the proportion of airflow in servers and air handlers. Textract Tsup ply Flowin servers RTI or RTI (10) T FlowinCRAH units server Both RTI and RCI are utilised to evaluate the capability and efficiency of the cooling system in the DC in various circumstances, especially by the drop of over of CRAH units. (c) 6
Min and max Min and max allowable recommended 18 o C 27 o C 15 o C 32 o C Table 2. Temperature values from guidelines (ASHRAE) 4.1 Failure scenarios Figure 4. Parameters for definition of RCI Two cases of failure scenarios are shown in Figure 5, CASE B and CASE C. The results of the failure situations are compared with the no failure CASE A in Figure 6. At this range of RCI values (60% to 70%), the inlet temperature of the servers are still below max allowable temperature. It guarantees uninterrupted working of the DC by avoiding overheating. Nonetheless, the intake temperature remains above max recommended values, meaning of poor rate of thermal performance. As can be seen, the trend in both RCI and RTI shows that the failure situation is properly handled by the CRAH standby units in the opposite side. Figure 5. Cases of failure scenarios in this study 7
RCI (%) RTI (%) 70 68 Rows numbers 41 40.8 66 40.6 CASE A CASE B CASE C 64 40.4 62 CASE A CASE B CASE C 40.2 60 1 2 3 4 Rack rows number (a) (13 rack cabinets in each row) Figure 6. Variation of RCI and RTI for each rack in the case of failure (the rows number has been annotated by the box). % opening area RCI RTI 10 65.2 40.34459 15 65.2 40.34459 25 65.35 40.37339 35 65.95 40.30638 45 66.7 40.37813 60 70.715 40.34941 Table 3. Effects of diffusers opening area on RCI and RTI for the entire DC 4.2 Tile opening area and aisles partitioning 40 1 2 3 4 Rack rows number One of the important geometrical parameters is the percentage of the opening area for raised perforated tile (diffusers), from 10% to 60%. The impact of diffuser opening area is presented in Table 3, and it illustrates the RTI values is slightly affected by changes in opening area. On the other hand, the thermal efficiency or RCI would enhance by 7.1% with an increase in opening area from 10% to 60%. The other method of improvement in thermal efficiency is to add partition to the DC room. Hence, the cold aisles will be isolated from the rest of the room and cold air should completely pass the rack. In this study, three cases are considered, first the entire DC room isolated from the CRAH units, second the entire DC room and cold aisles isolated, third only cold aisles isolated from the rest of the domain. The results for RCI and RTI values are presented in Figure 7. The ideal value for RCI is 100%, meaning the server intake temperature is similar to recommended one. This happens when cold aisle partition is employed in the design of the DC. Moreover, the absence of glass walls around the racks have no impacts on the thermal performance of the system. On the other hand, adding partition to the cold aisles does not significantly change the RTI, not related to the isolated room. The reason can be understood from Figure 8. Due to leakage in each server, the cold airflow partially passes the gap between servers in the rack and remains untouched from the generated heat. Then, it is mixed with the air in the hot aisle and reduces the return temperature to the CRAH unit. (b) 8
(a) (b) (13 rack cabinets in each row) Figure 7. RTI and RCI values for the case with and without cold aisle partition. Figure 8. Leakage of airflow through the gap between servers and mixing with hot air CONCLUSIONS Thermal performance and efficiency of a data centre with four rows of rack were numerically studied. Firstly, the robustness of the model was confirmed with the experimental measurements of the literature and subsequently, a new geometrical model was thermally analysed. The main criteria were considered the intake and outlet temperature from the racks and CRAH units as non-dimensional indexes RCI and RTI. Two failure scenarios were investigated by running the standby CRAH units in the opposite side of the main CRAH units. The results showed that the standby CRAH units could properly manage the thermal distribution in the DC in the case of the main CRAH interruption, without any relationship with the arrangement of the air handler units. However, the thermal performance of this arrangement has not met the expectations. In another situation, it was suggested 9
that thermal performance represented by RCI increased only by 7.1% due to expanding the opening area up to 60%. While, installing partition on cold aisle section could improve the thermal performance up to 100%, with no need for isolating the entire room with racks. In other words, it is recommended to only isolate the cold aisle instead of isolating the entire DC. This will prevent extra costs regarding partitioning the entire area. As a future study, different arrangement of CRAH units with respect to racks is analysed and changes are made in the racks to ensure passing maximum cold air going through the servers with optimum RCI and RTI. Furthermore, the energy analysis of the DC will be investigated. 10
(a) Books and handbooks REFERENCES 1. J. Dai, M.M. Ohadi, D. Das, M.G. Pecht, Optimum cooling of data centers, Springer, 2014. 2. Ashrae, Thermal guidelines for data processing environments, ASHRAE, 2015. 3. S. Patankar, Numerical heat transfer and fluid flow, CRC press, 1980. 4. J.O. Hinze, Turbulence McGraw-Hill, New York. 218 (1975). 5. B.E. Launder, D.B. Spalding, Lectures in mathematical models of turbulence, Academic press,london, England, 1972. (b) Journal articles and papers 6. Y. Joshi, P. Kumar, Energy efficient thermal management of data centers, Springer Science & Business Media, 2012. 7. H. Zhang, Z. Shi, K. Liu, S. Shao, T. Jin, C. Tian, Experimental and numerical investigation on a CO 2 loop thermosyphon for free cooling of data centers, Appl. Therm. Eng. 111 (2017) 1083 1090. 8. Z. Wang, X. Zhang, Z. Li, M. Luo, Analysis on energy efficiency of an integrated heat pipe system in data centers, Appl. Therm. Eng. 90 (2015) 937 944. 9. Y. Ma, G. Ma, S. Zhang, F. Zhou, Cooling performance of a pump-driven two-phase cooling system for free cooling in data centers, Appl. Therm. Eng. 95 (2016) 143 149. 10. J. Ni, X. Bai, A review of air conditioning energy performance in data centers, Renew. Sustain. Energy Rev. 67 (2017) 625 640. 11. S. Alkharabsheh, J. Fernandes, B. Gebrehiwot, D. Agonafer, K. Ghose, A. Ortega, Y. Joshi, B. Sammakia, A Brief Overview of Recent Developments in Thermal Management in Data Centers, J. Electron. Packag. 137 (2015) 40801. 12. H. Fernando, J. Siriwardana, S. Halgamuge, Can a data center heat-flow model be scaled down?, ICIAFS 2012 - Proc. 2012 IEEE 6th Int. Conf. Inf. Autom. Sustain. (2012) 273 278. 13. R.R. Schmidt, K.C. Karki, K.M. Kelkar, A. Radmehr, S. V. Patankar, Measurements and predictions of the flow distribution through perforated tiles in raised-floor data centers, in: Proc. IPACK 01 Pacific Rim/ASME Int. Electron. Packag. Tech. Conf. Exhib., Kauai, Hawaii, USA, 2001: pp. 1 10. 14. K.C. Karki, A. Radmehr, S. V Patankar, K.C. Karki, Use of computational fluid dynamics for calculating flow rates through perforated tiles in raised-floor data centers, HVAC&R Res. 9 (2003) 153 166. 15. M. Herrlin, R.G. Kluge, Thermal management in telecommunications central offices: The next steps, in: Telecommun. Energy Conf. (INTELEC), 32nd Int., 2010: pp. 1 8. 16. S. V. Patankar, Airflow and Cooling in a Data Center, J. Heat Transfer. 132 (2010) 73001. 11