PERFORMANCE OF TEMPERATURE CONTROLLED PERIMETER AND ROW-BASED COOLING SYSTEMS IN OPEN AND CONTAINMENT ENVIRONMENT

Transcription

1 Proceedings of the ASME 2015 International Mechanical Engineering Congress and Exposition IMECE2015 November 13-19, 2015, Houston, Texas, USA IMECE PERFORMANCE OF TEMPERATURE CONTROLLED PERIMETER AND ROW-BASED COOLING SYSTEMS IN OPEN AND CONTAINMENT ENVIRONMENT Kourosh Nemati State university of New York at Binghamton-SUNY Binghamton, NY, USA Husam Alissa State university of New York at Binghamton-SUNY Binghamton, NY, USA Bahgat Sammakia State university of New York at Binghamton-SUNY Binghamton, NY, USA ABSTRACT The continuous increase of data center usage is leading the industry to increase the load density per square foot of existing facilities. High density (HD) IT load per rack demands bringing the cooling source closer to the heat load in contrast to room level air cooling. For high density racks, the use of in-row cooling systems is becoming increasingly popular. In-row cooling can be the main source of cooling for a data center or work jointly with perimeter cooling in what is called a hybrid cooled room level system. Also, hot or cold aisle containment can be integrated with perimeter cooling and used throughout the data center to reduce the mixing of hot and cold air. Currently, there has not been much work comparing the performance of in-row cooling in open versus contained environments. The present work builds on a previous study where the interaction of perimeter and row-based cooling was evaluated for a cold-aisle containment (CAC) environment. Previously, the benefit of using row-based cooled in an aisle has not been compared with an aisle in open conditions. Here, we numerically investigate the performance of in-row coolers in both opened and coldaisle contained environments. Groups of IT equipment that differ in air flow strength are used to provide the heat load. Empirically measured flow curves for common IT equipment are employed to provide simplified models of the IT equipment in the CFD software used. The steady state analysis includes information provided in the manufacturer s specifications such as heat exchanger performance characteristics. The model was validated using a new data center laboratory with perimeter cooling. A single aisle of the data center is modeled to reduce the computational time, and the results are generalized. The cold aisle contains 16 racks of IT equipment distributed on both sides. In addition, the aisle contains 2 power distribution units. Full details are incorporated in the computational model. A single Liebert CW114 CRAC unit provides the perimeter cooling in the data center. The model captures the particular air flow behavior in the cold aisle when row-based cooling is utilized. Correlations are derived to predict the ability of air cooling units to maintain set points at different air flow rates. The effect of leakage is also considered. NOMENCLATURE ACU: Air Cooling Unit [CRAH, in-row, over-head and etc.] CAC: Cold Aisle Containment CFD: Computational Fluid Dynamics CRAH: Computer Room Air Handler (Chilled Water) HD: High Density K: Controller curve slop R: Three-way valve ratio T : Temperature ( C) ρ : Density ( kg ) m R 3 m : Valve ratio at specific CRAH airflow rate R min : Minimum value to satisfy set point temperature at specific room heat load ṁ: Mass flow ( kg s ) ε c : Cooling Effectiveness (air side) INTRODUCTION The rapid growth of data center capacity has necessitated the increase of load density per square foot. High density IT loads can be addressed by bringing the cooling source closer to the heat load. In-row coolers provide one method of localized cooling. In-row cooling can be the only source of cooling in the data center or they can work parallel with perimeter cooling in what is referred to as a hybrid cooled room level system. This 1 Copyright 2015 by ASME

2 approach is well-suited to treating hot spots. In-row coolers can provide more uniform cold air for High Density (HD) IT load racks. In addition, containment solutions have been integrated with perimeter cooling and used globally since they reduce the mixing of hot and cold air in the facility. In-row coolers are becoming commonly used in recently designed data centers. A disadvantage is that they typically occupy about one half of a rack width. In the case of perimeter-based cooling system, cooling air is provided through the plenum under a raised floor and delivered to the IT equipment vertically via perforated floor tiles. In-row air delivery comes out at 180 from the IT inlet and circulates back. Containment is a rapidly growing cooling solution in data centers, as it reduces mixing of hot and cold air in the facility. According to Gartner [1] about 80% of current data centers are using containment systems or are planning to install them. In-row technology and containment can be combined to form a very strong cooling solution for HD data centers. The containment systems can be installed on cold and/or hot aisles. In order to characterize cold-aisle containment, a multidimensional sensor array was used in [2] in order to measure air flow under different configurations: open aisle, partially contained aisle and fully contained aisles. This study recommended the use of over provisioning for fully-contained aisles or at least a partial containment. Based on the experimental results it was shown that either an under-provisioned or over-provisioned top only contained configuration is preferable in comparison with other partial containment systems. Dunlap et al. [3] concluded that the legacy room-oriented cooling is more effective and practical for lower density data centers. They showed row-oriented and rack-oriented cooling architectures are more successful with operating densities of 3 kw per rack or higher. It was concluded that these cooling systems can provide more flexibility and predictability, and reduce electrical power consumption and TCO. In [4] it was discussed that in-row cooling systems can be used as the only source of cooling for an entire data center. In this study an extensive CFD analysis was used to validate using pure row cooling systems is more beneficial in compare with a hybrid coolers and row cooler. Namek [5] presented main in-row products based on coolant, brand, capacity and mount location. Iyengar et al. in [6] mentioned that numerical models are one of the most popular techniques for designing new data centers. They investigated numerical error of temperature, flow rate and power of a simulated server rack in a small room under steady state condition. It was concluded the dominant sources of error are the use of the standard k-e turbulent model and uniform assumption of air flow at the exhaust airflow of the simulated rack. A simplified server model (SSM) that is based on an innovative black box approach is employed for incorporating the transport through the servers that does not require meshing the regions of the cabinet containing servers [7]. Alissa et al. [8] validated a numerical model of new research data center at Binghamton University. In the CFD model, they addressed details such as floor pedestal resistance, raised floor leakage, the use of CRAH flow curves, CRAH air supply vent locations and scoop and rack detailed structure. Each part of the model was experimentally validated. The effect of leakage under the racks on performance of the lowest server was investigated by using infrared thermography in [9]. Also, it was mentioned that in most CFD studies, insufficient grid resolution in capturing leakage can have significant effect on the results. In [10] the effect of cold aisle containment on increasing the ride through time in cooling failure cases was studied. In a specific test case, cold aisle containment results in a ride through time that is almost five times longer in comparison to the same test case without containment. Here, we numerically investigate the performance of in-row coolers in both opened and cold-aisle contained environments. Based on the model results, we will show that when extra cooling capacity is needed to handle ancillary loads the use of row-based coolers is a practical approach; this is one of the advantages of redundancy in row-based cooling configuration. In addition, the effect of leakage under the racks is addressed. NUMERICAL MODEL The layout of the Binghamton University data center lab is shown in fig. 1. In this data center 41 racks are distributed in four aisles. Aisles C and D have cold aisle containment and are fully populated with IT equipment. The raised floor depth is 0.91m(3 ft). A cutout is located at the right side of the room beside aisle D with depth 0.305m(1 f t). The ceiling height is 4.26m(14 ft) from the raised floor. The data center is cooled by a perimeter CW114 Emerson-Liebert unit (1930mm Height x 3099 mm Width x 889 Depth) that is located in front of aisle C. This unit provides cold air with three output vents. For the modeling in this study, aisle C was selected as a slice of whole data center to conduct the numerical study effectively (Fig. 1). This aisle contains 16 racks in two rows. A detailed model is constructed for each rack and 18 perforated tiles with 22% perforation are treated using a lumped model. All the components used in data center model are calibrated based on room level experimental validation [11]. FIG. 1: Selected numerical domain 2 Copyright 2015 by ASME

3 Using the section of the data center mentioned above, two numerical models were employed in this study. In the first model only the perimeter cooling system is employed (fig. 2). Containment was applied on the cold aisle. A 2.5 mm space between the CAC and rack frames was introduced as a source of leakage in the CAC. The CAC doors have three sources of leakage: under the doors (0.066 m 2 ), on the top of the doors (0.006 m 2 ) and between the doors (0.02 m 2 ). The main source of leakage in the CAC is under the racks(0.232 m 2 all along the containment). This leakage has a significant impact on the inlet temperate of the racks, especially on the IT near the bottom of the racks. The leakage dimensions are defined based on the real facility leakage of the data center lab. In modeling the leakage, care was taken to properly refine the mesh in the leakage regions to capture the variation in the flow field. In the data center, each rack is fully populated by twenty-one 2U Dell PowerEdge 2950 servers, each server generates 510 W that is assumed to be totally converted into dissipated heat. Under normal operation the rotational speed of the server fans is 7200 RPM. A simplified server model is used for the simulations and an experimentally measured effective flow curve is applied to the outflow of the server to correlate pressure and flow rate. The operating point of the flow curve for this fan speed is about m3 s (54 CFM). FIG. 2: Perimeter cooling system In the second numerical model, six in-row units are installed in staggered arrangement and the CRAH unit and the raised floor were removed. Sensible cooling curves are used to simulate the thermal performance of the in-row heat exchanger. The maximum cooling capacity of each in-row cooler is 28.7 kw [12]. They each have 8 blowers and flow curves were applied with the operating point being 0.22 m3 s. The supply air temperatures were controlled at 20 C with a dead band range of ± 0.1 C for both ACUs. The temperature sensors that are applied on the return of the ACUs provide signals to the controller systems. The chilled cooling water is controlled by a three-way valve which is characterized by the ratio of the current cooling capacity provided by the ACU to the maximum cooling capacity that can be provided by the cooling unit at the specified return temperature. The three-way valve controls the water flow rate of the cooling coil by shutting off water flow in pipe while opening water flow in a bypass pipe. FIG. 3: Perimeter cooling system RESULTS AND DISCUSSION Effect of containment on perimeter and row-based system As discussed above, effect of containing cold aisles in the legacy cooling scheme has investigated in previous studies. However, effect of containment on localized cooling systems (i.e. with in-row coolers) has not been studied. Fig. 4 shows the average inlet temperatures of the racks with containment and with an open cold aisle for both perimeter and row-based cooling systems. In all four cases, the racks were populated with twenty one 2U servers that are generating 171 kw. To decrease the effect of leakage recirculation on rack inlet temperatures all four systems are at 100% provisioning. When the open and contained perimeter systems are compared, the racks closer to the CRAH on both sides experience higher inlet temperatures that is due to hot air recirculation when the containment is removed. For the row-based case with containment, only the locations of racks C2-1 and C1-8 are subjected to higher inlet temperature. These two racks are adjacent to an in-row cooler and there is no cooling system in front of them. By removing the containment of the cold aisle, the average inlet temperature of all 16 racks increase significantly. In order to investigate the flow behavior in cold aisle, the velocity contours for both containment and open systems at 1m above the floor are shown in fig. 5. The velocity contours illustrate the signature of the horizontal jet coming out from the in-row blowers. The staggered arrangement of in-rows creates regions of high velocities that cause recirculation. This occurs in both the contained and open systems. In fig. 5a the jet trail of the airflow supply of the in-rows is evident. The higher air velocities are shown in red for the main velocity component x-direction. The edge in-row coolers loose a significant portion of their supply airflow rate to the room. Fig. 5b shows that by containing the cold aisle, the edge in-rows supply flow rate is restricted from leaving the cold aisle. This explains the zero stagnation points at the doors of the 3 Copyright 2015 by ASME

4 which explains the overall increase in rack inlet temperature as it was shown in fig. 4. In the contained case, the hot air recirculation source is eliminated by the containment roof. It can be seen in fig. 6a the pressure difference range in the cold aisle is lower in compare with the un-contained case. Different type of in-row arrangement may decrease the negative pressure in the cold aisle, but the in-rows can t maintain the cold air supply for most of the cases. FIG. 4: Average inlet temperature of racks for perimeter and rowbased cooling systems in contained and open aisle cold aisle. The non-uniform air flow pattern of the in-rows induces discrepancies in pressure. Pressure differentials are used to estimate the level of provisioning in CAC. However, the nonuniformity in the pressure field may yield inaccurate estimations. FIG. 5: Air Velocity contours Fig. 6a and 6b show the pressure contours of the uncontained and contained aisles at 1m above the floor. In both cases it can be seen that the pressure is highly non-uniform. When the supply jets of coolers collide, a region of high positive static pressure is created and is shown in red. At the regions between the jets, vortices are formed. The figures show three main vortex cells in the cold aisle. In the middle of each vortex, a pressure wake is formed in which negative pressure occurs. Those negative pressure regions can potentially draw hot air from the room. In the case of open system, the hot air is drawn from the room and mixed with the cold air in the aisle FIG. 6: Air Pressure contours Two vertical slices of the cold aisle show the pressure distribution relates to the flow structure. Fig 6a shows the location of the cuts, the red line on top is cutting through the positive pressure region, while the bottom line cuts through the negative pressure region. Fig. 14a demonstrates the velocity vector at the positive cut in the contained in-row system. The velocity vectors show the recirculation on top of the containment, but in this case the containment ceiling is blocking mixing between hot and cold air. On the other hand, Fig. 14b shows the flow pattern at the positive pressure cut for an open in-row system. It can be observed that the penetration of hot air stream occurs at the top of the cold aisle. Also, it is diluted as the height decreases. Fig. 15a and 15b demonstrate the flow pattern for the negative pressure cut in the contained system and open in-row systems. In both cases, the velocity vectors are directed downward in the middle of the vortex. Fig. 15a shows the cold air in the cold aisle containment is recirculating from top to bottom. The thermal impact of this flow behavior is minimized by containment. However, in the actual data center facility measuring pressure to determine the level of provisioning is a challenging task. Fig. 15b shows that the thermal impact of the negative pressure region is significant when the cold aisle is not contained. In this case, areas of hot air concentration are not limited to the upper region of the aisle, but rather infiltrates over the rack height. This explains why the temperatures difference of rack inlets were higher between open and contained row-based systems when compared to open and contained perimeter-based systems. Fig. 7a shows the pressure contours at 1m above the raised floor 4 Copyright 2015 by ASME

5 and the same level of provisioning. The pressure is more uniform in the cold aisle, hence all vertical cuts yield similar velocity patterns. In this case, fig. 7b shows that the vertical air flow supply limits hot air penetration from above the cold aisle. These results suggest that containing the ceiling of the cold aisle is very important for in-row based systems. FIG. 8: Average inlet temperature of racks at three airflow provisioning for both perimeter and in-row coolers FIG. 7: Air Velocity contours In order to further investigate this situation, three provisioning percentage for both perimeter and row-based cooing systems for the open systems were studied. Fig. 8 shows the average inlet temperature of the racks for 90% (under-provisioned), 100% (neutral-provisioned) and 110% (over-provisioned) air flow provisioning. The average inlet temperature for three provisioning percentage of perimeter cooling systems shows that although increasing the air flow provisioning decrease the inlet temperature of the racks but it is not a dominant factor and from 90% to 110% of provisioning the average rack inlet temperature only decreases by 1 C to 2 C. In in-row based cases, from 100% to 110% no significant temperature difference is observed and that is due to the extra cold air flow of the in-rows makes the local pressure discrepancies in cold aisle containment increases at higher levels of provisioning. Higher positive and negative pressure in the cold aisle cause more mixture of cold air of in-row blowers and room hot air. Over-provisioning can be used in perimeter systems to prevent recirculation on top of the rack, however, over provisioning a row-based aisle results in stronger jets from the coolers. By decreasing the level of provisioning to 90%, the positive pressure regions diminish and room hot air penetrates more the cold aisle that explains the significant increase in rack inlet temperature. Based on previous section, a fundamental difference is observed in the airflow behavior between the in-row and perimeter. Hence, each of the two systems is expected to follow a different pattern based on the type of IT mounted in the racks. To demonstrate the effect, another IT configuration was used in the numerical model. In this configuration, the racks were populated by thirty-four 1U Dell PowerEdge 1950 servers. Similar to 2U servers, the empirically measured effective flow curve of the 1U servers was applied for their outflow. The operating point of the flow curve is about m3 s (35 CFM), which is lower than 2U servers. The total air flow and power generation within the racks in both IT configurations are the same and this explains the number of 1U servers in each cabinet. The under-rack leakage was found to have largest effect on IT inlet temperature, which most directly affects the bottom servers. Fig. 9 shows the average inlet temperature of the lowest servers for the 16 racks with different under-provisioned arrangements. For both the perimeter and row-based systems, when the racks are populated with 2U servers, higher inlet temperatures for the lowest servers are obtained. This is explained by the higher negative pressure generated by the 2U servers when compared to operating point of the 1U servers. However, the type of IT is not the only factor that controls the leakage rate. Another important factor is the cooling system deployed. For an instance, the perimeter system has a more uniform leakage pattern when compared to the in-rows. Further discussion is carried out in the next section. Effect of leakage on Cooling system performance In previous section it was shown that the cold aisle in rowbased cooling has less uniformity in pressure through the aisle. This non-uniformity of local pressure gradient had a significant effect on the recirculation behavior in the open environment. On the other hand, in the containment, leakages are the main sources of recirculation. In this section, the effect of leakage is correlated to the cooling system deployed in the aisle. Due to the number of cases in terms of ACU, IT and provisioning arrangements, in this section, three cases will be presented. 5 Copyright 2015 by ASME

6 FIG. 9: Average inlet temperature of racks at three airflow provisioning for both perimeter and in-row coolers FIG. 11: Average inlet temperature of racks at three airflow provisioning for both perimeter and in-row coolers different behavior for the in-row based and perimeter cooling systems. This is mainly attributed to the resulting airflow direction. For the in-row case, the under-rack leakage brings surplus cold air into the room, which is shown in blue on the left in fig. 12 (the air flow direction is into the negative direction of x-axis). However, the under-rack leakage in the perimeter case results in entrainment. FIG. 10: Air Velocity contours First, the effect the cooling system has on leakage is discussed. Fig. 10a and 10b compare the x-velocity component at 22mm (mid-height under rack leakage) above the floor for perimeter and in-row based systems at 90% provisioning. From fig. 10, the velocity component has both positive and negative values ranging from 2.5 m s to -2.5 m s at the under rack leakage region. However, in case of the perimeter cooling system Fig. 10b, the velocity component at the leakage region is in one direction, which is into the CAC. For quantitative comparison, Fig. 11 shows the measured x-velocity values at six selected points in the leakage vicinity. This explains the behavior of the under-rack leakage in the row-based system where flow occurs in both directions, i.e. cold air escapes in to the room which reduces cooling efficiency and allows hot air to penetrate into the cold aisle. Second, at 110% provisioning the under-rack leakage shows FIG. 12: Air Velocity contours Third, when comparing the under-rack leakage for the inrow coolisg system betweeen 1U and 2U equipment, a different flow pattern is observed as was mentioned before. The results shown in Fig. 10a is an under-provesioned in-row system populated with 1U servers. The same conditions with the racks populated with 2U servers is shown in fig. 13. From a comparison of the two figures, it is evident that a higher amount of airflow leakage is observed into the CAC when the racks are populated with 2U servers. CONCLUSION In the current study, the performance of perimeter and inrow cooling systems was evaluated. Comparisons of pressure, 6 Copyright 2015 by ASME

7 FIG. 13: Average inlet temperature of racks at three airflow provisioning for both perimeter and in-row coolers velocity and temperature fields were made. The difference in the flow pattern between the two systems was used to characterize the effectiveness of each approach. Due to the non-uniformity in pressure and velocity in the in-row based cold aisle, larger recirculation is observed at the upper region of the row. Concentrated low pressure regions draw in higher temperature room air into the cold aisle increasing the average rack inlet temperature. Based on the results presented, at least ceiling containment is recommended when utilizing in-row coolers as the sole source of cooling. For both perimeter and row-based cooing, under-rack leakage was shown to have a significant effect on the average rack inlet temperatures. In the case of in-row coolers, the nonuniformity in the pressure field leads to improper air flow at the leakage locations. This behavior makes the prediction of leakage effect on IT harder to analyze. Finally, by utilizing the air side energy balance equation of the CRAH and the maximum cooling capacity curve equation, a correlation between airflow and threeway valve ratio of the chilled water is made. This correlation can be utilized to predict the upper airflow limit after which the ACU can no longer maintain the supply air temperature set point at a predefined data center load. ACKNOWLEDGEMENT We acknowledge the partial support from the National Science Foundation under Grant Also, we would like to acknowledge Prof. Bruce Murray from Binghamton University and Mark Seymour from Future Facilities for their advice and guidance in conducting this study. REFERENCES [1] S.K. Shrivastava, A.R. Calder, and M. Ibrahim. Quantitative comparison of air containment systems. In Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), th IEEE Intersociety Conference on, pages 68 77, May [2] V. Sundaralingam, V.K. Arghode, and Y. Joshi. Experimental characterization of cold aisle containment for data centers. In Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), th Annual IEEE, pages , March [3] K. Dunlap and N. Rasmussen. The advantages of row and rack-oriented cooling architectures for data centers. In White Paper 130, APC. [4] J. VanGilder and W. Torell. Cooling entire data centers using only row cooling,. In White Paper 139, APC. [5] Y. Namek, R. In row cooling options for high density it applications,. In TSS. [6] Hendrik Hamann Madhusudan Iyengar, Roger R. Schmidt and Jim VanGilder. Comparison between numerical and experimental temperature distributions in a small data center test cell. In ASME 2007 InterPACK Conference collocated with the ASME/JSME 2007 Thermal Engineering Heat Transfer Summer Conference, pages , July [7] D. King, M. Ross, M. Seymour, and T. Gregory. Comparative analysis of data center design showing the benefits of server level simulation models. In Semiconductor Thermal Measurement and Management Symposium (SEMI- THERM), th Annual, pages , March [8] H.A. Alissa, K. Nemati, B. Sammakia, K. Ghose, M. Seymour, and R. Schmidt. Innovative approaches of experimentally guided cfd modeling for data centers. In Thermal Measurement, Modeling Management Symposium (SEMI- THERM), st, pages , March [9] J. Barr von Oehsen Jim Pepin Yogendra Joshi Vaibhav K. Arghode Robin Steinbrecher Michael K. Patterson, Randall Martin and Jeff King. A field investigation into the limits of high-density air-cooling. In ASME 2013 International Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Microsystems, pages 68 77, July [10] Saurabh K. Shrivastava and Mahmoud Ibrahim. Benefit of cold aisle containment during cooling failure. In ASME 2013 International Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Microsystems, pages 68 77, July [11] H.A. Alissa, K. Nemati, B. Sammakia, M. Seymour, K. Schneebli, and R. Schmidt. experimental and numerical characterization of a raised floor data center using rapid operational flow curves model. In InterPACK 2015, March [12] H.A. Alissa, K. Nemati, B. Sammakia, A. Ortega, D. King, M. Seymour, and R. Tipton. Steady state and transient comparison of perimeter and row-based cooling employing controlled cooling curves. In InterPACK 2015, March Copyright 2015 by ASME

8 FIG. 14: Air Velocity contours FIG. 15: Air Velocity contours 8 Copyright 2015 by ASME

9 FIG. 16: Effectiveness and Three-way valve ratio of ACUs FIG. 17: Return and supply temperature of ACUs 9 Copyright 2015 by ASME