The Energy Advantages of Containment Systems David L. Moss Dell Data Center Infrastructure www.dell.com/hiddendatacenter
Summary The Uptime Institute estimates that the cost to build a new 20,000 square foot Tier 3 data center a facility with less than 1.6 hours of downtime a year is around US$26 million. However, due to changes in the global economy, funds for new data center construction are increasingly difficult to obtain, even as business demands on IT are increasing. Improved cooling techniques are essential to extend the life of existing data center facilities. The use of containment a method of separating supply (cold) and return (hot) air to more effectively cool the IT equipment in a data center -- ensures that room and equipment temperatures can be better managed. Containment allows for a reduction in cooling-related energy, much of which can be redeployed into either increasing the amount of IT equipment in the data center or refreshing older systems for more powerful and energy efficient IT equipment. Coupled with other techniques, containment can provide data centers with an energy reduction of greater than 20%. This paper presents data and analysis of a chilled water facility. A chilled water facility was chosen because it is easier to separately measure energy associated with the movement of air from energy associated with the chilling process. Overall facility efficiencies would be different for direct expansion (DX) equipment, but the improvement percentage estimates presented in this paper should be comparable. The Challenge One of the biggest challenges in the data center is determining if it has adequate cooling. Many data centers still maintain an open delivery-and-return method of cooling their facility. When the hot/cold aisle strategy was introduced, the industry started moving away from simply cooling the room and toward a strategy of cooling the racks directly, yet still in an open environment. While this strategy is an improvement, it is difficult to determine if it is being executed effectively: air often doesn t get where it s intended, and used hot exhaust air is often unintentionally ingested by the IT equipment. Remediation of hot spots is often addressed by simply adding more air handling units to increase the available airflow under the floor. Although this may fix hot spots, the resulting airflow increases elsewhere in the room may not have been needed and as an unintended consequence may unnecessarily waste the energy of the facility. In fact, it is not uncommon for data centers to be providing twice the air that is eventually consumed by the IT equipment. That s 100% waste at the CRAC units, energy that could have been used on IT equipment to get more work out of their facility. Containment can help address the challenges of hot spots and over-provisioning, challenges that if solved, can result in more IT gear deployed in the data center. Containment Definition Simply stated, containment is the intentional separation of either the supply (cold) or return (hot) air from, or to, the air conditioning system. Containment strategies are normally focused on either hot or cold air, but they can be on both. Return containment is probably more prevalent since it has been around the longest, but supply containment is gaining in popularity. Typical return containment systems are either aisle- or rack-based. In rack-based containment, each rack has a chimney; this usually is connected to a dropped ceiling which is also connected to the large raised floor units, CRACs or CRAHs. [Note: For the remainder of this paper, raised floor air conditioning units will be referred to as CRACs whether they are a true CRAC or a chilled-water-based CRAH.] Aislebased return containment is generally associated with InRow TM cooling and is a patented product from APC. Supply containment is most often done at the aisle by encapsulating the cold aisle on top and at the row ends. Although there are engineered systems, many data center owners are taking it upon themselves to come up with different methods to isolate the cold aisle. 2
No Containment Cold Aisle Containment Exhaust Containment Chimney Containment Advantages A containment system offers several opportunities to turn facility/cooling power into new IT systems; the examples below add up to over 20% regain in power that can be repurposed as new IT and its complement of cooling and power delivery. In some facilities, the opportunity can be much greater. Energy savings in servers: Without containment, hot spots are inevitable. By using containment, servers which would have previously sat in a hot spot can have their inlet temperatures reduced, which results in a lower level of fan power consistent with other equipment. IT equipment operating at consistently optimal cooler temperatures will require significantly less energy to power their internal fans. Energy savings in the facility chilling process: By using containment, the temperature delivered to IT equipment can be raised, and it s therefore possible to achieve significant facility energy savings. By ensuring that either supply or return air is isolated, containment systems provide for more consistent IT inlet temperatures. This is one of the most powerful advantages because once consistency is achieved; facilities can be empowered to raise the temperature of the air it delivers directly to the IT equipment. Resulting energy decreases in the chilling process and the elimination of energy wasted on moisture control can easily exceed 10% of the facility s overall electrical consumption. Energy savings in the facility air handling units: Containment allows for CRACs to be throttled or shut down. It is not uncommon for a data center without containment to find itself 60, 80, even 100% over-provisioned on air delivery in an attempt to cool hot spots. If a containment solution is introduced, the need for the over-provisioning is reduced. It now may be possible to turn off some CRACs or for the CRACs to throttle themselves if they are VAV equipped units. The energy savings opportunities associated with containment offer the ability to either add more IT equipment to the data center or to refresh IT equipment, both of which can increase the facility compute productivity. One of the intangible and immediate benefits is the ability to put equipment virtually anywhere. Containment corrects the delivery disparities of an open data center. Without containment, attention must be paid to how much cooling is being delivered to each rack. But with most containment systems, as long as the room level complement of cooling is satisfied, equipment can be deployed anywhere. High density racks can be located next to low density racks. The data center manager no longer has to worry about where they are going to put their next server. Predictability is restored to deployment planning activities. Consider the varied delivery volumes in the next figure. This is a top view of a data center engineering lab at Dell. The two outer CRAC units have been turned off to restrict the delivery to levels more typical 3
of most data centers. The blue squares represent vents, and the numbers represent the amount of air in Cubic Feet per Minute (CFM) exiting each vent 1. The average flow rate is 600 CFM, but there is obviously a wide variation. The wide variance is due to the natural dynamics under the floor. The rack s ability to support equipment is based on the facility s ability to deliver adequate volumes of chilled air to it. For instance, one rack of 40 1U servers consumes 1000-1200 CFM. In rack locations where the facility cannot deliver an adequate complement of chilled air, it is unlikely all 40 servers could be deployed. The disparity in flow rates in the figure above represents one of the challenges the data center manager faces in an open data center. There are obviously areas of very low flow where a high density deployment would be difficult. The areas of very high flow will likely result in some wasting of chilled air that escapes the aisle before it can be consumed by a rack. Imagine, however, if each of these racks were connected to a chimney and dropped ceiling plenum so that the only path back to the CRAC unit is through the racks. In this scenario, it really doesn t matter which vent the air comes from; it must go through a rack on its way back to the CRAC. With containment, high densities and low densities can be mixed easily and placed anywhere because the delivery to the rack is not dependent on the local delivery through the floor. Rather than having to comprehend the vent-to-vent flow inconsistencies, data center managers can plan around the aisle s or even the room s volumetric delivery. Quantifying the Energy Advantages of Containment How much energy can be saved by deploying a containment system? A portion of the savings has a lot to do with how much over-provisioning of air delivery existed prior to the containment system. If containment is the after picture, then one must accurately understand the before picture. It takes a reduction in the number of CRACs, or introduction of variable air movers inside the CRACs, to save energy related to moving air. Savings are also available at the chiller and potentially inside the IT equipment. The following are examples of potential advantages of containment systems. Any facility related savings are taken from experimentation first introduced by Moss/Bean 2 from testing done in the new Schneider Electric Technology Centre in St. Louis. 1 Flow rates were measured with a flow-hood. Five measurements were made for each tile, discarding the lowest and highest, and averaging the remaining three. The total flow rate through the tiles measured 16,817 CFM. Relative to the rated flow rate for the operating Liebert FH600C (17,100 CFM), the measurements represent a good correlation of 98%. 2 The Impact of Server Temperature on Energy Use; David Moss, John H. Bean, Jr., 2009 4
1. Server energy savings: Hot spots occur when not enough cool air is delivered to a rack and it consumes re-circulated exhaust. If a server is consuming air that is still within operational specifications, it will still work fine; but it is using quite a bit more energy because of the hot spot. Consider this graph of a typical 2U server. It is configured with quad processors, a generous compliment of memory, and it is running a heavy load. While operating at inlet temperatures in the 60s and low 70s, its power requirements are fairly constant. At the lower end, there is a slight drift upward due to leakage current. The fan control for this product triggers at about 73 degrees, at which point it starts an exponential increase in fan power. Below 73 degrees the contribution of the fans to the total power consumption is a constant at 10-15 watts (less than 5% of the server s power). If it were to operate at the max spec of 95 degrees, there would be about an 80 watt increase (increasing the server s fan power to about a 20% overhead). This is almost completely due to increased fan speeds. Containment allows the servers to be cooled consistently, removing hot spots and avoiding the consequences of increased IT fan power. 2. 3. Facility savings in the chilling process: The previous section discussed the significant IT power reductions possible if hot spots were eliminated. The consistent temperatures resulting from containment can also empower the facility to raise temperatures and achieve significant savings in the facility s chilling process. Consider this graph. In this test, a set of servers running a constant compute load was exposed to a wide range of inlet temperatures. Energy measurements were totaled for tests running at each inlet temperature point. These measurements included both the IT energy and the energy related to the cooling process (CRACs, chillers, pumps, and cooling tower). In the portion of the curve between 55 and 75 degree inlet temperatures, the total energy decreases. The energy use of the servers is essentially constant; the energy use at the chiller drops significantly, a total of about 13%. The second portion of the curve has an upward trend. Even though the chiller energy continues to decrease with greater temperatures, these decreases are outweighed by increases in fan power. In this instance, not only the server fan power increased, but there was also a corresponding increase in CRAC fan power reacting to the server increase. This reinforces the point that the optimal operating temperature for this particular server is about 75F, and containment would help achieve and maintain that consistent temperature in all locations. Energy Savings in facility air handling units: Unless grossly over-provisioned by several hundred percent, the energy associated with delivering air to the rack offers a modest energy reduction 5
relative to the facility s total energy consumption described above. Once contained, the facility may still provide a volumetric flow rate slightly higher than required by the IT equipment, but it can be a large reduction relative to an uncontained facility. A reduction in the over-provisioning for a typical 30 ton CRAC unit from 100% to 10% results in a 5% facility energy reduction if accomplished by turning off units. It makes almost a 9% difference if the reduction is due to a throttling back of the CRAC units through VFD. The server example showed a variance in system power use that showed a 25% increase from a low operating temperature to a high temperature. Assuming just 10% of a facility s systems were located in hot spots, and their power consumed increased only 10% due to additional fan requirements, that would make about a 1% difference in the facility energy use. So, the removal of hot spots by a containment system would have an immediate 1% benefit to the facility energy use. By stabilizing inlet temperatures, an environment is fostered in which the temperature can be increased. The cited example resulted in a 13% reduction in facility energy due to the temperature increase. This did not include any energy benefits for moving the operating temperature above the dew point. If it had, there would have been an additional benefit of 1-2% by avoiding re-humidification. The example reduction in facility flow rate concluded a 5-9% energy decrease. Combined, the reductions total 20-25%. Design Considerations There are definite opinions about which containment strategy is better, supply versus return. One obviously has to be careful of opinions from biased containment product manufacturers. The deployment of a containment system may require professional services to avoid some of the associated challenges. Here are a few considerations when considering a containment system: Will containment be applied to the entire facility or only in zones? If considering supply containment, are you prepared for an increase in room temperature? Segregation of any kind in the data center may have fire suppression considerations If the containment strategy involves using an existing false ceiling, any existing wiring may have new requirements to satisfy fire codes A tight coupling of racks to the air conditioning system can have effects on either system; the coupling cannot add too much resistance to either the racks or the air handling system A tight coupling of the racks to the air conditioning system can also have temperature/capacity challenges. Later model IT systems may have a high delta-t (temperature rise through the equipment) relative to the delta-t capability of the air handling unit. Mixing in a previously uncontained data center may have mitigated the risk of a large mismatch between IT delta-t and HVAC delta-t capability. 6
Summary Containment can enable energy savings within IT equipment, at the CRAC unit, and within the chilling process. It is an enabler of higher facility temperature, which can be the largest opportunity for energy savings. It can help data center managers achieve higher, more predictable densities and alleviate worries of putting too much IT equipment in the wrong place. If it is applied to a facility previously overprovisioned with cooling, it might allow for tapping into stranded cooling capacity and postponing the costs of new air handling equipment. Containment can be an integral part in the strategy of extending the life of a current data center and postponing the construction of a new facility. Definitions CFM- Cubic Feet per Minute; a unit of volumetric flow measurement CRAC- Computer Room Air Conditioner; large floor unit that chills and pressurizes the raised floor. This unit actually has chilling components built into it in contrast to the CRAH unit which requires chilled water that is cooled remotely CRAH- Computer Room Air Handler; large floor unit that chills and pressurizes the raised floor. This unit is fed with water that has been chilled in a central chilling plant Delta-T- the change in temperature as it travels through a device. The delta-t for a piece of air conditioning equipment is essentially a measure of its ability to cool the air. The delta-t of a piece of IT equipment is the amount it heats up the air. DX- direct expansion- in layman s terms, DX refers to the type of air conditioning unit that contains all or some of the chilling components, i.e. a CRAC unit; this is in contrast to a CRAH unit that accompanies a remote water chiller and pumps to circulate the chilled water to the CRAH unit InRow- APC trademarked name for their air handler; similar in function to a CRAC or a CRAH, this unit sits in the row between racks VAV- Variable Air Volume applies to air movers that are equipped to vary the amount of flow they deliver VFD- Variable Frequency Drive; variable speed option in CRAH blower motors 7