Deploying High-Density Zones in a Low-Density Data Center

Deploying High-Density Zones in a Low-Density Data Center By Neil Rasmussen Victor Avelar White Paper #134 Includes Interactive tool

Executive Summary New breakthroughs in power and cooling technology allow for a simple and rapid deployment of self-contained high-density zones within an existing or new low-density data center. The independence of these high-density zones allows for predictable and reliable operation of high-density equipment without a negative impact on the performance of existing low-density power and cooling infrastructure. A side benefit is that these highdensity zones operate at much higher electrical efficiency than conventional designs. Guidance on planning design, implementation, and predictable operation of high-density zones is provided. 2

Introduction High-density equipment such as blade servers, 1U servers, and multi-core high-end servers provide more computing per watt compared to previous generation servers. However, when consolidated, this new generation of equipment requires concentrated power and cooling resources. Data center operators and IT executives are often uncertain about the capability of their existing data center and whether a new data center must be built to support higher rack densities. Fortunately, a simple solution exists that allows for the rapid deployment of high-density racks within a traditional low-density data center. A high-density zone, as illustrated in Figure 1, allows data center managers to support a mixed-density data center environment for a fraction of the cost of building an entire new data center. Figure 1 Basic concept of high-density zone Low-density room High-density zone A high-density island in the room A mini data center with its own cooling Thermally invisible to the rest of the room (ideally) Hot/cool air circulation is localized within the zone by short air paths and/or physical containment Hot/cool air circulation is localized within the zone HEAT OUT To building s heat rejection system In this paper a high-density zone is defined as one or more rows of racks that support a per-rack density average of 4 kw or above. A high-density zone resides within the borders of a larger, low-density data center. The high-density zone is not the same as a high-density data center, which is a data center dedicated to supporting nothing but high-density racks. Managing for the deployment and operation of a high-density data center is not the subject of this paper. 3

High-density zone compared to spreading out strategy Although today s IT equipment operates at high power density that is, each individual server draws a high amount of power this does not always mean such devices must be deployed in a high-density manner by packing them together in a rack. In fact, a popular strategy has been to spread out high-density servers by installing fewer in each rack. If the equipment is dispersed like this, the data center s average power density will likely stay in the range that the data center was originally designed for. In this way, a variety of technical problems can be avoided. However, the spreading out strategy may not be viable for a number of reasons: Consumption of additional floor space, which may be difficult to justify or simply not possible Executive management perception that partially-filled racks are wasteful Increased cabling costs (because of longer runs) Increased cost and difficulty of maintenance cabling and mounting may be intertwined with other equipment in a nonstandard manner, scattered throughout the room Reduced electrical efficiency of the data center, because cooling-system air paths are longer and less well targeted this increases mixing of hot and cool air, which results in lower return temperature to the air conditioner (see later sidebar Why shorter air paths increase data center High density enables high efficiency efficiency) For these reasons, it is expected that data enter operators will begin to deploy IT equipment at its full density capability in zones rather than try to stay within an overall room power density by spreading out the load. With new power and cooling technologies, there is now a significant efficiency entitlement from concentrating high-density equipment into zones. In traditional data centers with room-based power and cooling, unmanaged high-density racks can cause destabilizing effects such as cooling inefficiency, loss of cooling redundancy, hot spots, thermal shutdown, and circuit overload. However, with today s new power and cooling technologies, high-density racks offer an opportunity for dramatically increased efficiency and predictability, if deployed effectively and supported by smart row-based power and cooling. This paper assumes the choice has been made to deploy The high-density zones described in this paper provide a way to deploy high density while at the high-density IT racks in a low-density data center. Rowbased cooling, as a technique to implement these high- same time achieving increased overall data center efficiency by targeted, scalable, localized power and cooling. density zones, is presented as a simple solution for addressing high-density power and cooling issues in both existing and new data centers. For more information regarding alternatives for deploying high-density equipment, including the option of spreading out IT equipment, see APC White Paper #46, Cooling Strategies for Ultra-High Density Racks and Blade Servers. 4

The Problem: Unmanaged High Density Traditional data center design uses a raised floor to distribute cooling to low-density IT equipment (Figure 2a). However, when high-density equipment is randomly installed throughout a low-density data center the cooling stability is upset and hot spots begin to appear (Figure 2b). Figure 2a Low-density data center Low-density room Figure 2b High-density hot spots Low-density room Concentrated highdensity IT equipment Stable cooling Unpredictable cooling Data centers designed for low-density racks (typically 1-2 kw / rack) vary dramatically in construction. Ceiling heights, raised floor depths, room geometry, power distribution, and raised floor obstructions are all quite different. In addition, IT managers vary in how they define a high-density rack. This paper defines a high-density rack as 4 kw or higher. Regardless of which number is used to denote a high-density rack, the following deployment issues need to be considered: Delayed server deployment uncertainty of knowing which rack can cool a newly provisioned server just adds to the already long delay by having to perform a cooling assessment Unplanned downtime due to overloaded power distribution circuits or thermal shutdown of IT equipment Unpredictable cooling throughout the data center no certainty that every high-density server will be properly cooled after every move, add, or change Loss of cooling redundancy as more high-density racks are added, air conditioning units that were once redundant are now required to supply the concentrated airflows. Some subsystems are extremely impractical or costly to instrument for power consumption (for example, PDUs due to number of output connections, or switchgear) Fortunately a solution exists that can neutralize these issues and is discussed in the following sections. 5

The Solution: High-Density Zones Placing high-density racks in an isolated, standardized self-contained area of the data center provides a low cost, viable solution to the challenges mentioned above. This high-density zone avoids dependence on the unpredictable nature of raised floor cooling and would not require complex computational fluid dynamics (CFD) analysis prior to installation. Figure 3 illustrates three high-density zone implementation methodologies all of which are capable of supporting independent power distribution, UPS, and cooling systems. This drop-in solution eliminates the hot spots in Figure 2b by simply moving high-density equipment into the zone. The heat generated from the high-density IT equipment within this zone is rejected to the outdoors with no impact to the existing data center cooling system or the surrounding low-density IT racks. In fact, the zone acts as its own high-density data center within an existing low-density data center and is thermally invisible to the rest of the room. Low-density room Figure 3 Isolated (room-neutral) high-density zone High-density zone Hot/cool air circulation is localized within the zone HEAT OUT to building s heat rejection system Three basic methods (Top view) Uncontained HOT-AISLE containment RACK containment What is room neutral? High-density zones operate on the idea of isolating server exhaust heat and directing all of that heat into the air conditioner intakes, where the air is first cooled before being redistributed to the front of the servers. By 6

isolating both hot and cold air streams, the high-density zone neutralizes the thermal impact that highdensity IT racks would otherwise have on traditional low-density data centers. In other words, the zone presents a room neutral load to the existing data center cooling system. Although this paper focuses on the cooling of high-density zones, it is also possible to power a zone with its own dedicated UPS and power distribution. This may be desirable in situations where the existing data center UPS is at capacity or is being phased out due to end-of-life or when targeted power availability is required for a specific zone. The system in Figure 4 integrates a cluster of high-density IT racks with a high-density row-based cooling system and high-density UPS and power distribution system in a pre-manufactured, pre-tested zone. Figure 4 Front view of a standardized modular multi-rack high-density zone (No containment in this example) Air conditioners return neutralized (ambient room temperature) air to the front of the racks Hot air is exhausted to the hot aisle and returns to the back of the air conditioners Integral row-based air conditioners Row-based cooling architecture A row-based cooling architecture makes it possible to have a room-neutral high-density zone. Row-based cooling is an air distribution approach in which the air conditioners are dedicated to a specific row of racks. This is in stark contrast with room-based cooling where perimeter air conditioners are dedicated to the entire room. Row-based air conditioners may be installed above IT racks, adjacent to IT racks, or in combination. An example of a row-based air conditioner is shown in Figure 5. stored in any retrieval system of any nature, without the written permission of the copyright owner. www.apc.com WP134 Rev 0 7

While most facilities and IT personnel understand the basic idea behind high-density zones, they question how the zone can be room neutral in the midst of constant moves, adds, and changes. Considering their past experience with the variability and at times perplexing nature of raised-floor cooling, skepticism toward the long-term predictability of highdensity zones is not surprising. Though raised floors and high-density zones are both governed by the same laws of fluid dynamics and Row-based cooling units Compared with the traditional roomoriented approach, the airflow paths of row-based air conditioners are shorter thermodynamics, one major aspect and much more predictable. In sets them apart standardization. addition, all of the rated capacity of the air conditioner can be utilized, and higher power density can be achieved. If raised floors were standardized so At the same time, the usable capacity of the perimeter (room-based) cooling that they all had the same depth, same system increases and in some cases its dimensions, same under floor obstructions, same under floor airflow pattern, same locations, and cooling redundancy is restored to the original design as IT load is removed from this system and placed into the zone. same air leakage from tile cutouts, they could more easily be modeled in real time so as to predict their behavior using design and planning software tools. If this standardization existed, IT Although not discussed in this paper, row-based cooling is also an effective method for entirely cooling small lowdensity data rooms (1-3 rows of racks). APC InfraStruXure InRow air conditioner managers would be able to predict the cooling impact of adding a blade chassis to a particular rack and make rational decisions based on the prediction. However, these raised floor attributes by their very nature are customized and are not conducive to standardization. Furthermore, the variability of all these attributes would make real-time computational fluid dynamics (CFD) modeling nearly impossible in a typical data center. In contrast, high-density zones use standardized hot / cold aisle widths, rack height, and air path distances to the rack. Row-based cooling also eliminates the variability introduced by the raised floor. These simplifications make it possible to design predictable high-density zones using standardized tools. These design tools provide the confidence that any design will capture and neutralize the expected amount of hot exhaust air. For more information on the row-based cooling architecture, and how it compares to roombased cooling, see APC White Paper #130, The Advantages of Row and Rack-Oriented Cooling Architectures for Data Centers. 8

Zone Containment Methods Server exhaust heat can be diverted back to the air conditioners in three ways: uncontained, hot aisle containment, and rack air containment (see Figure 5). All of these methods leverage a row-based cooling concept (e.g., the air conditioner is brought within a few feet of the IT rack). Figure 5 High-density zone containment methods Low-density room Three ways to create a room-neutral island in a low density room 1 Uncontained 2 HOT-AISLE containment Hot/cool air circulation is localized within the zone High-density zone Room-neutral island in a low-density room 3 RACK containment 1 Uncontained Uncontained zones rely on the standard layout and widths of the common hot aisle and cold aisle arrangement to keep hot and cold air streams from mixing. For this reason, uncontained zones depend on multiple racks in a row and are not effective in cooling stand-alone IT racks. The hot and cold aisles formed by rows of racks (and in some cases walls) are what isolate the hot and cold air streams as illustrated in Figure 6. The closer an IT equipment rack is to a row-based air conditioner, the greater the amount of exhaust air that is captured and cooled. As the distance between the IT rack and the row-based air conditioner increases, the more the hot exhaust air mixes with the surrounding air in the data center. Importance of blanking panels Proper row-based cooling depends on the isolation of hot and cold air streams. If any of the vertical space in a rack is not filled by equipment, the gaps between equipment allow hot exhaust air to flow through the rack and to the front of equipment such as servers. This mixing between the hot and cold air streams reduces the effectiveness of row-based cooling. For more information see APC White Paper #44, Improving Rack Cooling Performance Using Airflow Management Blanking Panels. 9

When to use this method When IT racks designated for the zone are moved and relocated frequently. When IT racks are used from a variety of different vendors Trade-offs More row-based air conditioners required at lower densities in order to properly capture hot exhaust air from all IT racks. Figure 6 High-density zone with NO containment WALL or ROW to help form hot aisle REAR Rack Rack Rack Rack Hot aisle FRONT Why shorter air paths increase data center efficiency With traditional room-based perimeter cooling, cool supply air must travel further to each rack, requiring extra fan power. In addition, if there is no raised floor there can be substantial mixing of the cool supply air with warm exhaust air in the room, which will require lowering the supply temperature far below what is needed at the IT racks. Hot spots can force lowering the supply temperature even further to control overheating. Too-low supply temperature can risk condensation on the cooling coil, which results in wasteful dehumidification-rehumidification and reduction of system cooling capacity. Long return paths similarly can cause air mixing, which lowers the return temperature to the cooling unit. Lower return temperature slows the rate of heat transfer to the coil, so heat is removed less efficiently. The much shorter air paths in row-based cooling dramatically lessen mixing of supply and return air (and with containment and blanking panels, virtually eliminate mixing). On the supply side, this allows operation at a higher coil temperature, which takes less chiller energy to maintain and is much less likely to cause wasteful condensation. On the return side, it produces a higher return temperature which increases the heat removal rate. Compared to long-path cooling systems, these combined short-path effects (1) increase operational cooling efficiency and (2) increase the cooling capacity of the heat rejection system. 10

2 HOT AISLE containment Hot aisle containment zones are identical to uncontained zones except for the fact that the hot aisle in every pair of rows is contained. The hot aisle becomes the hot exhaust channel by enclosing it with ceiling panels and a door at each end of the aisle (Figure 7). In addition, the racks rear doors are removed. The hot exhaust air is physically contained and unable to mix with the ambient data center air. A wall or another row of racks is required to form a cold aisle in order to isolate the cold supply air. When to use this method In cases where floor space must be conserved. This method is popular because it consumes the same space as two rows of low-density racks. In data centers with hot aisle / cold-aisle layouts Trade-offs Hot aisle containment panels increase capital cost Hot aisle containment may exceed work environment policies due to high temperature Incompatible with some types of cabling, power strips, labels, and other materials that are not rated for high temperatures Not possible with a single row of racks Authority having jurisdiction (AHJ) may require fire suppression in hot aisle Figure 7 High-density zone with HOT AISLE containment FRONT Rack Rack Rack Rack REAR Hot aisle Contained hot aisle FRONT Rack Rack Rack Rack 11

3 RACK containment Rack containment (also called rack air containment) is similar to hot aisle containment except that the hot exhaust air is contained using the back frame of the equipment racks and a series of panels to form a rear air channel. This channel can be attached to a single IT rack or to a row of racks (Figure 8). The panels used to create the hot exhaust air channel increase the depth of a normal rack by 20 cm (8 in). An optional series of front panels may be used on rack containment arrangements that require complete containment of hot and cold air streams as shown in Figure 9. This optional front containment adds an additional 20 cm (8 in) to the depth of the rack. When to use this method Trade-offs In cases where hot aisle containment is the preferred method, but a single odd row is left uncontained When frequent access to and easy management of communication cables is required For complete isolation in cases such as standalone open data center environments or mixed layouts only when optional front containment is used In wiring closets that lack any form of cooling, exposing high-density equipment to high temperatures only when optional front containment is used When sound attenuation is required only when optional front containment is used Why NOT use containment? It may appear that containment would be the clear choice for any row-based cooling scenario. However, this is not always the case. With row-based cooling, containment is more important at lower densities, where the ratio of IT racks to air conditioners is higher. The higher this ratio the greater the distance between IT racks and air conditioners, with more chance for hot exhaust air to escape. Higher densities, on the other hand, mean a lower ratio of IT racks to air conditioners, with shorter air paths and less chance for hot exhaust to escape in this case, containment is less essential because airflow is tightly targeted and tends to behave all by itself. In addition, there may be practical considerations that rule out containment, such as higher cost at certain rack power densities, company restrictions on hot work environments (i.e., a contained hot aisle), and incompatibility with existing racks. Front and rear containment panels increase capital cost In a single rack configuration, cost increases substantially when cooling redundancy is required 12

Figure 8 High-density zone with RACK containment (Also called rack air containment) re ar d oo rs Rack Rack Rack So lid FRONT Rack REAR Return air contained Figure 9 High-density zone with RACK containment plus optional front containment Solid front doors FRONT Single rack So lid r t do or s FRONT Rack ea rd oo rs Rack Rack Multiple racks fron Soli d Optional front containment REAR Solid rear doors Rack Rack REAR stored in any retrieval system of any nature, without the written permission of the copyright owner. www.apc.com WP134 Rev 0 13

An overall comparison of high-density zone methods is shown in Table 1. Table 1 Comparison of zone containment methods Selection Criteria NO containment HOT AISLE containment RACK air containment Comments Minimize footprint Good Good Moderate to poor NO containment and HOT AISLE containment provide minimum row spacing RACK air containment adds 8 inches to the depth of the rack but may be acceptable in consolidation applications Front AND rear containment adds 16 inches to the depth of the rack should be weighed against available floor space Ease of change management Good Moderate to poor Moderate to poor Taking racks in and out of an existing row is more difficult when containment systems constrain the rack with hardware, especially with front containment Minimize energy consumption Moderate Good Good NO containment layout is closely linked to the existing data center layout which could increase the number of row-based units Ease of redundancy Moderate Good Moderate to poor HOT AISLE containment row-based positions are independent of redundancy More row-based s needed to maintain redundancy in RACK air containment Minimize # of rowbased S (particularly at low density) Poor to moderate Good Moderate to good RACK air containment and RACK air containment with front containment may be limited since not all rack air can be shared among all row-based coolers as with HOT AISLE containment NO containment depends heavily on rack power density where high densities require less rowbased coolers RACK air containment and RACK air containment with front containment highly influenced by redundancy (more coolers needed) Poor to moderate with RACK air containment only Sound attenuation Poor Moderate to poor Good Good when using RACK air containment with front containment Will reduce the decibel level of the cooling equipment but will not completely eliminate the noise Installation in thermally unstable or non-data center space Cost Poor Poor Good Dependent upon variables such as rack power density and number racks Poor to moderate with RACK air containment only Good when using RACK air containment with front containment Examples include wiring closets, offices, and commercial spaces Although the HOT AISLE containment has additional panels that increase cost, it will require fewer row-based coolers than NO containment, particularly at lower rack power densities 14

Additional High-Density Zone Benefits The decision on whether to move forward with deployment of a high-density zone should also consider the following benefits: Standardization of design elements Compatibility with any data center, new or existing Configurability with dedicated UPS and power distribution Configurability with any level of redundancy Configurability with any number of IT racks Standardized design elements In order for high-density zones to provide predictable performance they must include standard design elements. This includes components such as air conditioners, power distribution, UPS, and racks. In addition, standard dimensions play a key role in predictably isolating hot and cold air flows. Standard dimensions include hot / cold aisle widths, rack height, and standard (short) airflow travel distances. Modularity is also a benefit of standardization and allows high-density zones to be quickly deployed, altered over time, and even moved to another data center. Standardized components and dimensions greatly simplify the design process. These pre-designed standard solutions may even be re-ordered for other data centers. Data center personnel can also leverage standardization by deploying predictable capacity and change management software that maintains the peak performance of the high-density zone (this is discussed later). For more information on standardization see APC White Paper #116, Standardization and Modularity in Network-Critical Physical Infrastructure. Compatible with any data center, new or existing High-density zones are modular and independent of room-based cooling architectures and existing UPS architectures. Therefore, few constraints exist to prevent their deployment in new or existing data centers. Sufficient floor space must be available and the floor must have enough weight-bearing capacity. All other aspects of a standardized high-density zone are replicable in multiple types of data centers. Configurable with dedicated UPS and power distribution The architecture of the high-density zone allows for deployment of zone-specific UPS and PDU configurations in cases where the existing data center UPS is at capacity or is being phased out due to endof-life. These systems are rack-based and designed to be modular and scalable. Configurable with any level of redundancy Redundancy levels vary depending upon the criticality of the IT assets. Traditional data center design is such that the entire physical infrastructure is built to satisfy the redundancy requirements of the most critical 15

set of assets. This type of design is extremely expensive both from a capital cost and operational cost perspective. A much more cost-effective design is to provide redundant power and cooling only where and when required. High-density zones allow for this targeted redundancy / availability approach by including redundant power and cooling modules when appropriate. Note that the core infrastructure such as chilled water piping and electrical service entrance must be designed and built on day one with the highest redundancy level required. Configurable with any number of IT racks High-density zones are scalable in that they accommodate the number of IT racks required at a specific power density. These zones can range in size from a single IT rack to 20 or more racks depending on local codes. Combining these characteristics results in a highly flexible high-density solution that can extend the life of a legacy data center and postpone the capital outlay required for building a new one. Table 2 Deploying high-density equipment: Traditional vs. zoned approach Attribute TRADITIONAL approach ZONED approach Comments Positioning the data center as a source of corporate competitive advantage Difficult Easier Simple economics cost of doing business is lower per unit of computational output Just-in-time IT deployments Very difficult Easy Deployments are highly dependent on modular and predictable power & cooling which affects management and ability to quickly deploy Energy cost per IT computation High Low Strongly linked to data center infrastructure efficiency Predictability of performance Low to no predictability Highly predictable via standardization Traditional approach too customized to allow predictable performance in constantly changing data center Likelihood of hot spots High Very low Management applications insure optimal placement of equipment in zones to prevent hot spots Cooling efficiency Poor Excellent Ability to plan Poor Excellent Room based cooling units are oversized to overcome under-floor obstacles, distance, air mixing, demand fighting, etc. Standardization / predictability facilitate what-if scenarios before moves, adds, and changes 16

In-House vs. Vendor-Assisted Deployment The data center owner has two options for the deployment of high-density zones: in-house deployment or vendor-assisted deployment. In both cases a solid project plan is required. More specific information regarding data center projects and system planning is available in APC white papers #140, Data Center Projects: Standardized Process, and #142, Data Center Projects: System Planning. In-house deployment IT managers can easily deploy smaller sized zones or smaller data centers (less than 20 racks) with no previous experience. A worksheet and checklist is provided in Appendix A. This worksheet can serve as a helpful guide and facilitates the collection of information required to specify and deploy a high-density zone. The worksheet assumes the project owner has knowledge of the IT equipment associated with the planned high-density zone (e.g. total power requirements, plug requirements, rack U-height requirements, and communications cabling requirements). If the worksheet is properly filled out, an educated decision can be made on which zone containment method to choose. APC TradeOff Tool #10, Data Center InRow Containment Selector, (see Figure 10) can help select the most appropriate zone containment method. The results generated by the tool are based on typical scenarios and in some cases the recommended containment option may differ from the actual final design. Once a containment type is chosen, a decision must be made on which components the zone will include. The worksheet helps data center staff determine whether to include a dedicated UPS, PDU, or chiller. In some cases, certain preferences and constraints dictate which components are included in a zone and which are not. Table 3 provides a list of possible constraints that could affect the ultimate configuration of the high-density zone. Table 3 High-density zone components under various constraints Constraint High-density zone requirement None No spare power distribution positions No spare power capacity on existing UPS system No spare cooling capacity on existing chiller No spare power or cooling capacity on existing UPS and chiller No spare power or cooling capacity on existing UPS and chiller or spare power distribution positions Racks and row-based cooling units Racks, row-based cooling units, and row-based power distribution unit (PDU) Racks, row-based cooling units, row-based UPS system, and row-based PDU Racks, row-based cooling units, and packaged chiller Racks, row-based cooling units, row-based UPS system, and packaged chiller Racks, row-based cooling units, row-based UPS system, packaged chiller, and row-based PDU 17

Figure 10 Interactive tool for containment method selection Even with the constraint of no spare UPS, chiller, or power distribution capacity, it is still possible to extend the life of an existing data center by installing a high-density zone with its own dedicated power and cooling resources. For example, the high-density zone in Figure 11 includes its own chiller plant, UPS, and power distribution. It is assumed that the data center s electrical service entrance has sufficient spare capacity to supply power to this packaged solution. In cases where a data center has run out of spare electrical service capacity, a decision must be made to install an additional utility feed(s) or build a new data center. Other factors beyond the scope of this paper such as available floor space, virtualization potential, business objectives, leasing contracts, and future growth plans factor into the buy-or-build decision. 18

Figure 11 Packaged standalone high-density zone Cooling distribution unit Packaged chiller From the time a need for a high-density zone is identified, IT and facilities personnel can expect to populate the racks in a given zone in one to three months, assuming the required budget is approved. However, internal company processes may extend the proposed timeline. Vendor-assisted deployment Although it is possible for data center staff to deploy high-density zones without outside assistance, projects involving data centers with 20 or more racks can be considerably more complex. In such cases consultation with design experts and project managers is recommended. Vendor-assisted deployment usually begins with an assessment of the existing data center or the design plans for a new data center. In either case an assessment provides the design experts with valuable information, including preferences and constraints, which allows optimum design decisions. Assessments help answer questions such as: Can an existing row be retrofit with row-based air conditioners to avoid downtime? If spare chilled water capacity is unavailable should a self-contained air conditioning unit be used as opposed to a packaged chiller? What steps can be taken to increase the speed of deployment of a future high-density zone? An effective assessment (such as APC s Blade Server Readiness Assessment) measures spare bulk power and cooling capacity as well as spare distribution capacity. Bulk cooling capacity is measured at the chiller while the distribution capacity is measured at the CRAH units on the data center floor. This data provides an estimate of cooling capacity and compares constraints against current and future requirements. Ultimately this will help answer the question, When will I run out of cooling capacity and require a high-density zone? After measuring and analyzing the data, a plan is created to meet future high-density needs. In the end, an effective design plan for mixed-density data centers should incorporate power, cooling and floor space 19

utilization efficiency. An effective design plan allows a data center to use up its power, cooling, and space resources all at the same point in the future, thereby avoiding stranded resources. Real-Time Management of High-Density Zone The architecture of row-based cooling makes real-time modeling of cooling performance possible. Design tools can configure racks, row-based air conditioners, UPS, and power distribution based on high-density zone specifications such as average and peak power density per rack, containment, redundancy, and plug types. Once a high-density zone is deployed, real-time planning and management tools allow IT personnel to maintain predictable operation even after moves, adds, and changes take place. Examples of appropriate design and planning tools include APC s InfraStruXure Designer and APC s Capacity and Change Manager. For more information on management and its critical role in predictable performance, see APC White Paper #150, Power and Cooling Capacity Management for Data Centers. Conclusion In the past it was a major challenge for IT personnel to successfully deploy a mix of high-density and lowdensity equipment in the same data center space. Traditional data centers were specified to cool a uniform rack power density and were not capable of predictably cooling a large number high-density racks. Now architectures such as row-based cooling allow for the rapid deployment of high-density zones within an existing or new low-density data center. Modular row-oriented power and cooling can now be added where and when high-density racks are required, without any effect on the existing room-level infrastructure. In combination with capacity and change management systems, zones offer a high-density deployment solution capable of maintaining a room-neutral and predictable operation even after moves, adds, and changes. 20

About the authors Neil Rasmussen is the Senior VP of Innovation for APC. He establishes the technology direction for the world s largest R&D budget devoted to power, cooling, and rack infrastructure for critical networks. Neil is currently working to advance the science of high-efficiency, high-density, scalable data center infrastructure solutions and is the principal architect of the APC InfraStruXure system. Prior to founding APC in 1981, Neil received his Bachelors and Masters degrees from MIT in electrical engineering where he did his thesis on the analysis of a 200MW power supply for a tokamak fusion reactor. From 1979 to 1981, he worked at MIT Lincoln Laboratories on flywheel energy storage systems and solar electric power systems. Victor Avelar is a Strategic Research Analyst at APC. He is responsible research in data center design and operations and consults with clients on risk assessment and design practices to optimize the availability of their data center environments. Victor holds a Bachelor s degree in Mechanical Engineering from Rensselaer Polytechnic Institute and an MBA from Babson College. He is a member of AFCOM and the American Society for Quality. 21

Appendix A Worksheet and Checklist for deployment of a high-density zone 22

High-Density Zone Worksheet and Checklist Data / Input Value Comments 1 Criticality level 1, 2, 3, or 4 Goal for the availability and reliability of the zone, consistent with the business mission. See APC White Paper #122 for guidance in choosing an appropriate criticality level. 2 Average rack power density (kw) 3 Peak rack power density (kw) Average IT load per rack. (The zone s cooling will be designed to handle this.) Maximum anticipated IT load in any rack of the zone. (The zone s cooling will be designed to handle this.) 4 5 6 7 8 9 10 11 12 Is there sufficient service entrance capacity to support this high-density zone? (Yes / No) Type of data center floor Raised floor Hard floor What floor-to-ceiling height is available for the zone s equipment, taking service clearances into account? (Indicate ft or m) Will the zone include a separate dedicated UPS? What is the available source input voltage to the UPS(s) or PDU(s)? How much current is available from the subpanel that will power the UPS(s) or PDU(s) (amps) How many 3-pole breaker positions are available in the panel(s)? What is the preferred UPS runtime? (minutes) Total spare capacity of all existing UPS system(s) dedicated to the data center Electrical service entrance capacity must be able to support the incremental power load added by the zone (IT load plus power/cooling infrastructure). The height available for proposed and future equipment, taking into consideration all applicable service clearances per local jurisdiction. For example, sprinklers will affect available height. If no, skip to item 12 Total spare current of the subpanel feeding the room must be shared with cooling equipment in item 21 Electrician is best source for this information Total number of spare 3-pole positions available to be used by UPS(s) and PDU(s) Electrician is best source for this information See APC White Paper #52 for guidance This will determine if there is sufficient existing UPS capacity to support the new high-density zone, taking into consideration desired redundancy and distribution. 23

High-Density Zone Worksheet and Checklist Data / Input Value Comments 13 14 15 How will the electrical distribution cables be routed from the PDU(s) to the racks? Overhead Underfloor Which method will this high-density zone use? No containment Hot aisle containment Rack containment What existing heat rejection methods are available at the site? Chilled water Glycol Refrigerant Water cooled Identifies the types of heat rejection systems that are available at the site. This will help in designing a high-density zone with a compatible cooling system. 16 What is the total sensible capacity (kw) of the existing cooling system? 17 What is the spare sensible capacity of the existing chilled water system? (kw) The total cooling capacity (in kw) available from the existing cooling system. For chilled water systems, this will be the capacity of the chiller plant. For DX systems, this will be the total of all units. To be filled out if chilled water spare capacity will be used for this high-density zone. 18 19 How is cooling system piping routed? Overhead Underfloor How will the chilled water piping be routed to the new cooling units? Overhead Underfloor Identifies how the following are routed: DX glycol, condensor water, humidification, and condensate lines Chilled water supply/return piping Routing for refrigerant, humidification, and condensate lines 20 21 22 23 What is the source input voltage to the new / CRAH unit(s)? (volts) How much current is available to power the new / CRAH unit(s)? (amps) Does the cooling solution require both critical and non-critical power inputs? What voltage feeds the critical power input of the cooling unit? Total spare current of the subpanel feeding the room must be shared with electrical equipment in item 9 Electrician is best source for this information. If NO, skip next two items Voltage for power to fans and controls 24

High-Density Zone Worksheet and Checklist Data / Input Value Comments 24 What voltage feeds the non-critical power input of the cooling unit? Voltage for power to compressor (DX only), humidifier, and pump. 25 What type of physical security system is required for the high-density zone? Door card Cameras Motion detectors 26 27 28 29 What building management system (BMS) does the existing data center use? Name of system None used What network management system does the existing data center use? Name of system None used What is the preferred level of instrumentation? Typical Full How is the structured cabling within the data center routed? Overhead Underfloor Identifies the preferred instrumentation level for the high-density zone, using various sensors such as temperature, humidity, water and motion Structured cabling refers to networking cables connecting infrastructure equipment 25