PROPER SIZING OF IT POWER AND COOLING LOADS WHITE PAPER

WHITE PAPER #23 PROPER SIZING OF IT POWER AND COOLING LOADS WHITE PAPER CONTRIBUTORS: JOHN BEAN, APC RON BEDNAR, EMERSON NETWORK POWER RICHARD JONES, CHATSWORTH PRODUCTS ROBB JONES, CHATSWORTH PRODUCTS PHIL MORRIS, SUN MICROSYSTEMS DAVID MOSS, DELL DR. MICHAEL PATTERSON, INTEL JOE PRISCO, IBM WADE VINSON, HP JOHN WALLERICH, DELL

PAGE 2 TABLE OF CONTENTS: I. Introduction...3 II. Today s Data Centers...3 III. Nameplate Power Rating Misconceptions...3 IV. Nameplate Versus Actual...5 V. Cooling Load Calculations...6 VI. Energy Ef ciency Focus New Tools...6 VII. Tool Links...8 VIII. Proper Use of Calculating Tools...9 IX. Summary/Conclusion...9 X. References...10

PAGE 3 I. INTRODUCTION Between 2000 and 2006, energy requirements for data centers doubled, and they are on track to double again by 2011. In the same time period, typical per-rack heat densities went from 1 kw to 7 kw, and they are estimated to exceed 20 kw per rack by 2010. This places tremendous importance on a data center facility planner s ability to properly budget for future power and cooling infrastructure requirements. In the past, facilities planners used IT equipment nameplate ratings values as a method to plan power and cooling capacity requirements, which resulted in over-built, under-utilized, inef cient infrastructure. Today s energy costs and ef ciency demands necessitate a more accurate method of determining those requirements. This white paper has been developed to introduce the reader to the many new and highly accurate software tools available for estimating power and cooling capacity requirements. II. TODAY S DATA CENTERS The inaccuracy of data centers standard process for estimating the power consumption of IT equipment has led to the under- and over-provisioning of circuits, power distribution units (PDUs), and uninterruptible power supplies (UPSs) as well as the cooling systems needed to remove the energy dissipated. This can have a signi cant impact on power distribution ef ciencies, total power consumption, service availability, and operational costs. In an attempt to simplify this process and to help improve power consumption accuracy, the majority of IT vendors have made power and cooling estimation tools available for their various equipment models and con gurations. Prior to the proliferation of those tools, the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE) suggested the use of the thermal report, which provided the end user with power-draw and cooling-load estimates heat removal and air ow rates for minimal, typical, and maximum load con gurations 1. But even that signi cant step forward was still subject to errors based on different server con gurations and the exact complement of hardware options (CPU, hard drives, memory, I/O con gurations, etc.) The minimum baseline has since been expanded upon by system manufacturers. III. NAMEPLATE POWER RATING MISCONCEPTIONS Prior to the ASHRAE thermal report, the buyer or data center architect had little-to-no information on the expected power consumption of a server, so the default value was initially the nameplate value. Yet the term nameplate is often misused. When discussing server power draw and cooling loads, nameplate is frequently used (albeit incorrectly) to describe the value from the server data sheet for the power supply (generally the output of the power supply listed in watts). The technically correct description of nameplate value is the value from the system rating label (see Figure 1). The nameplate on a piece of IT equipment is a marking from a third-party agency that indicates that the equipment complies with a set of requirements set forth by a product safety standard such as IEC 60950-1 (Information technology equipment - Safety - Part 1: General requirements). Figure 1 shows an example of a nameplate found on a piece of IT equipment.

PAGE 4 FIGURE 1. EXAMPLE NAMEPLATE The nameplate includes, at a minimum, rated input voltage, rated input amperage, kilovolt-ampere (kva), frequency, and phase information. It is important to note that the wattage does not appear on the nameplate. The purpose of the nameplate is twofold: To give information necessary for the preparation of the electrical installation (e.g., circuit breakers, receptacles, wiring methods) to ensure adherence with electrical codes. To provide assurance to the installer or user who plugs the equipment into the mains that the equipment is suitable for installation at a particular location within the data center. Manufacturers derive the nameplate value in many different ways to ensure compliance with a product safety standard. The value could be based on the line cord assembly that is used to provide power to the IT equipment or the DC output power rating on the power supply unit (PSU). The nameplate value chosen may be a result of the cost of manufacturing: many OEMs standardize their power supplies on a smaller number of PSUs, which can result in substantial gaps between nameplate values and actual power consumed for a given piece of IT equipment. Also, power consumption can vary based on device con gurations additional components such as memory can increase the actual power draw of equipment. The same power supply may be used in a server with low-voltage CPUs and minimal memory as in the same server model that is con gured with the highest memory and CPU con gurations. Regardless of how they are chosen, nameplate values are generally accepted as representing power consumption levels that exceed actual power consumption of equipment under normal usage. Therefore, these over-in ated values do not support realistic power prediction. Unfortunately, nameplate value (regardless of whether it is the incorrect use of the data sheet PSU description or the proper de nition above) is still frequently used by data center planners in the design, planning, and deployment of power and cooling systems. This practice often results in excessive capital expense and operational costs due to lower ef ciencies that result from overestimating resource requirements. Many of these planners mistakenly use the nameplate value because it is either on the data sheet when they are ordering or always available and clearly visible on any piece of IT equipment when installed. There is also a misconception that devices upstream from the IT equipment (such as PDUs and/or UPSs) must be sized to accommodate the sum of all the nameplate ratings. As discussed above, the nameplate rating generally does not re ect actual power consumption and using it as an indicator leads to grossly over-provisioned power distribution systems. Typically, any electrical loads hardwired into a system are, by code, limited to 80% of the upstream components capacity. However, because the server has a plug, it is

PAGE 5 considered portable and the management of the load then falls to the owner. In this case, a better design practice would be to use the peak-measured or maximum-measured manufacturer s reported draw for the speci c IT equipment con guration and ensure that the sum of those loads does not exceed 80% of the rating of the upstream panel. Also, it is important to check with the local code authority having jurisdiction for any speci c ruling that would preclude this practice. There are some speci c geographies, instances, and tools that can allow higher loadings when the diversity of the load (frequency of equipment peak power occurring at the same time) is known, but these are outside the scope of this paper. In general, the electrical loading should be sized based on IT equipment peak measured or maximum measured power consumption levels and not nameplate values. IV. NAMEPLATE VERSUS ACTUAL 1400 51 Servers, PSU Rating vs peak draw Actual "50%" PSU Nameplate (Watts) 1200 1000 800 600 400 200 0 0 200 400 600 800 Peak draw (Watts) FIGURE 2. PSU NAMEPLATE RATING VERSUS PEAK POWER DRAW The above gure shows 51 servers and their power supplies tested by the U.S. Environmental Protection Agency 3. The nameplate rating (Y axis) is plotted against peak draw (X axis). Consider, for example, the large group of 1000 W PSUs in the gure. There were 23 of these, which indicates the manufacturer s propensity to use standard-size components at the next-greater size rather than exactly matching the expected load. Also noteworthy is the range of actual power draw found for these servers. The low end was roughly 160 watts compared to a high end of about 580 watts all using the same 1000 W power supply. With only the knowledge of the nameplate value in hand, many data center managers use a method of de-rating in order to estimate actual power consumption. In doing so, they establish actual power consumption as a percentage of the nameplate value. Some use a simple nameplate divided by 2 (nameplate/2) convention for estimating power loads. If that method were accurate, then the peak draw values would be 50% of the PSU nameplate

PAGE 6 values and fall on the 50% line in Figure 2, yet review of Figure 2 clearly shows this is not the case. This example shows how the nameplate/2 practice could result in incorrect power and cooling load estimates, as well as the over-sizing (or under-sizing) of both electrical systems and cooling systems. Some of the data points fall on or near the 50% line, but the number that do is far less than the number that do not. The issue is further complicated by the redundancy (or lack thereof) in PSUs. A typical server operation with redundant PSUs has both PSUs operational and carrying 50% of the load. In the case of one failing, the functioning PSU can rapidly ramp to 100% of the server load. Two important aspects should be noted. First, the total connected load from the dual PSU nameplates is now double the non-redundant load and can add to far greater over-sizing if not clearly understood. Second, dual power supplies will typically increase server consumption, due to the PSU being lower down the utilization curve, which can mean lower ef ciency. This added power needs to be factored in. Clearly, rule-of thumb methods such as nameplate/2, PSU divided by 2, and others cannot account for this level of detail. V. COOLING LOAD CALCULATIONS Data center planners and managers also use IT equipment s power predictions to calculate cooling load. All of the inaccuracies involved in using nameplate data for power load calculations are repeated in the cooling load analysis. In sizing the cooling infrastructure, both the cooling load (kw or BTU/hr) and air ow rates (m3/ hour of CFM) need to be determined. To further complicate the problems created by using nameplate data, planners and managers use an estimated, historical, or rule-of-thumb differential temperature (deltat) across the servers. From that deltat and the estimated power load, they calculate an air ow rate. Unfortunately, that air ow rate is based on two inaccurate estimates (nameplate and an assumed deltat). VI. ENERGY EFFICIENCY FOCUS NEW TOOLS As a result of the focus on energy ef ciency, most IT vendors now provide power measurement and power estimation tools that can help to identify a more accurate power consumption value for their IT equipment. That value is based either on actual measurements (a power meter) or an estimation tool (hardware options and utilization). Ideally, measuring the power consumed by IT equipment provides the most accurate method for determining the power and cooling load that IT equipment places on a data center. IT equipment manufacturers provide power meters that are internal and external to IT equipment. These can be used to show trends in power consumption over long periods of time under many workloads. Measuring power consumption allows for realtime measurement of application intensity and other factors that have an impact on power consumption, such as inlet ambient temperature and/or operating voltage. Vendors power and cooling estimation tools are best for planning power and cooling loads for new equipment being placed in the data center. The values found in vendor planning tools are typically the result of using this approach, with an added margin for error that will yield much more useful power consumption estimates. Application loading or utilization within the planning tool is offset from a peak power consumption level based on some type of proxy application, such as an industry benchmark or stress program. It is the deviation from actual usage to this proxy that adds one level of inaccuracy. Another factor that increases inaccuracy is the variation of power consumption in electronic components that vary from lot-to-lot. Although vendors tools typically yield results that are far better than common de-rating methods, their results tend to be somewhat conservative as compared with actual power measurements from the IT equipment.

PAGE 7 An additional bene t of power and cooling estimation tools is that some tools provide more accurate estimations of the speci c air ow rates for the server con guration, utilization, and inlet air temperature. The air ow rates combined with the heat released by the equipment will provide the information that planners need to size the cooling infrastructure. Table 1 below is a typical example of the results obtainable by using a vendor estimation tool compared to the value that would likely have been derived from using a nameplate de-rating estimate. Equipment Design Input Tool Output 1U server Power 370 watts 208V AC power (power factor.98) Flow rate 32 CFM Dual socket quad core processor Weight 46 lbs 8 GB memory Current 1.78 A 2 disk drives deltat 37.3 F Dual power supplies Operating condition: 25 C server inlet temperature, typical workload TABLE 1. TOOL EXAMPLE Using the con guration above, the sample vendor s power estimate is 370 watts. Per the vendor s guidance, this estimate would still be conservative, even for a virtualized load that typically operates at a higher level of processor utilization. General-purpose servers running at lower utilization levels would measure up to 20% less in terms of total power consumed. The tool estimates a worst-case con guration for the same server to be 490 watts under full load, but that power estimate would increase to 555 watts if the server were running a more processor-intensive load, such as HPC (high performance computing) cluster computing. Furthermore, if the system were running a processor-intensive load at a maximum inlet temperature of 35 C (95 F), the power estimate would increase to 620 watts due to the electrical inef ciencies that result from running at high temperatures. This server has a power supply nameplate value of 670 watts. If, for example, the server power supply was actually delivering 650 watts of DC power, and it was running at 85% ef ciency, then it would be consuming 765 watts and releasing approximately 2610 BTUs (1 W of load requires 3.412 BTUs to cool) of heat into the room. This example demonstrates that there are many parameters that affect power consumption, including, but not limited to, server con guration, processor selection, memory amount and con guration, number of drives, peripherals, I/O, processor utilization, and inlet temperature. The sophistication of vendor estimation tools continues to grow and provide the end user with an increasingly accurate method of predicting power and cooling loads.

PAGE 8 VII. TOOL LINKS As mentioned above, actual power measurements are ideal, but many IT vendors produce power estimation tools and calculators that can be used for planning purposes. Cautionary note Updates to calculators may occur without notice. While The Green Grid will attempt to keep the reference links below as up-to-date as possible, readers should contact the supplier to verify they have the latest version. Company Link Comments HP TOOL: http://h30099.www3.hp.com/con gurator/powercalcs.asp HP also has a white paper on the tool (see second link) WHITE PAPER: http://search.hp.com/redirect.html?type=reg&qt=powe r+calculator&url=http%3a//h20000.www2.hp.com/bc/ docs/support/supportmanual/c00881066/c00881066. pdf%3fjumpid%3dreg_r1002_usen&pos=1 Dell PLANNING FOR ENERGY REQUIREMENTS: http://www.dell.com/calc This link includes instructions and Data Center planning tools IBM Calculator for the blades and modular product lines (x86 architecture server systems and rack storage), the tool can be downloaded at: http://www-03.ibm.com/systems/bladecenter/resources/ powercon g/ For the Power Processor-based server systems, an online tool is available at the following link: www.ibm.com/systems/support/tools/estimator/energy Sun Power Tool: Calculators http://www.sun.com/solutions/eco_innovation/ powercalculators.jsp Cisco Power Tool: Calculator https://tools.cisco.com/cpc/authc/forms/cdclogin. fcc?type=33619969&realmoid=06-00071a10-6218- 13a3-a831-83846dc90000&GUID=&SMAUTHREASON=0& METHOD=GET&SMAGENTNAME=$SM$GDuJAbSsi7kExzQD RfPKUItt%2bPcjKOjTGlbtk%2fRp7BdNYLiP9lyOBjXBU5PAxIX D&TARGET=http%3A%2F%2Ftools.cisco.com%2Fcpc%2F TABLE 2. POWER/COOLING ESTIMATION TOOLS Registration required

PAGE 9 VIII. PROPER USE OF CALCULATING TOOLS Remember that many of these vendor estimation tools are estimators. Because there are so many factors that affect power draw, it is nearly impossible to exactly match a tool to the actual results that will be seen in every data center. If more exact power or cooling planning information is needed, there is no substitute for real-time measurement using the particular system installed in a controlled fashion and running its normal workload and utilization levels. The tools may be used for electrical and cooling load planning instead of nameplate or maximum measured values, and they will yield far more accurate predictions than those derived from other estimating methods. However, detailed attention should be paid to individual server con gurations and their power systems and potential system upgrades. For example, changes to the memory con guration or additional PCI cards over the life of the server could affect the amps from an electrical circuit, the heat released, and the server air ow or deltat. Finally, when using tool estimators for data center planning and design purposes, keep in mind that while cooling systems can handle short increases in heat load in the room, an increase in amperage draw from a server that is over the limit of a circuit breaker will create a problem far more quickly. Servers can temporarily run on an increased intake air temperature, and typically a cooling system has the ability to react to that short temperature spike and provide extra cooling. If a spike in amperage draw over the circuit breaker limit occurs, that server and others around it may go down due to tripped breakers in the rack-level power distribution or in a oor PDU. Know what safety factors have been designed into your data center. Use the online tools to get good estimates for power draw and heat load, but also measure your units once installed to verify that the tools were correct. IX. SUMMARY/CONCLUSION Measuring power consumed by IT equipment provides the most accurate method for determining the power and cooling load that IT equipment places on a data center. However, vendors provide power and cooling calculators that can be used to determine a more accurate estimation of load based on hardware con guration and utilization. The Green Grid highly recommends the use of power meters and power calculators for ensuring that equipment resource infrastructures are sized properly to accommodate the actual electrical and heat loads. Due to planning calculator tools inability to re ect your speci c application, the accuracy of your planning/ deployment process can be increased by factoring in your understanding of the differences between the tools estimates and your own application-based measurements. If future capacity requirements are unknown, refer to ASHRAE 90427, Datacom Equipment Power Trends and Cooling Applications (2005) 2. In summary, using nameplate values to estimate power resource requirements or space-cooling loads can produce highly unreliable results, including substantial under-sizing (miscalculating) or over-sizing (using the label as is without understanding hardware options and equipment utilization) of electrical systems and cooling systems. Miscalculations also can result in loss of service or excessive cost of operations. The more accurate the power consumption data that can be established, the more ef cient and reliable the electrical distribution systems can be. In addition, accurate data helps improve service availability and optimize operational costs for power and cooling systems

PAGE 10 X. REFERENCES 1. ASHRAE 90431, Thermal Guidelines for Data Processing Environments (2004) https://eweb.ashrae.org/eweb/dynamicpage.aspx?site=ashrae&webkey=69c74d61-facd-4ca4-ad83-8063ea2de20a&listwhere=(prd_etab_ext%20like%20 %25datacom%25 ) 2. ASHRAE 90427, Datacom Equipment Power Trends and Cooling Applications (2005) https://eweb.ashrae.org/eweb/dynamicpage.aspx?site=ashrae&webkey=69c74d61-facd-4ca4-ad83-8063ea2de20a&listwhere=(prd_etab_ext%20like%20 %25datacom%25 ) 3. United States Environmental Protection Agency, Energy Star for Workstations http://www.energystar.gov/index.cfm?c=new_specs.enterprise_servers 4. Find an ENERGY STAR Compliant Computer: http://www.energystar.gov/index.cfm?fuseaction= nd_a_ product.showproductgroup&pgw_code=co 5. ENERGY STAR Framework Document www.energystar.gov/ia/partners/prod_development/revisions/downloads/computer/workstation_ Framework_Document081705.pdf XI. ABOUT THE GREEN GRID The Green Grid is a non-pro t trade organization of IT professionals formed to address the issues of power and cooling in datacenters. The Green Grid seeks to de ne best practices for optimizing the ef cient consumption of power at the IT equipment and facility levels, as well as the manner in which cooling is delivered at these levels. The association is funded by four levels of membership, and activities are driven by end-user needs. The Green Grid does not endorse any vendor-speci c products or solutions but seeks to provide industry-wide recommendations on best practices, metrics, and technologies that will improve overall data center energy ef ciencies. www.thegreengrid.org