White Paper - Infrastructure in an energy-efficient data centre

Transcription

1 White Paper - Infrastructure in an energy-efficient data centre Verena Schwalfenberg Dipl.-Ing. Markus Schmidt Copyright 2008 All rights reserved. Rittal GmbH & Co. KG Auf dem Stützelberg D Herborn Phone +49(0)2772 / Fax +49(0)2772/ V10

2 Contents List of figures... 3 Abbreviations... 4 Executive Summary... 5 Introduction... 6 Energy-related problems in data centres... 7 Infrastructure... 7 Development of electricity costs and consumption... 7 Reasons for the increasing energy consumption Concepts and options for optimization Power supply and distribution Cooling Room cooling LCP Air/water heat exchanger Free cooling Outlook

3 List of figures Figure 1 Development of electricity costs (industrial) in Germany... 7 Figure 2 Procurement costs for new servers compared to the costs for power & cooling... 8 Figure 3 Increase in server energy density... 9 Figure 4 Relationship between operating costs over 3 years and server procurement costs... 9 Figure 5 Watts per US$ 1,000 for 1U servers Figure 6 RimatriX5 efficiency monitor Figure 7 Efficiency of a 40kW UPS Figure 8 RimatriX5 Pay-as-you-grow concept Figure 9 Flexible scalability of PMC Figure 10 Raised-floor cooling with cold / hot-aisle design Figure 11 Cold-aisle containment Figure 12 Cold-aisle containment & LCP Inline cooling Figure 13 Average annual temperatures in Germany

4 Abbreviations CO 2 - Carbon dioxide COP - Coefficient of performance CPU - Central processing unit IDC - International Data Corporation IT - Information technology kw - Kilowatt kwh - Kilowatt hour LCP - Liquid Cooling Package MTTR - Mean time to repair PDM - Power Distribution Module PDR - Power Distribution Rack PMC - Power Modular Concept DC - Data centre SI-EER - Site infrastructure energy ratio UPS - Uninterruptible power supply 4

5 Executive Summary This white paper, focusing on the infrastructure in energy-efficient data centres, provides an overview of the basic energy problems in existing data centres. It explains which components in an infrastructure place the biggest load on the energy budget. Innovative solutions are needed as additional power consumption leads to increased waste heat, which in turn has be cooled using energy-intensive methods. The ever-rising electricity costs also make this a top priority. It is predicted that by 2010, electricity and cooling costs will account for 50% of the total costs of a data centre 1. There are a number of reasons for this rise in energy consumption, including the growing demands on computing power and the continuous increase in the packaging density in the server area. At the same time, there is also huge potential savings offered by e.g. IT chillers, which often remains untapped. RimatriX5 is one solution that helps improve energy efficiency. It is a modular concept from a single source that meets the demands for an energy-efficient data centre infrastructure. 1 Source: International Data Corporation (IDC)

6 Introduction Energy efficiency is one of the key terms used in IT today in conjunction with data centres. There are numerous reasons for this development. The increasing demands on computing power, rising server packaging densities, more waste heat, CO 2 emissions and, above all, the huge rise in energy costs automatically push for energy-efficient solutions to help cut electricity consumption. A few years ago, the day-to-day energy costs for data centre operation played a relatively insignificant role. More emphasis was placed on procurement costs for the IT hardware, such as servers, storage systems and network technology. Over time, this technology has become increasingly powerful, which has caused power consumption to rise accordingly. It is now clear that this level of power consumption leads to considerable costs and problems. That is why manufacturers of IT hardware have increasingly been integrating energy-saving mechanisms into their products. These include flexible clock rates, lower operating voltages and sections of processor chips that can be deactivated as necessary. So far, little attention has been paid to the data centre infrastructures used to provide and distribute the energy required by the IT hardware and to dissipate waste heat. In a worst case scenario, this energy consumption is higher than the consumption of the IT hardware itself. Therefore, innovative concepts and solutions are also required in the infrastructure to boost energy-efficient operation. 6

7 Energy-related problems in data centres Infrastructure The infrastructure of a data centre must solve two basic problems in relation to energy: Reliable provision of energy Dissipation of the heat that is generated These are both key cost factors in day-to-day operation and have a major impact on the efficiency of a data centre. In simplified terms, the efficiency of a data centre can be defined as follows: efficiency DC = power _ consumption power _ consumption IT DC power _ consumptionit = server + storage + network _ technolog y power _ consumptiondc = power _ consumptionit + ( USV + PDM + cooler + chiller) The efficiency of a data centre depends on the infrastructure technology required for the uninterruptible power supply (UPS), power distribution (PDM), coolers and water chillers. The lower the efficiency levels of these individual components, the poorer the overall efficiency of the data centre. If the infrastructure components exhibit a poor level of efficiency, this has a double impact on the overall system the energy costs rise and the server's power consumption increases. Development of electricity costs and consumption Figure 1 Development of electricity costs (industrial) in Germany 2 Electricity costs for the commercial sector have been increasing steadily since 2000, and the trend looks set to continue as a result of the worldwide demand for energy. 2 Source: Energieagentur NRW (Energy Agency NRW) 7

8 Furthermore, servers today consume more power than they did a few years ago despite state-of-theart architectures, which is causing the absolute consumption values for the infrastructure to skyrocket. Figure 2 Procurement costs for new servers compared to the costs for power & cooling 3 It is not surprising, therefore, that the power and cooling costs are assuming a bigger share of the overall costs of a data centre. In Figure 2, the IDC forecasts that energy costs will account for around 40% of the total IT budget in 2010, while other experts predict that this figure may even rise to 50%. 4 In addition to high energy consumption and the associated costs, increased energy consumption is giving rise to a further problem in that existing data centres can no longer handle the power requirements and waste heat. 3 Source: International Data Corporation (IDC) Source: ( CEO Guide to Green Computing, businessweek.com 2007/05). 8

9 Figure 3 Increase in server energy density 5 The energy density in data centres increased more than 10-fold between 1994 and 2004 alone. This rise places extreme demands on the infrastructure of the power supply (UPS), power distribution and cooling concept. This is because every watt of power supplied must also be removed from the data centre in the form of waste heat. Figure 4 Relationship between operating costs over 3 years and server procurement costs 6 According to calculations from the Uptime Institute, the server operating costs skyrocket compared to procurement costs when users continue to work with old concepts and structures. In the best case (projection B) 7 the increase is still considerable, but projection A 8 is realistic. However, the old infrastructures must be modernised to achieve these values. 5 Source: Datacom Equipment Power Trends and Cooling Applications, Source: Data Center Energy Efficiency and Productivity, Uptime Institute 7 State-of-the-art 9

10 Further evidence of an obsolete infrastructure is its SI-EER value. The site infrastructure energy efficiency ratio is the ratio between the energy supplied and the energy that reaches the IT hardware. It is the reciprocal value of Efficiency DC. According to the calculations of the Uptime Institute, this value came to an average of 2.5 in This means that of the 2.5 watts supplied, only 1 watt reaches the IT hardware. In a data centre covering 2,800 square meters, reducing the SI-ERR to 2.0 would cut energy costs by around US$ 1 million a year. Reasons for the increasing energy consumption To be able to implement cost-saving measures and therefore run a data centre more efficiently, it is important to identify the reasons behind the rising energy consumption. The first and most obvious reason for this is the fact that the performance in the server area is increasing all the time. Moore s Law 9 states that the complexity of the chip circuitry doubles every 24 months. This doubling of integration density goes hand in hand with a significant increase in computing power. As a result of the growing demands, more and more servers are being operated in each data centre. Therefore, the total power consumption of data centres rises even though the value for watt per computing power falls significantly. This phenomenon becomes clear if you consider how many server watts could be purchased over the years for US$ 1,000. Figure 5 Watts per US$ 1,000 for 1U servers 10 From 2000 to 2006 alone, the power per US$ 1,000 increased more than 13-fold. Even in best case scenarios, the possible developments (projections A and B) predict a rise to 157 watt/us$ 1,000. One reason for the considerable increase in recent years is the blade server. Its extremely compact design means that a very high power density can be generated in the rack. A rack equipped with several blade centres can easily result in power consumption greater than 20kW, which then has to be cooled. With an SI-EER of 2.5, the data centre has an input power of 50kW for just one rack! 8 Best Practice 9 A law devised by Gordon Moore in 1965 that states that that the number of transistors that can be inexpensively placed on an integrated circuit increases exponentially, doubling approximately every two years 10 Source: Data Center Energy Efficiency and Productivity, Uptime Institute 10

11 Many servers are bought to host one or two special services. These applications often only place a load of between 10 and 20 percent on the server, which means that 80% of the CPU s computing time is spent on idle operation. This would not be a major problem if power consumption was scaled linear to the computing load. Unfortunately, this is not the case. A server with only a 20% load needs around 75% of the energy that it needs in productive operation with a 70 80% load. This problem cannot be solved by the infrastructure, but it places a tremendous load on it. Figure 6 RimatriX5 efficiency monitor The RimatriX5 efficiency monitor illustrates the problem with infrastructures that are oversized or not operated to their full capacity. The zoom view shows that the efficiency of the data centre (bottom graph, blue curve) drops rapidly to less than 10% when the IT load (top graph, green curve) is far below the normal load. During the planning of a data centre, the infrastructure is often designed to cater for maximum server configuration. In other words, the capacities of the UPS and cooler in particular far outstrip the requirements in the first few years of operation. In addition to causing very high procurement costs, at worst the devices also operate at less than 30% of their maximum load and, therefore, with a significantly lower level of efficiency. 11

12 Efficiency 40kW UPS 95.0% 94.0% 93.0% Efficiency 92.0% 91.0% 90.0% 89.0% 88.0% 15% 30% 52% 76% 102% Load Wirkungsgrad Figure 7 Efficiency of a 40kW UPS When operated to 90% of its capacity, a 40kW UPS, for example, has an efficiency level of approx. 94.5%. However, if it is operated at just 25% of its capacity, the efficiency level drops to 91%. With a power supply of 100kW, a drop in the efficiency level of 3.5 percentage points increases power losses by 3.5kW. Over a year, this amounts to 30,660kWh or 3,066 (based on electricity costs of 0.10 per kwh). This loss of 3,066 is caused purely by poor sizing, which also has to be paid for at the investment stage (bigger UPS = higher procurement costs). Similar calculations can also be done for cooling. Solutions in this area must, therefore, ensure that components deliver optimum efficiency throughout the service life of a data centre. Cost savings Runtime kwh Saving [ ] 1 year 30,660 3,066 5 years 153,300 15, years 306,600 30,660 Potential savings from improved UPS efficiency Reduced power loss of 3.5kW 12

13 Concepts and options for optimization Power supply and distribution As mentioned in the previous section, two main factors impact on the power supply of a data centre efficiency and sizing. Sizing means choosing the right-sized components for the system, and it should be based on the demand and the final expansion stage of the data centre. When data centres were planned in the past, only the maximum configuration was factored into the equation to be on the safe side. This configuration is often only achieved in the final years of a data centre if at all. To enable infrastructure components, such as UPS and power distribution, to operate at maximum efficiency during the initial phase, they must be configured slightly smaller, with the option of expansion at a later date. This balancing act can be realised with modular systems such as RimatriX5 from Rittal Figure 8 RimatriX5 Pay-as-you-grow concept 12 Legend 1 Current power consumption 2 Scalable, demand-oriented and gradual adaptation of capacity (RimatriX5) 3 Overcapacity configured with conventional DC technology x-axis: Service life in years y-axis: Installed infrastructure in % / room capacity Figure 8 demonstrates the pay-as-you-grow concept from RimatriX5. The components for UPS, power distribution and cooling grow as the data centre expands. This modular concept can be illustrated using the example of Rittal PMC 200 UPS. 12 Source: Rittal GmbH 13

14 Figure 9 Flexible scalability of PMC 200 The PMC 200 can be operated using three 20kW modules as an (n+1)-redundant 40kW UPS. If new servers demand more power, there is no need to buy new or additional UPSs. The power output can be increased simply by inserting another module whilst the system is operational. These modules are available in several classes ranging from 8 to 40kW, enabling flexible adaptation to the power consumption required. The modular n+1 redundant design offers energy-saving benefits for both upgrades and operation. A second UPS is not required in the event of failure an additional module with less power loss is sufficient. The modular concept also boosts availability. During service scenarios, a faulty module can be replaced in just a few minutes. This results in a very low MTTR (mean time to repair). The state-of-the-art system design without a transformer cuts down on the use of copper in production and makes for above-average efficiency of more than 95%. The same modular concept with all its energy-saving benefits can also be realised for power distribution. Rittal offers such a system with its Power Distribution Rack (PDR) and Power Distribution Modules (PDM). Cooling Room cooling The second biggest factor that impacts on the efficiency DC value is data centre cooling. Even with an average SI-ERR of 2.0, a data centre with a 100kW IT load creates almost 200kW heat losses that have to be cooled. The traditional method is room cooling, whereby cold air is blown through the room or better, through a raised floor into the DC. In older data centres, this arrangement can cool a heat load of between 1 and 2kW per rack; in newer, better planned DCs, this value increases to 5kW. However, this solution demands very accurate implementation of the cold-aisle concept. Figure 10 shows this type of cold-aisle scenario. 14

15 Figure 10 Raised-floor cooling with cold / hot-aisle design 13 It is essential that hotspots or heat leakages, which can cause the cold and warm air to mix, are avoided. This reduces the efficiency of the cooling output considerably and threatens the reliable operation of the servers at these points. Figure 11 Cold-aisle containment 14 The next optimisation step is cold-aisle containment, whereby cold air is directed using additional ceilings, walls and doors. This method is some 10-20% more efficient than traditional cold-aisle cooling without containment. 13 Source: Rittal GmbH 14 Source: Rittal GmbH 15

16 Figure 12 Cold-aisle containment & LCP Inline cooling 15 LCP If the cold air supplied via the raised floor is no longer sufficient, LCP Inline i.e. cooling devices inserted between the racks is the next step to optimise the cooling efficiency. LCP Inline devices from Rittal are the ideal solution, offering a high level of efficiency and excellent reliability. With a Liquid Cooling Package (LCP), the air is cooled via an air/water heat exchanger, enabling very high cooling outputs. However, if several fully-equipped blade centres are installed in the racks, the cooling load in the enclosure is too high for traditional or optimised room cooling concepts. In this case, rack-based cooling is virtually compulsory. The encapsulated rack is cooled and does not affect the temperature in the server room. The LCP Modular or LCP Plus system can be used to cool the rack. The cooling output of these systems, which are also based on air/water heat exchangers, can be adapted and extended step-by-step via cooling modules, in line with the RimatriX5 pay-as-you-grow concept. The maximum cooling outputs for LCP Modular and LCP Plus are 20kW and 30kW respectively. In very critical applications, a redundant design can be realised with the LCP system with 2 LCPs being used to cool one rack. Air/water heat exchanger In cooling solutions using air/water heat exchangers, the cooling water must be made available with a specific inlet temperature and the heated water from the return flow must be cooled again. IT chillers are usually used for this purpose. The power of the IT chiller must be adapted to the waste heat that is generated. Rittal offers a finely-graded range with cooling outputs between 15kW and 462kW. These power levels in themselves indicate that cooling requires an enormous amount of energy, coupled with substantial costs and CO 2 emissions. 15 Source: Rittal GmbH 16

17 Thermometer values on the rise Annual average temperatures in Germany In degrees Celsius linear trend long-term average Free cooling Figure 13 Average annual temperatures in Germany 16 Free cooling offers good savings potential for IT chillers. A free cooling system uses ambient air to cool the heated cooling water. This method can always be used to generate cold water when outside temperatures are low. Free cooling aims to cut operating costs and reduce greenhouse gases. Hot air is extracted without the use of a compressor, which reduces the total electrical energy required by the IT chiller, thus making it much more efficient. In standard applications, the IT chiller can be operated in free cooling mode with an outside temperature of around 10 degrees Celsius. On average, the temperature in Germany is below 10 degrees for almost 50% of the year. As air-conditioning in a data centre usually runs 24/7, the savings potential is also almost 50%. In standard cooling mode, a 462kW chiller requires an input power of 96.25kW. Cooling efficiency Q=462kW; coefficient of performance 17 : COP=4.8 Required input power for the IT chiller: Q 462kW P = = = kW COP 4.8 When the IT chiller is operated in free cooling mode, the required input power drops to around 10kW. Based on an electricity price of 10 cent/kwh, this results in potential savings of almost 38,000 every year! 16 Source: DWD / FAZ graphic 17 Thermal efficiency of heat pumps, identical with the term coefficent of performance 17

18 Sample calculation for IT chiller costs with and without free cooling Power consumption of chiller with compressor [kw] Operating hours per yeear Costs [ ] Power consumption of chiller without compressor [kw] Operating hours per yeear Costs [ ] Total costs per year [ ] Without free cooling With 30% free cooling With 50% free cooling With 70% free cooling ,760 84, , ,132 2, ,628 2,628 61, ,380 42, ,380 4,380 46, ,628 25, ,132 6,132 31,360 Compared to costs without free cooling, a system that uses 50% free cooling saves 376,680 kwh a year, which is equivalent to 37,668/year. This system works by feeding the water via an air/water heat exchanger to the outside air where it is cooled by temperature balancing with the ambient temperature (<10 C). This process can be enhanced by fans. This method is efficient because it dispenses with the need for a compressor which would consume power. 18 The calculation is based on an electricity price of 10 cent/kwh 18

19 Outlook The importance of energy efficiency for data centre operators will continue to grow. The need for action will also increase in view of the costs, energy availability and environmental factors. RimatriX5 from Rittal is an innovative and modular concept from a single source that meets the infrastructural requirements for a modern, energy-efficient data centre. The modular design of RimatriX5 offers a wide variety of options for expanding and modernising existing data centres, regardless of whether the operator wants to install a complete new cooling system or just a single rack. The next optimisation steps for the RimatriX5 range are already in development and will ensure that existing and new installations will continue to be energy-efficient in the future, too.