Resilience, Efficiency and Scalability in Data Centre UPS Systems

Resilience, Efficiency and Scalability in Data Centre UPS Systems By Janne Paananen, Technology Manager, Large Systems Group, Power Quality Division Eaton EMEA Executive summary Operating in a fiercely competitive industry means that data centre operators are under enormous pressure to keep operating costs low and, in this environmentally conscious era, they are also under pressure to minimise the environmental impact of their operations. It is not hard to see that energy efficiency is the key to responding effectively to these pressures, but energy efficiency is not and can never be the sole goal of data centre operations. Data centre operators also want their systems to be readily scalable so that they can respond quickly and cost-effectively to the seemingly inexorable growth in demand for their services and, overarching all of these requirements is the need for near 100% availability. Resilience the ability of IT installations to tolerate faults without serious impairment of the service they provide and, crucially, without the risk of data loss is, therefore, a major focus in today s demanding IT world. The energy efficiency, scalability and resilience of a data centre depend on numerous factors, but there can be no doubt that one of the most important is its uninterruptible power supply (UPS) installation. There are, however, significant challenges involved in designing a UPS installation that combines efficiency with resiliency and scalability. This paper explores those challenges and explains how they can be addressed effectively and affordably by taking advantage of recent developments in UPS technology. Table of contents Scalability Issues... 2 Scalability and bypass... 2 Scalability and ease of expansion... 2 Scalability and reliability... 3 Reliability Issues... 4 Reliability and efficiency... 4 Reliability and cooling... 5 Reliability and battery management... 6 Reliability, resilience and availability... 6 Conclusion... 7 About Eaton... 7 About the author... 8 EPE0747 www.eaton.eu/powerquality August 2013

Page 2 of 8 Scalability Issues Scalability and bypass A major concern in the design of scalable UPS installations is making sure that adequate bypass provision is available at all stages in the development of the UPS installation. As its name suggests, the bypass function of a UPS installation routes power around the UPS and allows mains power to be fed directly to the loads in the event of a fault, or if the UPS has to be taken out of service for maintenance or upgrading. While the use of the bypass function is a comparatively rare occurrence, it is nevertheless an essential feature of every UPS installation. Scalable UPS installations invariably adopt a modular architecture so that more UPS modules can readily be added as demand grows. In many cases, each UPS module will have its own bypass provision in the form of a static semiconductor switch. Initially, this may appear to be a good arrangement, since the bypass provision automatically grows as the size of the installation is increased. With this type of UPS design (with one internal static bypass switch for each UPS) there is, however, a problem. Upon initial installation when only a few modules are installed, bypass capacity is limited. Because of the inherent limitations of the semiconductor switches, reliable co-ordination with the downstream protection devices (usually circuit breakers) can be difficult or even impossible to achieve. As a result, under fault conditions, the UPS system may not be able to provide enough let-thru current for downstream protection device in load distribution to quickly isolate the faulty circuit from electrical system thus jeopardising power availability for other load branches as well. One solution to this problem is to provide the UPS installation with a centralised static bypass switch, instead of relying on individual switches in each UPS module. This switch must be rated to suit the planned final capacity of the system and to provide enough let-thru current for proper co-ordination of protective devices. This solution is effective, but it is also costly and takes up valuable floor space. There is, however, now an attractive alternative systems that combine distributed bypass modules with modular UPSs. This approach, where all static bypass switches are installed on day one but UPSs are equipped with a fewer number of power modules, can ensure that adequate bypass capacity is available at every stage of the installation s growth, and greatly eases the problem of achieving protection selectivity. It also maximises scalability. Whatever solution for static bypass design is adopted, it is important to remember that, unless appropriate measures are in place, a single shorted static switch can paralyse the whole installation. It is essential, therefore, to ensure that the bypass systems incorporate backfeed protection so that the UPS does not attempt to feed power back into the supply system via a defective switch. This protection is has been required by legislation in most countries to ensure personnel safety and when combined with shorted SCR detection, can highly increase the system resiliency. It is also essential that each and every static switch in the installation can safely handle the full prospective fault current available from the mains supply as required by latest standards. Scalability and ease of expansion Scalable UPS systems are by definition expandable, but that does not necessarily mean that all scalable systems are easy to expand. In fact, the mechanics of expansion should always be considered very carefully during the initial design phase of the installation. Factors to take into account include the physical arrangement of the modules those requiring only front access are much easier to work on than those that require access at both the front and the rear. The number of connections required when adding the modules should also be considered, along with the accessibility of these connections. It is also a good idea to plan ahead for future expansion by, for example, fitting junction boxes for signal wiring.

Page 3 of 8 Figure 1: Scalable and modular UPS systems allow for ease of expansions and optimisation pf both CAPEX and OPEX. Additionally, it is worth bearing in mind that the UPS units themselves can be modular. This makes expansion particularly straightforward, as it is only necessary to install additional power modules in the UPS. This work can be carried out quickly, conveniently and inexpensively by the UPS supplier, without the need to involve other contractors to provide, for example, additional cabling. Another factor that should be considered at the design stage is provision for concurrent expansion, so that the capacity of the UPS installation can be expanded without the need to interrupt mains supplies to the loads. Whether or not this is possible, however, when the time comes for expansion, the work should always be carefully planned and executed in a controlled manner. In today s IT environment, there is simply no place for a plug and pray approach! Scalability and reliability Scalability in UPS installations is virtually synonymous with modularity and, on the face of it, adopting a modular architecture may not seem to have major implications for reliability. In practice however, when multiple UPSs are operated in parallel, as they are in modular systems, provision has to be made to ensure that the load is shared equally between them. When this is done by commonly adopted techniques, the overall reliability of the installation can be significantly degraded. The standard arrangement for load sharing is to use a master load controller and to arrange for each of the parallel-connected UPSs to communicate with this to share load-balancing information. Unfortunately, this arrangement has the following two significant weaknesses: 1. The complexity of the load sharing system increases rapidly as more and more UPSs are connected to it, and the more complex the system becomes, the more likely it is to fail. This problem is significant even when relatively small numbers of modules are operated in parallel and systems with large numbers of parallel-connected modules more than, say, 30 are very unlikely to deliver acceptable performance. 2. The master controller and its associated communications network become single points of failure for the UPS installation. The more connections and components there are associated with this network, the more likely it is to eventually fail due to component failure or servicing procedures. If the master controller or its network should fail, the whole installation is likely to go out of service, and its loads will all be dropped. Fortunately, a more resilient load-sharing technology is available. With this, each of the parallel-connected UPSs has its own self-contained load balancing system. This references only the UPS s own output to control the UPS in such a way that it fully handles its own share of the load but no more. In other words, this load-sharing system operates without the need for a master controller and communication network, thereby eliminating the single point of failure associated with traditional modular UPS architectures. Additionally, in principle any number of UPSs with internal load balancing can be operated in parallel without increasing the complexity of the load-sharing arrangements.

Page 4 of 8 Because the ability to dependably share loads between UPSs operating in parallel without degrading the reliability of the UPS installation is the foundation stone for successfully implementing systems with the high levels of scalability that data centre operators now routinely require, these are important points. Reliability Issues Reliability and efficiency The key to maximising resilience in UPS installations is to build in redundancy. Unfortunately, however, adding redundancy in traditional installations means that the load levels on the individual UPS modules are reduced, and lightly loaded modules deliver poor operating efficiencies. Once again, the emergence of new technologies has provided a solution by making it possible to concentrate the load on just enough modules to satisfy the immediate demand for power, rather than sharing it equally between all of the UPS modules in the system. This means that the on-load modules are well loaded and therefore operate efficiently, while the off-load modules can be put into a quiescent state where they consume almost no power. When the load on the UPS installation increases, the quiescent modules can be brought back into full operation almost instantly typically within two milliseconds ensuring that the loads supported by the UPS installation are completely unaffected by the transition. Further, the best systems incorporate internal monitoring which ensures that all of the modules are restored to normal operation if any kind of anomaly is detected, a provision that ensures system resilience is fully maintained. Figure 2: UPS efficiencies when operating in on-line mode, load optimised on-line operation and ecomode. Particularly well suited for use in dual-fed A and B applications, this approach to combining energy efficiency with resilience delivers maximum benefits in installations where the individual UPSs are made up of several power modules. In these cases, a very accurate match can be made between instantaneous power requirements and the number of modules that need to be in service to meet these requirements. This means that energy efficient operation can be achieved at virtually all load levels as the power supplied is determined by or matched to the demand of the IT load.

Page 5 of 8 Another development that is significantly helping to boost the energy efficiency of highly redundant UPS installations is the emergence of UPSs with energy saver functionality. This is most usually incorporated in the double-conversion UPSs that are now used for almost all critical data centre applications. It works by, in effect, feeding the load direct from the mains supply when this supply is within tolerance and problem free, but switching to full double-conversion mode within two milliseconds of a mains supply problem occurring. This transition is completely invisible to the connected loads. With energy saver mode, which has no negative implications for reliability, operating efficiencies of 99% or more can be achieved. Reliability and cooling Where highly efficient UPSs are used, data centre operators are increasingly finding it expedient to reduce cooling provisions as a way of making even more energy savings. Indeed, there is now widespread interest in free cooling where air at ambient temperature is used to cool the data centre hardware. In terms of reducing the environmental impact of data centre operations, these trends are laudable, but a consequence of their adoption is that temperature within many data centres is now significantly higher and less tightly controlled than it used to be. This makes it increasingly important to consider the way the UPS system behaves at elevated temperatures. Most UPSs are designed to operate normally up to a specified maximum temperature, which is often 40ºC, in line with IEC 62040. It is a good idea, however, to avoid products where the manufacturer qualifies this temperature limit with statements along the lines of conditions apply, which may suggest that derating will be needed as the specified temperature is approached. Figure 3: Most UPSs are designed to operate normally up to a specified maximum temperature, which is often 40ºC Even if the UPS can be used without derating at up to 40ºC, it is worth investigating further. What happens, for example, if the ambient temperature reaches 45ºC? Will the UPS trip and drop the load? There are many cases where such behaviour would be completely unacceptable; no one would pretend that operating at elevated temperatures is good for a UPS, but it is unlikely to lead to rapid failure and the increased risk of degradation is almost always going to be a better option than an unexpected interruption of the supply to the load.

Page 6 of 8 For these reasons, the best UPSs for critical applications can now be configured to cope gracefully with over-temperature conditions, as might arise in the case of cooling system failure, for example. In some installations, as the temperature rises, a switch to high-efficiency energy saver mode can be made as described in the previous section, which reduces heat generation. Even if energy-saver operation is not an option or has been disabled, a resilient UPS continues to operate until the temperature of the semiconductor devices at their heart becomes so high that failure is likely to be imminent, and only then do they trip to protect themselves from damage. These provisions mean that operation at ambient temperatures well above the nominal maximum specified in the data sheet is likely to be possible for surprisingly long periods, especially with partial loads where less heat is generated by UPS thus giving extra system resiliency and allows load protection even in case of cooling system failures or other abnormal conditions leading to elevated temperatures in the UPS room. Reliability and battery management Batteries are a crucial part of every UPS, which means that effective battery management is essential if high levels of reliability are to be achieved. In particular, it is important to understand that when batteries are subjected to a high rate of discharge for short periods, it is safe to discharge them to a comparatively low cut-off voltage. This is what typically happens in UPS applications, provided that the UPS is operating close to full load. Taking this low cut-off voltage into account means that, in many cases, smaller, lighter and less costly batteries can be used. Extra care must be taken, however, in applications where the UPSs may be lightly loaded for some or all of the time, as batteries that are discharged slowly require a higher cut-off voltage to protect them. This is unlikely to affect battery sizing as the batteries will be sized on the basis of delivering the required runtime when the UPS is fully loaded but it does mean that, in order to maintain good battery health, the battery management system must be able to automatically adjust the cut-off voltage according to the load on the battery. To achieve good battery life and maximum operating economy, the battery management system should also incorporate intelligent charging control that adjusts the charging parameters to take into account changes in the ambient temperature. In addition, the best systems constantly monitor the status of the batteries and charge them only when necessary. Compared with the widely used approach of continuous float charging, intelligent charging control has been shown to extend battery life by as much as 50%. Regular battery testing is another key requirement for reliable UPS operation. Without this testing the real condition of the batteries is unknown until the UPS is called on to support the load, which is too late to deal with any problems. Discharge testing is by far the most accurate way of assessing battery condition. Battery management systems that offer automatic discharge tests are recommended, as this type and frequency of testing gives a reliable early warning of battery deterioration. The power taken from the battery during the automatic test must be set at a level that will provide a true test of battery condition, but the depth of discharge must not be so high that it impacts that battery s service life. Reliability, resilience and availability Much has been written about reliability, but it is still a subject that is poorly understood, particularly in relation to redundant complex systems. For these, the simplistic calculation methods usually proposed are inadequate, and more advanced calculation methods, such as the Markov model, must be used to estimate mission reliability. It is also important to understand that the real concern is not with mean time between failure (MTBF) but with mean time between critical failure (MTBCF), where a critical failure is one that leads to the system being unable to perform its prime function. In a UPS installation, for example, the probability of failure of an LED indicator would be taken into account in the MTBF calculation, but not in the MTBCF calculation as the UPS would continue to support the load even if the LED failed. When considering system resiliency, it is useful to remember that the two most important words are what if. For example, what if communication is lost in a system based on parallel-connected UPS modules? Would this result in a transfer to bypass or even a shutdown, or would it be possible for the system to remain in double-conversion mode? Another example: what if a failure occurs when the UPSs are operating

Page 7 of 8 in energy-saver mode? Would this trip the system or would all of the power modules transfer back to double conversion mode? In a well-designed system, no single component or signal failure should lead to a critical (mission) failure, and full advantage will have been taken of new operational modes, such as self-contained load-balancing, which can deliver significant improvements in reliability by reducing system complexity and stresses on components. It also worth noting that a resilient system with concurrent serviceability for all equipment that is, the equipment can be serviced without taking the system off line will give maximum availability. Finally, in all matters relating to reliability, resilience and availability, there are factors that have a significant influence but are not always taken into account. These include the impact of human operators on the system, and the influence of the location in which it will be installed, with particular reference to mains reliability, the quality and availability of the local workforce, and the likelihood or otherwise of natural disasters. Conclusion Until recently, combining resiliency with efficiency in data centre UPS systems has been a major challenge. Redundancy could be used as a way of improving reliability, but led to higher capital expenditure on equipment and increased running costs. This meant that, in some cases, reliability had to be sacrificed in order to maintain high levels of efficiency and to minimise operational expenditure. And, although taking advantage of scalability offered a potentially attractive way of reducing capital expenditure, this also had negative implications for reliability, because of the extra complexity introduced by the load-sharing arrangements in conventional modular UPS systems. Today, thanks to recent technological advances, the situation is very different. Redundancy no longer has to impact efficiency because the latest modular UPSs can be configured so that the load is at all times concentrated on the smallest number of modules needed to support it, with the result that those modules are well loaded and therefore operate efficiently. This means that low PUE (power usage effectiveness) designs are now possible with highly redundant systems. The former drawbacks associated with scalability have also been addressed. UPS modules with a selfcontained load sharing function that does not rely on communication networks and master controllers eliminate the reliability issues associated with traditional modular systems. In fact, parallel connection of UPS modules that feature self-contained load sharing can actually increase overall reliability by providing an extra level of redundancy. All of this means that there are no longer any restraints on taking advantage of scalability to reduce capital costs pay as you grow is now a valid approach to equipping data centres. Economics, customer requirements and environmental regulations are all putting increasing pressures on data centre designers to deliver scalable, redundant and resilient solutions while simultaneously minimising environmental impact. New technologies, particularly in the UPS sector, are making this possible, but design and component selection must be carried out with meticulous care if availability is to be maximised. Providing this care is taken, it is possible to increase system resiliency without compromising any of the other key design criteria. This is no pipedream for the future all of the technologies discussed in this paper are already in use around the world, and all are delivering fully on their promised benefits. About Eaton Eaton s electrical business is a global leader with expertise in power distribution and circuit protection; backup power protection; control and automation; lighting and security; structural solutions and wiring devices; solutions for harsh and hazardous environments; and engineering services. Eaton is positioned through its global solutions to answer today s most critical electrical power management challenges. Eaton is a diversified power management company providing energy-efficient solutions that help our customers effectively manage electrical, hydraulic and mechanical power. A global technology leader, Eaton acquired Cooper Industries plc in November 2012. The 2012 revenue of the combined companies was $21.8 billion on a pro forma basis. Eaton has approximately 103,000 employees and sells products to customers in more than 175 countries. For more information, visit www.eaton.eu

Page 8 of 8 About the author Based in Espoo, Finland Janne Paananen is Technology Manager in Large Systems Group, EMEA region for Eaton. Janne specialises in large UPS system solutions for data centres and special applications. He has more than 10 years of experience with large three phase UPS products and has been working in after- and presales organisations providing tailored UPS solutions, support and in-depth product trainings for Eaton s personnel and customers world-wide.