Maximizing Data Center Reliability and Utilization Using Enterprise Energy Management Technology

Transcription

1 WHITE PAPER Maximizing Data Center Reliability and Utilization Using Enterprise Energy Management Technology by Hugh Lindsay, Bill Westbrock and Terrence Tobin, Schneider Electric Introduction Companies that own or lease data center space have the challenge of maintaining near 100% availability while keeping costs down. These goals are made more difficult by an environment of constant change, a minimum of operations staff, and an increasingly complex infrastructure of power distribution, reliability equipment, cooling systems and changing server loads. Enterprise energy management (EEM) technology offers a solution by delivering a unified view of the performance of all energy assets. Through unique reach and depth of analytics, EEM systems help maximize reliability, reduce risk, improve efficiencies, and cost-effectively manage growth of data center capacity. Figure 1 Enterprise energy management offers a broad range of risk and cost management benefits to companies that own or lease data center space Schneider Electric Page 1 of 9

2 The new quest for uptime and value IT infrastructure is not a commodity, it s a competitive weapon. Gartner, Inc. i Data centers wanting to stay competitive need to differentiate reliability and value of their services. Companies that lease expect peak performance from every square foot of raised floor space. Improving efficiency and lowering the cost of operations translate to maximized profits for both parties; however, this is often at odds with guaranteeing the appropriate level of availability. Pressures to accommodate growing demands on space and infrastructure are making this balance more of a challenge. What is needed is the ability to support fast and seamless growth without major disruptions and support new services without a major overhaul of infrastructure. With the resulting upgrades to legacy systems, additional servers and storage, and variable server loading to support business growth comes increased power density and cooling capacity requirements. These new server technologies are pushing power requirements up to 300 Watts, and in some cases beyond. Each new change requires careful consideration of the infrastructure resources that are available to support this growth while still maintaining the highest reliability. Such changes can imply significant additional cost; however, data center infrastructure is typically underutilized, often by more than 50% ii. With life-cycle costs of approximately $120k over 10 years to supply the necessary infrastructure to a single server rack iii, there is great incentive to find new ways to avoid capital costs. Unfortunately, a lack of accurate and reliable information has made the high cost a tolerable alternative. Managing this energy infrastructure is demanding on operations, maintenance and IT staff. With this level of intricacy, even in the best designed data centers a single failure can have catastrophic repercussions costing millions. A data center team must manage redundant system architectures and a myriad of equipment made by various vendors who employ dissimilar monitoring and information systems. These can range from no inherent monitoring, to manual device reading from local displays, to each system having proprietary software. The lack of the right information to support a comprehensive management strategy over this collection of components has an impact on the highest level of reliability that can be achieved, with failures leaving facility personnel in the dark as to exactly what happened. To achieve a dynamic, efficient and scalable data center infrastructure, a centralized information architecture is required to support decisions and leverage existing resources. With the right information, infrastructure can be rightsized, expansion planning can be optimized and further potential savings can be achieved through managing the variable costs of energy consumption. Uptime assurance and cost-effective design can co-exist, and in turn boost profits and client satisfaction. EEM: a layer of energy IT Enterprise energy management (EEM) is a process, supported by leading-edge technology that can help data center management improve operational reliability, manage risks and maximize equipment utilization. EEM systems combine intelligent energy and power quality meters, robust communication technologies and powerful software featuring the latest in business analytics. Together, these act as a layer of energy IT reaching across all energy assets. Component and vendor neutral, an EEM system unifies and simplifies data collection, enabling whole infrastructure analytic views and reporting that support quick and precise decisions. Support for a wide range of communication protocols and standards, along with extensive I/O options enable metering devices to tightly integrate with, and thoroughly monitor, all facility equipment. Leveraging existing facility networks, data from meters and other equipment is uploaded to the enterprise-level EEM software. Information feeds from building automation systems, weather services, UPS and PDU systems, electrical distribution equipment, generator controls, or other related sources are also integrated with the EEM system (Figure 2) Schneider Electric Page 2 of 9

3 Figure 2 Typical enterprise energy management system inputs and outputs The software provides real-time status, operational data, alarm notification, key performance indicators, analytics, modeling, benchmarking, and budgeting features. These give staff the ability to respond quickly to potential threats and independently audit performance to reveal opportunities to reduce risk, create value, measure results, and communicate needs to management. The best EEM systems offer multiple user access through fully customizable graphic interfaces and are built on modular architectures that can adapt quickly to dynamic requirements. Open data access can benefit operations and IT personnel by helping both groups understand the power metrics associated with each business unit s performance. Access is enabled by data aggregation and archival in industry-standard formats, providing interoperability with IT management and other systems through OPC, XML, SMTP, SNMP or other standards. This level of compatibility and data sharing can ultimately help standardize procedures and develop enterprise best practices. EEM administration tools provide control over user access levels as well as alarm and information hierarchy. Flexible database designs support complex reporting needs, redundancy, and the location of secured data within network operations centers. Finally, the availability of data quality tools can ensure that all information acquired from within the facility or from external feeds is complete, accurate and dependable. Unified management reduces risk Independent of the level of power redundancy a site has implemented, an EEM system helps facility staff easily keep on top of all active and passive distribution paths, simultaneously. It facilitates this through three complimentary methods of operational management: Active. An EEM system delivers a real-time state of the facility view at a glance. It provides comprehensive overview snapshots of the facility and detailed information at the equipment and load levels. It can also push regular status updates to staff. Reactive. An EEM system delivers immediate notification of critical conditions to the right personnel, wherever they are. Staff can respond quickly and trigger recovery strategy processes when, for example, a 2007 Schneider Electric Page 3 of 9

4 load nears critical limits, excessive heat is sensed, or when equipment or circuits fail. Key data is logged to enable users to perform root cause and fault analysis, pinpoint the causes and effects of an anomaly, fix the problem and keep it from happening again. Proactive. The EEM system also helps prevent potential problems by delivering performance metrics that indicate increasing risk. Examples include high levels of electrical harmonics that can adversely affect normal equipment operation, or a load trend that shows the possibility of a circuit or rack being overburdened if more equipment is added. For each process, the system provides macro and micro level views, delivering an appropriate breadth and depth of information where and when it s needed. This ability helps highlight and simplify the understanding of complex data center system interdependencies. Managing at the macro level From a macro perspective the goal is to provide a unified overview of all the operation and performance of the major pieces of the system, from the incoming utility supply to the parallel operation of the redundant power system to the individual racks and servers. Custom displays and reporting allow quick evaluation of equipment and system status in each zone of the data center. The EEM system offers a complete, integrated view of all key components and their most critical operating parameters (Table 1). Location / Data Source Measurements and Status Indicators Equipment Utility Feeds Meters and I/O Surge Protection, Ground Fault, Trip Unit, Power Quality (transients, sags, swells) Transformers Meters and I/O Fan failure, Temperature, Power Quality (harmonics), Output (voltage, current, power, energy) Generators PLC data output Battery Charger, Ambient Temperature, Sensors (water, fuel, etc.), Output (voltage, current, power, energy) Protective Relays Relay data output Reverse Power, Temperature, OV/UV, Over speed Paralleling Gear & ATS UPS Meters and I/O Meters and UPS software Status change, SCR Status, Alarm Condition, Summary Alarms, Test Position Alarms (battery, bypass, summary, online), Power Quality (even harmonics, input THD, output sag / swell), Ambient Temperature, Output Voltage Chillers & CRAC Meters and I/O Cooling system, Status, Input Power, Power Quality (input THD), Temperature PDU & STS Meters and I/O Transformer Over-Temp, Emergency Power Off, Input Power, Power Quality (input THD) Loads Branch Circuit Monitors Circuit Loading, Overload Limits, Power/Energy per Circuit 2007 Schneider Electric Page 4 of 9

5 Table 1 List of typical data center equipment and associated information monitored and logged by an EEM system Since 95% of all site infrastructure failures occur between the UPS and the computer load iv, the PDU is one of the most critical monitoring points. In cases where front panel indicators are provided by equipment, for example by an STS, facility personnel on the floor may not see a critical condition. An EEM system centralizes monitoring of all condition indicators so an alarm is never missed. The breadth of inputs monitored by an EEM system enables a complete real-time view of the facility, presented either as schematic or graphical diagrams. A variety of EEM screens can be created and linked, each showing different parts or aspects of the facility and its equipment, with all associated live measurements and indicator status represented. Key performance indicators offer a simple yet powerful way to track facility status and reliability trends (Figure 3). Analogous to KPIs used by IT management systems, EEM metrics can be developed for the power infrastructure. Facility personnel can view key indicators for risk conditions such as branch circuit loading, or view continuity metrics for number of nines or other industry-standard benchmarks such as mean-time between failure (MTBF) and mean-time to repair (MTTR). Figure 3 Reliability metrics such as PDU loading can be graphically represented in a variety of ways for quick visual identification of critical conditions Automated alarming and notification capabilities provide a coordinated early failure detection warning system that can be highly customized. Operational limits are user-definable, critical conditions can be prioritized, and specific personnel notified using any combination of pager, cell phone, , SNMP, on screen list, flashing light or other method. A system can measure and alarm on progressive limits (high, high-high, etc.) and on any combination of conditions. This could include power quality events, anomalies in mitigation or cooling equipment, communications failure, or circuit loading approaching breaker capacity ratings. Advanced alarm management capabilities include support for personnel scheduling and for escalation strategies Schneider Electric Page 5 of 9

6 Managing at the micro level From a micro perspective, the key is to recognize the metrics necessary to quickly and easily assess the risks for each underlying point of failure within the context of the relationship of that piece within the whole. EEM provides independent system auditing, going deeper that traditional analysis methods and allowing for real-time evaluations. It shows the relationships between equipment and conditions, revealing drivers that affect system stability and risk. A range of analytic tools helps verify the performance and tolerances of each critical point in the power delivery chain. Taking this one step further, the EEM system will help exercise the power distribution system to optimize the configuration. Load characteristics at each level can be compared to confirm that correct connections were made on installation. Individual circuit values are rolled up to the next level, identifying losses and available capacity. On the fly audits reveal the state of efficiency and reliability, help personnel optimize each point in the system, and essentially provide risk management insurance. Powerful visualization and reporting tools make the process easy for a skeleton crew of operations staff. During regular testing procedures to validate design or ensure correct operation of equipment, an EEM system can play an important role in verification and documentation. For example, high-speed trending of relevant electrical and physical data can be performed for generator start-up tests. Automatic transfer switch tests can be monitored to assess performance (including voltage and current waveforms as well as timestamps for switch operations). An EEM system can verify UPS output remains within specifications during switching tests and that the batteries are recharged correctly (Figure 4). In the event of a UPS failure, the operation of an STS inside a PDU can be verified, to ensure it properly switched to the alternate power source. During facility expansion or new construction the EEM system will verify and document the proper operation of the reliability equipment as it is commissioned, streamlining the process by providing consistency and decreasing the overall cost and time for testing. Figure 4 An extensive range of electrical and physical parameters can be monitored and analyzed to validate all aspects of equipment operation In the event of a power distribution or equipment failure, the intelligent meters throughout the EEM network will perform the same job as the black box flight recorders used on aircraft. In response to an alarm condition, facility staff can use the data captured and uploaded to isolate and understand the source of a failure and to determine the 2007 Schneider Electric Page 6 of 9

7 corrective actions required. Sequence-of-events, waveforms, and other relevant measurements are each accurately time-stamped, allowing the nature and propagation of conditions to be clearly connected with their effect. For example, the source of an occasional server malfunction can be correlated with power quality disturbances at the electrical output of a UPS or from the utility feed. Performance of the power reliability system as a whole as well as that of individual components can be benchmarked against design specifications and industry standards. For example, the electrical output of UPS equipment can be compared to the electrical input to validate whether the equipment is performing to specifications and able to mitigate disturbances. Disturbances that propagate through to loads can be analyzed to verify if they fall within acceptable by industry standard tolerances (e.g. on an ITIC curve), and if the load rides through the disturbance and continues to operate as expected (Figure 5). EEM systems can also normalize measurements for weather and other variables to allow reliability factors for different facilities, or different zones or equipment within a facility, to be benchmarked against each other. Availability, capacity or any other dimension can be comparatively profiled for similar time periods or seasons. Figure 5 Typical EEM power quality screens, including ITIC tolerance curve showing events clearly categorized by type and severity, and drill down to detailed time-aligned waveform analysis Comprehensive historical logs can also help improve operational procedures and reduce maintenance requirements. An EEM system will track equipment operations such as relay or breaker trips, notifying personnel when equipment requires maintenance instead of being driven by scheduled work that can sometimes be unnecessary. Extended utilization reduces costs Using the same extensive reach and analytic depth that support reliability improvements, an EEM system can support a number of efficient and effective cost management strategies. In analyzing loading to identify areas of risk, the system can also reveal equipment, circuit and/or CRAC capacity that is underutilized, and can do this on a real-time and predictive basis. This improved visibility gives operations staff the confidence to push more performance out of existing resources. Areas with available capacity can be quickly identified and circuits safely loaded up closer to their limits while still staying well under their breaker ratings. By managing infrastructure growth in this way, capital expenditures can be deferred by maximizing the use of existing capacity when new or redundant loads need to be added Schneider Electric Page 7 of 9

8 The EEM collects detailed load data down to the circuit level on an ongoing basis from the networked metering devices. After data quality tools assure the data is dependable, it is stored in a secure database, hierarchically organized and optimized for a variety of analytic processes. Key performance metrics for capacity utilization can be established and tracked (Figure 6). With metering at all important points from service entrance to racks, loading can be calculated in Watts and/or cost per square foot in real time at every level. This can include evaluating loading by the physical hierarchy of equipment (utility feed, generator, UPS, PDU, circuit) or by the logical business hierarchy (building, floor, zone, application, row, rack, etc.). Graphic load profile analysis can help quickly identify areas approaching maximum limits and other areas of underutilization. In this way, a more comprehensive and accurate risk profile can be developed and growth can be actively managed. Figure 6 An EEM system will present system loading metrics in a visual way to help staff actively manage capacity utilization The EEM system will set accurate baselines for loading and available capacity on a circuit for a specific number and types of loads. As servers are added, the new loading can be accurately tracked and compared against the baseline. If the data center is implementing on-demand computing, the EEM system will correlate energy use against application or server use. Ideal target loading benchmarks can be established and used to replicate configurations on other circuits. Using extensive historical data and forecasting capabilities, various growth scenarios can be modeled to evaluate decision criteria and analyze potential impacts. If an expansion or new facility is required, the EEM system can help control the cost of infrastructure. By incorporating improved loading factors and utilization within EEM forecasting tools, electrical systems can be right sized to avoid unnecessary overbuilding. The EEM system can also help in design evaluation modeling and design validation. Variable costs can also be recovered through accurate cost accounting and allocation. Detailed financial analysis capabilities support sub-billing for tenants or different properties of the same tenant. The EEM system will remove billing guesswork and allocate costs accurately by tracking the real energy usage for any data center operation or process. It will automatically monitor and allocate expenses for electricity, gas, steam and condensate, compressed air, chilled water, waste products and emissions Schneider Electric Page 8 of 9

9 Finally, the load profiles saved by the EEM system will help accurately characterize the power needs of one or more facilities. Along with the ability to develop "what-if scenarios based on different utility rates, and to aggregate multiple facility loads together for bulk energy purchasing, data center managers can build a strong negotiating position with their utility. Conclusion We are moving from isolated and primitive instruments in an increasingly complex distributed computing environment to more comprehensive instrumentation Dan Farber, ZDNET v In the quest for highest availability and value, companies that manage and lease data center space are under continuing pressure to increase load densities without increasing risk to availability. Meeting these goals requires a greater depth of intelligence about rapidly changing power requirements. Enterprise energy management systems help businesses achieve the highest aggregate value from data center capacity. By delivering a unified, real-time view of all energy assets and offering a unique breadth of efficient analytic capabilities, EEM technology helps reduce risk by isolating potential problems, optimizing system configurations, and benchmarking performance against established metrics. An EEM system reveals unused capacity to help operations personnel safely maximize infrastructure utilization and right size new construction. Variable costs can be recovered through sub-billing, cost allocation and more informed energy contract management. Ultimately, an EEM system presents the key drivers affecting reliability and efficiency, shares that information between stakeholders, supports critical decisions and helps develop best practices. When enabled through professional services as a fully integrated turnkey solution, EEM tools will quickly become part of an operations and IT workflow that can effectively manage the dynamics of the new data center environment. Author Biographies Hugh Lindsay is the Market Segment Manager - Critical Power at Schneider Electric. He has worked with Schneider Electric clients throughout the world to help them understand and leverage the benefits of accurate, timely and relevant energy information for maintaining uptime and maximizing system performance. Hugh holds a degree in Marketing and Communications from Simon Fraser University in British Columbia, Canada. Bill Westbrock is a Business Development Manager for Schneider Electric, focused on critical power and data center applications. Bill has experience is providing engineered solutions for energy management and has worked in control applications with Square D, Westinghouse and Johnson Controls as well as in power applications as a Regional Sales Manager with MGE UPS. He received his Bachelor of Science in Electrical Engineering at the University of South Carolina. Terrence Tobin is a Communications Specialist for Schneider Electric. Terry began his affiliation with Schneider Electric in 1988 and has held a variety of positions including Director of Marketing Communications and Brand Manager. He obtained his Diploma of Electronics Technology in Victoria and has worked extensively in the high technology sector in the fields of research, development and communications. Schneider Electric i ii iii iv v Get Real: The Future of IT Infrastructure, Gartner, Inc., December 2004 Avoiding Costs From Oversizing Data Center and Network Room Infrastructure, American Power Conversion, April 2003 Determining Total Cost of Ownership for Data Center and Network Room Infrastructure, American Power Conversion, March 2002 Industry Standard Tier Classifications Define Site Infrastructure Performance, W. Pitt Turner IV, Kenneth G. Brill, The Uptime Institute, 2001 Instrumenting the enterprise, Dan Farber, Tech Update, ZDNET, January Schneider Electric Page 9 of 9