Power System Event Reconstruction Technologies for Modern Data Centers

Transcription

1 Power System Event Reconstruction Technologies for Modern Data Centers October 2011/AT321 by Bill Brown, P.E., Mark Kozlowski, Square D Critical Power Competency Center Make the most of your energy SM Revision #3 10/11

2 Summary Introduction...p 3 Why Worry About Event Reconstruction?...p 4 Time The Crucial Factor...p 11 Get connected to management Recording Hardware/Software Considerations...p 14 Where is SER Required in a Data Center Power System?...p 17 SER as Part of an Enterprise-wide Power Monitoring Solution...p 20 Summary...p 21 References...p 22 AT321 2

3 Introduction Reliability principles indicate that, due to component degradation, no power system can operate 100% of the time indefinitely. Back up and recovery procedures are designed to save the critical load in an emergency condition. The key to incident recovery is in the ability to quickly understand what went wrong and implement corrective actions. New power monitoring systems can re-construct the sequence of events such as circuit breaker tripping, static switch transfer, etc., down to 1 ms resolution. This paper discusses this technology and the necessity of its application to modern data centers. Get connected to management AT321 3

4 Why Worry About Event Reconstruction? The reliability paradox Due to the need for reliable electric power stemming from the high economic impact of outages, the electric power systems employed in modern data centers are true marvels of optimization for reliability. Redundancy and maintainability are two hallmarks of these types of power systems. In their most advanced form, defined in the Uptime Institute s Tier IV classification [1], they have system + system (S+S) redundancy, two active power delivery paths, and must be fault-tolerant and concurrently maintainable. Get connected to management However, the principles of reliability put a limiting factor on just how reliable any engineered system, whether for electric power or any other function, can be. Simply stated, the limiting factor is that every system component, no matter how well-designed, has a failure rate that is greater than zero. This fact may be stated in a different way: No system component can operate indefinitely without failure, no matter how well designed. The consequence of non-zero component failure rates is the following: No engineered system can operate indefinitely without an incident, no matter how well designed. Even Tier IV systems, as well-designed as they are, are subject to this statement, as evidenced by representative site availability figures which are less than 100% [1]. The cold reality implied in such statements is at odds with the reassuring reliability built into data center power systems. Granted, in well-designed systems with reliable components the probability of an incident is quite low. However, it is non-zero and must be accounted for to give an electric power operating environment that will meet the goals of the facility. Two types of activities are used for this purpose. One of these is a rigorous testing and maintenance program which, if designed and executed successfully, will help to enhance the reliability of the individual system components. The other is the implementation of an incident mitigation strategy to properly deal with incidents as they happen. Incident mitigation The term incident, as used throughout this paper, will be in the context of an undesired operation of the power system which causes an outage or a lowering of the reliability of the system (i.e., increased risk of outage). Assuming that the system is properly-designed and constructed, incidents generally stem from component failure, whether from causes internal or external to the component. Simply stated, two crucial goals of any incident mitigation strategy, in the aftermath of an incident, are: 1.) Determine what happened 2.) Keep it from happening again In reality, both of these activities are challenging. Determining what happened requires a pre-designed infrastructure for this purpose, and has its own set of challenges, as will be explored herein. Keeping it from happening again implies a zero probability of the incident occurring in the future, which is not possible; instead, the probability of re-occurrence is minimized. This may require system re-design or component replacement, or both; on a practical basis this minimization of the probability of re-occurrence must take place within the economic confines of the installation and may not be a true minimization, only an economicallyfeasible minimization. The focus of this paper will be on the first goal mentioned above: Determine what happened! AT321 4

5 The need for event reconstruction Any power system incident may be categorized into a series of events. In this context, the term event refers to an occurrence of some type in the power system. Each event may trigger another event or events, which in turn trigger other events. This cascading series of events will eventually stop, leaving the system in either an operating, partially-operating, or non-operating condition. Such a series of events may be classified as shown in Figure 1: Figure 1: Visualization of a series of power system events The condition of the power system referred to above is termed the state of the system. The system state starts in the pre-event steady-state operating condition and changes after each event. The diagram of Figure 1 may be further augmented to explicitly show the system state after each event. In Figure 2, on the next page, the pre-event steady-state operating condition is referred to as State 0, and the post-event steady-state operating condition is referred to as State n: AT321 5

6 Figure 2: Augmented version of Figure 1, showing system states between events The term steady-state in the description of the starting and ending states above implies that the system is in a state that can be maintained for some period of time. The intermediate states, i.e. States {2, 3..n-1} in Figure 2, are, in general, non-stable or quasi-stable operating states than cannot be maintained without causing another event. This begs the question: What caused the first event? The answer could be one of two things: 1.) An occurrence external to the system 2.) An occurrence internal to the system An example of an external occurrence is a lightning strike close to the facility. An example of an internal occurrence is a feeder cable fault incident. In the case of an external occurrence, the initial operating condition could indeed be treated as a true steady-state condition. In the case of an internal occurrence, the system is actually changing states very slowly, so slowly that the system appears to be in a steady-state condition before the first recorded event. It should be noted that, for the example of a cable fault incident just mentioned, the case of cable damage due to a maintenance worker running into the conduit with a forklift is actually an external occurrence, even though it occurred inside the facility walls. The reason for this is that although it was internal to the facility, it was external to the power system itself. As confusing as the above explanation may appear, it is really only a simplified version of what can happen with regard to sequential events in a power system. In some cases, a single event causes the system to go into an operating state that triggers multiple events. Such multiple events add to the confusion as they may or may not be the root cause of the resulting system state, only a consequence of the system being in the previous state. Further, external occurrences while a system is in a non-stable or quasi-stable state can cause further complications. AT321 6

7 Adding to the inherent complexity of the shifting system states is the fact that, because the system in question is an electrical system, many of the events are not directly measurable with the human eye. In fact, many events are measurable only by their effects. Instrumentation is required to be able to measure these events and/or their effects, and this instrumentation must be built into the infrastructure of the system so as to be on-line all of the time, continuously monitoring those quantities it is designed to monitor. These are not limited to such quantities as current and voltage, but must also include the states of overcurrent protective devices, protective relay status, etc. The final, and often the crucial, complication is that the only way to put measured events in order of occurrence is via a record of the times at which they occur. Unfortunately, the timing of events in an electrical system is usually much faster than for their counterparts in the mechanical world, making fast measurements crucial. Just as crucial as fast measurements is the need for fast recording of the events for later analysis, since the analysis of the events will take place, in electrical terms, a very long time after the events themselves occur. In summary, it is necessary to reconstruct a series of events in an electrical system via recorded electrical measurements, and it is crucial that the measuring means be fast enough to record these measurements and when they happened in order to accommodate this. Once the measurements are ordered, the events themselves may be reconstructed so as to arrive at a root cause for the incident. To illustrate this with an example, consider the system of Figure 3: Figure 3: Example system AT321 7

8 Assume that at time t 0 this system is in a steady-state operating condition as follows: Utility Sources 1 and 2 at nominal voltage and frequency, circuit breakers CB-UM1, CB-UM2, CB-UF1, CB-UF2, CB-UF3, CB-UF4, CB-L1, CB-L2, CB-L3, CB-L4, CB-GF1, CB-GF2, CB-GF3 and CB-GF4 are closed, circuit breakers CB-T, CB-GM1, CB-GM2, CB-L1A, CB-L2A, CB-L3A and CB-L4A are open, and generators G1 and G2 are offline. Now assume that the following measurements events are recorded: Table 1: Sequence of measurements for example system of Figure 3 Measurement Time Per above t 0 CB-UM1 A-phase current increases to 200% of the steady-state value t 1 A-phase voltage on line side of CB-UM1 and CB-UF1 decreases to 88% of the t 1 steady-state value CB-UF1 current increases to 600% of the steady-state value t 1 A-phase voltage on the line side of CB-L1 decreases to 88% of the t 1 steady-state value CB-UM1 A, B and C phase currents increase to 1700% of their steady-state values t 2 All three phase voltages on the line side of CB-UM1 and CB-UF1 decrease to 1% t 2 of their steady-state values CB-UF1 A, B and C phase currents increase to 5100% of their steady-state values t 2 All three phase voltages on the line side of CB-L1 decrease to 1% of their t 2 steady-state values CB-UF1 trips on its short-time characteristic t 3 All three phase voltages on the line side of CB-L1 decrease to zero. All voltages t 4 on line side of CB-UM1 and CB-UF1 return to their steady-state values. CB-UM1 current drops to 66% of its original value CB-L1 opens due to ATO operation t 5 Generators G1 and G2 start t 6 Voltage at the generator G1 terminals is within nominal magnitude and t 7 frequency limits CB-GM1 closes t 8 Voltage at the generator G2 terminals is within nominal magnitude and t 9 frequency limits CB-GM2 closes t 10 CB-L1A closes due to ATO operation t 11 Note that these measurements, by necessity, come from more than one piece of equipment. Although they are shown as descriptive text, they will consist of numerical measurements and waveform captures that give information, which is equivalent to that shown. In Table 1, they have been ordered as to time sequence using time-stamp information for each event generated by the monitoring equipment. This time ordering of the recorded events from different pieces of equipment is the first part of the event reconstruction. The second part of the event reconstruction uses knowledge of the system to describe the cause of the event sequence, and how one event led to another. One potential format of the completed event reconstruction is as shown in Table 2: AT321 8

9 Table 2: Event reconstruction for example measurements in Table 1 Time Event Resulting system state t 0 None System is operating normally, with the utility services supplying power. t 1 t 2 t 3 Ground fault occurs on the feeder from CB-UF1 to CB-L1 Ground fault escalates into a three-phase fault CB-UF1 begins tripping on its short-time characteristic A ground fault is present on the feeder from CB-UF1 to CB-L1. CB-UF1 and CB-UM1 are timing to trip on long-time delay characteristic. A three-phase fault is present on the feeder from CB-UF1 to CB-L1. CB-UF1 and CB-L1 are continuing timing to trip, but now on short-time characteristic. CB-L1/CB-L1A automatic transfer scheme is timing to trip CB-L1. CB-UF1 is clearing the fault. CB-L1/CB-L1A automatic transfer scheme continues timing to trip CB-L1. t 4 CB-UF1 clears the fault CB-L1 and the portion of the system it supplies are de-energized. CB-L1/CB-L1A automatic transfer scheme continues timing to trip CB-L1. The rest of the system is operating normally. t 5 t 6 t 7 t 8 t 9 t 10 t 11 CB-L1/CB-L1A automatic transfer scheme times out and trips CB-L1 Generators G1 and G2 start and begin ramping the voltage and frequency at their terminals Generator G1 voltage and frequency reach nominal limits The generator control scheme times out and closes CB-GM1 Generator G2 voltage and frequency reach nominal limits and are synchronized with the Generator G1 voltage The generator control scheme times out and closes CB-GM2 The CB-L1/CB-L1A automatic transfer scheme times out and closes CB-L1A CB-L1 is open. The portion of the system supplied by CB-L1/CB-L1A is de-energized. The CB-L1/CB-L1A automatic transfer system is sending a start signal to the generators. The rest of the system is operating normally. Generators are running. Voltage and frequency at generator terminals are climbing. The portion of the system supplied by CB-L1/CB-L1A is de-energized. The rest of the system is operating normally. The generator control system is timing to close CB-GM1. Generator G2 continues to ramp its voltage and frequency and is now being synchronized with the generator G1 voltage. The portion of the system supplied by CB-L1/CB-L1A is de-energized. The rest of the system is operating normally. Generator G1 energizes the feeder to CB-L1A. The CB-L1/CB-L1A automatic transfer scheme is timing to close CB-L1A. Generator G2 continues to ramp its voltage and frequency to synchronize with Generator G1. The portion of the system supplied by CB-L1/CB-L1A is de-energized. The rest of the system is operating normally. The generator control system is timing to close CB-GM2. The CB-L1/CB-L1A automatic transfer scheme continues timing to close CB-L1A. The portion of the system supplied by CB-L1/CB-L1A is de-energized. The rest of the system is operating normally. Generators G1 and G2 are now supplying the feeder to CB-L1A. The CB-L1/CB-L1A automatic transfer scheme continues timing to close CB-L1A. The portion of the system supplied by CB-L1/CB-L1A is de-energized. The rest of the system is operating normally. Generators G1 and G2 now supply all of the loads for CB-L1/CB-L1A. The utility services supply all of the other loads. AT321 9

10 As can be seen in Table 2, the root cause of this sequence of events was a ground fault in the feeder from CB-UF1 to CB-L1. Assuming the measured system parameter history before time t 1 gives no indication of an external electrical cause for this fault (such as a history of transient overvoltages), this leads to the conclusion that this cable fault incident is most likely due to the cable insulation damage during installation, or an accident involving the cable/conduit system. Also apparent from Table 2 is the fact that the measured parameters have been coalesced into an ordered sequence of events and resulting system states, with each event leading to a new system state until the system reaches a state which is maintainable for a relatively long period of time. In this case the final system state at time t 11 is considered a steady state but, in reality, the system state is slowly changing since fuel is being consumed by the generators. Ultimately this could lead to a very noticeable state change if the generators run out of fuel! In reality, there is really never such a thing as a true steady state, only states where changes of system state are occurring slowly enough not to be noticed, or are minute enough not to be noticed. Equipment ages, generators consume fuel, etc., and all contribute to changes of system state. In fact, even at time t 0 in the example the system is changing state the minor imperfection in the feeder cable insulation is growing due to voltage stresses and heat, and it is only a matter of time before the system moves from the steady state into the sequence of events described above. Because the change of state is minute enough not to be noticed, it is therefore classified as a steady state. The conclusion is inescapable: Even the most well-designed system is constantly changing state, and ultimately the net result of these state changes is a sequence of events that has enough effect to be noticed or, worse, an incident. This points to the need for event reconstruction if system reliability is to be optimized in the aftermath of such an incident. On a practical basis human intervention is typically required to accomplish the second part of the event reconstruction described by Table 2 in the example above. This is due to the fact that knowledge of the system is required to make sense of the changes the system is undergoing. However, in order for the second part of the event reconstruction to be successful, the first part that described by Table 1 for the example must be completed successfully. The first part of the reconstruction is accomplished via sequence-of-event recording (SER), and due to the number of events and the time-frames involved this must be done automatically by power monitoring equipment built into the system infrastructure. This equipment must have the capability to make fast measurements of system parameters, usually in different pieces of equipment, and put them into time order. AT321 10

11 Time The Crucial Factor Having convinced ourselves of the need for event reconstruction, we now move to an investigation of just what it means to satisfy the basic requirement for SER stated above the ability to make fast measurements of system parameters, usually in different pieces of equipment, and put them into time order. How fast is fast enough? Just how fast is fast enough for measurements in the electrical environment? To help judge this consider the following table, which gives several electrical events and the representative timeframe for each: Get connected to management Table 3: Representative time-frames for various electrical events Event (1) 60 Hz power system cycle ms (1) Third-harmonic (180 Hz) power system cycle ms (1) Fifth-harmonic (300 Hz) power system cycle ms (1) Seventh-harmonic (420 Hz) power system cycle ms Voltage sag Lightning strike (return stroke) Low-voltage circuit breaker, instantaneous trip from initiation of overcurrent Low-voltage circuit breaker, short-time trip from initiation of overcurrent Overcurrent relay instantaneous trip Bus differential relay trip Lockout relay trip Low-voltage circuit breaker, time from trip to fault clearing (arcing time) Medium-voltage circuit breaker (3-cycle), time from trip to fault clearing (arcing time) Representative timeframe* ms 1 min 42 μs 44 ms 200 ms 35 ms 6 ms 8 ms 16 ms 32 ms * These timeframes are representative examples only. Consult manufacturer s literature for actual values for specific equipment. System-related phenomena time-frames are dependent upon the parameters of the system. From Table 3, it can be seen that most phenomena of interest in an electrical power system environment occur on timeframes that can be measured in milliseconds, while a few, such as lightning strikes, occur on timeframes measurable in microseconds. To explore this further, consider typical timeframes for the example incident given above. In doing this, the absolute times t 0, t 1, t 2, etc., are not as important as the differences between them, i.e., (t 1 t 0 ), (t 2 t 1 ), etc., since the differences represent the time between events: AT321 11

12 Table 4: Representative timeframes for the example of Table 1 and Table 2 Timeframe t 1 t 0 t 2 t 1 t 3 t 2 t 4 t 3 t 5 t 4 t 6 t 5 t 7 t 6 t 8 t 7 t 9 t 8 t 10 t 9 t 11 t 10 Time Hours, Days, Weeks or Years 50 ms 300 ms 12 ms 2138 ms 8 ms 6000 ms 500 ms 100 ms 500 ms 400 ms In Table 4, the smallest representative timeframe between events is 8 ms, and the largest is on the order of years. To account for more complex events which occur in different portions of the power system, and to account for more complex incidents where multiple events are triggered at close to the same time, a time accuracy of 1 ms between different recording devices is the generally-accepted norm for sequence-of-event recording. The complexities of measuring time Measurement of time is commonplace today. This was not always the case. As technology progressed to the point that mechanical timepieces became available, the one problem that always manifested itself was that of accuracy. With no standard of time to compare against, it was very difficult to quantify this problem. This was solved by the adoption of the SI definition of the second based upon the frequency of radiation from the Cesium 133 atom under given conditions [2]. With this definition, most modern quartz wristwatches lose, on average, ±1 second per day. This is sufficient for general use in the activities that most humans engage in, but not for measurement of time increments down to 1 ms as required for event recording in a modern power system. To solve this problem, a hyper-accurate clock must be used. An atomic clock, properly calibrated, can measure time with a loss of as little as ±2 ns per day. However, such clocks are prohibitively expensive for direct use on an SER system. Instead, the method usually employed is to synchronize a standard clock with a properly calibrated atomic clock. Such a system is implemented using standard Global Positioning System (GPS) technology in common use today. Originally developed by the United States Department of Defense, GPS allows easy time synchronization using a network of satellites. These satellites have atomic clocks on board, and by synchronizing a local clock in a GPS receiver to these atomic clocks, the local clock can be made to be accurate to within the 1 ms time accuracy required. Once the local clock in the GPS receiver has been synchronized, it must then synchronize the clocks in all of the power monitoring devices in the system. Serial codes have been developed for this purpose, the most commonly-used being IRIG-B. IRIG-B, and other codes like it, are broadcast on a serial network which includes all of the devices that require time synchronization. In some cases more than one time code is required due to specific device compatibility requirements; in this case a separate network for time synchronization is used for all devices that use the same time code for synchronization. The time synchronization configuration can be visualized to look something like that shown in Figure 4. AT321 12

13 In Figure 4, all of the devices to be time-synchronized can receive either an IRIG-B signal or a second, non- IRIG-B signal. The antenna, GPS receiver, IRIG-B signal generator, and other time-code generator may be in the same physical device or in separate devices. The figure shows the IRIG-B generator supplying the time signal for the non-irig-b time code generator; this device may take input from the GPS receiver directly in some cases. Figure 4: Conceptual time-synchronization scheme for SER It should be noted that larger systems will require more IRIG-B channels from the IRIG-B signal generator than shown in Figure 4, due to the maximum number of devices one IRIG-B output can service. While there are many variations on the scheme shown in Figure 4, one thing remains the same: Time synchronization is crucial for successful event recording in a modern data center power system. AT321 13

14 Recording Hardware/ Software Considerations Having acknowledged the requirement for time synchronization for SER and how it is implemented, we move to a discussion of the recording hardware to be used. Data types Before we can specify the characteristics for the recording hardware, an examination of the data types to be recorded must be made. In general, for sequence-of-event recording three specific data types are required for an effective system: Get connected to management 1.) Measurements of specific system parameters such as current, voltage, frequency, etc., as they rise above or fall below pre-determined thresholds. 2.) Status of devices which typically have a binary on/off, open/closed, pass/fail, etc., state. Example of such devices are overcurrent protective devices such as circuit breakers ( open/closed ), discrete alarms ( on/off ), etc. 3.) Voltage and current waveform capture, usually triggered by one of the occurrences from 1.). In examining these data types, the concept of a log must be introduced. In the context of power monitoring a log may be thought of as the repository for the records of all recorded system events. The size of the log is limited to the size of the physical memory in the recording device. It is therefore of important to take into account the amount of memory available when planning for the above data types. For data type 1.) above, it is important, in order to conserve memory and to make interpretation of the log data easier, to limit measurements to those times when a system parameter crosses the pre-set threshold. It is also important to consider the levels of such thresholds to avoid recording of events during normal power system operation. For data type 2.), because there are only two states it is only necessary to record when the device transitions from one state to another. For data type 3.), each waveform is an individual entry which consumes more memory than the other two data types, and therefore the conditions for waveform capture must be carefully defined. Typically the number of cycles of pre-event and event data for the waveform may be specified, along with the resolution in samples per cycle and what channels (currents, voltages, etc.) to capture. Often, the waveform capture can also show the status of discrete inputs as well, such as circuit breaker open/closed status. AT321 14

15 SER recording device configurations It should be no surprise, based upon the above discussion, that high-end digital power monitoring devices are typically used as the recording devices in data center power systems for sequence-of-event recording. The reason, of course, is that they combine the accuracy of measurement required with event logging and waveform capture. In general, top-of-the-line power monitoring equipment typically exhibits the following features: Instrumentation channels for three-phase current and voltage at the location of the device Current and voltage measurements with 0.1% accuracy or better Sampling of waveforms at up to 500 samples per cycle, with optional increased sampling rate for detection of medium-to-high duration of impulsive transients Calculation of almost all conceivable system quantities, including power quality indices, using the sampled data Data logging capability with 1 ms resolution, with a large on-board non-volatile memory for log storage Waveform capture capability, with adjustable pre- and post-cycle capture duration and sampling rate Discrete logic input capability, for sampling of status-type events Capable of time synchronization via IRIG-B or other industry-standard serial time code Communications capability via industry-standard serial protocol(s), with optional Ethernet connectivity These devices, in addition to this impressive list of features, also have local display of metered quantities, allowing them to be used in place of conventional instrumentation or lower-end power monitoring devices at the equipment where they are employed. Data type 1.) above is typically configured as an alarm in the device. The alarm setup includes specifying the thresholds at which the alarm will activate and, if active, when it de-activates. Data type 3.) is similar, only the alarm is generated from a state change for a discrete logic input. Data type 2.) is usually initiated via an alarm from data type 1.) or 3.). While a general discussion of power monitoring is beyond the scope of this paper, it should be noted that the devices employed should meet the IEEE standard [3] for the type of phenomena to be measured. In many cases the number of required status inputs exceeds the number of discrete logic inputs on the power monitoring device, even with optional add-on cards. This is easily solved by adding a dedicated discrete status input module. Such a module typically has the following characteristics: Discrete logic input capability, for sampling of status-type events Data logging capability, with on-board non-volatile memory for log storage Capable of time synchronization via IRIG-B or other industry-standard serial time code Communications capability via industry-standard serial protocol(s), with optional Ethernet connectivity Other sources of event data include microprocessor-based protective relays and low-voltage circuit breaker trip units. Although such devices do typically have metering, waveform-capture, and data logging functions, the metering functions are generally not as accurate as those available in a high-end power monitoring device, leading to the conclusion that their most practical use in a sequence-of-event recording system is to give discrete protective-element pickup and dropout data. This data may be logged internally by the device, in which case the device would have on-board memory, time-synchronization and network communication requirements similar to those above; alternatively, the data may be logged as a discrete input to a power monitoring device or discrete status input module. AT321 15

16 Network communications Network communications between recording devices is crucial, for two reasons: 1.) To obtain a big picture view of an incident, a summary of events from several different recording devices, in different locations of the power system, is required. 2.) A single operator interface should be able to give a master log of events, in time order without the need to gather event data from each recorder and manually order it. The network communications will ideally take the form of an industry-standard protocol over Ethernet, such as Modbus /TCP. However, due to cabling cost or hardware constraints in some cases, serial communications between devices in the same physical piece of equipment or same local area may be desirable. Any long runs of network cabling, such as between individual equipment rooms that are separated by large distances, should be Ethernet rather than serial communications. It is highly recommended that these long runs of cabling utilize fiber-optic cables rather than copper. Conveniently, these also guidelines apply for networked power monitoring systems in general. Software and operator interface No SER system would be complete without the proper software and operator interface. The software should be able to gather the events from the various recording devices and put them in time-order for ease of use. The use of network communications makes the personal computer the ideal means for the operator interface. Typically, the same software used for general power monitoring can also serve as the software for the SER system, since the data is generated by the same physical devices and SER functionality is usually incorporated into the software. In larger systems, more than one computer workstation may be able to access the recorded data, to make the data access as convenient as possible; in this case a central server or servers would typically be used. Physical equipment location In most cases, the recording devices themselves may be placed inside the electrical equipment. Examples of equipment which typically this type of arrangement are switchgear, switchboards and unit substations. For other types of equipment, the recording devices must be placed in a cabinet external to the equipment. Summary A conceptual sequence-of-events recording system, utilizing the principles above, is shown in Figure 5. In this figure, the point made earlier that the sequence-of-events recording system must be built into the system infrastructure becomes clear. Fortunately, with a few modifications (most notably the addition of the time synchronization system) a power monitoring system may be used for sequence-of-events recording, since the majority of the recording devices are power monitoring devices. This holds true for existing systems as well. It should be noted that in Figure 5 those devices labeled as Ethernet Gateway provide an interface from a serial communication protocol, such as Modbus, to an Ethernet-based protocol, such as Modbus/TCP. AT321 16

17 Where is SER Required in a Data Center Power System? Thus far, the need for data center power system SER has been established and a means for accomplishing it within the power monitoring system has been described. That leaves a crucial question unanswered: Where in a data center power system is sequence-of-events recording required? Unfortunately, there is no clear-cut laundry-list of locations in a typical data center power system that require SER. The cost of SER at each location must be weighed against the potential benefits to be gained for each particular case. With this said, the first step in identifying event recording locations is to identify the types of potential events to be recorded. Typical events that fall into this category are: Get connected to management Loss of utility voltage Generator start-up Frequency and voltage excursions while on generator power Automatic transfer switchgear/switch operation All UPS switchgear operations UPS input and output voltage abnormalities Cable faults This list is by no means exhaustive. However, it gives emphasis to locations higher in the power system, closer to the utility service and generators, over locations lower in the system, downstream from UPS s. In most cases, the cost/benefit balance is achieved by including sequence-of-events recording from the utility service and generators down to the UPS outputs only. The limiting factor is typically the cost of power monitoring devices that have fast event recording capability and can be time-synchronized in order to give 1 ms accuracy. Of course, in terms of the over-all power monitoring system the inclusion of events which are logged by devices which have slower data logging capability and are not time-synchronized is possible, but these records are of severely diminished value when determining the root cause of an event. AT321 17

18 Figure 5: Conceptual data center sequence-of-events recording system AT321 18

19 Recording locations for the example system of Figure 3 The example system Figure 3 was used above to generate a simple example of event reconstruction. In this example, in order to generate the ordered event log of Table 1 the following recording locations are required: Currents and line voltages for CB-UM1 and open/closed status High-end power monitoring device at CB-UM1 Currents and line voltages at CB-UF1 and open/closed status High-end power monitoring device at CB-UF1 Line voltages at CB-L1 and open/closed status High-end power monitoring device at CB-L1 CB-UF1 trip status interface with CB-UF1 trip unit (typically discrete contacts to High-end power monitoring device or to separate discrete input module) CB-L1 ATO operation status Interface to CB-L1 ATO (typically PLC) Generator status Discrete contact inputs and appropriate discrete input module(s) for generators CB-G1 line voltages and open/closed status High-end power monitoring device at CB-G1 CB-G2 line voltages and open/closed status High-end power monitoring device at CB-G2 CB-L1A line voltages High-end power monitoring device at CB-L1A To summarize the above and provide consistency, the following are therefore required for adequate capture of events for an incident like the example: High-end power monitoring devices at all of the circuit breakers shown Interfaces with all ATO PLCs, with PLCs programmed to time-stamp events Interfaces with all circuit breaker trip units Discrete input module(s) at generators Although this example is for illustration purposes only, it does serve to show the thought process behind the placement of SER devices. AT321 19

20 SER as Part of an Enterprise-wide Power Monitoring Solution As stated above, the physical and software platforms for SER are typically contained within an over-all power monitoring system. Because much of the infrastructure of the SER system is also required by the power monitoring system itself, only those costs which are above and beyond that required for the power monitoring system without SER need to be evaluated in comparison with the benefits of SER. These costs, as may be readily seen from the description of the hardware requirements above, are mainly due to: The addition of a time synchronization system Get connected to management The upgrade, in some cases, of power monitoring devices The addition, in some cases, of discrete input modules This is an important point, as it allows the economically-feasible realization of SER in many cases when the over-all cost of the power monitoring system is taken into account. The goal of an over-all power monitoring system is not only to maximize system reliability, but also to effectively utilize the capacity of the system. SER is an important part of this, as the effective maximization of reliability and utilization of capacity both require the ability to diagnose past incidents with an eye toward system expansion. A portion of the overall cost-savings associated with the power monitoring system, therefore, can be said to be attributable to its SER capability. This, also, aids in the economically-feasible realization of SER. For existing facilities, if a power monitoring system exists SER may also be realized in an economically-feasible manner. In addition to the costs described above, network communications in the power monitoring system may need to be improved, additional devices, such as servers, installed, etc. However, these also benefit the power monitoring system as a whole, and the cost/benefit analysis should take this into account. More detail on the benefits of an enterprise-wide power monitoring solution may be found in [4]. AT321 20

21 Summary In modern data centers, reliability is crucial. The reliability of the facility s electric power system is an important part of the over-all reliability of the facility. Because all electrical components, no matter how well-designed, have non-zero failure rates the electrical system as a whole will have an availability of less than 100%. Even if an incident does not lead to an outage, it can lead to an operating condition which is less reliable than desired. It is critical, therefore, to have the ability to diagnose past system incidents. Sequence-of-event recording technologies fulfill this requirement. Get connected to management The basic requirements of electric power SER systems is that they be time-synchronized to 1 ms, have recording devices which are able to time-stamp events down to 1 ms resolution, and have a network architecture which allows data to be gathered from a central location. A good deal of the cost of such a system is typically built into the facility s power monitoring system, which provides additional benefits in addition to SER. A carefully planned SER system, implemented with the proper placement of recording devices, will allow diagnosis of most power system incidents. This capability, the ability to see into the past with great accuracy and precision, is invaluable to keeping the data center s electric power system as reliable as it can be, to minimizing the possibility of repeat incidents, and to planning facility upgrades in the most reliable manner possible. AT321 21

22 References [1] W. Pitt Turner IV, John H. Seader, Kenneth G. Brill, Industry Standard Tier Classifications Define Site Infrastructure Performance, The Uptime Institute, [2] Bureau International des Poits et Mesures, The International System of Units, 7th edition, [3] IEEE Std , IEEE Recommended Practice for Monitoring Power Quality. [4] Hugh Lindsay, Bill Westbrock, Terrence Tobin, Maximizing Data Center Reliability and Utilization Using Enterprise Energy Management Technology, Power Measurement, Get connected to management AT321 22

23 Bill Brown, PE, is a principal engineer for the Square D Engineering Services group by Schneider Electric. On behalf of the Schneider Electric Data Center Solutions division, Bill Brown provides electric power consulting and engineering support for power system design, application and maintenance, providing innovative solutions to customers, consultants and end-users in critical power applications and data center facilities nationwide. Brown is a registered engineer in the State of Tennessee. Schneider Electric USA, Inc. Data Center Solutions 1010 Airpark Center Drive Nashville, TN sedatacenters.com 2011 Schneider Electric Industries SAS, All Rights Reserved. Document Number AT321 October 2011 tk