Designing Non-Deterministic PAC Systems to Meet Deterministic Requirements

Designing Non-Deterministic PAC Systems to Meet Deterministic Requirements 1 Abstract. This paper presents the results of the design and performance testing of a complete PAC system for a 230/115 kv generation substation using IEC 61850 Ethernet communication for protection and control. The station included multiple step-up transformers, busses, circuit breakers (45), synchronous condensers and auto transformers in a breaker and a half arrangement. More than 200 protection and control IEDs from different manufacturers, with 4,000 messages were part of the integrated system. Design techniques and best practices to avoid communication delays due to network routing and congestion are presented. Test results of the network reliability and performance with detailed results evaluated for suitability of the design for this highly important application are also provided. As an example for high traffic conditions (data storm), test data from a live bus differential scheme is given. Lessons learned from the project are detailed and future plans and opportunities to expand this technology are discussed. Index Terms--. IEC 61850 / Performance / Inter - device Interoperability / Substation Automation / Network structure I. NOMENCLATURE Generic Object Oriented Substation Events MMS Manufacturing Message Service (SV) Samples Values IED(s) Intelligent Electronic Device(s) II. INTRODUCTION In the past Ethernet has not been seen as a means of reliable communication for critical communication such as control and protection within a substation. Serial communication is said to be more deterministic as its polling cycles can be calculated. However, the bandwidth of serial communication and the fact that there is no peer-to-peer protocol makes it less suited for protection. For many years, hardwired signals from protection IED relay to relay was considered by many engineers to be the only deterministic means to provide the needed speed and reliability for protection and control of a substation. This paper describes how it is possible, with a modern Ethernet, to achieve similar reliability and shows best practices and lessons learned for designing a reliable system based on non-deterministic protocols (). Stefan Nohe, Oliver Hartmann and Farel Becker Siemens Raleigh, NC Cedric Harispuru SIEMENS Nuremberg, Germany III. CONCEPTS OF THE IEC 61850 STANDARD [2] According to the IEC 61850 standard the network LAN of a substation is divided into two parts. The first domain is the process bus, which consists of the interface between the process equipment at the instrument transformer level, such as breakers, disconnects switches, and intelligent protection relays (IED). The second domain is the station bus which comprises the interface between the digital protection relays and the supervision equipment, including HMI and SCADA. The standard defines the three protocols for communication between the Process Bus or Station Bus as follows: MMS (Manufacturing Message Service) Protocol Reporting services (Alarms/Events) are messages which are vital to the operation of the substation. This message type contains all of the necessary information for supervision of the control and monitoring signals. These types of messages communicate on the Station Bus between IEC 61850 Client and Server and the communication protocol used is MMS. It is a lower priority messaging and is usually located between the control/protection equipments (IEDs) and supervision systems such as a Data concentrators, local HMIs and SCADA systems. (Generic Object Oriented Substation Event) protocol is often used to replace the conventional hardwires for intra-relay interlocking and trips. The messages service of IEC 61850 is most often used for protection and automation of electric systems within the substation. In this application messages communicate horizontally and travel peer to peer between IEC 61850 IEDs or servers. At this level they pass commands and status data amongst protection relays (IEDs) which are associated with breakers/disconnect switches. These IEDs then execute the control/protection algorithms. When a message is generated by IED (IEC 61850 server), it uses a layer 2 (Figure III-1) multi-cast transmission to send the event on the network. The receiving devices, known as a subscriber, subscribe to the multicast address of the message to be able to quickly filter the information and execute the needed task(s). The performance requirements for a message are stringent no more than 4 ms is allowed to elapse from the time an event occurs to the time the message is received. During a steady state or normal operating conditions in the substation, the IED broadcasts and retransmits messages cyclically ( heartbeat ). The number of IEDs and the amount of information for a given functionality will all contribute to the number of messages that will be generated. The health of the messages are monitored closely to assure on-time delivery as described in the next

2 section. A missing message is an indication for the subscriber that the communication from the sender has become disturbed. messages are repeated rapidly to account for possible lost telegrams due to congestion in the network or failure scenarios. The IEC 61850 architecture in the case of status changes, guarantees the speed, and avoids the speed degradations of layer 3 which is used for normal Ethernet LAN traffic. messages are digital or analog signals. Samples Values (SV) Protocol are messages with stamps of the frequency (50 or 60 Hz) to build the phase vector (phasor) together with the amplitude of the measurement. This type of message is horizontal and it goes in the Process Bus between IEC 61850 servers, usually between the current and potential transformers and the protection equipments (IEDs) which will execute the protection algorithms. These messages contain phasor measurement magnitude data. A process bus implementation was not available at the time for implementation in the described application. Figure II-1 demonstrates how the services defined by the standard are applied to the ISO/OSI Model. Figure III-1 Full 7 Layer ISO/OSI Model Stack message validity and delivery Under normal conditions, all messages are cyclically retransmitted heartbeat with a predefined time called T 0 or T max. (Figure II-2). The normal range for T 0 is 5-100 ms. When a substation event occurs (i.e. Circuit Breaker trip signal) the transmission rate of the new messages increases to statistically assure that the message is delivered. This rapid fire mode called T 1 or T min has a range of.5 to 5ms. After the first rapid fire retransmission of the message, each successive message time doubles until again reaching T 0`. For example, if T 0 is set at a cyclical rate of 10ms and T 1 is set at a spontaneous rate of 1ms, the following message sequence would be expected: Message Sequence # Elapsed Time at Breaker Status Status # 1 0ms Closed 100 1000 2 10ms Closed 100 1001 3 20ms Closed 100 1002 4 22ms tripped 101 1 5 23ms tripped 101 2 6 25ms tripped 101 3 7 29ms tripped 101 4 8 37ms tripped 101 5 9 47ms tripped 101 6 Table III-I Mechanism Refer to figure III-2 for graphical representation In this example from the time the trip signal was initiated, four (4) rapid fire message transmissions would have been broadcast to the subscribing IEDs in just 7 ms. This would statistically assure that the message was received in time to cause the IED to trip. The IEC 61850 defines the mechanisms for tracking message frame exchange amongst the IED(s) in the network. The first mechanism is the tracking of message status. In this validation process, the message has a status number embedded in the frame header. At the moment when the status changes (e.g. circuit breaker goes from closed to tripped), the message status value increases by one. A second mechanism for tracking message frame exchange is to detect missing packages. Also imbedded in each message frame is a sequence number. Each message includes a number which increases with every message sent. In case of a change of state (going from cyclic to spontaneous transmission) the sequence number is set back to zero. If the receiver detects a gap in the sequence number it immediately knows that there were missing packages in between. The manufacturer s configuration software will allow one to annunciate an alarm message if this occurs. The final mechanism for tracking the validity of message frames detects if the time between messages has become extended beyond a safe transmission period. On the receiving side, if a non-sequential message character is received the message Receiver or Subscriber will wait a predefined time before it invalidates the signal. This waiting time is longer than a predefined time called Time Allowed to Live or T TAL, which is commonly defined as 1.5 times T max. If the T TAL is exceeded, the receiver will treat the message as suspect and therefore invalidate it. Once the message is invalidated, all subscribers on the network will receive updated information on the message status every T max time. The receiver will re-validate the message again after the next repetition after T TAL if and when it becomes valid. In combination, these three message validity mechanisms mathematically ensure delivery of the message even if there are temporary problems in the network.

3 10 ms 20 ms 23 ms 29 ms 22 ms 25 ms 37 ms 47 ms are Layer 2 multicast messages and broadcast to all ports without delay. This contributes to the fact that messages are faster then other Ethernet traffic and less likely to get lost in the event of a network reconfiguration. The use of VLAN and multicast filter is a possible concept to achieve separation of messages, but as discussed later, was not found to be necessary in this project. 0 ms IV. ENGINEERING AND DESIGN REQUIREMENTS Figure III-2 Mechanism Prioritization The IEC 61850 standard allows for the prioritization of messages over other Ethernet messages, thus allowing them to bypass the Ethernet data buffer. This requires Ethernet switches that support this feature. The message frame uses a layer 2 frame extension as defined by IEEE called 802.1Q, as shown in figure III-3. This makes the IEC 61850 different from most other protocols which do not use this Ethernet feature. This frame has an extra parameter that defines priorities for switches. Dependent on the priority set, the switch will bypass the buffer of non-prioritized telegrams and handle messages based on their priority. Figure III-3 uses the 802.1Q Frame The parameter that defines the priority tag is called 802.1p. frames used to bypass the normal traffic are shown in figure III-4. Figure III-4 Priority Tagging of Telegrams (Ethertype)[1] Since the message mostly exists in layer 2 of the ISO/OSI Model, the switch does not have to switch a message or learn any IP addresses. messages A. General In a conventional substation automation system, the design of the particular control or protection equipment is defined by the amount of inputs and outputs as well as the functions needed. All of this data can be found in respective device data sheets. Similar limitations apply when designing an IEC 61850 system. However, the limitations are not as obvious and will influence the network design decisions. Another focus in designing a reliable IEC 61850 based system is on the design of the telegrams. This can influence the number of required telegrams. Figure III-3 suggests that up to 1,500 bytes of data can be encapsulated into a single signal. Both control and protection signals could be put in the same telegram. A best design practice is for protection and control to be separated if possible so that the higher priority (protection) signals can be given a higher priority. One important design criterion is the network design, which is based on the performance of individual components that make up the backbone of the protection system. It must be understood that some devices have better performance than others in the IEC 61850 network system. This appears to be based on the design experience of the manufacturer rather than a comparison of the design philosophy of one component manufacturer to another. The know-how of the substation automation integration engineering staff designing a system is also an important criterion for success. To help avoid incompatibility, one should research and compare the performance data of a specific component vs. other components available on the market prior to incorporating a specific device into the network design. B. System For a protection system the most important requirement is reliability combined with secure execution of protection functionalities in the required time duration of a few milliseconds. During the design stage of the project it was beneficial to understand the selected devices as well as communication protocols and configuration tools in depth. For example, depends on implementation details like utilizing prioritization of events in the device. An event can be marked ( prioritized ) as non-time critical, e.g. control signal, or as time critical event, e.g. protection signal. This depends on the device capabilities and is based on the IEC 61850 data model.

4 The solution implemented in this project utilizes one field device ( I/O ) at the breaker to distribute status information and receive all breaker commands including trip signals. One of the main advantages of a system solely based on network communication is that the customer saves miles of copper cable. Additional features that require additional hardware and wiring in a conventional system can be done inside the relays, like a common lock-out for the breaker located at the I/O device. C. Network [3], [4] Reliability in a network-based application starts with the selection of the components and protocols used to ensure reliability. One major design aspect, besides choosing different manufacturers not only for the IEDs but also for the network switches, was to isolate the primary network and the secondary network. This assures that an issue causing a problem in one network does not affect the other network. Issues could include failure of a component which causes the network to reconfigure as well as congestion of the network due to an unexpected high volume of messages. This could be caused by design flaws as well as software defects in a device or a switch. A second design decision was to connect all relays in a star configuration to two switches and connect the switches in two independent rings are shown in figure IV-1. This assures not only protection against 1-out-of-n failures but provides protection against a large number of m-out-of-n failures. Switch Switch Switch Switch Relay Relay...n Relay Relay...n Relay Relay...n Relay Relay...n Switch Switch Switch Switch Figure IV-1 Overview primary protection network The selection of RuggedCom for the primary network and Hirschmann for the secondary network also supported the decision for the network redundancy protocol. RuggedCom switches were configured with the enhanced Rapid Spanning Tree implementation and on the secondary side Hirschmann s Hyper Ring protocol was utilized. Today, communication standards have introduced seamless protocols such as PRP and HSR to solve the issue in a generic way. D. Design Criteria From the start of the project until the site testing, the following main expectations needed to be fulfilled: 1. Reliability under any circumstance 2. Timely execution of trip functions 3. Secure behavior (lockout functionalities and interlocking) Reliability under any circumstance As mentioned above, the network design together with a fully redundant protection system ensures that no single failure will cause the system to fail. To ensure that both networks were separated as far as network protocols and messaging is concerned, but interconnected for all other purposes, a redundant set of routers is provided. This design not only protects against single failures, it also covers the following scenarios: - A failure in the DC power supply of either the primary or the secondary system causes a shutdown of the entire system. - Multiple network components on one side fail at the same time (e.g. an accident destroys multiple fiber cables). Another important design aspect is to ensure that if there is a problem in the system, the user knows about the problem and its extent. The substation monitoring and control system provides a full diagnostic system as part of the HMI for the entire network as well as for the communication. Timely execution of trip functions The timely execution of messages and respective commands mostly depends on the device capability as the network causes very little delay under normal circumstances. The timely execution was guaranteed by utilizing signals that the devices consider important (like protection trips) and sending them directly from the source (the protection function) and not processing them through internal logic. The test results presented at the end of the paper show that with this design: - High network traffic has no significant impact on the performance - Time delay of execution is experienced if devices are starting internal applications or if signals are used that are processed through internal logic rather then using the direct trip signal - Sending source signals (e.g. the internal trip signals) improves the timing Secure and selective behavior Secure behavior with IEC 61850 messages is a very important design criteria because unlike a hardwired signal, a relay will always keep the last status of a signal even if the communication fails. The signal will be marked as invalid but it keeps its last status. To ensure safe operation of the system, logic was put in place that set the signal to a safe state in case of a communication loss to prevent the system from underfunctioning. Another example for this is stub bus protection, which depends on the position of a switch/breaker. If the breaker position is unknown the safe assumption is to enable protection but this might cause the system to trip for no reason. In this

5 case the system will rely on the backup protection and disable the function to avoid over tripping.. V. TESTING The objective of this system building a system based on nondeterministic components with the same level of reliability as a system based on deterministic components leads to very extensive testing requirements. Based on this, the objective of testing has to verify that in normal operation the system performs within these limits and that it recovers from single failures to ensure that at any time the loss of redundancy is limited to an acceptable minimum. Therefore testing was conducted in the project to ensure reliability and to obtain the information so that design (theory) and implementation (reality) are aligned. Because of a high level of uncertainty, and because the system load can only be experienced with all components put together, the system was tested as a whole. There were two main stages of testing: - Performance Test - Reliability Test Performance Test The performance test was conducted in stages, adding additional load (number of messages) in every stage from simulating a single occurrence to the worst case a trip of the entire busbar. In the first test, a trip signal was developed using primary injection via a Dobel test kit generating a trip signal in the line protection relay of two vendors. The line protection relay of these two vendors and the associated protection scheme were tested. In turn, the line protection relays broadcast messages to the vendor s remote I/O devices to trip the breaker and two breaker failure relays. The results of when the messages occurred were recorded. Each time the test was run the results were consistent. The table below shows a summary of the test results for one of the test runs. The test shows the transmission time of the (different sender to different receiver; all equipment is mounted in one panel). This behavior is expected and not an issue to perform the most time critical function, clearing the fault, only the first message is needed. Test 5b - Latency (Only line) Broadcasting Relay Trip Sent Receiving - Time Received transmission time Line Distance protection (Vendor 1) 79ms I/O 1 (PBIO2202) 80ms 1ms Relay 1(P2202) 80ms 1ms Relay 2(P2208) 81ms 2ms Line Differential (Vendor 2) 133ms I/O 1 (PBIO2902) 144ms 11ms Relay 1(P2902) 146ms 13ms Relay 2(P2908) 145ms 12ms Table V-I: Goose Latency - same switch same panel The second performance test involved sending three messages at the same time to the same devices in this case, the trip of three phases of the same busbar. The results of this testing depend on the receiver capability. The first message is processed very quickly. The event list entry of the second message is delayed due to the fact the devices already executing the first command. This is an acceptable behavior because all trips are 3-phase trips, so the first message will already trip the breaker and clear the fault. The third test utilized 7 sending devices; in this case all 3 bus bars trip at the same time. This shows that even with an increased load on the network, the performance of the first received message is not affected. The performance of subsequent messages is slightly delayed due to a higher volume of messages that has to be processed in the relays. Sending I/O 1 I/O 2 Relay 1 Relay 2 Relay 3 Sect1. BBP Phase A 12ms 5ms 12ms 14ms 13ms Sect1. BBP Phase B 4ms 5ms 4ms 8ms 9ms Sect1. BBP Phase C 5ms 2ms 3ms 4ms 3ms Relay 4 Relay 5 Relay 6 Sect2. BBP Phase A 17ms 7ms Sect2. BBP Phase B 5ms 3ms Sect2. BBP Phase C 7ms 7ms Sect3. BBP Test 1a - 3 Phase Bus fault Bus 1A Receiving - Time Received Protectio n Relay 1 Relay 2 Test 2a - 3 Phase Bus fault Bus 1A, 2A, 3A Receiving - Time Received Relay 3 Table V-III: Latency 7 sending devices Relay 4 Sending I/O 1 I/O 2 Sect1. BBP Phase A 5ms 4ms 7ms 8ms 5ms 7ms Sect1. BBP Phase B 8ms 7ms 11ms 11ms 8ms 10ms Sect1. BBP Phase C 8ms 3ms 3ms 5ms 8ms 4ms Table V-II: Goose Latency - Three senders to one receiver The overall results of the performance test showed that independent of the network load the messages still reach the receiver in an acceptable time. The biggest delay can be seen from subsequent messages received by the same relay. This is caused by the relay reacting to the first received message. Reliability Testing The results of the performance test were promising and in line with the project requirements. The next step involved reliability testing. How reliable are the test results in different scenarios? Reliability testing in this case involved reproducing the test results from the performance testing under unusual, but possible scenarios. Theoretical scenarios with a minor possibility to occur in reality where disregarded. As part of reliability testing, we needed to see under which load conditions the network would start to become unstable and the delay in the communication would lead to a different 5ms

6 result. To be comparable with a deterministic system we did not consider any failure of components (switch/root switch) as this is covered by means of a redundant system. This is similar to deterministic solutions, where it is also not assumed that the wire will be loosened over the years from terminals (which is in reality happening). The load was increased further and more devices were added to the test, taking the results shown in Table V-III: Latency 7 sending devices as a reference. The outcome of this test was that no additional load (as it could occur in a realistic scenario) could affect the times. As a final stage additional load was generated by changing an existing signal with a high frequency to generate so called storms which can lead to delays in communication. Even in this extreme case, the test results were the same as shown in Table V-III: Latency 7 sending devices. Network reliability The other design aspect reliability of the network and downtime due to single failures was also tested extensively. To perform this test a ping was sent with high frequency to about 30% of the equipment and the number of times the ping signal was lost during several failure scenarios was recorded. The scenarios tested extended from a single network link failure up to failure of the root switch. The network showed the expected performance for the used protocols (erstp and Hyperring).[6] VI. LESSONS LEARNED The main lesson that was learned from the design and test of this system is that success depends upon the combined expertise in IEC 61850, relay programming and network design. This knowledge is needed to be able to design the system to be able to use all involved components in the best possible manner and to anticipate and avoid problems that can occur with this design. The design approach uses the IEC 61850 in a manner that the standard did not consider during the standardization process and therefore a careful selection of components is required. The complexity can be reduced if each system (primary and secondary) would only consist of one manufacturer for the protection and control devices because in this scenario all internal mechanisms that one vendor provides can be used to their full extent. Another lesson learned is that a fully developed and tested IEC 61850 process bus implementation would be the method of choice from an IEC 61850 point of view for such an application in the future. With regards to the network, technologies that provide a seamless (zero failover time) redundancy like PRP (parallel redundancy protocol) would increase the reliability beyond the point were the system has to rely on a secondary system in case of a network failure. VII. CONCLUSION The system described proves that it is possible to utilize, a non-deterministic application, to meet deterministic requirements of reliability, if the design techniques are taken into account. The test data presented proves that the concerns about network congestion can be avoided. The system performance depends on the utilized hardware and software architecture of the respective devices and other network components. This project serves as proof demonstrating that a system based only on fiber cables can replace a conventional system. The availability of faster CPUs and multiple communications ports enables this approach as an alternate for a conventional system. In the near future protocol based systems will bypass the conventional way of using miles of copper cables, not only in speed, but also in reliability. The lessons learned in this project will allow for the incorporation of better design techniques and the use of newer network technologies that were not available at the time this project was deployed. References: [1] Cesar Guerriero, McMahon Ryan, Winters Bill and Becker Farel, "Station integration for reduced costs and improved operational efficiency" presented at the Clemson Conference [2] IEC 61850-8-1 Communications networks and system in Substations First Edition 2005. [3] IEC 61850 -Communication Networks and Systems in Substations: An Overview of Computer Science, Jianqing Zhangand Carl A.Gunter, Illinois Security Lab. [4] RuggedCom: The Communications Backbone for IEC 61850, IEC 61850 Seminar, derived from UCA Users Group Meeting CIGRE 2006 [5] DistribuTECH 2009 - Setting your network for IEC61850 Traffic, Rene Midence. [6] Ruggedcom white paper - Redundancy in Substation LANs with the Rapid Spanning Tree Protocol (IEEE 802.1w). Michael Galea, Marzio Pozzuoli