Importance of Interoperability in High Speed Seamless Redundancy (HSR) Communication Networks Richard Harada Product Manager RuggedCom Inc. Introduction Reliable and fault tolerant high speed communication in the substation leads to more efficient operation and cost savings for Smart Grid operators. Substation networks require communications that are fast, reliable, with built-in redundancy and time synchronization. The substation process bus is the Ethernet network that carries the information for automated intelligent protection and control of the Smart Grid. The protection algorithms are time sensitive and time synchronization is critical for successful operation. In an HSR network, the end nodes themselves are connected together to form a ring without an external intermediary switch to manage the connections. The fact that the end nodes are now communicating directly to each other puts an emphasis on the interoperability between different vendors. Mission critical control networks cannot afford delays and therefore efficient handoff and processing of messages between nodes is increasingly important. Utilities that are considering new or upgraded communications in their substations and are considering HSR, the new addition to the IEC 62439 standard for High Availability Automation Networks, need to ensure that interoperability requirements are met so the performance of their network is not compromised. The interest and need for high performance, high reliability Ethernet networks is not new; the IEC 62439 standard was originated to address the need and this standard continues to evolve. The motivation has always been to create a fast, reliable, predictable network architecture that is low cost, easy to maintain and interoperable. The goal has been to create an architecture that is easy to implement so that devices from many vendors can be connected together and communicate on a common network infrastructure. Problems that reliable Ethernet need to solve: Speed delays or latency The challenge in process control applications is that messages from one machine to another need to arrive within a certain time to control the process. In industrial automation where networks are use to control robots, parts placement, welding or label printing can be affected if messages arrive late or don t arrive at all. In protection and control, it could mean getting trip messages to breakers in time to prevent cascading outages. For some manufacturing processes the critical time can be 100ms or less, in a substation environment the critical time can be 10ms or less. Predictability Ethernet was originally designed as a shared medium architecture with mechanisms designed to sense when the medium was clear and it was OK to send a message. While these mechanisms could sense the medium and deal with collisions, there was no way to predict when other devices on the shared medium would be using the available bandwidth. While the mechanisms could resolve the collisions and get the message through, they could not predict when the collision would be resolved. With today s switched networks, collisions can be avoided however unless QoS and prioritization schemes are built in there is still no guarantee that a message would arrive a given time. Control applications rely on messages getting through to manage the function, movement and monitoring of automated processes.
Redundancy / Failover High reliability networks require redundancy and switchover time to the backup system that is fast enough so that it does not negatively impact the operation of the application. As above this can be below 100 or even 10 ms depending on the application. It is often the case that the physical layer is the part that can break, the cables, the connectors or the network hardware itself may fail and cause expensive problems for the process application. Reliable Ethernet needed reliability built in to the architecture itself. To further complicate the requirement, control applications are often needed in uncontrolled environments such as those found in industrial manufacturing or substation automation. Interoperability In addition to the above problems are interoperability requirements. Interoperability is necessary due to the fact that there are many functions in automation and not all vendors are specialists in all areas, so to build a complete high performance multifunctional application may not be possible with a single supplier. High levels of performance and precision is often easier to achieve where interoperability is not required; in these cases a single vendor can create optimized but proprietary schemes to optimize communications and improve response times. Industry organizations such as the IEEE or IEC serve to gather requirements from specific groups and form standards that are published so all potential vendors can build products that will work together. Existing High Availability Ethernet Solutions In order to address the need for high reliability Ethernet, the IEC has created IEC 62439 to standardize protocols that attempt to meet the requirements. To date there are a number of protocols within the standard that are recognized to meet many of the high availability Ethernet requirements: 1) Field Bus Redundancy Protocol (FRP) It uses two separate networks, to which devices are attached through two network adapters. The networks are used alternatively rather than in parallel. +: provides cross-redundancy (double fault network and node) +: provides protection against adapter failures - more than double network costs with respect to non-redundant networks - large effort for building doubly-attached nodes. - switchover time not specified 2) Medium Redundancy Protocol (MRP) Uses a ring based architecture with a Medium Redundancy Master or MRM that checks the integrity of the ring by sending in both directions test frames. These test frames are forwarded by all intact switches and inter-switch links. If the MRM does not receive its own frames over its alternate interface, it closes the ring at its location to re-establish traffic. + fast switchover (< 200ms worst case) + no impact on the nodes + no increase in network infrastructure.
- MRP switches are not compatible with RSTP switches, limited market - limited to ring topology - open but licensed standard 3) Rapid Spanning Tree Protocol (RSTP) RSTP is applicable to any network architecture, ring, mesh etc where redundant paths are physically wired together. The network switches learn the architecture and block the redundant paths to prevent network loops. If an existing path becomes broken, failover occurs so that the redundant path is unblocked to re-establish communication. RSTP has undergone improvements and optimizations from its original form to improve failover time. In original form this failover could take several seconds but in its optimized form, failover can now occur in milliseconds, depending on the size of the network. + IEEE standard, field proven, large market, cheap + no impact on the end nodes (all end nodes are singly attached) + can be implemented in the nodes if the nodes contain a switch element - RSTP is infamous for being rather slow (some seconds switchover time). However, if the topology is fixed, RSTP switches can learn the topography and calculate alternate paths in case one should fail. Some manufacturers claim recovery delays <100 ms for selected configurations - Failover time can grow depending on the size of the network (at least 5ms per switch hop) - no redundancy between end nodes and switch Parallel Redundancy Protocol (PRP) PRP is an architecture where each node is physically connected to two parallel networks that provide complete redundancy down to the packet level. The sending nodes are responsible for generating duplicate packets and the receiving nodes are responsible for receiving the packets and eliminating the duplicates. Sending Receiving Network A Network B
+ PRP allows bumpless switchover, no frames are lost + During normal operation, PRP reduces the loss rate + PRP checks the presence of nodes by periodical supervision frames that also indicate which nodes participate in the protocol and which not - double network costs - doubly attached devices are costly to build - frame size must be limited to prevent frames from becoming longer than the IEEE 802.3 - maximum size (but most switches and Ethernet controllers accept frames up to 1536 octets) High Availability Seamless Redundancy (HSR) Principles of Operation HSR is based on ring type architecture to achieve its network path redundancy. Duplicate packets are sent in opposite directions of the ring to achieve redundancy down to the packet level. When a packet arrives at a node, the node will determine if the packet is addressed to it or to another destination. If the packet is for another destination, the node will simply forward the packet on to the next node in the network. If the packet is addressed to the node it will either process it or if it is a duplicate packet, discard it. Unlike protocols such as MRP or RSTP, the paths are always active and there are no decisions to be made to reconfigure the network, therefore there is zero recovery time from a single point of failure. Instead of the network switches having to figure out duplicate paths and adding or removing links, the end nodes are responsible for handling duplicate packets. The end nodes simply process the first packet it sees and discards the duplicate. If the end nodes do not receive the second packet within a time period, it knows there is an open path and can inform the network administrator. There are additional management packets that nodes circulate to monitor the network. Non HSR Network IEEE 1588 v2 Timing Source HSR Redundancy Box HSR Redundancy Box HSR Redundancy Box HSR Redundancy Box C B A Sending Receiving TM IED with Embedded HSR Module TM IED with Embedded HSR Module TM IED with Embedded HSR Module HSR is similar to PRP in the fact that there are duplicate packets sent and received however HSR does not come with the overhead of duplicated network cabling and network switches making it much less costly to install and maintain than PRP.
The HSR protocol provides redundancy at layer two making it useful for implementing industrial designs that leverage layer two features such as multicast. Multicast is useful for high speed communication of messages from one to many nodes. As an example in the utility market, IEC 61850 uses multicast extensively for GOOSE messaging and merging unit communications on the high speed process bus. Combining Network Features There are improvements in networking that are not specific to HSR but can be employed in conjunction with HSR to provide a high performance network. Some of these features include: Virtual LANs (VLANs) provide network segregation to keep application specific traffic on separate virtual networks. On the same physical medium separate networks for mission critical control applications can exist with less critical SCADA or administration networks. These VLANs can be given different priorities and bandwidth limitations to ensure critical applications always have enough network capacity to get their messages through. Medium Priority VLAN High Priority VLAN Low Priority VLAN Best Effort VLAN Cut-through Switching Most network switches use store and forward switching, allowing them to examine the whole packet for errors before forwarding it onto the network. This prevents the propagation of bad packets but at the cost of time. Cut-through switching on the other hand looks only at the first few bytes of the packet to determine the destination address before forwarding the packet. Cutthrough switching is the preferred method in an HSR network due to the low latency. IEEE 1588 v2 Precision Timing Many control applications that will take advantage of HSR will also have a requirement for high accuracy timing. 1588 v2 is capable of providing accuracy on a switched network of better than 1us. HSR will take advantage of this with onboard hardware time stamping to provide the precision needed for these applications. Packet is HW Packet
Linking Non-HSR s The HSR Redundancy Box (Redbox) is used to connect non-hsr nodes to an HSR network. The non-hsr node does not need to be HSR aware, there is a simple Ethernet connection from the Redbox to the non-hsr node. A Redbox can be designed with a single non-hsr interface or several, providing a gateway for several nodes onto the HSR network. In this way HSR can accommodate using HSR in some parts of the network and using RSTP or PRP in others providing tremendous network flexibility, scalability and interoperability. Linking HSR Rings The HSR Quad-box provides the facility to link adjacent HSR rings. The quad-box again provides flexibility and scalability to allow multiple HSR rings in a site to be linked with different physical sites or locations. Sending Receiving Group Multicast Registration Protocol (GMRP) In some process control applications, there are devices that distribute information that multiple other devices are interested to receive. In a Smart Grid application this could be a monitoring instrument or merging unit that looks at the condition of a particular section of the power system and publishes this information for adjacent IEDs that may have to activate protective relays. GMRP allows devices to subscribe to multicasts, providing the information to network elements that will automatically route the multicasts only to areas of the network where required, to prevent unnecessary traffic flooding on the network. In this way GMRP optimizes network communication and reduces the burden of manual configuration. HSR Interoperability Managing interoperability in an HSR network can potentially be very challenging due to the fact that the topology is so different from a typical Ethernet network. In a typical Ethernet network, the end devices connect to a network switch that manages the sending and receiving of packets to other nodes. In contrast HSR network end devices connect to each other in a peer to peer ring
without the intermediary Ethernet switch. With just a basic HSR network, the interoperability can be challenging since it forces vendors that are potentially competitors to work together to sort out problems and differences. In a switch environment it can be the switch vendor that is the neutral party that can work with vendors and customers to manage the issues. Additionally networking features such as security, VLANs, QoS etc are generally configured on the Ethernet switch. An IED with HSR is like having a three port switch embedded with all the networking features that need to be managed now becoming part of the IED configuration. From a customer perspective it can potentially be very challenging to know what vendor devices support what features and which vendors work well together. This is in addition to knowing how to configure all the different devices. For this new technologies to be introduced and be successful there will have to be an independent third party established that will be able to bring vendors together for initial testing and then certification. The third party will require a test bed of devices that are known to work together with a test plan to introduce and certify new devices. The initial testing of HSR has been done in Switzerland at the University of Zurich. The University has also participated in the development of the HSR technology. Benefits of HSR HSR provides end users many benefits over traditional Ethernet networks. The original weaknesses of Ethernet have been the unpredictable behavior under heavy load and under failure conditions. Modern Ethernet technologies such as switched networks, VLANs, prioritization, quality of service etc have served to address many of these concerns. Switched networks eliminate the unpredictability of packet collision domains. VLANs, prioritization and QoS serve to classify, isolate and prioritize different network types to ensure that there is enough bandwidth for high priority messages to get through. HSR provides a solution for the remaining issues by providing the predictable behavior and zero recovery time under failure conditions. While RSTP has become the most popular and widely accepted redundancy method with recovery times down to 5ms per node, the recovery time increases with the size of the network and even 5ms may not be acceptable for all applications. HSR is able to tolerate single points of failure without any interruption and comes at a much less expensive cost than PRP. The end user benefits of HSR s higher network uptime and predictable network behavior are therefore less outages and less cost than other Ethernet redundancy algorithms. Applications HSR can have many applications where high speed, high reliability and low latency are requirements. In the utility industry, the thought is that the IEC 61850-9-2 process bus will be the initial application that has the most impact. The process bus has several characteristics that make it suitable for HSR topology: 1) the process bus could have many merging units that continuously sample current and voltage readings from the power system, digitize these samples and multicast them on the network for other IEDs to monitor and react to; HSR provides the bandwidth and prioritization for these messages 2) HSR can support GMRP allowing self subscribing of multicast to reduce device configuration 3) HSR can support high precision IEEE 1588 v2 time synchronization that is critical for protection and control applications 4) HSR has low latency to provide delivery of critical information in the shortest amount of time Conclusions
+ bumpless failover in case of failure of a node or reinsertion of a repaired node + constantly supervises the redundancy + monitors actual topography (over network management / SNMP) + application-protocol independent + an open standard, no license fee + reuses most of the concepts of PRP (IEC62439) + can be used for other industrial Ethernet applications - can halve the available network bandwidth for multicast messages - non-hsr devices can only be inserted over a RedBox - limited to a layer 2 broadcast domain - requires a hardware implementation (ASIC or FPGA) to meet the time constraints - HSR is a new addition to IEC 62439 High Availability Ethernet that addresses reliability and predictability of Ethernet - HSR is a peer to peer ring architecture therefore vendor configuration and interoperability are important considerations References: 1) Standard Redundancy Methods for Highly Available Automation Networks: Rationales behind the upcoming IEC 62439 standard Hubert Kirrmann ABB Switzerland Ltd, Corporate Research