automation network Prof. Dr. Hubert Kirrmann Diana Ilie ABB Research Center, Baden, Switzerland Claudio Honegger

Size: px

Start display at page:

Download "automation network Prof. Dr. Hubert Kirrmann Diana Ilie ABB Research Center, Baden, Switzerland Claudio Honegger"

Silvia Summers
5 years ago
Views:

Performance of a full-hardware PTP implementation for an IEC 62439-3 redundant IEC 61850 substation automation network Prof. Dr. Hubert Kirrmann ABB Research Center, Baden Switzerland hubert.

sotiropoulos@ch.abb.com Abstract Seamless redundancy and precise clock synchronization are integrated fully in hardware (FPGA) with no processor support.

Measurement results confirm the validity of the concept. Performance of the full-hardware implementation is compared with that of a conventional software-hardware implementation.

1 Performance of a full-hardware PTP implementation for an IEC redundant IEC substation automation network Prof. Dr. Hubert Kirrmann ABB Research Center, Baden Switzerland hubert.kirrmann@ch.abb.com Claudio Honegger ABB Research Center, Baden Switzerland claudio.honegger@ch.abb.com Diana Ilie ABB Research Center, Baden, Switzerland Ioannis Sotiropoulos ABB Research Center, Baden, Switzerland ioannis.sotiropoulos@ch.abb.com Abstract Seamless redundancy and precise clock synchronization are integrated fully in hardware (FPGA) with no processor support. The interaction between redundancy and clock synchronization is explained. Redundant synchronization messages are not be discarded, but used to improve clock accuracy. Measurement results confirm the validity of the concept. Performance of the full-hardware implementation is compared with that of a conventional software-hardware implementation. Keywords: IEEE 1588; IEC 61588; IEC 61850; IEC ; PRP; HSR; clock redundancy; FPGA; power profile; I. INTRODUCTION Substation automation networks requiree high availability and precise clock synchronization. Until now, synchronization and redundancy were treated separately. IEC [1], the communication standard for Electrical Substation Automation, will specify 1 µs precise time distribution over an IEEE 1588 profile [3] in its next edition. Since IEC (GOOSE) and IEC (SV) transmit time-critical data over Ethernet IEEEE [4] layer 2 (physical and link layer), they also specify redundancy on layer 2 [5], by using IEC [2] as explained in [6]. IEC consists of two redundancy protocols with zero switchover delay, PRP (Parallel Redundancy Protocol) and HSR (High-availability Seamless Redundancy), that operate on the same principle of parallel transmission over two indepent paths. This scheme works welll for usual traffic, but presents a challenge for clock synchronization since 1588 does not consider redundant PTP messages [7], [8]. IEC Annex A defines how to handle redundant PTP messages when using IEEE 1588 profile suited for substation automation. This paper describes a first implementation of IEC Annex A and shows that this concept holds its promises. This paper shows the benefits of an integrated HSR/PRP/PTP transparent and ordinary clock in FPGA. This paper show that the whole stack can be handled in the FPGA with benefits in terms of resources and performance. II. NETWORK REDUNDANCY PRINCIPLES A. Parallel Redundancy Protocol (PRP) Principle Figure 1 shows a typical duplicated PRP network. Redundancy-enabled s, called DANP (doubly attached s with PRP) have two ports connected to two indepent LANs, LAN_A and LAN_B. Both LANs are switched Ethernet with an arbitrary number of s and bridges operating with RSTP. Each bridge executes the clock synchronization according to IEEE 1588 Annex J.4 (default peer-to-peerr profile, one or two steps). The devices within the LANs are unaware of the PRP protocol. Figure 1 Parallel redundancy Protocol (PRP) principle (IEC ). The source of a frame ss a copy of it over each port, an A -frame and B -frame. The frames are sent at about the same time and travel indepently. The DANP destination accepts the first frame and discards the duplicate (if it arrives). In case of loss, the application on the destination operates with the remaining frame undisturbed. To identify duplicates, each frame carries a sequence number in a redundancy control trailer apped to the frames. Since a normal application ignores this trailer, it is unaware of

PRP. This allows to connect Singly Attached Nodes (SANs) such as printers or laptops, to one LAN without modification.

RedBoxes allow to attach single devices or complete LAN segments and also interface to HSR (see below). B. PRP structure C.

As PRP, it relies on duplication of frames and discarding of duplicates (Figure 3). Figure 3. Example of HSR ring (IEC 62439-3). Figure 2.

This allows all protocols above layer 2 to operate as if there were no redundancy and simplifies engineering.

upper layers as a single Ethernet interface. In a source, the LRE duplicates the frame it received from the layers above and ss the frames over both adapters at nearly the same time.

It keeps track of the duplicates through the duplicate tables, that are typicallyy arranged as hash tables.

2 PRP. This allows to connect Singly Attached Nodes (SANs) such as printers or laptops, to one LAN without modification. Singly attached devices that need redundancy are attached through RedBoxes (redundancy boxes) that duplicate / merge the frames for them. RedBoxes allow to attach single devices or complete LAN segments and also interface to HSR (see below). B. PRP structure C. High-availability Seamless Redundancy (HSR) principle HSR (IEC , Clause 5) provides seamless redundancy with only one additional link. As PRP, it relies on duplication of frames and discarding of duplicates (Figure 3). Figure 3. Example of HSR ring (IEC ). Figure 2. Node structure of PRP (IEC ) Each DANP (Figure 2) has two Ethernett adapters with the same MAC address and present the same IP address(es); therefore, PRP is a layer 2 redundancy. This allows all protocols above layer 2 to operate as if there were no redundancy and simplifies engineering. An additional sub-layer is introduced in the (otherwise unmodified) link layer, the LRE (Link Redundancy Entity), that handles both Ethernet controllers and presents the same interface towards the upper layers as a single Ethernet interface. In a source, the LRE duplicates the frame it received from the layers above and ss the frames over both adapters at nearly the same time. In a destination, the LRE receives the same frame from both adapters within some time skew since the delays in the networks are different. It keeps track of the duplicates through the duplicate tables, that are typicallyy arranged as hash tables. The LRE discards the second frame of a pair, mainly to offload the application (it would work without duplicate discard but not as well). If a link or a port is damaged, the LRE will still receive frames over the other path, data keep flowing over the healthy path without influence on the application, the LRE can take note that the duplicate did not arrive for diagnostics purpose. The LRE for PRP could be implemented in software, e.g. within the driver or the communication processor, since it introduces a negligible delay. Each redundancy-enabled, called a DANH (doubly attached with HSR) has at least two ports of similar capabilities. The s are daisy-chained into a ring, but other topologies also exist. Each DANH implements the forwarding capability described in IEC [2] according to IEEE 802.1D [9]. The ser of a frame inserts a copy of it into the sing queue of each port. The two frames are sent at nearly the same time and travel in opposite direction. In the fault-free state of the ring, each destination receives two identical frames, passes the first frame of a pair to its application and discards the duplicate. A DANH should not forward a frame which is received from the ring, when: - the is the sole destinationn of the frame - the itself injected the frame into the ring - the already forwarded the frame. Every is responsible to detect duplicates and remove them from the ring. Thus, communication can sustain the loss of any frame, any link or any that is neither source nor destination. The network delay is the product of the individual bridging delays (max. 5 µs) by the number of transit s. Although the average delay in a healthy HSR ring is half that of a simple ring, the worst-case delay in case of loss of a link or is given by the total number of s in the ring. To keep this delay at a rate acceptable for time-critical transmission of sampled values (4.8 khz, or 208 µs interval), IEC prescribes that each in the ring forwards the frames within 5 µs when there is no other traffic. This requires that s provide a cut-through bridging that can only be provided by hardware support. Cut-through reduces the average forwarding delay, but it does not improve the worst case propagation delay, which happens when all s at the same time inject a maximum

size frame, and delay the forwarded frames, even though the ring traffic has priority. D. HSR Node structure As Figure 4 shows, the structure of an HSR is similar to that of a PRP (Figure 2).

This condition can be occasionally defeated in RSTP during reconfiguration after a link or bridge failure, but in PRP and HSR, this is the normal case but this is not foreseen in IEEE 1588.

The PRP duplicate discardd method cannot apply to PTP messages because the bridges in the LAN are not PRP-aware.

Node structure of HSR (IEC 62439-3) In HSR, there is addition to PRP a bridging logic that forwards frames from port A to port B and vice-versa.

Such a low delay requires a cut-through mode, i.e. a frame is forwarded before it is entirely received.

pre-allocated time window (TDMA), for instance using a common precision clock (seee [2]).

There may also exist a duplicate table for each port, a solution that scales better with the number of ports. III.

3 size frame, and delay the forwarded frames, even though the ring traffic has priority. D. HSR Node structure As Figure 4 shows, the structure of an HSR is similar to that of a PRP (Figure 2). IEEE 1588 assumes that the Announce, Sync and Follow_Up take the same path. This condition can be occasionally defeated in RSTP during reconfiguration after a link or bridge failure, but in PRP and HSR, this is the normal case but this is not foreseen in IEEE The problems stated in [7] have been addressed in IEC Annex A, as follows. In PRP, the GrandMasterr is assumed to be a DANP connected to LAN_A and LAN_B, as Figure 5 shows. The PRP duplicate discardd method cannot apply to PTP messages because the bridges in the LAN are not PRP-aware. Thus, they do not preserve the PRP trailer when forwarding the Sync messages and app no trailer to their Follow_Up. Figure 4. Node structure of HSR (IEC ) In HSR, there is addition to PRP a bridging logic that forwards frames from port A to port B and vice-versa. IEC prescribes a forwarding delay of less than 5 µs when there is no other traffic, to keep the sum of the propagation delays along the ring low. Such a low delay requires a cut-through mode, i.e. a frame is forwarded before it is entirely received. Therefore, if an Ethernet protocol for a hard real-time communication system utilizing HSR wants to exploit fully its cut-through properties, it has to constrain the s to s their frames in a pre-allocated time window (TDMA), for instance using a common precision clock (seee [2]). The duplicate detection is more elaborated in HSR than in PRP since a malfunction of the duplicate detection can cause a ring flooding. Figure 4 shows the duplicate detector as a central component. There may also exist a duplicate table for each port, a solution that scales better with the number of ports. III. IEEE 1588 CLOCKS SYNCHRONIZATION Clock synchronization in PRP and HSR relies on IEEE 1588 V2, using the profile of Annex J.4 [3], [10]. For PRP and HSR, IEC Annex A defines a PTP profile suitable for time-critical systems in substations. All network devices between GrandMaster and Ordinary Clock are Transparent Clocks, using: Ethernet layer 2 communication (no UDP) Multicast PTP messages (no unicast) Peer-to-peer path delay (no -to- delay) One-step or two-step both allowed Best Master Clock with default settings Figure 5. Clocks in PRP The three frames (Announce, Sync and Follow_Up) must belong to the same LAN, otherwise the correction field will not be consistent. The Sync messages have different contents whether they come over LAN_AA or LAN_B. Although Announce messages should be identical on both LANs, some derived standards allow dynamic modification and Announce could be different on LAN_A and LAN_B. Pdelay_Req and Pdelay Resp are link-specific and therefore LAN-specific. A DANP sees two different link delays on LAN_A and LAN_B. To simplify logic, it converts two-step Sync and Pdelay to one-step if it receives such messages. Therefore, the ordinary clock in the DANP treats PTP messages from each port separately, as coming from different clocks, and listens to only one of them, or accepts both. When the DANP detects that PTP messages come from the same GrandMaster, it uses both Sync messages to reduce jitter and improve the clock accuracy. Indeed, although Sync A and Sync B come from the same source, using both improves accuracy since the two Syncs

suffered different stochastic network delays due to PHY jitter, clock inaccuracies or path variations.

Figure 6 shows the effect of losing one LAN on the clock synchronization when a large number of transparent clocks have been crossed in each LAN.

A -frame GPS MC redundant master clocks B -frame MC interlink switch OC OC OC Figure 7. Clocks in HSR Figure 6.

4 suffered different stochastic network delays due to PHY jitter, clock inaccuracies or path variations. In hardware measurements the network jitter has a normal distribution, which is used to approximate the real value. Figure 6 shows the effect of losing one LAN on the clock synchronization when a large number of transparent clocks have been crossed in each LAN. In a healthy PRP/HSR network, the jitter is smaller due to a statistical effect. For a known distribution of random variable, the more values are considered, the better the approximation. A -frame GPS MC redundant master clocks B -frame MC interlink switch OC OC OC Figure 7. Clocks in HSR Figure 6. Clock accuracy in PRP (normalized) It could also be that the clocks are different in LAN_A and LAN_B. The loss of the link to the grandmaster in one LAN causes the election of a backup master on that LAN. In that case, the DANP considers only the best quality master. So, as an extension to IEEE 1588, the DANPs also execute the Best Master Clock algorithm to select among different masters. A. HSR and clocks In HSR, the time distribution is similar to PRP. The PTP messages injected into the ring by the GrandMaster travel in both directions (Figure 7). They have the same HSR and PTP sequence number. Each DANH has both a Transparent Clock and an Ordinary Clock for itself. This is called a Hybrid Clock. While the preferred transmission mode is one-step, twostep are also allowed. A one-step capable DANH converts twomessages for each step messages to one-step. As in PRP, each DANH treats the PTP direction separately. The Ordinary Clock uses both Syncs to improve accuracy when both come from the same clock. A situation in which two different masterss s Announce messages is handled as in PRP. In principle, PTP messages need no HSR tag, but it has been maintained for removing truncated messages. Compared to a non-redundant clock, an HSR has to handle double as many PTP messages and execute the BMCA. This is especially felt when the GrandMaster fails. The back-up masters cause a PTP-storm of messages that most processors have difficulty to cope with. B. PRP to HSR connection. A ring can be connected to a duplicated PRP network as Figure 8 shows. Two RedBoxes are needed to avoid a single point of failure. Since each RedBox injects two frames into the ring, four copy of a frame would circulate, but as soon as a frame reaches a port that has already sent a copy, it will be discarded, so that the overall traffic consists just of the pair of frames. On reception, a RedBox only forwards one copy. The PRP trailer, resp. the HSR tag carry an identifier of the originating HSR ring or PRP network to prevent reinjection. This information is lost for the PTP messages. Therefore, HSR s must handle four Syncs and detect from which GrandMaster they come from. An HSR RedBox therefore could perform an exted duplicate discard using the sequence number of the PTP messages to reduce traffic, but here also, using all four Syncs reduces jitter. Figure 8. Clocks in PRP and HSR coupling

IV. NODE IMPLEMENTATION The close coupling between bridging and clock calls for an integrated implementation of HSR and PTP in hardware.

Three transparent clocks compute the residence delay between each port pair. Each port keeps track of the peer delay.

The choice of FPGA and processor is a convenience due to the available hardware. The FPGA is an Altera Cyclone III with a 66 MHz clock, 55 000 LE and 256 x 9K of RAM.

5 IV. NODE IMPLEMENTATION The close coupling between bridging and clock calls for an integrated implementation of HSR and PTP in hardware. Although PRP can be realized in software only, once the HSR and PTP logic is in place, adding the PRP logic is trivial. Figure 9 shows a combined HSR-PTPP with three ports. Three transparent clocks compute the residence delay between each port pair. Each port keeps track of the peer delay. The ordinary clock is syntonized and keeps the exact time, it can serve as a real-time clock for the application processor and generate the sampling pulses for the ADC and the 1PPS output. The choice of FPGA and processor is a convenience due to the available hardware. The FPGA is an Altera Cyclone III with a 66 MHz clock, LE and 256 x 9K of RAM. The Micrel KSZ8001 PHY operate at 100 Mbit/ /s with a 25 MHz clock. Time-stamping is done in the FPGA. The challenges were to reduce the amount of logic elements, cope with different clock domains (each PHY has its own clock), and provide a future-proof architecture that can accommodate higher speeds and a larger number of ports. The inaccuracy introduced by the transparent clocks is currently below 50 ns per and could be improved with PHYs that recognize the time-stamping point in the messages. logic. To accept indifferently one and two-step clocks, a port is connected to a 2-to-1 step conversion block. Therefore, only 1- step Sync frames are handledd internally. Each port is also connected with a block of Transparent Clock. The PTP Ordinary Clock that is synchronized by the PTP Sync messages is implemented in hardware. The parts implemented in software are the management stack and the Best Master Clock Algorithm. The CPU receives Announce and Sync frames, decodes them and executes the BMC algorithm. In case the current is selected as a master, the CPU generates the Sync and Announce frames, which pass through the HSR/PRP core, which tags them and registers them in the duplicate discard tables and then transmits them through the other ports, so this s clock is distributed over the network. The CPU can be an external hardcore CPU, connected with the FPGA, or a softcore CPU (NIOS) instantiated directly in the FPGA. Both approaches have been implemented. In the first approach, the same CPU that executes the application layer also executes the BMC algorithm. Figure 9. Node layout (one direction) A. Hybrid PTP Implementation A hybrid software-hardware implementation is the state-of- both HSR/PRP the-art implementation for a system that uses and PTP ([8]). In such a system, a part of the functionality is implemented in logic, and the rest is implemented by a CPU. A block diagram of the hybrid implementation is presented in Figure 10. The has 3 external ports, 2 of them used as HSR ports and one as a non-redundant port. This port is connected with a CPU that executes the application, in particular substation automation protection. The HSR/PRP core that identifies and discards duplicates for each port is implemented in hardware Figure 10 - Hybrid PTP implementation with redundancy Retaining part of PTP functionality in a CPU is convenient, since it can be easily implemented if the network stack of the CPU can handles layer 2 messages. However, reception or generation of PTP traffic can easily overload the CPU. B. Full hardware PTP implementation A hardware-only PTP switch with redundancy (HSR/PRP) was also implemented. The block diagram of this design is the same as in Figure 10, only that the NIOS is replaced by the BMC-/Management logic directly implemented in VHDL. C. Comparison of the hybrid and the hardware-only implementation The hybrid approach provides an easy maintenance and update of the BMC algorithm, as only the software code has to

6 be changed. On the other hand, the layer 2 handling of the netstack requires a lot of effort and intensive testing. The hardware-only approach relieves the processor from the PTP related tasks, which are significant. It provides also better area and memory utilization, faster system response and less programming effort. The softcore processor can be removed, leaving place for more RAM blocks and logical elements. System response is better since the hardware BMCA reacts faster. Table 2 shows the resource utilization of a hybrid solution, when a softcore CPU (Altera s NIOS II) is used. Saving the resources of the CPU and its communication with the FPGA core is an important optimization, as shown on Table 3 Another advantage of the hardware-only implementation is portability. While adding PTP to a CPU stack requires many changes and additions to the existing stack, integrating the PTP IP requires only the configuration and interface signal connections on the FPGA. HSR s and transparent clocks both forward the ingressing traffic from one port to another. When a supports both modes, the two modes can be treated separately/indepently. In this case, the forwarding delay (measured between the time-stamping point of the ingressing and egressing frame) is the sum of each mode s delay (which is caused by the frame buffering, until required information is decoded from it or controllers responses about forwarding or dropping the frame). The residence delays can be reduced by unifying operations that buffer the ingressing frames. (Table 1). TABLE 1 - AVERAGE FORWARDING DELAY PER NODE (FULL HW) Components Forwarding Delay HSR ~ 4 µs Transparent Clock ~ 2.5 µs HSR+Transparent Clock (Indepent Functions) ~ 6.5 µs HSR+Transparent Clock (Unified Functions) ~ 4.2 µs TABLE 2 RESOURCE UTILIZATION OF A HYBRID HW-SW SOLUTION Components Logic Elements Block RAMs (M9K) HSR/PRP (2 Red Ports 2 Interlinks) 24, PTP Transparent Clock (3 Ports) 16, PTP 2-1 Step Converter (3 Ports) 3,700 6 PTP Ordinary Clock 4,000 2 PTP BMC + PTP Frame Generator on CPU 3, Communication of CPU with PTP on FPGA 3, TABLE 3 RESOURCE UTILIZATION OF FULL HW SOLUTION Components Logic Elements Block RAMs (M9K) HSR/PRP (2 Red Ports 2 Interlinks) 24, PTP Transparent Clock (3 Ports) 16, PTP 2-1 Step Converter (3 Ports) 3,700 6 PTP Ordinary Clock 4,000 2 PTP BMC + PTP Frame Generator 3,000 3 D. Scalability This concept allows to build devices that connect to multiple HSR rings, replicating the logic for each port, with practically no overhead. V. CONCLUSIONS A full-hardware implementation of a combined redundancy IEC with IEEE 1588 clock synchronization is cost effective and can be used for new devices and retrofit. It provides synchronization and redundancy with no additional burden on the application processors. The modular design allows to build HSR s with a large number of ports, as well as RedBoxes. As special advantages, this design is technology and operating system indepent, has a good form factor and eases maintenance while providing better stability. Future development would benefit from PHY that provide a synchronization pulse according to IEEE A 1 Gbit/s design is no real challenge, it has only a cost impact since faster logic and larger memories are needed. References [1] IEC Ed.2: (Part 5, 8-1 and 9-2) Communication networks and systems for power utility automation, International Electrotechnical Commission, Geneva, 2012 [2] IEC High Availability Automation Networks PRP & HSR, International Electrotechnical Commission, Geneva, 2012 [3] IEC 61588, Precision Time Protocol (PTP), version 2, International Electrotechnical Commission, Geneva, [4] The Institute of Electrical and Electronic Engineers, CSMA/CD access method and physical layer specifications. IEEE Std 802.3, [5] H. Kirrmann, C. Hoga, O. Kleineberg; H. Weibel HSR: Industrial Ethernet redundancy with zero recovery time and low cost (High availability Seamless Ring, IEC ), ETFA 2009, 2009 [6] Kirrmann, H.; Dzung, D.; Selecting a Standard Redundancy Method for Highly Available Industrial Networks, 2006 IEEE International Workshop on Factory Communication Systems, June 27, 2006 Page(s): [7] Abdul Amin, Integration of HSR and IEEE 1588 over Ethernet networks, ISPCS [8] Hans Weibel, Sven Meier, IEEE 1588 applied in the environment of high availability LANs, International IEEE Symposium on Precision Clock Synchronization for Measurement, October [9] The Institute of Electrical and Electronic Engineers, ANSI/IEEE Std 801.2D, Media Access Control (MAC) Bridges, [10] Tournier J.C, Weber K., Hoga C. Precise Time Synchronization on a High Available Redundant Ring Protocol, 2009 IEEE International Symposium on Precise Clock Synchronization. Brescia, Italy, September 2009.

SoCe. Introduction to HSR/PRP/ IEEE 1588(PTP) System-on-Chip engineering V: UCA STICK

SoCe. Introduction to HSR/PRP/ IEEE 1588(PTP) System-on-Chip engineering V: UCA STICK SoCe System-on-Chip engineering Introduction to HSR/PRP/ IEEE 1588(PTP) V:140626 UCA STICK SoCe Index: Introduction: PRP (IEC 62439 3 Clause 4) HSR (IEC 62439 3 Clause 5) IEEE 1588 IEC61588 (PTP) HPS IP: