IP/MPLS PERFORMANCE AND PATH ANALYTICS FOR TELEPROTECTION IN POWER UTILITIES

Transcription

1 IP/MPLS PERFORMANCE AND PATH ANALYTICS FOR TELEPROTECTION IN POWER UTILITIES TECH BRIEF

2 Table of Contents Introduction 3 Adapting legacy technologies and devices to an IP/MPLS network 4 IP/MPLS networks introduce new challenges 4 IP/MPLS path and performance reporting with Packet Design 6 A utility use case 9 Time-stamped alerts for tunnels 11 Evolving the network 13 Page 2 of 14

3 Introduction Power utilities face major cost and technology challenges when building out and modernizing their communication networks. New IP applications in the Smart Grid need to be rolled out and the existing power grid that uses old TDM technologies must be integrated with modern technologies, such as IP/ MPLS networks. Utilities can t afford to operate parallel networks such as one SONET/SDH network and another IP/MPLS network to carry both legacy and new applications. Maintaining reliability of supply and human competence are other key factors that drive the migration to IP/MPLS-based transport infrastructures. At the same time, utilities must continue to comply with strict specifications for TDM-based equipment, such as E1/C37.94, to maintain electric power delivery and control of grid equipment under all circumstances. This paper explains how to monitor and simulate the behavior of highly flexible IP/MPLS networks that carry legacy E1/C37.94 information and Smart Grid applications from a performance and routing redundancy perspective. It is essential to monitor performance metrics, such as IP/MPLS latency, jitter and packet drops, in Inline Differential Protection power systems that use E1/C37.94 equipment. It is also critical to monitor redundancy in various MPLS Traffic Engineering protection schemas, such as Fast Reroute (FRR), Secondary Tunnels and Secondary Paths, to ensure the reliable transport of E1/ C37.94 time-sensitive information from power grid protective relays. Page 3 of 14

4 Adapting legacy technologies and devices to an IP/MPLS network Traditionally, utility companies have relied on TDM technologies, such as SONET/SDH. These technologies deliver carrier-class performance and support the deterministic traffic critical for grid operations. However, they no longer adequately support the long-term needs of utilities. TDM infrastructures are often built and operated for specific applications, thereby creating siloed solutions. Utilities are introducing new IT systems and Smart Grid applications. These new systems cause traffic patterns in the network to change rapidly, requiring packet-switched networks that provide much more flexible network topologies based on multipoint connections (Figure 1). Figure 1. A new network for the Smart Grid IP/MPLS networks introduce new challenges Packet-based networks offer more and better functions and higher levels of service for the new IP applications while continuing to deliver reliability and deterministic (real-time) traffic support for old TDM applications and devices. However, packet-based networks use a statistical multiplexing technology (store-and-forward) that may introduce delay and delay-variations (jitter) that can severely affect legacy TDM applications. These new IP/MPLS networks need to be closely monitored and properly designed to continue to allow for proper operation of the integrated TDM equipment. One example of a critical TDM application with very stringent requirements of the IP/MPLS network is Inline Differential Protection. This technology uses relays that connect E1/T1 (electrical) or C37.94 (Optical TDM) interfaces and measure and compare the AC current of the two remote ends of a power line (Figure 2). When comparing the two ends, it is critical that they are synchronized in timing. If not, this could lead to currents from different time instances being compared, producing a phase angle shift that ultimately leads to degraded protection sensitivity and incorrect tripping of the power line. Page 4 of 14

5 Correct tripping of protected lines is a good thing since it isolates failures, preventing them from spreading across the power grid. Figure 2. Inline Differential Protection of Powerlines If the IP/MPLS network introduces too much latency variation (jitter), it reduces protection sensitivity. So, it is critical to measure the jitter caused by the IP/MPLS network. Delay that is constant and symmetrical usually does not introduce any functional issues, but it does add to the time it takes for the relays to convey a failure and trip the line. Therefore, constant delay is an important metric to measure. The relay specifications for latency tolerance and jitter are vendor specific, but some typical values are: 50ms maximum delay tolerance; 4ms latency asymmetry; 0.5 percent error rate. 3-4ms jitter To handle latency asymmetry, many implementations now clock the relays locally using the Precision Time Protocol (PTP) derived from the IEEE specification. Still, maximum delay and jitter need to be handled properly over the IP/MPLS communication channel. Another complexity introduced by the IP/MPLS network is the transport and protection of the TDM traffic. In SONET/SDH technologies, there is a built-in, carrier-class protection using a 1+1 protection scheme. IP/MPLS offers a variety of protection mechanisms, like FRR protection of nodes and links, secondary pre-signaled tunnels, secondary path-options and/or fallback to native IP routing, to mention some of the more commonly used techniques. Once these TE-tunnels and protections are introduced, there is a critical need to monitor the behavior of the primary and protected tunnels. In a large network, there may be several thousand tunnels, creating complexity and making troubleshooting very difficult. This is where route analytics technology can be very helpful. Page 5 of 14

6 IP/MPLS path and performance reporting with Packet Design To monitor TE-Tunnels in IP/MPLS networks, Packet Design s Route Explorer physical or virtual machine appliance delivers real-time, historical and simulated insights into the network control plane. By connecting to OSPF/IS-IS using a passive neighbor relationship, all routing messages for both the plain IPv4 and the Traffic Engineering LSA/LSP types are captured and stored in the Route Explorer database. This database is then used to create reports in real time and for any point in time. Reports start at an overview level (Figure 3), summarizing the network topology and its primary and protected tunnels, and enable drill-ins to very detailed views about TE attributes, such as ERO objects, Affinity Bits and Reserved Bandwidth, to mention a few (Figure 4). Comparison reports on FRR and Secondary Tunnels and Paths are easily available and graphed on the HTML5-based Web interface (Figure 5). Dashboards showing TE tunnel information can be easily customized on an individual user basis. Figure 3. TE Tunnel Overview Page 6 of 14

7 Figure 4. TE Tunnel Details Page 7 of 14

8 Figure 5. Comparing an active Primary Tunnel (blue) with a stand-by Secondary (yellow), here also showing dynamic delay information for each link on both paths. Reporting on paths, their symmetry and protection is important to operations and engineering teams within utilities. In addition, performance metrics, such as latency, jitter and errors, overlaid on the IP/ MPLS information, are also invaluable to network engineers. The Packet Design Performance Explorer collects SNMP latency metrics using Cisco IP SLA tests and the equivalent for other major router vendors. This information is then projected on top of the routing and path topology, producing a coherent picture of the most important network performance attributes for a utility company (Figure 6). Page 8 of 14

9 Figure 6. A Primary TE-Tunnel crossing a high-delay link A utility use case A regional utility operator in the Nordics uses Route Explorer to monitor its entire L3 routing infrastructure, including OSPF and RSVP-TE tunnels. The operator encountered stability issues when integrating the TDM systems into the new IP/MPLS network the line protection application (the relays monitoring and comparing AC current at substations) proved to be very sensitive to partial or bad connectivity. The relays can handle no connectivity or full connectivity but not asymmetries. After conducting thorough testing using the various RSVP-TE options, the engineering team configured all tunnels in the network to use a multiple path-option statement to achieve redundancy, instead of more traditional FRR techniques. This proved to mimic the legacy SDH network better than other FRR configurations. With path-option, the protection tunnel (or more precisely, the higher path-option tunnel) is not pre-signaled, but rather it is set up on a new explicit path, once the primary path has failed. Page 9 of 14

10 An example configuration using the path-option statement is shown here: interface Tunnel1234 ip unnumbered Loopback0 mpls ip tunnel source Loopback0 tunnel mode mpls traffic-eng tunnel destination tunnel key 1234 tunnel mpls traffic-eng path-option 1 explicit name PATH1 tunnel mpls traffic-eng path-option 2 explicit name PATH2! ip explicit-path name PATH1 enable <------Primary Path-Option index 1 next-address ! ip explicit-path name PATH2 enable <------Redundant re-signaled Path-Option index 1 next-address index 2 next-address index 3 next-address index 4 next-address index 5 next-address index 6 next-address ! pseudowire-class LDS-1234_PW encapsulation mpls preferred-path interface Tunnel1234 disable-fallback < don t route using vanilla IP if tunnel fails Page 10 of 14

11 Time-stamped alerts for tunnels Route Explorer can alert whenever a tunnel goes down and is re-signaled on the second path-option, giving the utility operator exact time-stamped information on tunnel down events (Figure 7). This information is logged locally in the Route Explorer database, displayed on a dashboard and sent via SNMP/Syslog/ to external systems. NOC staff and engineers can focus on the time of the event simply by clicking the alert message (Figure 8) to easily and quickly troubleshoot and monitor for symmetry. Before route analytics were implemented, engineers had to sift through thousands of lines of router logs to try to find relevant events. Figure 7. Time-stamped alert with URL link in message for drill-down into failed tunnel Figure 8. Drill-in to alert URL to see details of tunnel and compare to IGP if need be Page 11 of 14

12 Figure 9. View of critical jitter measurements over 48 hours Page 12 of 14

13 Evolving the network With rapidly changing requirements on the network to enable more functions in the Smart Grid, the regional utility operator is looking at evolving network functionalities, including the ability to monitor and analyze new technologies using Packet Design s Explorer products. Among the planned changes are: Performance measurements - latency, jitter (Figure 9) and packet drops, projecting IP SLA information on paths and routes, to assure the inline differential protection mechanisms for substations. L2 pseudo wires, L3 VPNs (using MP-BGP) in combination with labeled BGP, creating a seamless MPLS transport network. Segment Routing. Packet Design s Explorer Suite already supports these technologies, greatly easing design and operations in the ever-evolving network. Page 13 of 14

14 To learn more about Packet Design and the Explorer Suite, please visit Page 14 of 14