Teleprotection over an IP/MPLS Network

Teleprotection over an IP/MPLS Network Technical validation ir. Lieven LEVRAU Alcatel-Lucent IP Division April 4 th, 2011

1 IP/MPLS-based Utility Networks

IP/MPLS-Based Utility Network Power Generation OS6855 7705 SAR LAN IED Corporate HQ / NOC Internet Omni PCX 5620 SAM SCADA 7750 SR 7750 SR IP/MPLS Network 7705 SAR 7750 SR Servers Collaboration Tool NOC Substation Substation Transmission 7750 SR 7705 SAR 7705 SAR IED RTU LAN 7705 SAR Video Surveillance TPR LAN TDM Video Surveillance OS6855 TPR 4 MPLS Network for Teleprotection Sep 2009

How IP/MPLS Address the challenges of the Utility Networks? Reliability High Availability, Fast Reroute, end-to-end, Traffic Isolation IP/MPLS based - L2-VPNs, VLL, L3-VPNs, PWE Multiple Legacy technologies Security Unified management Quality of Service FR, ATM, TDM, Serial, analog voice, synchronous, Traffic isolation, Rtng protocols authentication, L2/L3 Encryption + dynamic secure Key distribution, DoS End-to-end secure management of network and services Guaranteed bandwidth, low latency and jitter ` Enhanced QoS on different network layers

2 teleprotection Challenges for IP/MPLS in

Using MPLS Network for Teleprotection Substation kv Substation G.703 E1/T1 MUX TPR E&M RS-232 Ethernet 7705 SAR 7710 SR t IP/MPLS 7750 SR TPR E&M RS-232 Ethernet 7705 SAR MUX G.703 E1/T1 Teleprotection relay (TPR) signals must be transferred reliably and fast with low latency End to end delay = telecom network latency + teleprotection equipment delay TDM over MPLS for legacy support (requires integration of legacy interface in MPLS node to limit and control end to end delay) VPLS for IEC 61850 (requires high reliable Layer 2 transport) 7 MPLS Network for Teleprotection Sep 2009

Main Challenges for Teleprotection End to End Delay Jitter Asymmetry Resiliency Denial of Services Synchronization Quality of Service Depending on vendor equipment May include packetisation/depacketisation Variation of delay in certain circumstances such as traffic burst on network Delay variations between transmit and receive The impact of a failure in the network shouldn t be noticed by application How DoS/DDoS can affect applications TDM applications need end to end synchronization Guaranteed bandwidth, low latency and jitter Enhanced QoS on different network layers

End-to-End TDM Transport Model (only left-to-right direction shown) TDM Packets moving in this direction DS1 Access Circuit DS1 LIU Data Si g Packet Switched Packetization GigE GigE Jitter Data Network Buffer (PSN) Si g DS1 LIU DS1 Access Circuit Packetization As TDM traffic from the Access Circuit (AC) is received, it is packetized and transmitted into the PSN Two modes of operation: CESoPSN (RFC5086) for structured nxds0/64k channels SAToP (RFC4553) for unstructured T1 Fixed delay Network Packet transfer delay based on link speeds and distances from end to end Variable delay the number of and type of switches queuing point in the switches QoS is key to ensure effective service delivery Playout TDM PW packets are received from the PSN and stored into its associated configurable jitter buffer Play-out of the TDM data back into the AC when it s at least 50% full 9 MPLS Network for Teleprotection Sep 2009

Using MPLS Network for Teleprotection Latency Substation kv Substation G.703 E1/T1 MUX TPR E&M RS-232 Ethernet 7705 SAR 7750 SR t IP/MPLS 7750 SR TPR E&M RS-232 Ethernet 7705 SAR MUX G.703 E1/T1 TPR relay signals must be transferred in < t T = 1 cycle at 50Hz or 60Hz = 20ms or 16ms End to end delay = telecom network latency + teleprotection equipment time latency = packetization delay + network delay + jitter buffer delay The total end-to-end latency is calculated by summing the packetization delay (PD), network delay (ND) and jitter buffer delay (JBD) as shown here: Total Latency = PD + ND + JBD e.g. PD of 2 ms (16 T1 frames/packet), ND of 3 ms, JBD of 4 ms Total Latency = 2 + 3 + 4 = 9 ms 10 Teleprotection over IP/MPLS - Validation - March2011

Calculation of Latency for Teleprotection Latency Latency is mainly at the edge where low speeds are present (Serial / E1 / 100 FX) Latency in the core depends on number of nodes but mainly transmission delays Each node adds a maximum of : 150µs (eqpt latency) 10µs (transmission of 1500 Bytes over a Gigabit link) 3µs / km : speed of light transmission over fiber Example of a connexion between 2 TPRs : Over 1000km traversing 10 nodes : 4ms 12 Teleprotection over IP/MPLS - Validation - March2011

Jitter Jitter in normal operations may come from the equipments themselves This is minimal (less than 1% of router latency) and compensated by jitter buffer. Jitter happens in non normal conditions such as congestion. This can be solved by applying the correct QoS parameters to the node. The implementation should allow total control of the bandwidth required per application, minimizing jitter.` Rate Limit TPR to 100Kbps TPR gets to EF class Rate Limit CCTV to 6Mbps CCTV to BE Class TPR Per SAP policing Per interface queuing Ingress Make Sure that no application can go higher than expected bandwidth. Via Rate limit per SAP Egress Ensure that TPR application always gets the priority H-QoS 13 Teleprotection over IP/MPLS - Validation - March2011

Impact of failure Today s teleprotection applications were developed towards SDH/PDH 50ms failover time. The impact of a failure (node or link) can have huge impact in case the failure exceeds the 50ms. MPLS FRR technology allows 50ms failover time in any failure scenario. These 50ms apply only in the core of MPLS network, but not in the case of the Access switch failing. 14 Teleprotection over IP/MPLS - Validation - March2011

Conclusion on network failure impact Failover in the backbone can be limited to 50ms with FRR FRR has to be implemented in the first node to minimize failure risks. All ALU Service routers support FRR. Integrating TDM in MPLS (as in 7705 SAR) brings the FRR to the application and limits the failure risks (compared to a 2 box solution) Providing extended rapid failover scenarios may be key in many applications. Support very rapid convergence time even in case of non direct connectivity (BFD), for example with Microwave links Implementing MC-LAG, G.8032 and BGP-MH for IEC 61850 greatly enhances total availability of the applications.

3 Other design considerations for Teleprotection 16 Teleprotection over IP/MPLS - Validation - March2011

Packet flow asymmetry LSPs are unidirectional Topology changes or mis-configuration may result in different path being used in both directions. Only trouble shooting tools may discover that. 5620 CPAM allows this detection and raises alarms. Bypass actual path Green and Red LSP follow Active Path Bypass Logical Link different Path Tracking of LSP status and history Tracking of LSP Path for a given service 17 Teleprotection over IP/MPLS - Validation - March2011

Synchronization Information need to be delivered with time precision from an application standpoint. Many TDM applications require clock synchronization (as SDH) Some applications require Time of Day type of synchronization Packet networks can deliver clocking through different techniques : 18 Teleprotection over IP/MPLS - Validation - March2011

4 Summary

Conclusion Teleprotection is the most stringent application that can be transported over networks due to : Low delays requirements, very low jitter requirements Impact of a failure in the application Alcatel-Lucent demonstrates that its IP/MPLS can be used as the next generation network for such applications with : Native TDM integration (incl. interfaces, synchronization, ) Very high resistance to potential failures in network Several management tools to anticipate, control and trouble shoot network IP/MPLS is the foundation to prepare the Smart Grid data explosion, and manage the transition of existing mission critical applications 20 Teleprotection over IP/MPLS - Validation - March2011