We Energies Case Study A Converged IP/MPLS Network Suitable for Critical Teleprotection Requirements: Solutions beyond PDH and SDH/SONET TDM

Transcription

1 We Energies Case Study A Converged IP/MPLS Network Suitable for Critical Teleprotection Requirements: Solutions beyond PDH and SDH/SONET TDM Jeff Polan, DNV GL Chris Staszak, Alcatel-Lucent 2013 Utilities Telecom Council Transforming Critical Infrastructure

2 Agenda We Energies Network Development Network Design & Development Challenges Facing We Energies Network Bandwidth Requirements Teleprotection Requirements Voice Requirements Principal Options Reviewed New Network Architecture Wide Area Connectivity Migration & Cyber Security Network Implementation & Lessons Learned Services MPLS Implementation Quality of Service Network Synchronization Migration of TDM and DATA Network Management Key Lessons Learned 2

3 We Energies Service Area 3

4 Issues & Challenges Facing We Energies Network Uses Multiple Technologies: Gigabit Ethernet and Fiber Channel over DWDM between Data Center (DC) and Business Center (BA) SONET OC-48 rings connecting 19 sites to Data Center and Business Center Gigabit Ethernet and TDM over SONET Gigabit Ethernet over dark fiber connecting 15 sites to Data Center and Business Center TDM microwave (nxds1, mxds3) links connecting about 100 sites to DC/BA and other network locations Multiple Service Types: Voice, ip- and Ethernet- Data, SCADA, Teleprotection Looming Capacity constraints Extension to as many as 300 additional Substations New high-bandwidth services including video surveillance Equipment Obsolescence: DWDM and SONET gear years old, Manufacturer Discontinued (MDC) and no longer supported Immediate need ethernet data traffic on SONET network needs to be offloaded due MDC interface Cost and looming obsolescence of 3 rd -party T1/T3 services 4

5 Bandwidth Requirements Network traffic is mostly data (ip and Ethernet) traffic Requirement to connect substations to the network - Primary bandwidth driver at substations currently: Real-time Video surveillance: Bandwidth requirements are a function of video quality: TV quality video (30 fps) ~ 2-3 Mbps per video stream (MPEG4/H.263) Good quality video (5 fps) ~ kbps per video stream Communications requirement is to support 6-7 real-time video surveillance streams at each substation Derived Bandwidth Requirements: Mbps per substation for TV quality video 2-3 Mbps per substation for video at 5 fps Network should be designed to accommodate any future requirements for next years 5

6 Teleprotection Requirements Teleprotection traffic poses particular challenges Latency Requirements: 5 ms one way latency, 1 ms asymmetric latency difference for CDP (Vendor S); more typically 3ms one-way latency ( ) and 0.5ms asymmetric latency max for CDP Interfaces: T1/fractional T1, 4-wire E&M; more typically RS- 232/422, X.21, E1, Ethernet (GOOSE), ip, Availability: >99.999% at less than 1 E-06 BER desired Low-speed interfaces (i.e. 9600bps) generally pose the most serious challenges because of packetization delays; SEL mirrored bits poses a particular challenge 6

7 Voice System Requirements Current system reaching end of life We Energies will have a project in 2014 to upgrade of its Voice System DNV GL recommended moving to a more VoIP based infrastructure As part of the project to upgrade, We Energies considered SIP trunking SIP to integrate Microsoft Communicator/Lync SIP to support video conferencing to desktops E911 services for IP phones Develop a migration plan to move voice traffic to the IP/MPLS network 7

8 Principal Options Reviewed Replacement OC-48/OC192 SONET Carrier Ethernet Transport (CoE) PBB-TE Native IP (TDMoIP) IP/MPLS Network MPLS-TP OTN (G.709) 8

9 DNV GL s Experience: Entirely feasible to support teleprotection traffic Well designed and dimensioned MPLS network with appropriate high speed connections within the MPLS network (e.g., 1 Gbps over fiber) Controlled number of hops Proper edge-router interfaces, Proper QoS provisioning including strict-priority scheduling, and Proper traffic engineering 1588v2 time synchronization performance can provide further protection, against latency asymmetry for current or future CDP teleprotection devices and/or PMUs without the need to deploy separate GPSdisciplined PPS/ToD clocks at each of these devices 9

10 Example: Actual Test Results Teleprotection performance under 100% congestion conditions Channelized DS1 (TDM) carrying critical CDP and TPR traffic, even under extreme stress conditions: over 6 nodes/5 hops, with one hop fully 100% congested with BE in-policy traffic, channelized DS1 traffic (at EF in-policy) was preserved without error using the recommended QoS, with maximum 1-way latencies never exceeding 3.8ms (with typical per-node forwarding delays under 20us), and maximum indicated latency asymmetry never exceeding 200us in all testing observed over 5 days (test conditions: 8 frames or 1ms packetization [192 bytes at DS1], and 5ms jitter buffer); these numbers substantially exceed the WE Energies requirements of <5ms and <0.5ms respectively, and as this 6 nodes/5 GiGE hops test is likely to exceed any practical teleprotection connectivity demands even after FRR/End-to-End protection switching, this test is an extreme test of latency/latency asymmetry performance of teleprotection services. 10

11 Recommendations for Wide Area Network Transition to a single converged network using IP/MPLS Single network to design, operate, manage and maintain Capable of carrying both data traffic and legacy TDM traffic (Voice, SCADA, teleprotection) Low latency, path deterministic, supports traffic engineering, QoS, Fast Re-Route in case of link failures (<50 ms typical), primary/secondary path protection and/or dual pseudowires Segment network between different applications (l2/l3 VPNs) for different QoS or security requirements Mature Technology with a large vendor ecosystem Hierarchical design of the network with well organized core, aggregation and access layers 11

12 Wide Area Network Design 12

13 Migration How to get from the current system to the planned network? Installation and deployment, provisioning, test and verification MUST have no impact/minimal impact on current operations NO free fiber available on existing SONET rings, uncertain availability on most dark fiber segments Chosen solution: Overlay new MPLS network over existing SONET and GigE networks using banded 1550nm CWDM Essentially zero impact on current network, highly cost-effective 5xGigE capacity in MPLS network, Both MPLS and SONET networks can co-exist indefinitely Allows migration to be performed only after new network is fully deployed, provisioned, tested, commissioned Migration can be performed incrementally, service-by-service, on asneeded basis Extend IP/MPLS network to new sites 13

14 Cybersecurity NERC CIP requires additional cyber-security protections and controls on cyberassets associated with identified critical assets (therefore critical cyber assets or CCAs) with qualifying connectivity - i.e., using routable protocols MPLS offers some benefits as fits within non-routable protocol exception (forwarding decisions based only on MPLS labels) Recommendations include: Enable routing/switching control plane security Enable data encryption on transit (data plane security) Use network segmentation based on VPNs Enable Network Admission Control. Enable embedded security for virtualized data centers. A comprehensive and detailed cyber-security framework, architecture, controls, and implementation analysis will need to be performed for the chosen communications solution and its deployment 14

15 Importance of Independent Lab Tests to confirm ability to support critical services Comprehensive testing performed on a scaled-down facsimile of actual network - approximately 20 nodes (about 50%) using actual core and edge routers, network architecture, serial/tdm interfaces, L2/L3 VPNs (data segregation), provisioning and protection schemes, and NMS Verify effective QoS management is maintained for critical legacy TDM services (teleprotection) under a wide range of simulated impairments (packet jitter, packet drop, clearing jitter-buffer under-run/over-run etc.), failover actions (FRR and primary/secondary path protection), and 100% traffic congestion conditions On critical TDM services, must verify packetization and jitter buffer provisioning appropriate to required latency Must verify IEEE 1588V2 network time synchronization accuracy is achieved, and exhibits proper transient settling behavior on primary-secondary clock failover to same synchronization accuracy 15

16 Summary & Conclusion: One converged network to design, operate, and maintain Properly designed and dimensioned, supports all legacy voice, SCADA, and Teleprotection TDM services, with failover equal or better than existing SONET (<50ms), sub 5 ms latency, sub 0.5ms latency asymmetry, and asymmetry protection using IEEE 1588v2 Extended IP/MPLS network to new sites 40 sites today, as many as 350 sites in the future Extend capacity throughout to 5GE from OC-48 and GE 16

17 Network Implementation Detail Aggregation sites Optical Management 31 sites each with an aggregation router ( 7705 SAR-18) Sites are interconnected w/ five pair of fibers with each pair fiber carrying a single GigE CWDM wavelengths will be multiplexed together by a Finisar optical add/drop passive CWDM multiplexer Two units used between the two Data Centers Additionally, 3 paths have been identified where the loss exceeds the link budget of the available SFPs so We Energies is using DWDM 1830 PSS-16s instead of Finisar 5620 Service Aware Manager (SAM) 5650 Control Plane Assurance Manager (CPAM) Service Portal Express 17

18 L km optics 120km optics Core Core We Energies IP/MPLS Network Work started in March 2013 Deployment starting from edge and working to Core Total actual duration on completion likely to be slightly over 1 year 18

19 Core Detail 5 x 1 GigE 7705 SAR-8 Existing OC SAM and 5650 CPAM Redundant configuration, one system located at 2 core sites Connected to 7750 SR SR-7 Finisar FWSF- M/D- 1550/ CWDM Service Portal Express for Utilities (SPE) - One Web Server Portal located at a core Site, IP Connectivity required to 5620 SAM and client browsers. CWDM and GigE Signals Copper GigE 1471nm 1491nm 1511nm 1591nm 1611nm Single Fiber direction shown. Other Connections not shown. Unamplified Composite CWDM Signal over Dark Fiber Notes: 1531nm, 1551nm, and 1571nm not available for use. Used by existing OC-48. Local drops not shown SAM and SPE Clients Located anywhere with IP connectivity to 5620 SAM and SPE Servers

20 We Energies Services Provided Service MPLS Svc Type Comments Voice Over IP VPRN-Voice SIP signaling and Payload Corporate Data Service VPRN-Data Regular corporate IP services to include Internet Access. Traditional TDM service VLL (C pipe) Network Synchronization is being implemented by Sync-E Management Service VPRN-Data In-band IP traffic STILL TO BE DEPLOYED Teleprotection VLL (C pipe) Will require RSVP TE Video Traffic VPRN-Data Corporate video traffic which will be deployed in future. 20

21 MPLS Implementation Overview OSPF is used as Interior Gateway Protocol (IGP) for the MPLS network Label Distribution Protocol (LDP) is used as the protocol to build the LSPs. Multiprotocol-BGP will be used as VPRN service label distribution protocol. LDP provides a standard methodology for label, distribution by assigning labels to routes that have been chosen by the IGP routing protocols. The resulting labeled paths, called label switch paths (LSPs), forward traffic across an MPLS backbone. In regard to VLL services (VPLS/Cpipe/Ipipe etc.), Target LDP (T-LDP) is used for inner label distribution. A key objective was to implement the network with minimal disruption to edge devices using EGRIP. Bidirectional Forwarding Detection (BFD) is used for failure detection. RSVP TE to implemented with Teleprotection Multi Area OSPF is being used with Area 0 and Areas 1&2 to allow network growth to eventual size 21

22 Quality Of Service Forwarding Class (FC) FC FC name Class type Notes NC Network control Real time For network control traffic. H1 High-1 Real time For delay/jitter sensitive data. EF Expedited Real time For delay/jitter sensitive data. H2 High-2 Real time For delay/jitter sensitive data. L1 Low-1 NRT Assured For assured traffic AF Assured NRT Assured For assured traffic L2 Low-2 NRT Best effort For BE traffic. BE Best Effort NRT Best effort For BE traffic. 22

23 Quality Of Service Deployment (FC To Queue) Service name Priority FC Queue SAP Ingress Policy SAP Egress Policy PIR (mbps) CIR (%LR) Voice Payload Platinum EF SIP Gold H Video Gold H Management Silver L Best Effort Bronze BE

24 Network Synchronization: Sync-E & 1588V2 Clock synchronization is needed in order to ensure proper operation of the TDM services (C pipe). Both 7750 SR7 and 7705 SAR8/18 can be configured with an External clock source as well as up to 2 other timing references. Time synchronization is needed for TPR, PMU, D/A and SCADA Four sites in We Energies' network provide GPS disciplined clock sources Nodes running SyncE (Synchronous Ethernet) synchronize their clocks to the Ethernet bit stream. Synchronous Ethernet replaces a portion of the standard Ethernet preamble with a 4-bit synchronization pattern and a 4- bit coding violation pattern. All nodes in this network will be configured with 2 peers under SyncE configuration. SyncE will automatically decide which peer has a better quality of clock to use. Should this peer fail to provide higher quality clock the node will switch to secondary peer. These quality measurements happen automatically. 24

25 Network Synchronization: Sync-E Synchronous-Ethernet (Sync-E) is considered a line-timed source for Service Synchronization Unit (SSU) it operates at the physical layer and is immune to PDV or packet loss at higher layers. The sender derives a reference frequency from the node s SSU to be used by the Ethernet transmission clock. The receiver will then synchronize its SSU to the received frequency on the Ethernet port. When using Sync-E all nodes/links between the source and destination nodes MUST support and have Sync-E enabled. 25

26 Migration Of Services TDM Based Service (ONLY LEFT-TO-RIGHT DIRECTION SHOWN) REMOTE SITE TDM Packets moving in this direction CONTROL CENTER N x 64 kb/s or T1/E1 Access circuit IWF Data Sig Performance IP/MPLS Network Jitter buffer GigE GigE Data Sig IWF N x 64 kb/s or T1/E1 Access circuit PACKETIZATION As TDM traffic from the Access Circuit (AC) is received, it is packetized and transmitted into the IP/MPLS network 26 NETWORK Fixed delay Packet transfer delay based on link speeds and distances from end to end Variable delay The number and type of switches Queuing point in the switches Network QoS is key to ensure effective service delivery PLAYOUT TDM PW packets are received from the IP/MPLS network and stored into its associated configurable jitter buffer Playout of the TDM data back into the AC when it s at least 50% full

27 Pre Migration Network Corporate Data Site X Access Distribution Access Layer 2 or Partial Layer 3 Site A Distribution Access Access Access Switches Mgmt IT Data Voice Video 27

28 Migration into MPLS IP/MPLS PE VPRN Access Access Access Switches Mgmt IT Data Voice Video 28

29 Migration Of Services Data 29

30 Network Management Three Elements 5620 Service Aware Management (SAM): provides enhanced service provisioning and assurance capabilities for operation, administration, maintenance, and provisioning functions Control Plane Assurance Manager (CPAM) is a multi-vendor route and path analytics tool to provide realtime visualization, surveillance and troubleshooting Service Portal Express: Provide utility specific network management capabilities 5620 SAM & 5650 CPAM 7701 CPAA Alcatel-Lucent

31 Key Lessons Learned Early involvement of all parties is important Value of an effective Network Management System Multi Area OSPF Implementation Migration of customer sites to OSPF Initial planning efforts were focused on the transport group IT group joined when full potential of the new architecture was realized Additional planning time (3 months) was required Early number of users was underestimated The GUI allowed more than CLI only users to have a view of the network Managers could view and understand network events Multi area OSPF was implemented to accommodate planned network growth This also facilitated the integration of the Access switch sites into the larger network Original plan was to keep these sites on EGRIP, but conversion to OSPF proved to be less work than expected Migration of sites to OSPF allowed network simplification to occur and to also allow removal of some nodes

32 Key Lessons Learned Reduce numbers of devices in the network Quality of Service Capabilities Test Lab Capabilities of the new aggregation equipment allowed the elimination of some devices at Access switch sites In addition to simplifying the network, this action also improved latencies in the network Enhanced QoS capabilities provided more opportunities Previously only voice received high priority, now more options are available. This should be planned for during the initial design meetings Lab testing prior to deployment was found to be very effective Testing validated move from EGRIP to OSPF was not as difficult as originally expected Staging of equipment for POC in the vaults was a major benefit Resident Engineer Provided by the vendor; this individual became a major help Involvement even earlier in the process would have been good Resident Engineer s expertise on the products and vendors organization quickly made him a valueable resource

33 Any further questions or comments? Thank you for your time and attention Mark Burke (703) Jeffery Polan (267) Chris Staszak (978) SAFER, SMARTER, GREENER 33

34 General Lessons Learned (Beyond KCP&L and We Energies) Packet based networks can support the requirements of Utilities when properly engineered TDM to Packet migration can be done successfully, with little or no service impacts, when the new network is entirely separate and overlaid the old, and service migration is done in stages Potentially, the number of devices (boxes) in a network can be reduced Bandwidth can be more effectively utilized in a MPLS based network Reliability can be maintained through the use of MPLS Traffic engineering and enhanced QoS opens up new potential Legacy interfaces can be supported, without requirements for whole scale changes at customer sites Movement to MPLS can reduce CAPEX & OPEX without compromising reliability