The enabler of IT/OT convergence a converged network

Transcription

1 The enabler of IT/OT convergence a converged network Ivan Schonwald April 27, :00 3:00 PM Room 320A 1 <Change information classification in footer>

2 The two silos in critical industries OT ERP IT IT OT mandate Operate critical infrastructure with optimal efficiency, high reliability and safety SCADA LMR/PMR Remote asset monitoring PTC Teleprotection Asset mgmt Business communication Customer mgmt Mobile workforce mgmt IT mandate: Support business information and communication systems Communication network Communication network 2

3 Two silos, two disciplines Attribute OT applications IT applications Mandate Control and monitor critical infrastructures Corporate cyber assets, resources and customer management; business data and communications User base Specific technical groups (sometimes could be unionized) Diverse and companywide Time of use 24 x 7 x 365 Often business hours only Communications type Frequent, regular and real time; mostly machine-to-machine Periodically random; human-to-human/machine Uptime required Typically (mission-critical) Often or less (business-critical) 3

4 Two disciplines, two networks Attribute OT applications IT applications Technology base Interface Legacy and proprietary transitioning to open standard-based (often IP) Serial/analog interface transitioning to Ethernet and IP/Ethernet Open IP-based (exceptions in data center like storage/fibre Channel) moving to cloud-based IP/Ethernet; non-ip such as Fibre Channel in data center Response time From milliseconds From hundreds of millisecond to seconds Downtime tolerance Very low (when packet stops, traffic/oil/gas stops) Moderate (affects workforce productivity and customer satisfaction) Packet discard tolerance Zero in some applications (e.g. teleprotection) High (resilient upper layers) 4

5 Compelling events in critical infrastructure operations Compelling OT events Compelling IT IT events Communications migrating to packet Desire to adopt new technologies like cloud, analytics and IoT Security becoming top priority Hosting new OT applications in DC Applications starts to embrace cloud technology Convert OT data into business info/analytics Communication network Converged network Communication network 5

6 Multi-faceted challenges in this network of one OT challenge 3. Does the current mode of operation remain intact? 4. Does it support legacy interface with SONET performance? 5. How to attain mission-critical grade resiliency? 6. How to protect critical traffic (TP, SCADA, IEC61850)? 7. How to transport precise synchronization across networks? Converged network IT challenge 1. Is the network optimized for cloud? 2. How to extend connectivity to IoT? Operations center Compute and storage Datacenters 6

7 1. Cloud readiness scale up network bandwidth With optical transport Datacenter Public cloud Cross layer management IP/MPLS Private cloud Optical DWDM 7

8 1. Cloud-optimized Network automation 1. Upon new compute instantiation, automate new data center fabric connectivity BSS/orchestrator 2. Provide an additional 10G bandwidth to an enterprise site Datacenter SDN Nuage VSD Infrastructure SDN (NSP) Nuage VSC 3. NSP enables: Automated bandwidth and connectivity provisioning between DC and office users Hypervisor Apps Hypervisor Apps Programmable private WAN Enterprise site 8

9 2. Extend IP/MPLS-based services over LTE to reach IoT LTE extends MPLS connectivity IP/MPLS-based services LTE uplink LTE IP/MPLS network NOCs LTE field router Backhaul VPN Applications VPN Integrated Wi-Fi AP LTE uplink LTE field router 9

10 3. Retaining current OT best practices in a converged world One network Two slices, two administrators Multi-applications SCADA Renders users with: Operations LMR/PMR Ability to rapidly define and instantiate services Private WAN IT AMI Teleprotection Mobile dispatch VoIP Full administration privileges of their pre-assigned network resources Complete network isolation to ensure performance not affected by other slices Seamless convergence of OT applications into IT realm when necessary Internet 10

11 3. Multi-tenant for IT/OT convergence (network slicing) NOCs NOC gateway LAN Microwave IT slice Public Internet Copper/fiber Aggregation Core Data centers Depots OT slice Offices DC gateway SDN LAN OT traffic IT traffic Compute and storage One network, two slices, many applications 11

12 3. Each slice contains its own set of MPLS-based VPN for different applications Virtual Leased Line (VLL) Point-to-Point pseudowire (TDM, Ethernet, FR etc.) Teleprotection SCADA LTE SCAD A Ethernet service 7705 SAR ATM service IP/MPLS network Virtual Private LAN Service (VPLS) Layer 2 bridged multipoint Ethernet service TDM service Virtual Private Routed Network (VPRN) Layer 3 IP VPN R R R 7705 SAR VPRN service 1 VPRN service 2 R R R Data VPLS service B B 7705 SAR VPLS service Virtual router IP/MPLS network Security B B IP/MPLS Network B B Voice B Virtual bridge 12

13 4. Graceful legacy migration SCADA LMR SONET/ SDH mux RS-232/V.24 DS1 MultiDrop T1/E1 Optical fiber LAN, HMI, MMS, CCTV GOOSE/SV Ethernet IP IP/MPLS Dry contacts FXO/ FXS C37.94 G.703, X.21 RS-232/V.24 TPR Teleprotection E&M 13

14 4. Graceful migration building blocks Legacy migration 1 2 Terminating low-speed interface on legacy devices Transporting TDM traffic over IP/MPLS reliably C37.94, E&M, G.703, V.24 etc. Packetizing TDM data with circuit emulation services (CES) Meeting stringent quality of service (QoS) control Delay, jitter and delay symmetry (for differential protection) P2P and multipoint (Data bridging and PCM voice bridging) 3 Synchronization Requires primary reference clock transport over network for frequency synchronization 14

15 5. Attaining mission-critical grade resiliency Almost zero tolerance as every packet counts A single packet loss in a differential protection relay communications channel could leave the line unprotected for at least 2 cycles Packet loss could conceivably disrupt emergency communications 15

16 5. Attaining mission-critical grade resiliency With multi-layer protection Corporate HQ/NOC 7705 SAR IP/MPLS network Internet Omni PCX SCADA E&M LMR ERP Service layer MPLS layer IP layer Link layer Equipment layer PW redundancy MC-LAG/MC-APS Non-stop service MPLS FRR/Standby LSP/ECMP Non-stop signaling IP ECMP Non-stop routing VRRP 802.3ad Ethernet LAG Microwave 1+1/N+0 ML-PPP/ATM IMA SONET/SDH APS CSM/fan redundancy Dual DC power 16

17 5. Attaining mission-critical grade resiliency With multi-fault tolerance Linear/chain Ring Meshed Reality: Multi-ring Vulnerable to failure Rapid switching Vulnerable to double failure Rapid switching Highest resiliency Perhaps unrealistic as network scales up # of link = N(N-1)/2 17

18 5. Attaining mission-critical grade resiliency Don t take multi-fault tolerance for granted G.8032 in normal state Ring recovers from link failure RPL RPL unblocked One link failure occurs X Continuity check Remote-APS Link failure X 18

19 5. Attaining mission-critical grade resiliency Don t take multi-fault tolerance for granted Traffic blackholed 1. RPL becomes active 2. Traffic restored RPL RPL Ring 1 Ring 2 Ring 3 Command center Ring segmentation Second fault occurs First fault occurs RPL: Resilient Protection Link Blocking traffic to break loop in ring Connectivity broken despite physical reachability exists (see G.8032 Appendix X) 19

20 5. Attaining mission-critical grade resiliency IP/MPLS provides multi-fault tolerance RPL Ring 1 Ring 2 Ring 3 Command center Second fault occurs First fault occurs Connectivity can be restored as long as physical reachability exists, thanks to IP routing 20

21 5. Attaining mission-critical grade resiliency With the non-stop paradigm Hot standby protection with zero second interruption Non-stop services IP VPN, VPLS, Ethernet, TDM etc. Non-Stop Forwarding Non-Stop Routing/SIgnaling Non-Stop Services Non-stop control plane Active control BGP, OSPF, IS- IS MPLS, PIM IPv4, IPv6, Ethernet label switching Configuration and state hot synchronized Non-stop data plane HOT standby control SONET/SDH-like reliability Line card Control hardware failure: Standby takes over immediately with full state information Other routers don t even know it happened Zero impact on protocols, and services 21

22 5. Attaining mission-critical grade resiliency Not every paradigm affords you multi-fault tolerance Graceful restart (GR) OSPF (RFC3623) BGP4 (RFC4724) RSVP-TE GR control plane BGP, OSPF, IS- IS MPLS, PIM IPv4, IPv6, Ethernet label switching Non-stop forwarding Active control Configuration and state NOT hot synchronized Standby control Line card Control hardware failure: Graceful restart process kicks off Retain routes learned before GR Inform peering routers that I ll be back Routing updates from other nodes will cause GR to exit Router will flush everything and start from a clean slate 22

23 5. Attaining mission-critical grade resiliency How GR fails during multi-fault scenario 1 2 Command center 1. Control complex switch; GR kicks in; forwarding continue 2. A second fault occurs in another side of the ring 3. The GR node revert now to normal restart, flushing out the forwarding table 4. Connectivity to the radio site is now lost until the network converges Label Switched Path GR paradigm does not render multi-fault resiliency 23

24 5. Attaining mission-critical grade resiliency Non-stop paradigm still ace during multi-fault tolerance 1 2 Command center 1. Control complex switch; GR kicks in; forwarding continue 2. A second fault occurs in another side of the ring 3. The second fault does not impact non-stop routing 4. Connectivity to the radio site is never in jeopardy Label Switched Path Non-stop routing withstands second fault 24

25 6. Network Group Encryption (NGE) for OT and IT traffic SCADA RS-232/V.24 DS1 MultiDrop LMR T1/E1 SONET/ SDH mux Optical fiber Tunnel Service TDM payload Tunnel Service Ethernet payload Tunnel Service IP payload MPLS packets carrying multiprotocol payload LAN, HMI, MMS, CCTV GOOSE/SV Ethernet IP IP/MPLS MPLS Dry contacts FXO/ FXS C37.94 G.703, X.21 RS-232/V.24 E&M But how to secure multiprotocol?? encrypted services TPR Teleprotection 25

26 7. Precise synchronization transport Frequency and time-of-day synchronization 1. Integrated GPS 2. Line timing Synchronous Ethernet SDH/SONET, PDH (E1/T1) Timing over packet IEEE 1588 (master, boundary clock, slave) Differential Clock Recovery (DCR) Adaptive Clock Recovery (ACR) Timing over packet Timing over packet 2 PRC 26

27 IT/OT convergence founded on a converged network Convergence benefits are significant 1. Capitalize on the latest IT technology 2. Make better business decision 3. Attain higher efficiency and safety 4. Have access to a larger talent pool IT + OT >> 1O The converged network of one Converged network essential trappings 1. Optimized for cloud 2. Ready for IoT 3. Retaining OT best practice 4. Graceful TDM migration 5. Critical-grade resiliency 6. Network security 7. Precise synchronization 27

28