WEI Conference SDG&E TCRI Project April 25, 2018 Mark Fowler, CISSP

Transcription

1 WEI Conference SDG&E TCRI Project April 25, 2018 Mark Fowler, CISSP

2 San Diego Gas & Electric Subsidiary of Sempra Energy 4100 Full-Time Employees Regulated public utility Providing clean, safe & reliable energy service to 3.6 million consumers 1.4 million electric meters (and growing) 17,125 miles distribution lines (10,606 underground) 1,984 miles Transmission Lines (154 underground) 873,000 natural gas meters 237 miles transmission pipelines 8,000 miles distribution pipelines 4100 square mile service territory in San Diego & extending east into Imperial and north into southern Orange Counties (25 cities) Renewable Portfolio Standard (Electrical): 2016 Total: 43% Current Estimated Total: ~45%

3 What we ll cover Discussion of Transmission Communications Reliability Improvement Project History What was the problem we noticed? History - What were the initial goals? Response to outages Reliability Evolution What are the goals today? SCADA Teleprotection Reduction of Serial Comms through Common Carriers (T1, etc.) Reliability Phase 1 Actualization Phase 1 Progress Phase 2 Goals Importance of Encryption in IP Network Traffic Meeting NERC-CIP and NERC-CIP intentions Overview of New Technology, CES-21 Initiatives

4 History What Problems did we notice? How to respond to numerous outages? Effect of using Common Carrier Serial Data Circuits (T1, etc.) Cost Reliability Time to Repair Use of old physical protocols & old physical infrastructure Reduction in reliance on carrier physical infrastructure Acknowledgement that the current methods were outdated Determination to do better

5 History of TCRI Initial Goals Build A Better WAN Infrastructure Faster, Redundant, Enhanced Reliability Reduce Round-Trip Latency in new infrastructure Build A Better MPLS Serial Packetization Network Need better all around performance Reduced Round-Trip Latency Real-Time Management and Event Visibility Identify Successful Projects in the Solution Space

6 Phase 1 Goals Evolution (2017) Create a basic connected network Identify Critical communication convergence points Fill Critical points with new high performance router network nodes Includes Data Center, Control Centers, Network Connectivity Hubs Maximize use of Fiber Plant Minimize hop counts router to destination Upon completion of Phase 1, we have a backbone of connectivity

7 TCRI Phase 1 Actualization Introduce an OT-Only router network using High Performance, High Availability MPLS Routers and MPLS Switches; MPLS using IS-IS Route Discovery Complements the existing Corporate MPLS router network (mostly Juniper) Allows mixed router environment where that is efficient Allows Low-Latency, Limited Hop Connections where useful Makes maximum use of the extensive fiber network in place Gives the ability to encrypt everywhere using Symmetric Key Group Encryption (Cyber Security High Five ) Designs using Templated Services, data services that are from an offthe-shelf repository of service types (integration vendor creates all templates) Expected reduction in troubleshooting Realtime network management gives a real-time view of network health Use of Templated Services gives troubleshooters a better understanding for repair

8 TCRI Phase 1 Progress Phase 1 has Transmission Substation network sites, each with minimum two routers (environmentally hardened, fanless, DC Powered) plus Downtown SCADA Phase 1 has (minimally) two separate physical and logical data paths, always ready to use Physical Redundancy (mostly fiber) and Data Path Redundancy (partially microwave) Two fiber paths out of each router for primary path, two for secondary path Phase 1 provides a Minimal Redundant Path Network Includes Transit Only sites (mountaintop radios, comm shacks to link fiber) Includes Transmission Facilities with greatest impact (High Voltage Ties) Improves Downtown SCADA network immediately

9 Phase 2 Goals (beginning 2018) Complete Transmission Substation Network 47 transmission substations (about 20% are telecom only) Some Microgrid, Energy Storage Sites Additional High-Visibility or Critical Infrastructure Sites Teleprotection Falling Conductor Study Synchrophasor sites High Capacity or Critical Distribution Substations Hospital, First Responder Government, Military Total Buildout (~3 years) 183 sites (estimated)

10 Evolution What are the future goals today? Become Future Proof Pick technologies that scale well into the future SCADA move Serial Scada to Packetized Serial Data Using Raw Sockets to packetize serial data streams Non-Realtime performance Teleprotection Low-Latency, High Availability, Redundancy, Reliability Fast Path Switchover Predictable Performance in Path Switchover Reduction in Cost, non-reliability in Common Carrier Serial Circuits T1 costs have vary dramatically, some more than 5 times Reduction in response time to repair (redundancy moves downtime, reduced to subseconds) Improved Reliability with MPLS fast-path discovery Ability to determine fastest paths through network using network mgmt. tools Auto-Reroute Creative Commons CC0 1.0 Universal Public Domain Dedication.

11 Conclusion Encryption More secure data transmission requirements are just around the corner, we believe NERC CIP of the Future Ready Future Readiness Moving more data transmission applications into the TCRI network Support for additional reduction in Serial Circuits Support for Emerging Technologies Low Cost LTE Routers Small Form Factor, DC, Fanless, 1613 Compliant Compatibility to Private LTE Network in our near future 6 times cost factor improvement where suitable Falling Conductor (<= 1.37 seconds de-energization) Requires dozens of open-space mounted network connections on poles, small structures

12 Other Considerations High Value, High Resolution Integrated Test Facility The Integrated Test Facility was conceived in 2010 in conjunction with Smart Grid development efforts. It opened in 2014 and currently consists of seven labs. Foundational/Communications Lab This lab tests Smart Grid communications systems using exact replicas of what can be found out in the field. Power Systems Lab This state-of-the-art lab allows for real-time simulation of power systems using 10 high-powered simulators. Distributed Energy Resources Lab Testing of equipment and devices like inverters, energy storage systems and their impact on the grid happens at this lab. Smart Garage Lab Researchers here are evaluating vehicle charging station metering accuracy and photovoltaic system inverters. Home Area Network Lab Work includes testing of smart meters and appliances, smart home thermostats, and technologies being developed by the Demand Response Team. Information Security Lab Cybersecurity is the primary focus of the work being done at this lab, but research is also underway here on things like GPS Jamming Recognition and Recovery. Situational Awareness Conference Room This presentation room provides a central location for the visualization of data from the six outlying labs at the Integrated Test Facility.

13 California Energy Systems for the 21 st Century (CES-21) Initiatives GPS Spoofing Study SDG&E partnered Southwest Research Institute (SWRI) Demonstrated current utility GPS hardware is vulnerable Failure modes vendor & model specific DHS indicates all GPS vendors/models are vulnerable Demonstrated latest vendor spoof resilient hardware is still vulnerable Demonstrated concept which can detect, alert, and mitigate against spoofing attacks

14 EPRI GPS Security Supplemental Project Growing industry problem Continue research to determine industry impacts Interest from other EPRI members EPRI is great platform for industry collaboration Great opportunity to bring more awareness and conduct further research

15 Quantum Key Distribution Qubitekk DataLoc Quantum Entangled Photon Key Distribution creates a continuous stream of one time pad DataLoc capable of 28,800 secret key bps over 20km of optical fiber Compatible with standard telecom optical fiber True Quantum Entanglement Developing new network protocols for configuring QKD network devices To date has been used to authenticate secure sessions for ICS application at SDG&E ITF