SCADA Network & Data Redundancy at Guelph Water

Transcription

1 SCADA Network & Data Redundancy at Guelph Water Ensuring Security of Supply for Operations and Compliance Graham Nasby Water SCADA & Security Specialist, Guelph Water Services Derek Wong Principal Consultant, Symbiotic Networks OWWA Conference Apr 29 to May 2, 2018 Niagara Falls, Ontario, Canada 1

2 About the Speakers Graham Nasby, P.Eng., PMP, CAP Water SCADA & Security Specialist City of Guelph Water Services 10 years in the consulting sector Joined Guelph Water Services in 2015 Co-chair of ISA112 SCADA Systems standards committee Member of the OWWA Automation Committee Contact: Derek Wong Principal Consultant, Symbiotic Networks Founder, Zabu 20+ years building and managing IT infrastructure Extensive experience supporting Communication Service Providers Active member in the information security community Outspoken advocate of open source software Contact: 2 2

3 Presentation Outline SCADA Systems Review Guelph SCADA Network Pre 2017 Datalogging Requirements: O.Reg. 170 Impact of SCADA Network & Data Logging Outages SCADA network design goals Guelph Water Solution: Fibre & Wireless with BGP SCADA Data logging design goals Guelph Water Solution: 3 layers of data logging Lessons Learned Best Practices Moving Forward 3 3

4 City of Guelph Water Services Guelph, Ontario, Canada 140,000 residents 21 groundwater wells 3 water towers 549 km of water mains 49,000 service connections 2,750 fire hydrants 35 unmanned facilities 46,000 m 3 /day [12 MGD] 4 4

5 Map of Guelph Water 35 Unmanned sites Central Ops Centre is staffed 8am 4pm Mon Fri On call operators on evenings and weekends minutes to drive across the city (with traffic) ARKELL SPRINGS 5 5

6 Water Supply Facilities Woods Station (Chlorine Building, UV Building, Booster Station) Arkell 1 & 7 Arkell 6 Arkell 8 Arkell 14 Arkell 15 Diversion Chamber Water Street Well Emma Well Park Well Clythe Station Paisley Station Membro Well Carter Well Downey Well Verney Tower Burkes Well Arkell Well Clair Tower Clair Booster Robertson Booster Helmar Well University Well Dean Well Calico Well Speedvale Tower Scout Camp Queensdale Well All sites are linked together with SCADA Network for remote control, automatic control, monitoring and logging. 6 6

7 What is SCADA? SCADA = Supervisory Control and Data Acquisition 7 7

8 Typical SCADA Architecture 8 8

9 Typical SCADA Network 9 9

10 Regulatory Requirements O.Reg. 170 Drinking Water Systems Free Chlorine Residuals Must be Logged Minimum Every 5 minutes Filter Plants: Turbidity must be logged every 15 minutes Low chlorine alarms must be communicated promptly Most IT Systems only reach 90 95% uptime (outages at night ok) Very challenging to meet this uptime for a SCADA network SCADA system must be online 24/7, 365 days/year Less than 5 min downtime/year = % uptime SCADA system has to be 10,000 X more reliable than an IT system 10 10

11 From O.Reg. 170, Section 6.5: 11 11

12 Nines Availability I.T. SCADA 12 12

13 Guelph SCADA Network: Pre

14 Impact of SCADA Network Outages Data Logging Meeting the 5 minute data logging requirement Process Visibility Ability of Operators to see the status of remote sites Speed of Response Ability to make adjustments remotely vs. Driving a Truck to Site Automatic Control of Pumps based on Water Tower Levels Requires PLC to PLC communications to work Can be very disruptive to Operations Operator has to be on site to physically write down residuals every 5 min Operator stationed at a Water Tower to call in pump starts/stops via radio 14 14

15 Goals of SCADA Network Upgrade Maintain SCADA Network Connectivity at All Times Be able to keep operating if a Fibre Cable is damaged/cut Be able to keep operating during fibre provider maintenance Reduce reliance on store/forward data loggers during outages Reduce the impact on Operations during a Fibre Outage Reduce the compliance risk associated with SCADA outages Implement continuous network monitoring for SCADA network Gather network statistics to monitor SCADA network performance 15 15

16 SCADA Network Technology Options Existing MPLS on Fibre (dedicated private network, via a Service Provider) Options to Supplement Back up Fibre Connections DSL Ethernet over cable Cellular Private Cellular Radios (unlicensed) Radios (licensed) How do we activate the backup network link when it is needed? 16 16

17 Redundant SCADA Network Design Hub Site MPLS Primary & Secondary Fibre Optic Connections Two different cable routings to the building( diversity ) HSRP = Hot Standby Router Protocol Two Routers that use HSRP to create a floating IP Address that is shared between routers UPS backed power, with Diesel Generator Backup Remote Sites MPLS Primary Link: Fibre Backup MPLS Link: Encrypted 3G/LTE Wireless on Private Cellular Network One Router Used running BGP: Border Gateway Protocol BGP is configured to switch to backup link in 45 seconds if Primary Fibre connection lost BGP will automatically switch back to fibre when fibre connectivity is restored UPS backed power 17 17

18 HSRP: Hot Standby Router Protocol Technology developed by Cisco Provides redundant Gateway using 2 identical fibre network links Used for Main Site because two links are both fibre & identical Does not do routing (could not be used for Remote Sites) Router One: xxx.xxx.0.1 Router Two: xxx.xxx.0.2 Virtual Network Address that shows up at: xxx.xxx.1.1 Takes about 2 minutes to switch from Primary to Secondary Does both fail over and fail back automatically 18 18

19 BGP: Border Gateway Protocol BGP is the Protocol that runs the Internet Used by Internet Service Providers to connect their networks together For example, the connection between Bell and Rogers uses BGP We use BGP to transparently and automatically use fibre & backup wireless links BGP switches from fibre to backup wireless within 45 seconds of a failure Border Gateway Protocol (BGP) is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet.The protocol is classified as a path vector protocol. The Border Gateway Protocol makes routing decisions based on paths, network policies, or rule sets configured by a network administrator and is involved in making core routing decisions. (source: Wikipedia) 19 19

20 SCADA Network - Before 20 20

21 Upgraded Network MAIN SITE 21 21

22 SCADA Network Monitoring Dashboard 22 22

23 SCADA Network Monitoring Graphs 23 OWWA 2018 Niagara Falls, Ontario, Canada 23

24 SCADA Network Uptime Results 2016 SCADA network uptime: 99.6% 2017 SCADA network uptime (post upgrade): % + increasing Pre upgrade when fibre provider had to do network maintenance it was very disruptive and required sites to be shutdown/manned Post upgrade when fibre provider has to do network maintenance, the network at that site automatically switches to backup wireless Fibre failures are no longer a major operational inconvenience Network monitoring allows us to proactively fix problems 24 24

25 Back up Data-Logging No technology or person is perfect So we have backup data logging to guard against failures/mistakes Primary Logging Facility PLC SCADA server reads data in real time Backup Data logging QuickPanel store/forward data logger Continuously logs into a buffer Pushes data up to SCADA server If no connectivity, stores timestamped data When network connectivity restored, forwards logged data up to server DNP3 based Data loggers New Technology 25 25

26 Existing Data Logging Infrastructure Historian SCADA Network Site PLC QuickPanel Data logger Flowmeters, Analyzers, Level, Pressure 26 26

27 New Data Logging Infrastructure Backup Historian Main Historian SCADA Network DNP3 Data logger (store & forward) Site PLC QuickPanel Data logger (store & forward) Critical Instrumentation: Flow, Analyzers Flowmeters, Analyzers, Level, Pressure 27 27

28 Wiring the DNP3 Logger Into the Loop 28 28

29 DNP3 Protocol for Data-Logging Automatic store/forward data logging is built into the protocol Timestamping data at the device/instrument Automatic Time synchronization Both Change on Value and Polling Interval data gathering Data Quality Flags Supports both Encryption and Authentication Automatic Error checking of data packets Prioritized Communications (Tags can be grouped in classes 0 4) Variety of networks supported: Ethernet, serial, ATM, cellular, etc. Supports wide range of analog, digital and structured datatypes 29 29

30 New Data Logging Infrastructure Backup Historian Main Historian SCADA Network DNP3 Data logger (store & forward) Site PLC QuickPanel Data logger (store & forward) Critical Instrumentation: Flow, Analyzers Flowmeters, Analyzers, Level, Pressure 30 30

31 Accessing your Logged Data 31 31

32 Take Aways A functioning SCADA system is critical for operations SCADA Network allows for remote data logging & control Data logging is needed to meet requirements of O.Reg. 170 SCADA outages can be very disruptive to operations and have compliance risks Uptime requirements for SCADA systems are very high A typical SCADA system has to be 10,000X more reliable than an IT system O.Reg. 170 requires us to log chlorine residual every 5 minutes (no exceptions!) 1. Consider using a redundant wire area SCADA network with automatic failover 2. Use redundant data logging to guard against equipment failures 3. Look at DNP3 protocol for doing back up store & forward data logging 4. Think about how your operations/compliance team will get access to the data 32 32

33 Any Questions? * Not a High Performance SCADA System Graham Nasby graham.nasby@guelphca Derek Wong dwong@symbiotic.ca 33 33