SCADA System Failure Impact Study at Lorne Park and Lakeview WTPs

Transcription

1 SCADA System Failure Impact Study at Lorne Park and Lakeview WTPs OWWA Conference 2014 May 5 th London, ON Shawn Xiong - Cole Engineering Jeff Hennings, Jason Little, Claudio Cuffolo - Region of Peel Dale Barker, Eric Zhang - OCWA

2 Presentation Outline Project Background Project Objectives Analysis Approach Recommendations

3 Background Lorne Park WTP and Lakeview WTP Supplies water to South Peel and York Region Owned by Peel Region and operated by OCWA

4 Background Employs advanced treatment process: membranes, UV, Ozone, BACC, GACC, and conventional filters Advanced SCADA system for automation

5 Background Failures have been experienced at both plants Adverse impact on the plants ability to maintain water production

6 Project Objectives Maintain three objectives during a SCADA failure event: Water quality Water production Compliance reporting

7 Project Objectives Establish critical process areas at each plant that are essential to the plant s ability to meet water quality and quantity requirements. Identify critical instruments for MOE compliance reporting. Review each critical process area to identify operational and data reporting problems that could occur if SCADA was disrupted.

8 Project Objectives Recommend operation procedures to follow during a SCADA failure for each of the problems identified above in order to minimize risks to water quality or quantity, or compliance reporting. Conduct network or SCADA failure modes and effect analysis (FMEA). Document the evaluation and recommendation in an organized format.

9 Analysis Approach A systematic analysis approach is required to analyze such a complex system with a multitude of subjects CAUSES FAILURE IMPACT DESIGN IMPROVEMENT EMERGENCY PREPAREDNESS PLAN

10 Analysis Approach AWWA G440 - Emergency Preparedness Practices AWWA M19 - Emergency Planning for Water Utilities AWWA J100 - Risk and Resilience Management of Water and Wastewater Systems

11 Analysis Approach Risk Management Framework Identify Risks Analyze Risks Evaluate Risks Treat Risks

12 Analysis Approach Risk Analysis Processes and Tools RAMCAP - Risk Analysis and Management for Critical Asset Protection HAZOP - HAZard and OPerability study FMEA - Failure Modes and Effect Analysis

13 Analysis Approach Combination of RAMCAP and FMEA FMEA *From AWWA, J-100

14 Analysis Approach Project Methodology Step 1 Asset Characterisation (RAMCAP Step 1) Step 2 Threat Characterization (RAMCAP Step 2) Step 3 FMEA Analysis (Consequence, Risk Analysis and Management, RAMCAP Step 3-7)

15 Asset Characterization Lorne Park MUG MEMBRANE AOUV GAC LOW LIFT RESERVOIR HIGH LIFT CONVENTIONAL FILTERS FILTER 9-12 UV

16 Critical Process Areas - Lorne Park Low Lift Pumping Plant 1 - MUG - Membranes Plant 1 MUG - AOUV System Plant 1 MUG - GAC Contactors Plant 2 - Filters No Plant 2 - Filters No UV High Lift Pumping Chemical Systems Coagulant, Bisulphite for WW supernatant dechlorination, Hypochlorite for disinfection

17 Compliance Intruments Lorne Park MEMBRANE/UV/GACC CONVENTIONAL FILTERS 9-12 & RESIDUAL MANAGEMENT CHEMICAL SYSTEM HIGH/LOW LIFT PUMP STATION CL2/pH FL CL COAG BW GAC BW SUPERNATANT CONTROL ROOM RESERVOIR LIT GAC FLOW LIT TIT TIT 87511

18 PLC & CCP Lorne Park

19 Threat Characterization SCADA System Failure Modes SCADA NETWORK FAILURE INSTRUMENT FAILURE SECURITY BREACH BUS NETWORK FAILURE RING SWITCH FAILURE CABLE FAILURE - FIBRE RING PANEL SWITCH FAILURE CABLE FAILURE - CAT6 CABLE CONTROLNET FAILURE MODBUS FAILURE DEVICENET FAILURE BOTH SERVERS FAIL VIRTUAL MACHINE CRASH PRIMARY SERVER FAILURE SECONDARY SERVER FAILURE CPU FAILURE I/O MODULE FAILURE RACK FAILURE POWER SUPPLY FAILURE SCADA SYSTEM FAILURE SCADA SERVER FAILURE POWER FAILURE PLC FAILURE

20 Threat Characterization Failure Modes SCADA Network Failure PLC Hardware Failure - Major ControlNet Failure Instrument Failure Failure Causes Detection Impacts Response /Remediation Ring Switch Software Failure (RSTP) - Unidirectional link - Duplex Mismatch - Software bug - etc Power Supply Failure - Remote Rack SCADA freezing All lights off at rack - Shutdown of the plant or process areas - Loss of compliance reporting - Loss of control of the Plant Loss of control of some equipment Remote Rack Failure All lights off at rack Loss of control of some equipment ControlNet Module Failure ControlNet Media Failure Compliance Instrument Failure - OK Status LED steady red, display "FAIL" - Module indicates "NET ERR" - Network Channel Status A/B red or flashing red - Fault alarm - Out of range alarm - Loss of Echo Loss of control of the affected process area Loss of control of the affected process area Shutdown network and reboot switches one by one Replace power supply Replace rack Test, reboot, or replace module Reconnect media - Loss of compliance data - Manual sampling and recording Criticality High Medium Medium High High High

21 FMEA Analysis Select Process/System for Evaluation Select Team Diagram system blocks Identify failure modes Identify failure causes Evaluate failure severity Evaluate failure probability Evaluate failure detection & rectification Calculate risk priority number (RPN) Recommend actions to reduce RPN

22 FMEA Analysis Failure impacts Water production Water quality Compliance reporting Operational resources

23 FMEA Analysis

24 FMEA Analysis

25 FMEA Analysis

26 FMEA Analysis Scoring of Failure Severity Score Severity Rationale 1 not perceptible No Effect 2 small 3 medium 4 significant This failure is not significant. Treatment process can continue with no interruption. This failure is significant enough to affect the treatment process system operation, compliance, water production, and water quality. Additional resources for plant operation are required. Inoperability of a significant process in the treatment plant. Significant requirement of additional resources to operate the process in local-manual mode. 5 very significant Total inoperability of the treatment plant

27 FMEA Analysis Scoring of Failure Probability Score Probability Rationale 1 improbable MTBF > 25 years 2 very small MTBF [ 10-25] years 3 Small MTBF [ 3-10] years 4 Medium MTBF [ 1-3] years 5 High MTBF < 1 years, or has occurred previously

28 FMEA Analysis Scoring of Failure Detection and Rectification Score Detection /Rectification 1 very probable 2 high 3 medium 4 small 5 very small Rationale Very probable and immediate detection and rectification of failure (<1 hr) Chance of discovery high, time to detect and rectify failure is short. (<4 hr) Time to detect and rectify the failure within a day. (<24 hr) Failure may be undetected for extended period of time or it may take days to troubleshoot and rectify the failure. (< 1 week) Very small chance of discovery or it will take a long time to rectify the failure (> 1 week)

29 Recommendations Design Improvement I/O distribution Redundant network cabling Environmental control for network equipment Local data recording Local operation of critical process equipment Signal communication for process control

30 Recommendations Operations Improvement Treatment process response to SCADA system failure Instrumentation response to SCADA system failure SCADA system maintenance

31 Recommendations Low Lift Pumps for Conventional Filters 9-12 Process Criticality: High Rationale: Supplies raw water to Plant 2. SCADA Communication Dependency: Local-Manual Operation: SCADA Failure Response: Additional Operational Resources Requirements: None. Low Lift Pump Operation Operate pump in Local-Manual mode from pump Control Gallery or Local Control Station. Start the required number of pumps based on filters feed flow requirement. Adjust flow control valve FCV16301/16302 to meet the flow setpoint and also to maintain a target level in Settled Water Conduit. (1)If SCADA network fails a.low Lift Pumps for Plant 2 can stay under SCADA control as there is no communication requirement between PLC-116 and PLC-398 to allow the operation of Low Lift Pumps (2)If PLC-116 fails a.check the feed flow demand for Plant 2 and start the appropriate number of low lift pumps. b.manually adjust the position of flow control valves FCV16301/16302 to achieve the target feed flow demand and also to maintain a target level in Settled Water Conduit. One operator is required to monitor and control the operation of the low lift pumps to maintain water supply to Plant 2 when operating the pumps in Local Manual mode.

32 Recommendations Emergency Preparedness Plan (EPP) Objectives: Emergency Response Team Communications Chart Plan Activation Component Specific Plans (process & instrumentation) Emergency Response and Recovery Training, Testing & Drilling

33 Recommendations Abnormal Event SCADA Problem? Actual/Probable Adverse Impact to Water Supply? NO OCWA designate contact Peel On-call SCADA Tech Peel SCADA follow with OCWA on action taken/ required YES OCWA ORO/Designate Contacts Peel On-call Compliance Rep. Activate EPP Communicates with internal/ external contacts per Communication Chart Engage the Emergency Response Team Determine Failure Impact and Affected Process Areas OCWA Follows Appropriate Treatment Process and Instrumentation Plans Calls in additional operations resources if required Treatment Process Recovery Event Documentation OCWA/Peel SCADA Troubleshoots the SCADA System Isolate event causes and apply remedial actions

34 Acknowledgement Regional Municipality of Peel Ontario Clean Water Agency AECOM CH2M Hill