EMS Managing Availability 24/7

Transcription

1 EMS Managing Availability 24/7 Mathew Malm Real Time Systems - SCADA Team Lead Sept 19,

2 BCH EMS Background Info Real Time Systems Group and EMS EMS Team ( Real-Time Systems) consists of about 25 engineers Divided into teams (SCADA/Power System Apps/Infrastructure/Generation/DTS) Provide Afternoon and Standby shift coverage (24/7) Target 99.95% availability for unplanned outages and never any outage greater than 30 min. Primary/Backup control centers located 400km apart, both running redundant Alstom EMS (i.e. quad redundant system). 2

3 BC Hydro EMS What has changed? Early Days (10 years ago) Problems were usually application related (ex. SE/SCADA). Security Requirements were not as stringent. Systemic problems were normally caused by human error. Fewer external interfaces and distributed manned control centers. EMS support was handled by a single standby support person on call 24/7 with best effort to contact SMEs as required. Today Moved to a more integrated system with many interfaces. Security requirements have significantly increased resulting in more possible points of failure. Refresh cycle increased (i.e. more patching/updates/etc.) Full regression testing not always possible. Currently have 30 Firewalls, 70 switches. Use VPN, Network Address Translation, static routes (for ease of troubleshooting). Processes in place (ex. Change Management) to minimize potential failures. 3

4 EMS Unplanned Outage 1 Complete Loss of EMS March 2011 Outage Details Occurred during regular business hours while updating the Firewall OS. Had successfully updated 24 out of 26 Firewalls before failure. Cause was a Cisco bug in the firewall associated with IPS config where devices configured that did not exist should have kept the path CLOSED allowing traffic, but the bug caused the path OPEN blocking traffic. Resulted in Loss of Closed Loop Control and visibility of the EMS for 55 min. Immediate solution was to rollback the OS to previous version. Action Taken as Result of Outage Configured a Test Environment (6 FW/4SW) for testing. Event Occurred during regular business hours and was believed to be isolated. No modifications were made to the Standby EMS support structure. Cisco resolved the bug that caused the path to remain OPEN in subsequent versions. 4

5 EMS Unplanned Outage 2 Partial Loss of EMS Nov 2011 Outage Details (Cause and Impact) Work was being performed to separate production/non-production domains. RTS team member was working non-business hours (Sunday). Within minutes of completing the work, team member left the building. DNS table was deleted by replication of new (empty) table to old (good) table. Control Room contacted EMS standby person to complain of problems calling up EMS displays. Closed Loop Control (AGC) was not impacted as it did not require DNS to continue running. Total time of the partial outage was 80min. Action Taken as Result of Outage Updated Process to require staff to wait 1 hour after completing updates and receive verification from System Control Manager before leaving. Checklist created and mandatory to be completed by the control room prior to the infrastructure staff leaving after a significant change is executed. Local hosts on each control room console updated to match DNS table (backup to DNS not working). This incident triggered discussions to modify the EMS support structure. General EMS standby support personnel. Additional Infrastructure standby added on weekends. 5

6 EMS Unplanned Outage 3 Complete Loss of EMS April 2013 Outage Details (Cause and Impact) Control Room placed a call to EMS Standby Support around 3:45AM complaining about loss of visibility/control of EMS. The problem was not properly diagnosed as systemic over the phone, and EMS support personnel drove to the control center to diagnose. Once at the control center (15min later) the problem was quickly diagnosed as loss of complete visibility and control of the EMS. Infrastructure personnel was called. Forced failover to standby firewall resolved the problem. Resulted in Loss of Closed Loop Control and visibility of the EMS for 85 min. What caused the outage? Problem was determined to be a Cisco bug in a critical firewall (VPN tunnel) which blocked all traffic and did not auto failover. The problematic firewall communicates with backup control center and data collection firewalls. Created two islands Active EMS/Passive EMS with no communication between them. 6

7 EMS Unplanned Outage 3 Continued Why so long to restore? Early call with problem not properly diagnosed over the phone, time wasted driving (~15min). Infrastructure Team Lead Engineer (not on standby) drove from home (~35min) to diagnose/manually failover firewall. Activation of the emergency switchover procedure took too long (approx. 45min and did not succeed). Recent implementation of Alstom Multi-Host software and related changes to emergency EMS switchover procedures. Insufficient training on remote execution of the EMS Switchover procedure. Testing of the Multi-Host failed to detect problem with the procedure causing it to fail. Time wasted attempting to contact RTS staff to assist in troubleshooting. Action Taken as Result of Outage Firewall SLA over the corporate network (VPN) created as an automatic backup path. Updated/Improved the emergency switchover procedure and created new EMS displays to facilitate the execution of the switchover. Switchover execution training improved and frequency of testing is now a once per month compulsory test. Created a new Team high availability EMS support structure to utilize existing resources to maximize coverage. Moved the primary switchover responsibility from the control room operators to the high availability Team EMS support. Created a callout list which is executed by a single call to Hydro Restoration Center (24/7 shift) to call RTS staff and provide quick summary of problem. 7

8 Team High Availability EMS Standby Support Structure. Problems with the existing EMS support structure. Standby Personnel has general EMS knowledge and not necessarily the required skills to diagnose and quickly execute EMS switchover. No requirement for EMS standby to be near EMS access (i.e. work or VPN from home). Only a few of the standby support staff had EMS laptops/vpn access from home. Much time is often wasted trying to reach personnel with the skills required to resolve problems Team EMS high availability (24/7) shift coverage. EMS standby support now consists of 4 teams each with 3 members with different roles. Primary Standby: This person is the primary contact for control room calls off-hours On call 24/7 and must be within 5 min of EMS access from home (VPN) or in the office. Specialized in executing EMS switchover procedure and restoring the system. For all other EMS problems locates the appropriate EMS person to resolve the issue. Backup Standby: This standby person is utilized to provide relief to Primary when needed. In the event that the Primary standby cannot be within 5min of EMS access, this person is used to provide coverage (same switchover skillset as the Primary). On call 24/7 and must be within 1 hour of EMS access. Application Standby: This standby person is utilized to provide EMS application support. This standby is used to provide support with EMS applications that are not within the Primary/Standby personnel skillset. On call 24/7 and must be within 1 hour of EMS access. 8

9 Team High Availability EMS Standby Support Cont d. Problems identified with previous standby structure resolved. Implemented a call bridge/callout list to free up time of support personnel to focus on resolving the issue. The new standby structure maximizes the available skill sets of EMS applications. All EMS support staff now have laptops with VPN access from home. Primary Standby personnel is required to be within 5min of EMS access. This eliminated the possibility of 1hr delay before accessing the EMS during a disturbance. 9

10 Questions? 10