Agenda Event Analysis Subcommittee Conference Call

Transcription

1 Agenda Event Analysis Subcommittee Conference Call October 23, :00 a.m. 1:00 p.m. Eastern Ready Talk Conference Call and Web Meeting Information: Dial-In: Access Code: Security Code: Webinar: (On the left side of the screen click Join a Meeting enter the above access code and security code to join the webinar) NERC Antitrust Compliance Guidelines Call to Meeting and Roll Call Agenda 1. EIDS Project Status (10 minutes) LaCreacia Smith Objective: Provide an update on the EIDS project to date. 2. Lessons Learned (30 minutes)* Mark Vastano Objective Final Review of lesson learned prior to publishing. a. LL93 IT Communications Disabled b. LL101 Loss of SCADA c. LL109 Failure of Energy Management System (EMS) 3. EMS Task Force Update (10 minutes) Paul Johnson Objective: Update on EMSTF activities. 4. Trends Working Group Update (10 minutes) Jacquie Smith Objective Update on TWG activities. 5. Roundtable Discussion (10 minutes) All 6. Adjourn Hassan Hamdar *As needed, background materials attached.

2 Loss of EMS IT Communications Disabled Primary Interest Groups Generator Operator (GOP) Reliability Coordinator (RC) Transmission Operator (TOP) Transmission Owner (TO) Problem Statement Transmission System Operators lost abilities to authenticate to the EMS system, resulting in a loss of monitoring and control functionality for more than 30 minutes. Details Scheduled control center server maintenance was being performed, which required the local authentication server to be taken out of service. By design, control center EMS application authentication should have rerouted automatically to a remote authentication server when the local server was taken out of service. Contrary to expectations and design, the automatic rerouting of authentication traffic did not occur and the EMS application was impacted. As a result, maintenance on the local authentication server was curtailed and was brought back on line. Once local authentication was re established, full EMS functionality was available. The root cause analysis determined that a specific firewall policy allowing authentication failover from the local authentication server to the remote authentication server was inadvertently deleted. Corrective Actions The following corrective actions were implemented: IT and business teams worked together to develop a test plan template to ensure that application functionality would be retained and supporting infrastructure components would function as designed. The IT Change Management process was immediately modified to ensure that comprehensive test plans are executed regardless of change classification. The past practice for work believed to be low risk was to allow test plans as a streamlined process when implementing a change. The prior streamlined process for low risk firewall policy changes required only that an engineering analytical review be performed. IT personnel were retrained on a revised Change Management process that included, but was not limited to, use of comprehensive test plans for all change classifications. A redundant local authentication server was installed at the primary control center.

3 EMS network design should, where possible, include a redundant local authentication server on the same internal network as the primary local authentication server. Having the primary and redundant local authentication servers on the same internal network (i.e., behind the same firewall) eliminates the dependency on a firewall rule for internal communications to both the primary and redundant local authentication servers. The IT Change Management process should consider applying the following principles: o Apply a thorough test process that is reviewed with the client for all changes that could affect EMS function. o Test the design redundancy or back out plan prior to implementing a change. o Test plans need to be comprehensive and include regression level testing. NERC s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the Bulk Power System. NERC requests your input on this lesson learned by taking the short survey provided in the link below: Click here for: Lesson Learned Comment Form For more Information please contact: NERC Lessons Learned (via ) NPCC Event Analysis Source of Lesson Learned: Northeast Power Coordinating Council Lesson Learned #: Date Published: October XX, 2013 Category: Bulk Power System Operations This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing Reliability Standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned Loss of EMS IT Communications Disabled 2

4 SCADA Failure Resulting in Reduced Monitoring Functionality Primary Interest Groups Reliability Coordinators (RC) Transmission Operators (TOP) Balancing Authority (BA) Problem Statement An entity s primary control center SCADA Management Platform (SMP) servers became unresponsive, which resulted in a partial loss of monitoring and control functions for more than 30 minutes. Because this loss of functionality was a result of a conflict between security software configuration changes and core operating system functions, a cybersecurity event was quickly ruled out, and no loss of load occurred during this event. Details The primary control center SMP servers ceased network functionality and were unresponsive to login attempts from the local console. Physical reboots of the servers were only able to resolve the problem momentarily. Recovery plans were immediately activated, and predefined decisions and procedures were followed as designed. The entity s primary control center Energy Management Platform (EMP) servers automatically began using the available SCADA signals provided by backup control center SMP servers and multisite IP routable SCADA. Manual actions quickly restored additional SCADA functionality for critical non IP routable circuits by moving those circuits to the backup SMP servers. Key generation facilities and substations were staffed to ensure that any needed control operations could be performed. Once the primary SMP servers were stabilized, they were used to operate only noncritical SCADA circuits until root cause was established and full remediation was completed. This was possible due to the multisite redundancy design of the overall Energy Management System, which allowed the entity s primary control center EMP servers to operate in a mixed mode, combining available SMP capabilities at both primary and backup control centers. Having this multisite redundancy meant the operators did not need to physically travel to the backup control center during this incident, and it also lowered risk during root cause analysis. The entity discovered that the root cause stemmed from a planned change to the security policy configuration of the host based intrusion detection (HIDS) and intrusion prevention (HIPS) software. As an unintended result, the HIDS/HIPS security software on the SMP server hosts began to block certain core operating system processes when those processes executed in a specific order that coincided with the HIDS/HIPS policy change. The block did not occur until several days after the change was implemented, when the SMP servers performed the specific functions that triggered the conflict and caused the HIDS/HIPS security software to lock down the core operating system.

5 Once the root cause was identified, the entity created a new HIDS/HIPS security policy configuration that allowed the HIDS/HIPS security software to handle the core operating system functions on the SMP server hosts properly. The entity then conducted the necessary testing and implementation to restore functionality to the SMP systems. Corrective Actions The entity engaged the HIDS/HIPS security software vendor to review and implement policy changes to better manage the balance between custom configurations and secure threat detection and protection. Processes for implementing HIDS/HIPS security policy changes while also maintaining system integrity are being reviewed for enhanced functionality and reliability. Solutions from these reviews are being implemented. Lesson Learned This event brought forward several positive lessons learned that minimized the extent of the outage: Security software configurations need careful analysis, design, testing, and implementation, as they may impact reliability in unpredictable ways. Registered entities should consider a multisite hosting configuration. This configuration provides flexibility and convenience for rapid recovery capability of EMS and SCADA functions. Frequent exercise of and training on recovery plans ensures that actual event responses go according to plan and promptly mitigate operational impacts. NERC s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the Bulk Power System. NERC requests your input on this lesson learned by taking the short survey provided in the link below. Click here for: Lesson Learned Comment Form For more Information please contact: NERC Lessons Learned (via ) Steve Ashbaker (via ) or (801) Source of Lesson Learned: Western Electricity Coordinating Council Lesson Learned #: Date Published: October XX, 2013 Category: Communications This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing Reliability Standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned SCADA Failure Resulting in Reduced Monitoring Functionality 2

6 Failure of Energy Management System While Performing Database Update Primary Interest Groups Reliability Coordinators (RC) Transmission Operators (TOP) Transmission Owners (TO) Problem Statement Failure of Energy Management System (EMS) while performing a database update. Details While performing edits to the EMS database, the entity received alarms that indicated errors for the communications servers. A decision was made to restore the database to its original state. While performing the restore procedure, the standby communications server in the Primary Control Center (PCC) was manually restarted. This caused the reversal of the database edits to fail and create faulty data files that synchronized across the integrated system servers. Although alarms were received for all communication servers, only the standby communications server in the PCC failed; the EMS remained fully operational. The faulty data files were manually removed from all servers, and a SCADA server failover was completed. An attempt to enable the standby communications server at the PCC failed. The EMS group executed a system warm restart, but since the EMS is an integrated system, the system warm restart resulted in the faulty data in the database being loaded into the remaining two communications servers, whereby all three communications servers failed. At this point, the EMS lost functionality and was operational on a sporadic basis. At no point was the EMS off line for a period exceeding 30 minutes. With the failure of the three communications servers, incremental system scans were performed. Subsequently, the substantive issues with the EMS were resolved and the EMS was restored with a minimum server requirement configuration with full functionality. Once the limited server system was verified as stable, all remaining servers were successfully brought back manually into synchronization with the EMS. Corrective Actions Training documents will be developed to document revised steps for database updates and communication server restarts to eliminate the failure mode experienced during this incident as a result of the integrated system. Database update testing procedures and documentation will be reviewed to ensure that testing requirements are clear and concise. Although the database updates were implemented first on

7 both the Product Development System (PDS) and the Dispatcher Training System (DTS), error logs were only partially reviewed. Therefore, the testing procedures will be updated to include step bystep instructions to ensure that the procedures are completely carried out, thus simulating the production environment that includes separate windows used for log viewing and update time log tracing. EMS analysts will receive training on the existing and new procedures. A proposal was requested and has been received from the EMS vendor to upgrade the EMS to a new EMS server environment in which the PCC and the Alternate Control Center (ACC) databases will be separate. In the new system, database updates will be required to be performed independently on the PCC and ACC to reduce the risk of any anomalies at the PCC from being propagated to the ACC. This will provide increased reliability to the EMS system. Lesson Learned When the EMS was purchased, the vulnerability of an integrated system architecture was unknown. To eliminate this now exposed vulnerability, it is recommended that functional separation of the PCC from the ACC be implemented. NERC s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the Bulk Power System. NERC requests your input on this lesson learned by taking the short survey provided in the link below: Click here for: Lesson Learned Comment Form For more Information please contact: NERC Lessons Learned (via ) NPCC Event Analysis Source of Lesson Learned: Northeast Power Coordinating Council Lesson Learned #: LL Date Published: October XX, 2013 Category: Communications This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing Reliability Standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned Failure of Energy Management System While Performing Database Update 2