Agenda Event Analysis Subcommittee Conference Call

Transcription

1 Agenda Event Analysis Subcommittee Conference Call September 11, :00 a.m. 1:00 p.m. Eastern Ready Talk Conference Call and Web Meeting Information: Dial-In: Access Code: Security Code: Webinar: (On the left side of the screen click Join a Meeting enter the above access code and security code to join the webinar) NERC Antitrust Compliance Guidelines Call to Meeting and Roll Call Agenda 1. EIDS Project Status (10 minutes) LaCreacia Smith Objective: Provide an update on the EIDS project to date. a. Schedule a demonstration of the EIDS tool before go-live. 2. Duke Energy OC Presentation (10 minutes)* Laura Lee/Sam Holeman Objective: Review revisions to the Duke Energy presentation of Events Analysis and Lessons Learned Program (OC agenda Item 6p) 3. Lessons Learned (30 minutes)* Mark Vastano Objective- Review next Lesson Learned for publishing. a. LL93 IT Communications Disabled b. LL101 Loss of SCADA c. LL109 Failure of Energy Management System (EMS) 4. COM-003 discussion (15 minutes)* Sam Holeman Objective: EAS has been asked by the OC to offer our opinion on communications impacts on the events we have seen.

2 5. EAS Access To Event Reports (15 minutes) Sam Holeman Objective: Discussion of EAS access to event reports submitted through the ERO EAP. 6. EAS Quarterly Report (10 minutes)* Sam Holeman Objective: Review EAS Quarterly Report to OC (agenda Item 5c) 7. EMS Task Force Update (10 minutes) Paul Johnson Objective: Update on EMS TF activities. 8. Trends Working Group Update (10 minutes) Jacquie Smith Objective Update on TWG activities. 9. Roundtable Discussion (10 minutes) All 10. Adjourn Sam Holeman *As needed, background materials attached. Event Analysis Subcommittee Conference Call Agenda September 11,

3 Appendix D Lesson Learned Template Lesson Learned IT Communications Disabled Category: Bulk Power System Operations Primary Interest Groups Generator Operator (GOP) Transmission Operator (TOP) Reliability Coordinator (RC) Transmission Owner (TO) Problem Statement Transmission System Operators lost abilities to authenticate to the EMS system resulting in a loss of monitoring and control functionality for more than 30 minutes. Details Scheduled control center server maintenance was being performed which required the local authentication server to be taken out of service. By design, control center EMS application authentication should have rerouted automatically to a remote authentication server when the local server was taken out of service. Contrary to expectations and design, the automatic re routing of authentication traffic did not occur and the EMS application was impacted. As a result, maintenance on the local authentication server was curtailed and was brought back online. Once local authentication was re established full EMS functionality was available. The root cause analysis determined that a specific firewall policy allowing authentication failover from the local authentication server to the remote authentication server was inadvertently deleted. Corrective Actions The following corrective actions were implemented: IT and business teams worked together to develop a test plan template to ensure that application functionality would be retained and supporting infrastructure components would function as designed. The IT Change Management process was immediately modified to ensure that comprehensive test plans are executed regardless of change classification. The past practice was to allow test plans for work believed to be low risk to use a streamlined process when implementing a change. The prior streamlined process for low risk firewall policy changes required only that an engineering analytical review be performed.

4 IT personnel were retrained on revised change management procedures that included, but were not limited to, use of comprehensive test plans for all change classifications. A redundant local authentication server was installed at the primary control center. Lesson Learned A. EMS network design should, where possible, include redundant local authentication server on the same internal network as the primary local authentication server. Having the primary and redundant local authentication servers on the same internal network (i.e., behind the same firewall) eliminates the dependency on a firewall rule for internal communications to both the primary and redundant local authentication servers. B. The IT Change Control Management process should consider applying the following principles: Apply a thorough test process that is reviewed with the client for all changes that could affect EMS function; Test the design redundancy and/or back out plan prior to implementing a change; and Test plans need to be comprehensive and include regression level testing For more information please contact: Michael Moon Senior Director, NERC Reliability Risk Management Michael.moon@nerc.net John Mosier Assistant Vice president-system Operations jmosier@npcc.org Source of Lesson Learned: NPCC This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing reliability standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned Title February 17,

5 Lesson Learned SCADA Failure Resulting in Reduced Monitoring Functionality Primary Interest Groups Reliability Coordinators (RC) Balancing Authority (BA) Transmission Operators (TOP) Problem Statement An entity s primary control center SCADA Management Platform (SMP) servers became unresponsive, which resulted in a partial loss of monitoring and control functions for more than thirty minutes. While this loss of functionality was a result of a conflict between security software configuration changes and core operating system functions, a cyber security event was quickly ruled out and no loss of load occurred during this event. Details The primary control center SMP servers ceased network functionality and were unresponsive to login attempts from the local console. Physical reboots of the servers were only able to resolve the problem momentarily. Recovery Plans were immediately activated, with pre-defined decisions and procedures followed as designed. The entity s primary control center Energy Management Platform (EMP) servers automatically began using the available SCADA signals provided by back-up control center SMP servers and multi-site IP routable SCADA. Manual actions quickly restored additional SCADA functionality for critical non-ip routable circuits by moving those circuits to the back-up SMP servers. Key generation facilities and substations were staffed to ensure any needed control operations could be performed. Once the primary SMP servers were stabilized they were used to operate only non-critical SCADA circuits until root cause was established and full remediation was completed. This was possible due to the multi-site redundancy design of the overall Energy Management System, which allowed the entity s primary control center EMP servers to operate in a mixed mode, combining available SMP capabilities at both primary and back-up control centers. This multi-site redundancy avoided the need for the operators to physically travel to the backup control center during this incident, and also lowered risk during root cause analysis. The entity identified root cause as stemming from a planned change to the security policy configuration of the host-based intrusion detection (HIDS) and intrusion prevention (HIPS) software. As an unintended result, the HIDS/HIPS security software on the SMP server hosts began to block certain core operating system processes when those processes executed in a specific order that coincided with the HIDS/HIPS policy change. The block did not occur until several days after the change was implemented, when the SMP servers performed the specific functions that triggered the conflict and caused the HIDS/HIPS security software to lock down the core operating system.

6 Once the root cause was identified, the entity created a new HIDS/HIPS security policy configuration that allowed the HIDS/HIPS security software to handle the core operating system functions on the SMP server hosts properly. The entity then conducted the necessary testing and implementation to restore functionality to the SMP systems. Corrective Actions The entity engaged the HIDS/HIPS security software vendor to review and implement policy changes to better manage balance between custom configurations and secure threat detection and protection. Processes for implementing HIDS/HIPS security policy changes, while maintaining system integrity, are being reviewed for enhanced functionality and reliability. Solutions from these reviews are being implemented. Lesson Learned This event brought forward several positive lessons learned which minimized the extent of the outage: Security software configurations need careful analysis, design, testing and implementation, as they may impact reliability in unpredictable ways. A multi-site hosting configuration provides flexibility and convenience for rapid recovery capability of EMS and SCADA functions. Frequent exercise of and training on Recovery Plans ensures that actual event responses go according to plan and promptly mitigate operational impacts. For more information please contact: Source of Lesson Learned: Regional Contact Steve Ashbaker Title Director Operations ashbaker@wecc.biz Phone # This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing reliability standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned SCADA Failure Resulting in Reduced Monitoring Functionality 2

7 Appendix D Lesson Learned Title: Failure of Energy Management System (EMS) Primary Interest Groups Reliability Coordinators Transmission Operators Transmission Owners Problem Statement Failure of Energy Management System (EMS) while performing a database update. Details While performing edits to the Energy Management System (EMS) database, alarms were received indicating errors for the Communications servers. A decision was made to restore the database to its original state. While performing the restore procedure, the Stand-by Communications server in the PCC was manually restarted. This caused the reversal of the database edits to fail and create faulty data files that synchronized across the integrated system servers. Although alarms were received for all communication servers, only the Stand-by communications server in the PCC failed and the EMS remained fully operational. The faulty data files were manually removed from all servers and a SCADA server failover was completed. An attempt to enable the Stand-by Communications server at the PCC failed. The EMS group executed a system warm restart but since the EMS is an integrated system, the system warm restart resulted in the faulty data in the database to be loaded into the remaining two Communications servers whereby all three Communications servers failed. At this point the EMS lost functionality and was operational on a sporadic basis. At no point was the EMS off-line for a period exceeding 30 minutes. With the failure of the three Communications Servers, incremental system scans were performed. Subsequently, the substantive issues with the EMS were resolved and the EMS was restored with a minimum server requirement configuration with full functionality. Once the limited server system was verified as stable, all remaining servers were successfully brought back manually into synchronization with the EMS. Corrective Actions Training documents will be developed to document revised steps for database updates and communication server restarts to eliminate the failure mode experienced during this incident as a result of the integrated system.

8 Database update testing procedures and documentation will be reviewed to ensure that testing requirements are clear and concise. Although the database updates were implemented first on both the Product Development System (PDS) and the Dispatcher Training System (DTS), error logs were only partially reviewed. Therefore, the testing procedures will be updated to include step-by-step instructions to ensure that the procedures are completely carried out simulating the production environment that includes separate windows used for log viewing and update time log tracing. EMS analysts will receive training on the existing and new procedures. A proposal was requested and has been received from the EMS vendor to upgrade the EMS to a new EMS server environment whereby the PCC and the ACC databases will be separate. In the new system, database updates will be required to be performed on both the PCC and ACC independently to reduce the risk of any anomalies at the PCC from being propagated to the ACC, providing increased reliability to the EMS system. Lesson Learned When the EMS was purchased, this vulnerability of an integrated system architecture was unknown. With this vulnerability having been exposed, it is recommended that functional separation of the Primary Control Center from the Alternate Control Center be implemented to eliminate this vulnerability. For more information please contact: Earl Shockley John Mosier Director of Event Analysis and Investigations Assistant Vice president-system Operations earl.shockley@nerc.net jmosier@npcc.org Source of Lesson Learned: Northeast Power Coordinating Council, Inc. (NPCC) This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing reliability standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned Title February 17,

9 Sam, Not sure if you have been following the COM-003 discussions from the NERC Board meeting. Please see the OC meeting material section I have attached below. Would you have some time to chat next week? It is my understanding that there has never been a BES event caused by miscommunications during non-emergency conditions. I just wanted to have a short discussion from an EAS perspective. Could you call my cell at your convenience? Cell (518) c. NERC Board Resolution: Operating Personnel Communication Protocols COM 003 1* Chair Castle Action: Approve Objective: Review, discuss and formulate an action plan to address the NERC Board s resolution regarding COM Background: The Board s resolution concluded with: FURTHER RESOLVED, that the RISC, the Independent Experts Panel, and NERC management are hereby directed to prepare responses to the questions set forth in the foregoing resolution and transmit a copy of their responses to the Chair of the NERC Board of Trustees and Chief Executive Officer of NERC no later than September 6, 2013, at which point the responses shall be transmitted by the Chair to (i) the Standards Drafting Team for the draft COM Reliability Standard, and publicly posted on the NERC website on the COM Reliability Standard development page, with a request for industry comment and (ii) the Operating Committee, with a request that the Committee review the questions and responses and provide their input to the Board. Presentation: No Duration: 30 minutes Background Items: The three reports will be provided to the OC as they become available. In addition, see OC Agenda Item 6.k. Jim Castle New York ISO (518)

10 Attachment 5.c OC Meeting September 17-18, 2013 NERC Operating Committee Sub-group Status Report Group: Event Analysis Subcommittee (EAS) Purpose: The Event Analysis Subcommittee is a cross-functional group of industry experts that will support and maintain a cohesive and coordinated event analysis (EA) process across North America with industry stakeholders. EAS will support development of lessons learned, promote industry-wide sharing of event causal factors and assist NERC in implementation of related initiatives to lessen reliability risks to the Bulk Electric System. Last Face-to-Face Meeting: June 10, 2103 Location: Atlanta, Ga. Duration: 1 Day Next Meeting: September 16, 2013 Location: Denver, CO Duration: 1 Day Bi-Weekly Conference Calls on Wednesdays from 1100 to 1300 (EDT) Chair: Sam Holeman Duke Energy Vice-Chair: Hassan Hamdar FRCC Pending OC Approval Items: None at this time Key issues for OC Resolution: None at this time Key Issues for OC Information: Event Analysis and Lessons Learned Program at Duke Energy Laura Lee, Manager, System Operations Implementation of Event Analysis Process Version 2 on October 1, 2013 o Summary of Webinar (August 28, 2013) Monitoring and Situational Awareness Conference September 18-19, 2013 in Denver, CO Cold Weather Event Report finalized and on the NERC site under Previous Cold Weather Event Analysis Current Initiatives/ Deliverables: Energy Management System (EMS) event review/summary

11 Future Initiatives/ Deliverables: Attachment 5.c OC Meeting September 17-18, 2013 EMS Event Task Force final report Event Trending Task Force ongoing Lesson Learned accountability model Presentation at the December, 2013 OC meeting from Florida Power and Light EAS will continue to review and address reliability issues that pose a threat and risk to the reliability of the BPS. Information obtained from the review will be shared with the OC and industry. External requests to group: Collaboration meetings being set up with North American Transmission Forum and North American Generator Forum Internal requests to group: PER Ad Hoc Team COM-003 standard development Coordination with Operating Committee on PER Ad Hoc request Coordination with Personnel Subcommittee on PER Ad Hoc request Group s recurring deliverables: EAS continues to manage the ERO Event Analysis Process Document update process Action oriented Lessons Learned posted on NERC website Any NERC Programs Oversight Responsibility for the Group: No Any NERC Document (non-reliability Standard) Responsibility for the Group: ERO Event Analysis Process Document