Agenda Event Analysis Subcommittee Conference Call

Transcription

1 Agenda Event Analysis Subcommittee Conference Call August 14, :00 a.m. 1:00 p.m. Eastern Ready Talk Conference Call and Web Meeting Information: Dial-In: Access Code: Security Code: Webinar: (On the left side of the screen click Join a Meeting enter the above access code and security code to join the webinar) NERC Antitrust Compliance Guidelines Call to Meeting and Roll Call Agenda 1. EIDS Project Status (10 minutes) LaCreacia Smith Objective: Provide an update on the EIDS project to date. 2. Duke Energy presentation to NERC OC in September (15 minutes)* Sam Holeman Objective: Review outline of Duke Energy Events Analysis and Lessons Learned Program presentation. 3. EAS Agenda Items for September OC Meeting (15 minutes)* Sam Holeman Objective: Review Draft OC Meeting Agenda and identify EAS Presentations and presenters. a. EAS Quarterly Report to OC (agenda Item 5a) b. EAS Lesson Learned (agenda Items 6p and q) 4. EAP Update Team (EUT) (10 minutes) Hassan Hamdar Objective: Confirm dates for Industry ERO EA Process Webinar. a. Proposed webinar dates b. Additional EAP updates 5. EMS Task Force Update (10 minutes) Paul Johnson Objective: Update on EMS TF activities. 6. Trends Working Group Update (10 minutes) Jacquie Smith Objective Update on TWG activities.

2 7. Lessons Learned (20 minutes)* Mark Vastano Objective- Review LL s updated with comments from July 31 st meeting. a. LL 89 - Inappropriate System Privileges Caused Loss of SCADA Monitoring b. LL 91 - Indistinguishable Screens during a Database Update Led to Loss of SCADA Monitoring and Control 8. Roundtable Discussion (25 minutes) All a. Can an entity choose not to participate in EA and not have any consequences from the Regions or NERC (i.e. is the process voluntary)? 9. Adjourn Sam Holeman *As needed, background materials attached. Event Analysis Subcommittee Conference Call Agenda August 14,

3 Duke Energy Events Analysis and Lessons Learned Program Presentation Outline draft ERO Support and Event Analysis Organization a. Purpose b. Responsibilities c. Organization 2. Description of Duke Energy s NERC Event Analysis process a. Flowchart b. Plans to engage NATF Challenge Board 3. Description of Internal Event Analysis process a. Flowchart b. Criteria 4. Description of Lessons Learned process a. Flowchart b. Sources c. Internal process document 5. Communication Plan a. Formal training for NERC Event Analysis process b. Change Management Guide for Internal Event Analysis and Lessons Learned processes c. Update for revision to NERC EA process 6. Metrics a. How many NERC EAs completed b. How many Internal EAs in process, completed c. How many Lessons Learned reviewed and shared d. How many actions items identified and tracked e. Qualitative evaluation of experiences to date

4 2. Consent Agenda Chair Castle a. June 11 12, 2013 Draft OC Meeting Minutes* b. June 20, 2013 Draft OC Webinar Meeting Minutes* c. July 17, 2013 OC Action without a Meeting Action: Approve Presentation: No Duration: 10 minutes Objective: Approve consent agenda items as a block. Background Items: 1. June 11 12, 2013 OC Meeting Minutes 2. June 20, 2013 OC Webinar Meeting Minutes 3. July 17, 2013 OC Action without a Meeting (The OC approved posting the Reliability Guideline: Operating Reserve Management by electronic ballot.) 3. Chair s Remarks a. Report on August 14, 2013 Member Representatives Committee Meeting and the August 15, 2013 Board of Trustees Meeting 4. Election of Nominating Subcommittee Chair Castle Action: Approve Objective: The Operating Committee will elect a Nominating Subcommittee as required by the committee s charter. The newly elected Nominating Subcommittee will recommend to the OC candidates for the executive committee s four at large members. Presentation: No Duration: 10 minutes Background Items: None 5. Subgroup Status Reports* a. Operating Reliability Subcommittee Chair Colleen Frosch b. Resources Subcommittee Chair Don Badley c. Event Analysis Subcommittee Chair Holeman d. Personnel Subcommittee Chair Neil Lindgren 6. Committee Matters a. Revised OC Strategic Plan* Vice Chair Case Action: Approve Objective: Review, discuss and amend the OC Strategic Plan where necessary to get committee approval. Objective: Vice Chair Case will provide a summary of the comments received and an overview of upcoming drafting team activities. Background: At its June 2013 meeting, the OC reviewed a draft of its revised Strategic Plan. The Stragetic Plan task team considered the Committee s comments, amended the plan consistent with the Agenda Operating Committee Meeting September 17 18,

5 along with Larry Kezele met with the intent to streamline the OC AI list. We propose that as items are are completed, they are archived following the next NERC Board meeting. This will help keep the list crisp and focused on current issues. Presentation: Duration: 10 minutes Background Items: Revised OC Action Item List Yes OC Reviewers: o. Planning Committee s Performance Analysis Subcommittee Jessica Bian i. Metric Enhancements ii. Load Loss Due to Transmission and Distribution Related Outages Whitepaper* iii. AC Substation Equipment Task Force Scope* Action: Approve Objective: Review and discuss an overview of activities of the Performance Analysis Subcommittee. Background: The PAS recommends an enhancement to ALR6 2 (Energy Emergency Alert 3 (EEA3). The PAS will also request OC endorsement of the AC Substation Equipment Task Force scope. Presentation: Duration: 15 minutes Yes OC Reviewers: p. Discussion EAS Lessons Learned TBD Objective: Background: None Presentation: Duration: Z minutes Yes OC Reviewers: q. Discussion EAS Lessons Learned TBD Objective: Background: None Presentation: Duration: Z minutes Yes OC Reviewers: Other possible agenda topics include: Background Items: 1. Load Loss Due to Transmission and Distibution Related Outages Whitepaper 2. AC Substation Equipment Task Force Scope Background Item: None Background Item: None a. NERC and NATF Coordination on Misoperations b. Lessons Learned from industry c. Coordination with Reliability Assessment and Performance Analysis Long Term Reliability Assessment Survey of OC and timeline for report development (may be good Joint Session Topic) Energy Emergency Alert Duration ALR Metric Agenda Operating Committee Meeting September 17 18,

6 Lesson Learned Inappropriate System Privileges Caused Loss of SCADA Monitoring Primary Interest Groups Balancing Authorities (BAs) Transmission Owners (TOs) Transmission Operators (TOPs) Reliability Coordinators (RCs) Problem Statement An entity experienced a loss of SCADA telemetry specifically a loss of the channel status indicators for 76 percent of its transmission system. This problem occurred during the implementation of a scheduled SCADA database update that caused one of the front-end processors to be in an abnormal state. An incorrect command was used to remedy the situation, which resulted in the channel status indicators being set to a failed state. The RC and neighboring entities were promptly contacted and field personnel were dispatched to staff critical substations in order to verbally transmit data and maintain operating capability, per the entity s Critical Facilities Staffing Procedure. The SCADA support team resolved the problem, and full SCADA functionality was restored 42 minutes later. Details An entity s SCADA support staff was deploying a change to one of the SCADA system s front-end processors. The change was required in order to remove a field device from the front-end processor s scan list. During the implementation of the change, the SCADA support staff did not use the appropriate system privileges to stop and start the front-end processors scanning processes. This resulted in two scanning processes operating simultaneously. The command used to stop the second set of processes was not appropriate for the situation, resulting in the scanning processes assuming that the field device channels were not active, thereby disabling a portion of the SCADA system. Analysis by the SCADA support staff and the SCADA system vendor determined that the command used to stop the errant scanning processes was not correct since the scanning processes required an immediate termination, not a controlled shutdown. The scanning processes were designed to set the channel status to a failed state for a controlled shutdown. As detailed above, the RC and neighboring entities were promptly contacted while field personnel were dispatched to staff critical substations in order to verbally transmit data and maintain operating capability, per the entity s Critical Facilities Staffing Procedure.

7 Corrective Actions The incident, including the correct steps required to implement the front-end processor change, was communicated to the SCADA support team. Training on the proper steps to use when making any change to the production SCADA system was conducted. The entity reviewed this incident with its vendor and is putting additional preventative measures in place, including error checking to prevent a user from stopping or starting the scanning program using an incorrect command. Lesson Learned Entities need to ensure that they fully understand and verify with their EMS vendors, the correct procedures and commands required in all situations. Specifically, entities need to better understand the behavior of the system to various commands. This was an unanticipated event, but could have been prevented if the entity had implemented better controls. Entities should also consider: Reviewing the training with respect to change management to ensure that it includes a checklist of steps required; and Educating SCADA support staff on global impact of commands on the entire SCADA system. NERC s goal in publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the Bulk-Power System. NERC requests that you provide input on this lesson learned by taking the short survey provided in the link below. Click here for: Lesson Learned Comment Form For more information please contact: NERC Lessons Learned (via ) or Jacquie Smith (via ) Source of Lessons Learned: RFC Lesson Learned #: 89 Date Published: Category: This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing reliability standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned Inappropriate System Privileges Caused Loss of SCADA Monitoring 2

8 Lesson Learned Indistinguishable Screens during a Database Update Led to Loss of SCADA Monitoring and Control Primary Interest Groups Balancing Authorities (BAs) Transmission Owners (TOs) Transmission Operators (TOPs) Reliability Coordinators (RCs) Problem Statement During a planned database update and failover, an Energy Management System (EMS) operations analyst inadvertently changed an on-line SCADA server database mode from remote (online) to local (local offline copy), which caused a loss of SCADA monitoring and control of Bulk Electric System (BES) facilities. Details An EMS operations analyst notified an on-duty system coordinator of a desired database update and failover. After concurrence with the on-duty system coordinator and notification of affected parties, the operations analyst proceeded to set up the maintenance workstation in preparation for the failover from the on-line SCADA server to the backup SCADA server. The maintenance workstation is connected to the real-time system, but has the ability to separate its EMS functionality from the real-time databases for validating and staging database updates. The operations analyst had two remote desktop sessions open, one for on-line SCADA server and other for backup SCADA server. In addition, the analyst had a third desktop window that was integral to the maintenance workstation. The operations analyst proceeded to change the maintenance workstation s database mode from remote (online) to local (offline) to complete final database modifications before the failover. Due to the identical screens, the mode change and database modifications were inadvertently performed on the on-line SCADA server database instead of the maintenance workstation s database. As a result, the front-end processor (FEP) processes stopped polling remote terminal units (RTUs) and failover functionality was disabled. This caused a loss of SCADA monitoring and control of BES facilities for 31 minutes and 44 seconds. Corrective Actions Database modifications were immediately suspended until the following mitigation actions were in place: 1. Implementation of unique background identifiers to distinguish each SCADA server from the local maintenance workstation to provide heightened awareness. 2. Development and implementation of a formal procedure and checklist for SCADA database staging and implementation. This procedure and checklist should include, but not be limited to: a. All steps required to implement a SCADA database update on the on-line system; and b. Notification protocols throughout the EMS on-line SCADA maintenance activities.

9 Lesson Learned The root cause of the event was that the EMS operations analyst mistakenly switched the on-line SCADA server database to local (offline) mode from remote (online) mode, which caused the FEP processes to stop and disabled the failover functionality. The initial cause of this action was that the remote desktop screens to each SCADA server, which the operations analyst set up on the maintenance workstation, were identical in appearance. This made it difficult to differentiate between screens and contributed to the analyst mistakenly selecting the inappropriate screen. Another cause was the lack of a formal written process or checklist for implementing database changes. Without a checklist or a procedure, the operations analyst was unable to follow proper protocol and failed to verify that he was working on the appropriate device prior to initiating the database modification. Discussion with the EMS vendor indicated that switching between the local and remote database modes was not an intended feature for the SCADA servers. The vendor did not identify any situations in which it would recommend changing the database mode on a server and indicated that a future release of EMS software would eliminate the ability to switch database modes on a server. NERC s goal in publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the Bulk-Power System. NERC requests that you provide input on this lesson learned by taking the short survey provided in the link below. Click here for: Lesson Learned Comment Form For more information please contact: NERC Lessons Learned (via ) or Hassan Hamdar (via ) Source of Lesson Learned: FRCC Lesson Learned #: 91 Date Published: Category: This document is designed to convey lessons learned from NERC s various activities. It is not intended to establish new requirements under NERC s Reliability Standards or to modify the requirements in any existing reliability standards. Compliance will continue to be determined based on language in the NERC Reliability Standards as they may be amended from time to time. Implementation of this lesson learned is not a substitute for compliance with requirements in NERC s Reliability Standards. Lesson Learned Indistinguishable Screens during a Database Update Led to Loss of SCADA Monitoring and Control 2