2 nd Annual NERC Monitoring and Situational Awareness Conference: FPL s Operational Technology Center

Transcription

1 2 nd Annual NERC Monitoring and Situational Awareness Conference: FPL s Operational Technology Center Ed Batalla Director of Grid Control Systems Florida Power & Light Company Sept. 24, 2014

2 Florida Power & Light Company (FPL) Overview FPL EMS and Control Center Facilities Overview FPL Technical Support Services Overview Problem Statement What is an effective technology support structure to enhance grid reliability? FPL Approach Support Structure Evolution Operational Technology Center (OTC) Q&A Agenda Technology Monitoring Examples 2

3 FPL is the largest electric utility in Florida and one of the largest rate-regulated utilities in the United States FPL is a subsidiary of NextEra Energy, Inc. (NEE) One of the largest U.S. electric utilities Vertically integrated, retail rate-regulated Approximately 4.7 million customer accounts 24,273 MW generating capacity $10.4 B in operating revenues $36.5 B in total assets 3 FPL Overview NOTE: All data extracted from NextEra Energy, Inc., Annual Report 2013.

4 FPL has geographically diverse and redundant control centers which contain the Grid Control Systems (GCS) an integral part of our business continuity and recovery plans Backup Control Center (BUCC) Facility Transmission Control Centers and EMS Facilities (FPL EMS is also used by FRCC Reliability Coordinator) Distribution Control Centers (use and access same EMS) Fully functional backup control center, capable of being activated per NERC requirements and has tools that closely replicate the primary control center System Control Center (SCC) Facility Local Backup Control Center Remote Backup Control Center The control room has emergency communication methods for loss of the primary communication tools; also has remote access to the Backup Control Center 4

5 Facilities and Systems Overview Geographically diverse primary and backup control centers facilities Interim provision control centers Redundant building power supply (with emergency generators) Facilities Redundant systems at both facilities with fastfailover capability Upgraded: November 2012 Linked to Distribution Management System Energy Management System Fully redundant with voice recording capabilities Backup Systems: (a) Satellite Phones (b) Hoot and Holler, and (c) Corporate Phone System Voice Communications Real-Time Data Links (a) FPL data direct from FPL s internal sources Data Links and SCADA Data Full EMS/SCADA applications suite Customized Wide-Area views and situational analysis displays Reliability Tools FPL's EMS models the pertinent FRCC BES elements within the FRCC footprint and neighboring reliability area to provide Wide- Area situational awareness Power System Model 5 Confidential Information

6 Technology support and services are provided to continuously improve the operational certainty of the operator tools, applications, and infrastructure Technology Support and Services PEOPLE Operator tools and applications support provided by Grid Control Systems (GCS) team Infrastructure support provided by FPL s Information Management (IM) team Mission Priority: Availability Integrity Confidentiality PROCESS 24X7 Operational Technology Center for GCS applications and infrastructure monitoring Six Sigma concepts applied to daily work and performance monitoring continuous improvement Rigid change management and configuration control process TECHNOLOGY Fast-failover using multi-site technology Advanced reliability tools Cybersecurity technology improvements to further protect control networks Technology upgrades underway to provide improved resiliency and robustness 6

7 What is an effective technology support structure to enhance grid reliability? Need to be able to PREVENT, MITIGATE, RESTORE technology-related events/incidents Problem Statement Technology support structures for EMS/SCADA systems have been prevalently accomplished through callout Most have dedicated technical support staff for the EMS/SCADA systems (and reporting up through the business unit responsible for grid monitoring and control -- transmission operations) Some have integrated the technology support structure with information technology (IT) personnel Few deployed 24X7 technology support To optimize costs, most EMS/SCADA vendors are moving toward commercial off-the-shelf (COTS) technology infrastructure, which is common to most IT environments Support structure needs to adapt to business needs 7 No silver bullet need to consider business alignment with both grid operations and technology support objectives

8 FPL s support model continues to evolve in order to keep up with technology complexity and to continuously improve the operational certainty of its Grid Control Systems FPL s Support Model Evolution Proactive 8 Reactive Situational Awareness Callout Support ( ) Off hours and weekends supported by callout SMEs Operational Technology Center ( ) 24X7 dedicated shift personnel Still complemented with callout support as necessary Incident response and triage Technology surveillance and monitoring (KPIs) Enterprise Integration (2014 Beyond) 24X7 OTC integrated with corporate Network Operations Center (NOC) and Information Security Proceed to incorporate proactive monitoring of events - Applications - Infrastructure - Cybersecurity

9 FPL continues to implement initiatives to improve the service level for the availability of the Operators monitoring and real-time assessment tools 24/7 Technology Surveillance via OTC 9 Initiatives to Improve Service Level Applications and infrastructure performance monitoring Improved incident response for: GCS application issues Technology-related issues Plans are in place to further improve capability in infrastructure and cybersecurity monitoring FPL s IM team is now adopting this support model enterprise-wide GCS specific training of OTC operators These initiatives will provide service level margin with regards to the Operators monitoring and real-time assessment tools

10 OTC is currently positioned to complement grid operations (TOP and RC) through active technology surveillance OTC: Current and Planned Initiatives Current (On-Production) EMS (and related infrastructure) monitoring Technology surveillance Alarms Telemetry data source monitoring Communication circuits (substation data) Ticket Management Contingency Analysis (CA) performance Critical interfaces / throughput monitoring Planned OTC integration with NextEra Energy NOC Coordinated cybersecurity monitoring (between control and business networks) Dashboards / Control Charts Analytics for technology-related events Event analysis Predictive capability As we move forward, FPL plans to enhance OTC capabilities to improve technology situational awareness and analytics capabilities 10

11 Health of the system is monitored continuously by 24/7 technology personnel via technology specific alarms System and Application Health Check Technologyspecific alarms Coordination with grid operations (TOP and RC) is essential to ensure that operator tools are functioning as designed 11

12 FPL implemented improved monitoring process capabilities (with alarming) on Contingency Analysis (CA) failures Support processes are triggered based on CA last run elapsed time 12

13 OTC provides for the continuous monitoring of the technology components of SCADA data availability 13 As necessary, OTC will contact circuit vendor for repairs

14 OTC relieves technology-related issues from operators so operators can concentrate on grid monitoring and control functions Ticket Management Tickets are monitored for resolution 14 Trends are analyzed as necessary for continuous improvement

15 OTC is integrated with grid operations provides the opportunity for operators and field personnel to be aware of technology issues Status Report Pending or Overnight Issues # Incident Report # Issues and Actions Taken to Date Date & Time Issue Began RTU s Down # Site Name Incident Report # Service Provider Customer Count Issues & Actions Taken to Date Date & Time Issue Began 1 BARBV LTP (0) IR BARBVLTP (0) 30FDXX Circuit is down. ATT#sf ATT TOK initially, OTC requested to re-test cct and dispatch if need it. Still being worked by AT&T 8/11/ :14 PM 1 MELRO SE TAP IR AT&T AT&T circuit being retested to confirm last mile line is good. 8/17/ Scheduled Maintenance # Provider Description Status 1 2 Weekly Scheduled Events # Event Date & Time of Event 1 Failover Schedule: EMS failover 8/27/ Stations on EDGE (Wireless Backup) Circuit 1 TRAIN (807) IR RESOLV ED 2 JETPOR T (573) IR MOTOR OLA (405) IR RESOLV ED AT&T Century Link TRAIN (807) Circuit down. Wireless ok. RTU currently on EDGE; ATT tkt# (/DHEC/888232/ ATI/) Century Link update: Tested ok to the OCU; Need to notify P&C Fibernet Circuit Down - MOTOROLA (405) - FNET CCT=CV80804 TKT=FN /12/ :42 PM 8/16/2014 6:54 AM 8/21/2014 6:07 AM 15

16 FPL believes that the concept of the Operational Technology Center (OTC) keeps up with technology complexity and continuously improves the operational certainty of its Grid Control Systems Final Note Entities need to continue to strengthen the capabilities of their technology support structure Essential component to maintain reliability Need to deal with technology complexity Cybersecurity is a major part of the mix No silver bullet need to consider business alignment with both grid operations and technology support objectives Entities must continue to share best practices 16

17