B O N N E V I L L E A D M I N I S T R A T I O N B O N N E V I L L E P O W E R A D M I N I S T R A T I O N

Transcription

1 B O N N E V I L L E A D M I N I S T R A T I O N B O N N E V I L L E P O W E R A D M I N I S T R A T I O N NERC Monitoring and Situational Awareness Conference Bonneville Power Administration Self-Monitoring: Network and System Operations Center (NSOC) Title Sub Title September 30,

2 Today s Topics Agency Overview; NSOC: Definition, Purpose and Services: Monitoring system health; Monitoring network and telecommunication health; and Monitoring cyber security. Benefits of an NSOC; and Challenges of Developing an NSOC. 2

3 Agency Overview Bonneville Power Administration (BPA) is a federal agency established in 1937; BPA is self-funded and recovers all Agency costs through rates for wholesale power sales and transmission services; and BPA s service territory spans 15,000 transmission miles in the Pacific Northwest and consists of Oregon, Washington, Idaho, and parts of California, Montana, Nevada, and Utah; BPA is registered with NERC for the following functions: BA, TOP, TO, TP, TSP, PC, PSE, and RP 3

4 Agency Overview, continued BPA: Assures resource adequacy to meet the Pacific Northwest region s firm power requirements; Markets, but does not own, wholesale power from federal dams and acquired from non-federal generation; Builds, owns, operates and maintains a high-voltage transmission system to integrate and deliver power from federal and non-federal generation to regional customers in the Pacific Northwest and between the Northwest and the Southwest; and Protects and enhances fish and wildlife in the Columbia River basin. 4

5 BPA Statistics BPA Statistics Balancing Authority (BA) Installed Generation 32,157 MW Amount BA Peak Load, Feb. 6, 2014 Average Load, Jan. 1 May 27, 2015 Total Exports 2014 Total Imports 2014 Interchange Points with 17 BAs ~247 Transmission Customers ~500 10,643 MW 6,200 MW 83,267,202 MWh 25,221,117 MWh Operates and manages the federal transmission system (over 15,000 circuit miles over 11,000 circuit miles 230kV or higher, over 260 substations). 5

6 Agency Overview Power Services Transmission Services 6

7 Agency Overview BPA markets electric power from 31 federal Power Services dams, the Columbia Generating Station Nuclear Plant, and several small nonfederal power plants: About 80 percent of the power BPA sells is hydroelectric; and BPA accounts for about 30 percent of the electric power consumed within the region. 7

8 Agency Overview Power Services Transmission Services 8

9 Agency Overview Transmission System Operating voltage Circuit miles 1,000 kv * 500 kv... 4, kv kv kv.. 5, kv kv kv.. 3,509 below 115 kv Total 15,156 Transmission Services BPA SUBSTATIONS. 259 *BPA s portion of the PNW/PSW direct-current intertie. The total length of this line from The Dalles, Ore., to Los Angeles, Calif., is 846 miles. BPA s transmission system contains more than 15,000 miles of high voltage lines. About 75 percent of the high-voltage grid in the Pacific Northwest. 9

10 Transmission Services System Operations System Operations is responsible for the safe, reliable, open access operation and dispatch of the high voltage transmission system and interconnected generation. This is accomplished through the operation and management of two independent control centers. These two control centers, one in Vancouver and one in Spokane, are connected via BPA-owned and operated, fully redundant, telecommunication systems comprised of both fiber optic and radio systems, with the same BPA-owned communication systems also used to communicate to most remote sites. Both centers are staffed and operating on a 24/7 basis. During normal operations, each control center has access to all transmission data, most through independent communications, but responds to only that portion currently under their jurisdiction. Either the locally hosted systems or those of the alternate control center can be used. During emergency operations, jurisdictions can be reassigned between control centers based on communication connectivity or a single control center can assume jurisdiction of the entire BPA footprint. 10

11 Network and System Operation Center Provides continuous Network and System monitoring, incident response, IT support, remedial action, and incident coordination. 11

12 What Is an NSOC? The N-SOC is a combination of a Network Operation Center (NOC) and a Systems Operation Center (SOC); The N-SOC support identifies problems as a result of alerts processed by monitoring tools, customer complaints, or other actionable information; and The N-SOC support group is staffed by people with broad technical skills, so they can address complex operational issues with little or no assistance. 12

13 NSOC Services Provide single point of contact for work affecting the control center computers, software, networks, and building systems; Perform on-site system response, mitigation, and recovery; Provide immediate IT/OT response, support, and maintenance; Support the coordination of work in the Control Center s data center to ensure no concurrent outages of critical systems; Remote OTDR system allows for immediate response; Personnel tracking at remote sites; Provide damage assessments and begin system recovery immediately after a critical failure; continued 13

14 NSOC Services, continued Advise dispatch on system availability, path degradation, and schedule work impacts to the BES; Perform detection and reporting of potential cyber instances; and Centralized log collection correlation and monitoring. 14

15 NSOC 15

16 System Health: Monitoring Infrastructure status and functionality; Functionality alarming and response; Stale-data detection; Performance utilization; and Availability metrics. 16

17 SYSTEM HEALTH Infrastructure Status and Functionality SCADA alarms for Control Center infrastructure status Infrastructure status: HVAC, UPS, Generator, etc., status; and Rack power availability. Infrastructure functionality: Server status and health indications: Drive space; CPU utilization; and Power supply health. Server status and functionality 17

18 SYSTEM HEALTH Functionality Alarming and Response EMS System Alarm Screen Procedure Storage Functionality alarming: EMS systems provide alarming related to the function of the system; and Alarming provided for metering and curtailment utilities along with various systems used by our dispatchers and schedulers. Response: Document store contains over 900 procedures for immediate response: Documents searchable and tagged or manually grabbed from store. 18

19 SYSTEM HEALTH Stale-Data Detection SCADA Datamon alarms: Watches various indications on data from RTUs and ICCP; and Alarms with data has not changed in a predetermined time-frame. SCADA alarm for stale data Displays ICCP Data status; and Graphical alert if a single dataset has bad data. ICCP Dataset Status 19

20 Network and Telecom Health: Monitoring Remote communication and environmental monitoring: SCADA RTU status; Fiber monitoring; Microwave monitoring; Telemetry data Network availability: Between Control Centers; Between primary and secondary systems; and Between ICCP and points. Network utilization; and Network status mapping. 20

21 NETWORK AND TELECOM HEALTH Remote Site Communication and Environmental Monitoring Communication Alarming: Communication status; Fiber status; and Microwave status. Environmental: Door alarms; HVAC status; and Battery and generator status. Telecommunication Alarming 21

22 NETWORK AND TELECOM HEALTH Network Availability and Utilization ICCP Network availability and utilization Network availability: Graphical representation of control center networks and services. Network utilization: Graphical path utilization alerting. 22

23 Monitoring; Response; and Security controls testing. Cyber Security 23

24 CYBER SECURITY Monitoring Malware detection; Intrusion detection; Login failures; Account changes and creation of elevated accounts; Predictive firewall analytics; and Security controls testing on control systems. 24

25 CYBER SECURITY Response Cyber Security Response: Centralized log collected for systems with-in the control center; Tiered Cyber Security approach: Functional groups maintain cyber response and visibility; and Dedicated Cyber Security and Analysis Center helps identify APTs and performs forensics. Splunk Alert Manager 25

26 CYBER SECURITY Security Controls Testing Security controls testing overview: Security controls testing: Approximately 460 security controls locked, enforced, and tested to ensure security; Security controls exemption by exception and justification only, other mitigations may be enforced; and Servers and workstations with-in the Control Centers monitored. 26

27 Benefits; and Challenges to Development. NSOC Final Comments 27

28 NSOC Benefits The benefits of the N-SOC include improved reliability, shorter Mean-Time-To-Recovery (MTTR); Centralized tools requires a lower lifetime investment and increases human resources utilization and operational efficiencies; and The N-SOC operates around-the-clock and is physically located next to our Dispatch customers, making it an unmatched option at an ideal location. 28

29 NSOC Challenges to Development Implementing the N-SOC required a large investment of time and resources. A significant investment in equipment, facilities infrastructure, staff hiring and employee training was required. Time was required to develop the skill sets needed to manage the new technologies and tools, cross-train, and develop equipment maintenance programs. To enable the N-SOC functions, new network, hardware and software infrastructure were needed. Additional communication lines and video conferencing equipment, were required for full N-SOC functionality. 29

30 Questions? NERC Monitoring and Situational Awareness Conference Self-Monitoring: Network and System Operations Center (NSOC) 30