DRAFT February 19, 15 BES Security s Working Group Page 1 of 7 Chapter X Security Performance s 1 3 3 3 3 0 Background The State of Reliability 1 report noted that the NERC PAS was collaborating with the BES Security s Working Group (BESSMWG) to develop security performance metrics. During 1, the BESSMWG developed an initial set of five draft metrics, with an additional two undergoing further development. This chapter introduces these new cyber and physical security metrics, provides results based on preliminary data collected during 1, and proposes next steps to further refine these metrics and validate the results. Purpose For some years now, NERC and the electricity industry have taken actions to address cyber and physical security risks to the BPS as a result of potential and real threats, vulnerabilities, and events. NERC s Critical Infrastructure Protection Committee established the BESSMWG to develop a comprehensive set of security performance metrics. These metrics would complement other NERC reliability performance metrics by defining lagging and leading indicators for security performance as they relate to reliable BPS operation. Methodology The BESSMWG, composed of subject matter experts from the electricity sector with experience in physical security, cyber security, power system operations, and NERC s ES-ISAC (Electricity Sector Information Sharing and Analysis Center) began their work by sharing how they measure and manage security performance within their own organizations. Members discussed the processes used to manage their security programs, as well as their own experiences with real security incidents and their potential impact on the BPS. Through these discussions, members considered the available sources of data that might provide insights to Lagging indicators are results-oriented, measure security performance for the electricity sector as a historical events, and tend to be easier to measure. whole. They also discussed the relative merits of Leading Indicators contribute to, or precede events lagging and leading indicators, with the goal of and tend to be more difficult to measure. developing metrics that address both attributes. The BESSMWG identified the following challenges related to developing security performance metrics: Note: This draft has been prepared by the BESSMWG for CIPC s review and endorsement during their March 15 meeting, and will provide the basis for a new chapter in NERC s 15 State of Reliability Report. Limited Historical Data: To-date, there have been relatively few security incidents with the potential to affect the BPS. Physical security incidents, such as vandalism and sabotage, have occurred infrequently for decades, typically with little or no impact on BPS reliability. However, recent high profile events have increased awareness regarding the potential for physical security incidents to significantly impact the BPS. Risks associated with cyber security appear to be rapidly evolving as the nature of cyber intrusions becomes increasingly sophisticated. Limited Ability to Normalize Available Data: Ideally, metrics provide a proportional indication by statistically sampling a known fraction of the whole, instead of measuring all events. Unfortunately, nobody knows the magnitude or number of constantly changing security threats and vulnerabilities with any degree of certainty, particularly as they relate to BPS reliability. Therefore, security performance
DRAFT February 19, 15 BES Security s Working Group Page of 7 metrics are limited to absolute numbers, rather than statistically valid percentages of the whole. While absolute numbers may indicate trends, these trends may not be statistically significant. Changing Threat Landscape: The frequency of occurrence of physical and cyber security threats, vulnerabilities, and incidents, while historically low, is changing rapidly. In particular, the risks associated with cyber security appear to be increasing dramatically. The Internet has no borders, and global cyber criminals and political activists alike are becoming increasingly sophisticated in using malware to attack their targets of choice. Sensitive Information: Information that details security threats, vulnerabilities, and real incidents is highly sensitive. In the wrong hands, this information can expose existing vulnerabilities to new and sophisticated exploits, create additional vulnerabilities, and limit effective response. 1 3 3 3 3 0 Consistent with the Development of Reliability Performance s The BESSMWG adopted the same approach to develop security performance metrics as that used to develop the reliability performance metrics. The BESSMWG noted that the definition of Adequate Level of Reliability and NERC s Reliability Principles recognized security as an integral component to maintaining the reliable operation of the BPS. The definition of Adequate Level of Reliability (ALR) for the Bulk Electric System 1 includes five performance objectives. The fourth of these objectives is relevant to security. Adverse Reliability Impacts on the BES following low probability Disturbances (e.g., multiple contingences, unplanned and uncontrolled equipment outages, cyber security events, and malicious acts) are managed. The definition of ALR also identifies events beyond the scope of predefined disturbances and include specific events that fall into the following two categories. Both of these categories reference security-related events (underscore added for emphasis).. Severe events resulting in the removal of two or more BES elements with high potential to Cascade: Two or more dependent or separate events leading to an unplanned loss and/or failure of elements on the BES that results in the loss of multiple Facilities not common to a single zone of protection. The events occur simultaneously or in close time proximity. An example is the loss of three circuits due to a lightning strike causing the outage of one element followed by a protection misoperation that causes an outage of another element followed by an overload that causes an outage of another element. Severe events can be the result of widespread cascading elements initiated by equipment failure, lightning strike, foreign intrusion, human error, the environment, contamination, sabotage, vandalism, or fire, or multiple element outages initiated by regional disaster events (e.g., hurricane or volcano) on Transmission infrastructure. 5. High Impact, Low Frequency Events: HILF is class of improbable events with the potential to significantly affect the reliability of the BES and cause long term, catastrophic damage to BES Facilities. The probability and magnitude 1 Filed with the Federal Energy Regulatory Commission, May 13. http://www.nerc.com/filingsorders/us/nerc%filings%to%ferc%dl/informational_filing_definition_adequate_level_reliability_15.pdf
DRAFT February 19, 15 BES Security s Working Group Page 3 of 7 1 3 3 3 3 0 of these events occurrence is uncertain but can result in Cascading, voltage collapse, or system instability, leading to uncontrolled separation. An example is a tornado or hurricane resulting in the failure of multiple BES elements (e.g. transformers) and voltage collapse. HILF events include coordinated cyber, physical, or blended attacks, pandemic illness, major earthquakes, Electromagnetic Pulse (EMP), and severe weather events. Disturbances that belong to categories and 5 cannot be predefined. Reliability Performance Objectives and 5 address responses to minimize and recover from Adverse Reliability Impacts on the BES resulting from conditions beyond the scope of predefined Disturbances. It is the ALRTF s general expectation that predefined Disturbances are determined through technical studies and the standards development process. Recognizing that reliability risks due to specific causes or events may be identified from time to time, the list of predefined Disturbances may need to be revised or expanded if industry stakeholders and regulatory authorities reach agreement that a particular risk should be mitigated with due regard to such causes or events. NERC Reliability Principles NERC s Reliability Principles have been developed to help ensure NERC s reliability standards are developed in a consistent manner to support the reliability of the BES. While the development of security performance metrics are not limited to the scope of NERC s reliability standards, the eighth principle is relevant.. Bulk power systems shall be protected from malicious physical or cyber attacks. Assessing the Value of the Draft s In order to assess the relative value of the security metrics being considered, the BESSMWG used the same SMART rating criteria used for developing the reliability performance metrics. As the SMART criteria were developed specific to BPS reliability, the BESSMWG found the criteria helped ensure that the impact of security on the reliability of the BPS was kept foremost in mind. From that perspective, security metrics that did not have an apparent link to BPS reliability received a lower SMART rating score, while those that have an apparent link received a higher score. The BESSMWG considered several general categories related to security performance: Actual Physical and Cyber Events: NERC s reliability standards require entities to report cyber and physical security incidents or events according to certain criteria. The BESSMWG considered metrics that would summarize these historical events as lagging indicators of security performance. Of importance is not just the number and frequency of these events over time, but also the extent to which they may have resulted in a loss of Load to customers. Information Sharing: The ability of entities to quickly and effectively share information with each other is an important capability when responding to new or rapidly evolving emergency situations. NERC s ES-ISAC provides a central clearing house to receive, analyze and share information with member entities. The BESSMWG considered metrics that would provide leading indicators of the extent to which entities are actively engaged with the ES-ISAC to contribute and receive security-related information. Global Cyber Vulnerabilities: Cyber security is not a concern limited to the electricity sector. The BESSMWG considered publicly-available metrics that describe how cyber vulnerabilities at the global level SMART: Specific/Simple, Measurable, Attainable, Relevant, Tangible/Timely
DRAFT February 19, 15 BES Security s Working Group Page of 7 1 affecting all information technologies are changing over time. While these metrics do not provide a direct measure of the impact on the BPS, they may provide leading indicators relevant to the electricity sector. The BESSMWG considered an initial set of more than security performance metrics. Detailed definitions have been developed for the top five of these metrics based on available data. An additional two are being considered for further development during 15. All security performance metrics are reported on an aggregated basis at the North American level, as there is no evidence to suggest that details at the Interconnection or Region levels would be meaningful. Draft Security Performance s and Preliminary Results This section provides the five draft security performance metrics recommended by the BESSMWG for implementation. BES Security 1: Reportable Cyber Security Incidents This metric reports the total number of Reportable Cyber Security Incidents 3 that occur over time and identifies how many of these incidents have resulted in a loss of Load. It is important to note that any loss of Load will be counted, regardless of direct cause. For example, if Load was shed as a result of a loss of situation awareness caused by a cyber incident affecting an entity s energy management system, the incident would be counted even though the cyber incident did not directly cause the loss of Load (e.g., through an unauthorized breaker operation). This metric will provide an indication of the number of Reportable Cyber Security Incidents and the resilience of the BES to operate reliably and continue to serve Load. This metric is based on data reported to and analyzed by NERC s ES-ISAC by Responsible Entities as required by the Critical Infrastructure Protection (CIP) cyber security standard CIP-00 Incident Reporting and Response Planning. Given the current relatively low number of such incidents, it is anticipated that the data will be gathered and summarized quarterly. Preliminary Results Based on Data Collected During 1 Table 1: Reportable Cyber Security Incidents Total number of Reportable Cyber Security Incidents Total number of Reportable Cyber Security Incidents resulting in loss of Load 1 15 Q1 Q Q3 Q Q1 Q Q3 Q Q1 Q Q3 Q 3 0 3 Ref. NERC Glossary of Terms: A Cyber Security Incident that has compromised or disrupted one or more reliability tasks of a functional entity.
DRAFT February 19, 15 BES Security s Working Group Page 5 of 7 1 BES Security : Reportable Physical Security Events This metric reports the total number of physical security reportable events that occur over time and identifies how many of these events have resulted in a loss of Load. It is important to note that any loss of Load will be counted, regardless of direct cause. For example, if Load was shed as a result of safety concerns due to a break-in at a substation, the event would be counted even though no equipment was damaged to directly cause the loss of Load. The metric will provide an indication of the number of physical security reportable events and the resilience of the BES to operate reliably and continue to serve Load. This metric is based on data reported to NERC s Bulk Power System Awareness group and analyzed by NERC s ES- ISAC by Responsible Entities as required by the reliability standard EOP-00- Event Reporting. Given the current relatively low frequency of such incidents, it is anticipated that the data will be gathered and summarized quarterly. Preliminary Results Based on Data Collected During 1 Table : Reportable Physical Security Events Total number of reportable events as a result of physical security threats to a Facility or BES control center Total number of reportable events that cause physical damage or destruction to a Facility Total number of reportable events as a result of physical security threats to a Facility or BES control center, or cause physical damage or destruction to a Facility, that result in a loss of Load 1 15 Q1 Q Q3 Q Q1 Q Q3 Q Q1 Q Q3 Q 7 9 0 BES Security 3: ES-ISAC Membership This metric reports the total number of electricity sector organizations and individuals registered as members of the ES-ISAC. ES-ISAC member organizations include NERC Registered Entities and others in the electricity sector including distribution (i.e., it is not limited to the BES). Given today s rapidly changing threat environment, it is important that electricity entities be able to quickly receive and share security-related information. This metric provides the number of organizations registered, as well as the number of individuals. Increasing ES-ISAC membership should serve to collectively increase awareness of security threats and vulnerabilities, and enhance the sector s ability to respond quickly and effectively. Reportable Events are defined in reliability standard EOP-00- Event Reporting, Attachment 1.
DRAFT February 19, 15 BES Security s Working Group Page of 7 This metric is based on data available from the ES-ISAC. It is anticipated that the data will be gathered and summarized quarterly. Preliminary Results Based on Data Collected During 1 Table 3: ES-ISAC Membership Total number of electricity sector organizations registered as members of the ES-ISAC Total number of individuals in ES- ISAC member organizations who have ES-ISAC accounts 1 15 Q1 Q Q3 Q Q1 Q Q3 Q Q1 Q Q3 Q 9 557 57 7 151 770 1 BES Security : Industry-Sourced Information Sharing This metric reports the total number of Incident Bulletins (currently known as Watchlist entries) published by the ES-ISAC based on information voluntarily submitted by ES-ISAC member organizations. ES-ISAC member organizations include NERC Registered Entities and others in the electricity sector, including distribution (i.e., it is not limited to the BES). Incident Bulletins describe physical and cyber security incidents and provide timely, relevant, and actionable information of broad interest to the electricity sector. Given today s complex and rapidly changing threat environment, it is important that electricity sector entities share their own security-related information, as it may help identify emerging trends or provide an early warning to others. This metric provides an indication of the extent to which ES-ISAC member organizations are willing and able to share information related to cyber and physical security incidents they experience. As ES-ISAC member organizations increase the extent that they share their own information, all member organizations will be able to increase their own awareness and ability to respond quickly and effectively. This should enhance the resilience of the BPS to new and evolving threats and vulnerabilities. This metric is based on data reported to and analyzed by the ES-ISAC. Given the current relatively low frequency of such incidents, it is anticipated that the metric data will be gathered and summarized quarterly. Preliminary Results Based on Data Collected During 1 Table : Industry-Sourced Information Sharing 1 15 Q1 Q Q3 Q Q1 Q Q3 Q Q1 Q Q3 Q Total number of ES-ISAC Incident Bulletins based on information provided by the electricity sector. 1
DRAFT February 19, 15 BES Security s Working Group Page 7 of 7 1 BES Security 5: Global Cyber Vulnerabilities This metric reports the number of global cyber security vulnerabilities that are considered to be high severity. This metric is based on data published by the National Institute of Standards and Technology (NIST). NIST defines high severity vulnerabilities as those with a common vulnerability scoring system 5 (CVSS) of seven or higher. As the term global implies, this metric is not limited to information technology typically used by electricity sector entities. As a result, this metric received a relatively low score using the SMART rating criteria. However, the BESSMWG recommends that this metric be adopted as it provides a leading indicator of the extent of constantly evolving cyber vulnerabilities and identifies trends beyond the electricity sector that may be relevant to the sector. The data will be gathered and summarized quarterly. Preliminary Results Based on Data Collected During 1 Table 5: Global Cyber Vulnerabilities 1 15 Q1 Q Q3 Q Q1 Q Q3 Q Q1 Q Q3 Q Number of global cyber vulnerabilities considered to be high severity 99 557 Next Steps Throughout 15, the BESSMWG will finalize these five preliminary metrics, and continue to develop and define additional metrics that can be developed with readily available data. In addition, the BESSMWG will develop a Phase plan to explore other metrics that would be valuable, regardless of the extent to which the data is readily available. 5 Ref. NIST http://nvd.nist.gov/cvss.cfm