Ben Christensen CIP Enforcement Analyst. Root Cause Analysis for Commonly Violated Requirements June 6, 2013 CIPUG

Transcription

1 Ben Christensen CIP Enforcement Analyst Root Cause Analysis for Commonly Violated Requirements June 6, 2013 CIPUG

2 Agenda Methodology Individual Standard Requirements o Root Causes o Effective Solutions Additional Resources 2

3 Purpose Share common root causes as observed by WECC Share effective solutions Prevent occurrence/recurrence of these violations o Learning from known root causes o Solutions based on effective plans and WECC s experience 3

4 Methodology WECC analyzed all CIP violations reported between 1/1/2011 4/29/2013 Each violation was individually assessed & the root cause was analyzed Effective solutions were developed based on plans submitted by entities and WECC s experience in these standards 4

5 Methodology WECC narrowed the analysis to 3 Requirements that are commonly violated o CIP-007 R2 o CIP-007 R3 o CIP-007 R5 5

6 6 Root Causes

7 Root Causes - Importance Identify the Root Cause o Always evaluate why the violation occurred o Involve all related staff in the process o Use Toyota s 5 Whys methodology o May be more than 1 root cause Use the Root Cause to mitigate and prevent future recurrence of the violation 7

8 Root Causes - 5 Whys Example CIP 007 R3: 4 Security Patches for 2 Windows Servers were not assessed within 30 days o Why? No one checked to see if patches were available o Why? There were no clear procedures to assess patches o Why? No one knew for sure who was responsible for assessing patches 8

9 Root Cause Categories 3 Categories of Root Causes o Administrative Programs, Processes, or Procedures Entity s patching program doesn t identify the assessment timeframe required o Logical Cyber Assets, access points Entity s access point ports and services do not match the baseline 9

10 Root Cause Categories o Human Personnel, Unintentional Changes An IT technician forgot to follow the change management procedures for testing changes 10

11 11 Solutions

12 Effective Solution Determination Directly related to root causes Common themes for the 3 requirements o Lack of understanding of the Requirement o Lack of defined processes and procedures o Lack of training of those responsible o Lack of understanding of device capabilities 12

13 13 Root Causes

14 CIP-007 R2 ensure that only those ports and services required for normal and emergency operations are enabled. 14

15 CIP-007 R2 Historical Root Causes Administrative o Process did not require specific ports and services to be documented o Incomplete documentation to support compliance 15

16 CIP-007 R2 Historical Root Causes Logical o Lack of vendor support to identify ports and services on devices 16

17 CIP-007 R2 Historical Root Causes Human o Did not understand ports and services required for plant system devices o Depended only on vendor documentation to document ports and services o Did not know which dynamic port ranges were used 17

18 18 CIP-007 R2 Effective Solutions

19 CIP-007 R2 Effective Solutions IT personnel should review your list of cyber assets on a regularly scheduled interval o Review Checklist: Create review process that lines up with annual CVA During device configuration changes, have personnel review all similar cyber assets at that ESP Compare to the existing cyber asset list 19

20 CIP-007 R2 Effective Solutions Create an accurate baseline of ports and services for each cyber asset or cyber asset type 20

21 CIP-007 R2 Effective Solutions Use up to date vendor documentation Perform your own validation of ports and services o Don t just rely on the vendor to tell you what is open 21

22 CIP-007 R2 Effective Solutions Account for all dynamic ports and ranges o Common applications which use dynamic ports Java enabled Antivirus Applications which use UDP ports 22

23 CIP-007 R2 Effective Solutions (cont.) Use system tools to monitor for changes in port usage o Example of tools Tripwire Netflow IDS/IPS Access points 23

24 24 CIP-007 R2 Effective Solutions

25 CIP-007 R2 Effective Solutions Enhance individual training for personnel responsible for ports and services o Include device settings and port requirements in your training o Identify and include individual cyber assets and applications in training o Provide continuous training to all IT personnel on a regularly scheduled interval Annual CIP-004 R2 training may not be enough 25

26 CIP-007 R2 Effective Solutions 26 Peer review when adding or changing cyber assets and baselines o Use IT personnel that were not involved in the configuration of baseline to review the results o Use Compliance personnel to review the lists and baselines o Conduct peer reviews before and after implementation of changes

27 Patches - CIP-007 R3.1 document the assessment of security patches and security upgrades for applicability within thirty calendar days of availability 27

28 CIP-007 R3 Historical Root Causes Administrative o The security patch management procedure does not specify the required timeframe to assess patches o No procedure to verify assessment was occurring within 30 days o Lack of clarity on what documentation to maintain for evidence of assessment and implementation 28

29 CIP-007 R3 Historical Root Causes Human o Thought CIP-007 R3 only applied to OS level patches and not to other software or applications o Individual was not aware of their responsibility to perform patching o Did not know how to perform security patching assessments or implementation 29

30 CIP-007 R3 Historical Root Causes Logical o Personnel were not getting notifications of available patches 30

31 31 CIP-007 R3 Solutions

32 CIP-007 R3 Effective Solutions Specify the personnel responsible for assessing and implementing patches Clearly require that all applicable patches must be assessed within 30 days of availability 32

33 CIP-007 R3 Effective Solutions If possible, use an automated system to track availability of patches for all devices or device types o Examples WSUS Languard Secunia 33

34 CIP-007 R3 Effective Solutions (cont.) List devices that use a manual method to track available patches o Define manual method for each device type List devices with OS and applications installed o Include version information 34

35 Example of Patch Tracking Device OS Tracking Method Server 1 Windows Server 2003 Location Applications Automated WSUS server EMS server Workstation 1 Windows 7 Automated WSUS server EMS, adobe Console 3 Linux Manual Vendor site EMS client 35

36 CIP-007 R3 Effective Solutions Peer review when creating and maintaining evidence of assessment and implementation of patches o Use IT personnel that were not involved in the assessment to review the results and evidence o Use Compliance personnel to review the evidence o Conduct peer reviews before and after implementation of changes 36

37 CIP-007 R3 Effective Solutions Enhance individual training for personnel responsible for patching o Include device patching methods in your training o Identify and include specific cyber assets, applications, and patches in training o Provide continuous training to all IT personnel on a regularly scheduled interval Annual CIP-004 R2 training may not be enough 37

38 38 CIP-007 R5

39 CIP-007 R5 R5.2.2 o identify those individuals with access to shared accounts. R5.2.3 o shall have an audit trail of the account use (automated or manual) R5.3 o shall require and use passwords 39

40 CIP-007 R5 Historical Root Causes Administrative o Did not enforce password management procedures o Infrequently used accounts were not reviewed 40

41 CIP-007 R5 Historical Root Causes Logical o Inadequate tools to list all user accounts o Password aging reports not available o Assumed user activity logs were working properly 41

42 CIP-007 R5 Historical Root Causes Human o Didn t know that the accounts existed o Didn t focus on tracking shared accounts o Incomplete list of devices led to a failure to manage accounts on these devices 42

43 43 CIP-007 R5 Effective Solutions

44 CIP-007 R5 Effective Solutions Audit trail for shared account usage o Require initial log-in using individual credentials prior to using a shared account o Require a manual sign-in for accounts where automated solution is not available o Add banner or pop-up reminding individuals to follow necessary steps to create and audit trail 44

45 CIP-007 R5 Effective Solutions Password Management o If possible, technically enforce password changes at least annually o Create reminders about password changes for both individual and shared accounts o Upon each password change, verify all users who have access to that shared account 45

46 CIP-007 R5 Effective Solutions Enhance individual training for personnel responsible for account management o Include account management methods in your training o Identify and include specific types of accounts in training o Provide continuous training to all IT personnel on a regularly scheduled interval Annual CIP-004 R2 training may not be enough 46

47 47 Additional Resources

48 Additional Resources The 5 Whys o W.htm SANS 20 Critical Security Controls o CIP-002 R3 & CIP-007 R3 Root Cause Analyses by Joe Baugh o sentations/1/2011%2010%2020%20cipug%20ci P-002_CIP-007%20-%20Baugh.pdf 48

49 Summary Common violated standards o Root causes o Solutions to root causes Additional resources WECC is here to help 49

50 Questions? Ben Christensen (801)