May SCADA Testbed Cyber-Security Evaluation. Iowa State University. Advisor: Members: Manimaran Govindarasu

Transcription

1 Iowa State University SCADA Testbed Cyber-Security Evaluation Members: Justin Fitzpatrick Rafi Adnan Michael Higdon Ben Kregel Advisor: Manimaran Govindarasu May 1013

2 Project Overview Problem/Need statement Since the early 90 s, experts have become more and more concerned about the threat of cyber attacks on the supervisory control and data acquisition (SCADA) systems used to monitor and manage infrastructure systems. Most SCADA system designs from the past never anticipated the security threats which exist today. SCADA systems are designed to provide an efficient solution to monitoring, regulation and control of various utilities. These systems compose a significant portion of the nation s infrastructure and they are a potential target of attack for this reason. With many of the SCADA systems being significantly dated, security was little concern prior to today s internet age. For this reason most control systems are open to attack from the outside. Design and implementation of SCADA test beds for use in security evaluation, testing and simulations is necessary to guarantee the safety of our critical infrastructure and utilities. A SCADA network consists of three major components and levels of abstraction. A control center and all the resources contained within is used to process data required to operate field devices. Field devices do not make decisions. They merely report data to the control centers via communication methods and receive instructionss based on the parameters of control software. Control Center RTU Relays The high level diagram pictured below is an illustration of how the hierarchy of the SCADA network is configured. Each element serves a specific purpose. Objective Critical infrastructure systems, such as electric power grid and water distribution systems, use SCADA (Supervisory Control and Data Acquisition) system for varieties of sensing, decision making, and control associated with real-time operation of the infrastructure systems. Cyber security of SCADA and hence critical infrastructure systems is a timely R&D challenge due to growing concerns for cyber attacks. The ECpE department at ISU has acquired SCADA system with necessary security software/hardware to setup a SCADA-Security Testbed. The testbed will be used to conduct attack-defenstheir potential impacts on the performancee and stability of the power system. The basic version of the testbed is already exercises to study the various vulnerabilities of SCADA systems and running. The goal of the project is to integrate real-time power system simulation capabilities into the SCADA testbed, and conduct cyber attack-defense evaluations on the integrated system.

3 Hierarchy of our project/system In a high-level sense, the objective was broken into several different, smaller projects. Due to the large scale of the final goal, keeping a focus on many of the smaller, individual pieces of the project proved to be of benefit to the group. Viewing each task or smaller project as an experiment, the group was able to tackle each problem individually. Taking a close look at each particular element of the project allowed the group to organize our time, efforts and other resources in an efficient manner. Accomplishing one experiment at a time, the group was able to build upon a solid foundation. Experiments (Stages) Experiment 1 System Familiarization The following are all pieces which lead to the overall operation and understating of the SCADA test bed. Siemens PowerTG Modes and Launch The following control panel can be used to specify the different modes and operation for the Siemens software. Initially, this host mode panel can be launched by double-clicking the PowerTG icon seen in the taskbar. The logo icon displayed in the taskbar is representative of the mode in which the system database is operating. In illustration below, the database is running in primary mode. Alternate modes of operation would be DTS or Stop for example. Selecting the Start Workstation button will launch the PowerTG workstation interface. Siemens Power TG Operations Index This can be regarded as the Intro or Home screen for the use of a Siemens PowerTG workstation. Upon logging in, this screen seen below will be the first presented to the user. It may be returned to at any point in time.

4 This operations index panel is the heart of an operational SCADA system. It allows the user to configure and monitor all aspects of the system. The page is useful for a system engineer to set alarms, view historical data and view live system statistics. Alarms can be set to warn the user in case of certain circumstances. In the case of over-currents, voltage derivations, outages and faults, alarms can be displayed to the user. These indications represent live data in the field, which may be undesired activity. Trending can be used to monitor the performance or characteristics of the system with respect to time. Graphs, plots and statistics can be used to give the user a visual and quantitative representation of the activity going on elsewhere on the power grid. For example, one might find a graph to display increasing current trends on a warm summer day. System Maintenance Index Following a similar pattern to that of the Operations Index, the System Maintenance Index is used to establish changes to the configuration of the system. A system engineer can use this panel to monitor a greater portion of the system devices and functionality.

5 Local settings such as the display board, console and printer can be found in the Equipment column. These parameters simply change preferences for the user. DNP Server As part of understanding our Test bed, we have to look into the communication protocols used by the different systems. One of which is the Distributed Network Protocol (DNP). Its primary use is in communication between the master station and the rest of the remote terminal units (RTUs), intelligent electronic devices (IEDs) etc. Among other things, DNP provides multiplexing, data fragmentation, error checking, link control and prioritization. These properties help create a more robust and reliable network connection between all the devices. Setting up the DNP server in SICAM Since this is a very important aspect of the SCADA system, proper understanding of it is quite crucial. Fortunately its application, for our part, is quite simple and entails setting up the server at each of the substation computers and making sure the master station can communicate the substation DNP server and poll for data from the RTUs. This is done with the help of the SICAM software and making sure the SCALANCE devices are working properly. By the end, upon reading our guide a user will be able to easily set up a DNP server and get a network up and running between the workstations and the RTUs. The DNP server is up and running

6 Databases The PowerTG Source Database (SDB) is an integral part of the PowerTG software. It contains definitions for everything from network information to host system configurations. Setting up stations in the SDB client The database is manipulated using the SDB client and all the information is stored in the SDB server. A basic setup of PowerTG requires the following information to be set up: Define all the stations Define RTUs at each station Define the network over which the system communicates Define individual computers that make up the system Define PowerTG host systems Define HMI consoles Define RTU communication servers After setting up these basic elements, you have to perform a database installation to transfer this information to the PowerTG real time database.

7 Setting up an RTU in the SDB client Our guides will contain a detailed yet brief look into carrying out all these steps including setting up communication lines, creating substations, defining RTU status/control points etc. After going through our guide the user will be able to set up their own complete system and carry out their own set of experiments on the SCADA test bed. The guide will contain an index of glossary, step by step configuration of each element with pictures and diagrams to help out at each step. The guides will be modularized enough so that any individual component will be able to stand on its own and yet be relevant to the rest of the system. Installing the information to the real time database

8 Experiment 2 At this stage of our project, we hope to have a basic and initial set up of our test bed. This would include all the RTUs, substations and master stations working properly and have them all interconnected using simple TCP/IP without any sort of firewall or security devices. This is to ensure at minimum reliable and proper test bed operations allowing us to conduct our later experiments which will include higher level security and attack/defense exercises Host 1 Host 2 WWW Relay 1 Relay Sicam 2 Sicam Topology of a basic networked SCADA system Topology The figure above shows a simple topology of the system. Each host, substation and remote terminal units will be connected to the internet and will be able to communicate with each other and as well as with devices from outside, which is a bad thing. After setting them up we can perform simple tests, such as closing and tripping circuits at the RTUs using the host systems and collecting analog data from them as well.

9 Closing a circuit breaker on a relay We believe with a proper and thorough understanding of the system from Experiment 1 will allow users to easily and quickly set up this system and move on to the next Experiments in our design process. Overview of the RTUCS, comm lines and RTUs Deliverables Deliverables for this stage will contain guides on how to accomplish the following tasks: Initial network hardware setup o Host systems o Database management o Substation and RTU configuration o Switches, Ethernet etc.

10 Internetwork communication o PowerTG to substation and back o Substations to RTUs and back Testing In addition to the previous, our guide will also contain a section on how to troubleshoot and test the final system. This will include simple exercises on how to connect to the DNP server and trip specific RTU led units. Experiment 3 Security Devices The SCALANCE devices are designed to protect individual devices or even entire automation cells from data espionage, data manipulation and unauthorized access. Security features available to us Firewall o IP firewall with stateful packet inspection o Bandwidth limitation Communication made secure by IPsec tunnels o SCALENCE devices are configured to form groups These groups can communicate securely with each other through these tunnels. Can also use SOFTNET Security Client to use establish secure IPsec tunnel communication in the VPN (Virtual Private Network). Protocol-independent o Both IP and non-ip frames are transmitted through the IPsec tunnel Router Mode o The SCALENCE device becomes a router that separates the internal network from the external network. The internal network becomes a separate subnet. Protection for devices and network segments o The firewall and VPN protective function can be applied to the operation of single devices, several devices, or entire network segments. No repercussions when included in flat networks o Just plug the SCALENCE device in and it will automatically find all the internal nodes, with no configuration required to nodes.

11 Our Current Setup - Tunneling Diagram Tunneling Please not that our groups project file for tunneling is located on the computer WORKSTATION in C:\Program Files\siemens\Security_Configuration_Tool\Projects\VPN\. Load this and transfer to all modules and you will have our current setup. ( see tunneling diagram) The following diagram shows what the SIEMENS Security Configuration Tool will show for our tunneling setup.

12 SCALANCE S (Tunneling) in productive operation The configuration is commissioned and the three SCALANCE S modules can now establish a communication tunnel over which network nodes from the two internal networks can communicate (green dashed line in diagram). No other communication can come from the outside network nor can anything in the tunnel talk to an external node outside the tunnel. You can only talk between SCALANCE devices. You can however set up IP address rules to allow other computers from the outside the network. Pretty much the same as just using the SCALANCE device as a firewall, except you do not have to set up all the IP address rules. SCALANCE Modules Tunneling Vulnerabilities The nice thing about the firewall is, only the rules set by the user are allowed through the SCALANCE devices. Substation 1 does not talk to Substation 2. This is good so that if one of the SCALANCE devices was compromised, it does not compromise both substations. This limits the amount of traffic between the certain SCALANCE devices as well and could improve performance All data being sent through the tunnel is encrypted even when going on the external network to get to the next SCALANCE device. Testing the tunnel function Testing phase 1 (All computers between each SCALANCE Module) 1. Enter the Ping command from Host 1( ) to SICAM 1 ( ) You will then receive the following message:

13 Result: You should see the Sent packets at 4 and the received packets at This means that since no other communication is permitted, these packets must have been transported through the VPN tunnel. 3. You can check all other nodes behind each SCALANCE device. Each node behind each SCALANCE device should be able to talk to the other nodes behind SCALANCE devices. Tunneling Test phase 2 (Computers outside the tunnel, on an external network) 1. Enter the Ping command from the outside computer to SICAM 1 ( ) You will then receive the following message: Result: You should see the Sent packets at 4 and the received packets at This means that the IP frames from the outside computer did not reach SICAM 1 since neither tunnel communication between these two devices is configured nor is normal IP data traffic permitted. A device inside the tunnel cannot talk to an outside computer not in the tunnel. Firewall In the firewall, IP traffic can only be initiated from the internal network; only the response is permitted from the external network. Please note that our groups project file for firewall is located on the computer WORKSTATION in C:\Program Files\siemens\Security_Configuration_Tool\Projects\firewall\. Load this and transfer to all modules and you will have our current setup. (see diagram)

14 Firewall setup diagram The following diagram shows what the SIEMENS Security Configuration Tool will show for our tunneling setup.

15 Configuring the firewall 1. The following are the rules needed for SUB1 to communicate to all other nodes necessary. 2. The following are the rules needed for SUB2 to communicate to all other nodes necessary. 3. The following are the rules needed for CONTROL to communicate to all other nodes necessary. SCALANCE S in productive operation The configuration has now been commissioned and the three SCALANCE S modules are now protecting the internal network with the firewall according to the configured rules. The rules are shown above and follow the diagram for who can communicate with who. Basically CONTROL SUB1 and CONTROL SUB2 Tunneling Vulnerabilities Since the SCALANCE devices are set up to tunnel Substation 1 can talk to Substation 2, which in a real world application, this would not be the case since there is no need for that.

16 This could cause vulnerabilities if somebody got into the tunnel because then it would compromise the whole network since the tunnel would let it talk to anybody inside the tunnel. o This can be fixed by creating 2 separate tunnels. 1 for CONTROL SUB1 and 1 for CONTROL SUB2 There is no way to encrypt the data being sent in this method. All data sent over the external network is not encrypted and could be sniffed if done correctly. Testing the firewall function Testing phase 1 (All computers between each SCALANCE Module) 1. Enter the Ping command from Host 1 to SICAM 1 (IP address ) You will then receive the following message: Result: You should see the Sent packets at 4 and the received packets at Due to the configuration, the ping packets can pass from the internal network to the external network. The PC in the external network has replied to the ping packets. Due to the "stateful inspection" function of the firewall, the reply packets arriving from the external network are automatically passed into the internal network. 3. This test will work for all IP rules set above. You can test each one. Test phase 2 (Computers outside the tunnel, on an external network) 1. Enter the Ping command from the outside computer to SICAM 1 ( ) You will then receive the following message: Result: You should see the Sent packets at 4 and the received packets at 0.

17 2. The IP packets from the outside computer must not reach SICAM 1 since the data traffic from the "internal network" (SICAM 1) to the "external network" (outside computer) is not permitted. 3. This will be the same for any computer not set in the above IP rules. NAT (Network Address Translation) Router Mode 1. The common use case in which all internal nodes send packets to the external network and keep their IP addresses hidden by the NAT functionality is preconditioned on the SCALANCE S. 2. This could be used for each SCALANCE device so that the IP addresses would be hidden of each node behind the SCALANCE device. 3. This would make it a little harder to sniff the data since it is not encrypted in this method. 4. We will be learning more about this method throughout the implementation part of the project. Checking to make sure the test bed works The major test of both all 3 of the methods above is: 1. We need to make sure RTUs. the 2 RTUs connect to the DNP server which gives us the ability to trip the a. This can be tested by logging onto the Power TG software and tripping the relays. b. The light should turn red as show in the picture below. If it does, you have configured everything correctly. c. As an example for Relay 1, IP 201 should talk to 217 when pinging 2. Make sure you cannot talk to the external network based on the rules you set forth or the tunnels you have set up. Along with this making sure all unauthorized connectionsare blocked. Checking status of SCALANCE Modules One of the things you can do is log the packets coming in and out of the SCALANCE device Allows you to see what is coming in and out of the SCALANCE device and what is actually being blocked. This allows you to see if someone from the outside is trying to get to your nodes behind the SCALANCE.

18 SCALANCE Device Summary Ideally we would use a combination of all these 3 methods. Each has their own benefits and flaws. When implementing this we have to make sure the testbed is set up like a real world situation. As of now everything is talking to each other the way they should be and we will be starting with the basics and trying to hack into that and then working our way up to higher security. Experiment 4 - Adjustable load on Relays Relay Overview The SIPROTEC series relays being used on this SCADA test bed are flexible and have the ability to serve in a variety of applications across the system. Primarily, relays are used on a power distribution network to open and close loads or various branches of the power grid. The Siemens SIPROTEC 7SJ61 relay modules have the ability to monitor current flow on all three phases of whatever particular node they may be connected to. Above is an illustration of a Siemens relay monitoring a distribution line. Current Monitoring Monitoring current becomes the secondary focus and implementation of these relays. Data can be logged from the activity of the relay as an analog point of the live performance. An analog point simply becomes a variable of some particular characteristic within the test bed. Say, current at a particular branch of a long three-phase transmission line, or the phase voltage at the end of a lengthy span of conductor.

19 Above is a relay on the test bed displaying data from a single analog point. A constant load has been connected to this relay, drawing a current of 5 amps. Deliverables In this scenario, the practice of over current protection will be the focus of our efforts. In the field, transmission lines are subject to handling excess amounts of current in times of high load demand or weather conditions. As a result, equipment can be damaged, transmission lines can heat up to levels compromising the reliability of the grid. To protect against these potentially harmful or dangerous situations, relays can release load from the distribution network, helping preserve the integrity of the grid. Modeling in PowerTG Data from the analog points of a relay can be monitored from within the Siemens PowerTG software. Naturally, an engineer on a regional power grid is going to use such relays to report data and protect their infrastructure. Over currents can be monitored in real-time, and set to be rapidly acted upon. In this case, we will be opening relays whose lines exceed chosen current values.

20 Above is a large scale representation of the analog points which a PowerTG control center can be monitoring. In a fully configured system, the values would not read 0. Above is a screenshot indicating two circuit breakers indicated by CBA 1 and CBA 2 contained within a single relay. These both display status trip which means that they have been opened by the PowerTG software for reasons related to protecting the system. Execution of action

21 Above is another screenshot displaying some configuration parameters from the PowerTG database. This particular configuration indicates how a section of the relay is currently configured to remain in a tripped or closed state based on a monitored analog point. Testing A simple method of testing can be used in the scenarios of a SCADA test bed such as the model used in this design project. With a fully functional network of equipment, it is unnecessary to have relays connected to an actual power transmission line to verify the switching and protection of an over current. Although actual testing procedures have not yet taken place, it is an intended focus of the Spring 2010 semester. In principle, the testing on our test bed will be composed of introducing a variable load to a relay. The load will be selectable and can range in value. Initially, a small load can be subjected to the relay. Slowly, load can be increased to such a level that the relay will execute an opening, or trip action as defined in PowerTG parameters. When the observed load exceeds this current threshold, the action will take place. Implementing such a test can be done by using some simple components and Electrical Engineering principles. We will be designing a panel containing perhaps a half dozen resistors. In this case, simple 100 watt light bulbs will suffice adequately. With a light switch in series with each bulb, current ranging from 0 to 5 amps can be produced when connected to a 120v source. Experiment 5 The final stage of our project consists of the security evaluation of our SCADA testbed. All 4 previous experiments have lead to the development of an operational, small-scale SCADA network. The primary goals of this evaluation will be to conduct an investigation into Critical Infrastructure Protection (CIP) protocol standards to determine whether our SCADA network adheres to these standards and if not, how can the system be modified to do so? The secondary objective will be to examine various types of known attack schemes and determine if any of them can be used effectively to compromise our SCADA network. In conjunction with this effort our team will also examine the SCADA system and software to determine if any security vulnerabilities exist which could potentially be exploited. Criticial Infrastructure Protection (CIP) Compliance The North American Electric Reliability Corporation (NERC) defines the reliability requirements for operation of bulk power systems such as those that might be controlled by our SCADA network testbed. The CIP standards, and their explanations as taken from NERC, consist of 9 requirements that all bulk power operators are required to follow: CIP-001-1: Purpose: CIP-002-2: Purpose: Sabotage Reporting Disturbances or unusual occurrences, suspected or determined to be caused by sabotage, shall be reported to the appropriate systems, governmental agencies and regulatory bodies. Critical Cyber Asset Identification Provide a framework for the identification and protection of Critical Cyber Assets to support reliable operation of the Bulk Electric System. Management and maintenance of Bulk Electric Systems relies on communication between Cyber Assets that support critical reliability functions and process. Development of a risk-based methodology to identify and document Critical Cyber Assets.

22 CIP-003-2: Purpose: CIP-004-2: Purpose: CIP-005-2: Purpose: CIP-006-2: Purpose: CIP-007-2: Purpose: CIP-008-2: Purpose: CIP-009-2: Purpose: Security Management Controls Development and documentation of a security policy that addresses the issues of cyber security policy exceptions, information protection and access control. Personnel & Training Establishment, documentation, implementation and maintenance of a security awareness program to ensure that personnel having authorized cyber or authorized unescorted physical access to Critical Cyber Assets, including contractors and service vendors, have an appropriate level of personnel risk assessment, training, awareness of security practices. Electronic Security Perimeter(s) Identification and protection of the Electronic Security Perimeter(s) inside which all Critical Cyber Assets reside, as well as access points on the perimeter. This evaluation should include an assessment of cyber vulnerability at access points as well as electronic access controls and monitoring of these access control mechanism. Physical Security of Critical Cyber Assets Establishment of a physical security program for the protection of Critical Cyber Assets. The Physical Security Plan should include documentation of physical access points, protection physical and electronic access control systems, monitoring and protection of physical access controls, and logging of physical access. Systems Security Management Definition of methods, processes, and procedures for securing those systems determined to be Critical Cyber Assets, as well as the other (non-critical) Cyber Assets within the Electronic Security Perimeter(s). Process and procedures should include port and service necessity, security patch management, malicious software prevention, account management, security status monitoring, and vulnerability assessment. Incident Reporting and Response Planning Identification, classification, response, and reporting of Cyber Security Incidents related to Critical Cyber Assets, including but not limited to an Incident Response Plan consisting of procedures to classify incidents, response actions, communication plans, reporting incidents and updating and ensuring the Incident Response Plan. Recovery Plans for Critical Cyber Assets Ensure that recovery plan(s) are put in place for Critical Cyber Assets and that these plans follow established business continuity and disaster recovery techniques and practices. Recovery plans should plans and procedures for response actions, exercising/drilling of the plan, making changes, and backup and restoration of information used to restore Critical Cyber Assets. CIP Compliance Evaluation During our security evaluation we will use the CIP Standard documents from the NERC website to take a detailed look at which requirements our SCADA network and lab adhere to as well as where the holes are and how they can be

23 modified. We will look to determine where the requirements leave the electronic realm and begin to enter the physical. Additionally, we plan to employ the Cyber Security Evaluation Tool (CSET) development by the Department of Homeland Security US-CERT. The purpose of the software is guide the user through a step-by-step process to evaluate the control system security practices against industry standards. This software incorporates standards from organizations such as NIST and NERC; our hope in using this software is to gain additional insight into industry standard control system security best practices. Attack Development After our evaluation of the SCADA testbed with respect to CIP compliance our focus will move more towards software and network security. While some of these concerns are addressed in CIP-007-2, Systems Security Management, the reliability standard does not seem to completely cover the topic. The main goal of this phase of the security evaluation will be to look mainly at network and some software level attacks that could potentially be deployed against our SCADA network. At the same time we will be examining the Siemens software looking for potential security vulnerabilities within the software. What follows is a few broad categories characterizing attacks we hope to develop and deploy: Denial-of-Service Prevent access to a network resource using various DoS attack schemes Invalid Data Sending packets into the network with invalid headers or data Physical Attacks Attacks to network connections or physical devices Remote Access Targeting applications and protocols used to remotely access devices and software Information Theft obtaining information using network traffic sniffing or social engineering Social Engineering Development of social engineering based access scenarios Testing Testing the effectiveness of these various attacks would be relatively simply and consist mainly of the development and execution of attacks against the SCADA network. It is likely that we will relax some of the security settings on our system in order to prove that an attack works without security measures in place and fails with security in place. Follow-up The last element of this phase will be an examination of any vulnerabilities discovered during the evaluation process. If software or network vulnerabilities are found then we will determine the risk presented by the vulnerabilities and look for solutions to correct the problem presented.