Expert Reference Series of White Papers. Cisco Security Troubleshooting: Part III Intrusion Prevention Systems

Transcription

1 Expert Reference Series of White Papers Cisco Security Troubleshooting: Part III Intrusion Prevention Systems COURSES

2 Cisco Security Troubleshooting: Part III Intrusion Prevention Systems Douglas B. McKillip, P.E., CCSI, CCSP, CCIE #1851 Introduction This paper is the third in a three-part series of white papers on Cisco security troubleshooting, each of which examines the challenge of implementing network security on equipment from Cisco Systems while maintaining the connectivity requirements of the business or enterprise. The focus of this third paper is primarily on troubleshooting the proper sensing operation of a signature-based Intrusion Prevention System (IPS). The first two papers in this series are: Troubleshooting Part I Connectivity Through ASA or PIX Firewalls and Troubleshooting Part II Virtual Private Networks Troubleshooting Scenario The following scenario will be used to illustrate many of the concepts discussed in this Cisco security white paper. If the names and IP Addresses look familiar to some of you, this is because the diagram represents a portion of the lab topology used in Cisco Security classes offered by Global Knowledge. Perimeter Router DMZ Subnet / /24 ASA/PIX Outside-PC DMZ-Srv NAT: Admin-PC Copyright 2009 Global Knowledge Training LLC. All rights reserved. 2

3 Intrusion Prevention Systems Before we illustrate the effective use of sensor (CLI) commands as well as IPS Device Manager, some basic principles of operation first need to be discussed. For all deployments of sensor appliances and modules, the interfaces belonging to these devices can take one of three roles. 1) Command and Control/Management - one per device; only interface w IP Address 2) Promiscuous - multiple per device; sensing interface 3) Inline (as part of a pair) - multiple per device; sensing interface While the Command and Control/Management interface is used for Management and Monitoring, the Promiscuous or Inline Pair interfaces are the ones into which the packets to be sensed arrive. The following sequence of events occurs with a properly operating and configured signature-based IPS. 1) A packet arrives at a sensing interface (promiscuous or part of an inline pair). 2) The packet is captured by what is known as the sensor app or analysis engine. 3) The sensor app invokes independent signature engines for matching of patterns. 4) If a pattern match is found against the signature database, an alert is generated. 5) Additional deny, blocking, capture, or alert actions are taken, if configured. The steps outlined above are an oversimplified example of how the IPS operates, assuming that the signature being matched is enabled and that the alerting behavior is configured. Secondly, the pattern match described in the fourth step could be the final packet in a multi-packet or fragment stream versus a single atomic capture. This Cisco white paper will focus on troubleshooting the failure of any of the five steps shown above. Promiscuous Mode Operation Single Switch with SPAN port The diagram to the right represents a modification to the troubleshooting scenario depicted earlier, illustrating the implementation of a promiscuous sensing interface on an IPS plugged into a switch. A Switched Port Analyzer (or SPAN port, as it is commonly known) has been configured for copying packets that enter the switch port connected to the DMZ interface of the ASA or PIX firewall. IOS-based switches must be configured using the monitor global configuration command using the following syntax. Copyright 2009 Global Knowledge Training LLC. All rights reserved. 3

4 monitor1session 1 source interface <hw-interface-name#> both monitor1session 1 destination interface <hw-interface-name#> The keyword both is used above to indicate packets being both received and transmitted by the interface. The commands show monitor and debug monitor can be used to verify proper configuration and operation, respectively. A sample display of show monitor is shown below. IPS-Sw# show monitor Session Type : Local Session Source Ports : Both : Fa0/1 Destination Ports : Fa0/4 Encapsulation : Native Ingress: Disabled Promiscuous Mode Operation - Multiple Switches with RSPAN Occasionally, an IPS will need to be operated in promiscuous mode connected to a switch other than the one through which the interesting traffic is flowing. In this case, a Remote SPAN or RSPAN VLAN can be used to copy packets from the source switch to this VLAN, carried by a trunk port between the switches, and then to a SPAN port on the destination switch. Several key steps are required on the two switches to accomplish this. On both switches: Switch(config)# vlan 900 (900 is an example; any unassigned VLAN # here is fine) Switch(config-vlan)# remote-span Copyright 2009 Global Knowledge Training LLC. All rights reserved. 4

5 On Switch2 (see previous diagram): Switch2(config)# monitor session 1 source interface <hw-interface-name#> Switch2(config)# monitor session 1 destination remote 900 On Switch1 (see previous diagram): Switch1(config)# monitor session 1 source remote 900 Switch1(config)# monitor session 1 destination interface <hw-interface-name#> Once these configurations are in place and spanning-tree for the RSPAN VLAN has been disabled, the reception of packets by the IPS promiscuous interface can be verified by a simple show interface command. Sensor1# show interface Interface Statistics Total Packets Received = 417 Total Bytes Received = Missed Packet Percentage = 0 Current Bypass Mode = Auto_off MAC statistics from interface GigabitEthernet0/0 Interface function = Sensing interface Description = Media Type = TX Default Vlan = 0 Inline Mode = Unpaired Pair Status = N/A Hardware Bypass Capable = No Hardware Bypass Paired = N/A Link Status = Up Link Speed = Auto_100 Link Duplex = Auto_Half Missed Packet Percentage = 0 Total Packets Received = 417 Total Bytes Received = While this command will verify that the sensor is receiving packets, it does not verify that the bi-directional flow between each session endpoint is seen. A more useful command on the sensor to verify this sessionized flow is the packet capture command as seen below (which can also be done for inline operation). Copyright 2009 Global Knowledge Training LLC. All rights reserved. 5

6 Sensor1# packet capture GigabitEthernet0/0 Warning: This command will cause significant performance degradation tcpdump: WARNING: ge0_0: no IPv4 address assigned tcpdump: listening on ge0_0, link-type EN10MB (Ethernet), capture size bytes 115 packets captured 115 packets received by filter 0 packets dropped by kernel Sensor1# packet display packet-file reading from file /usr/cids/idsroot/var/packet-file, link-type EN10- MB (Ethernet)...(other output omitted)... 21:24: IP > : icmp 1208: echo request seq :24: IP > : icmp 1208: echo reply seq :24: IP > : icmp 1208: echo request seq :24: IP > : icmp 1208: echo reply seq :24: IP > : icmp 1208: echo request seq :24: IP > : icmp 1208: echo reply seq :24: IP > : icmp 1208: echo request seq :24: IP > : icmp 1208: echo reply seq 4096 As can be seen from the trace above of a 1200 byte ping from to , both the echorequest and echo-reply packets can be seen. The observation of both directions of flow is necessary for the IPS sensor to properly reassemble fragmented packets in addition to its ability to observe well-behaved TCP connections (the completion of 3-way handshake & FIN exchange, for example). Copyright 2009 Global Knowledge Training LLC. All rights reserved. 6

7 InLine Mode Operation Both Interfaces on the Same Switch The diagram shown above is a more detailed depiction of the Troubleshooting Scenario seen at the beginning of this paper. If the two interfaces on an IPS sensor inline pair are connected to the same switch, the DMZ Server and the ASA/PIX dmz interface must be on different VLANs; otherwise, the sensor will be bypassed. Since each VLAN is traditionally thought of as its own broadcast domain, this means that VLAN 3 and VLAN 13 now have been effectively combined into one. Verification of Sensor Analysis Engine Proper Operation Now that we have examined ways to confirm the proper capture of traffic by using a promiscuous interface or an inline pair, our next focus will be on the analysis engine itself, or sensor app. There are several ways to confirm that the analysis engine is running. The first of these is a simple CLI-based command, show version, whose output is shown below. show version Application Partition: Cisco Intrusion Prevention System, Version 6.0(2)E1 Host: Realm Keys key1.0 Signature Definition: Signature Update S Virus Update V (output omitted) --- MainApp 2007_MAR_29_14_06 (Release) T14:44: Running AnalysisEngine 2007_MAR_29_14_06 (Release) T14:44: Running CLI 2007_MAR_29_14_06 (Release) T14:44: Copyright 2009 Global Knowledge Training LLC. All rights reserved. 7

8 As can be seen previously by the bolded text, the AnalysisEngine is operational. On the next page, an analogous screenshot of IDM is shown of the Monitoring --> System Information area. The display in the text window also indicates a running status. Verification of Operational Status of Signature Engines The third step in the five-step sequence of events outlined in the beginning of this paper was the analysis of captured packet(s) by the various signature engines. A signature engine is a category or family of pattern matches with common tunable thresholds and parameters. An example of this category would be ATOMIC. IP where the signatures belonging to that family all are matched by examining the packet headers of an IP packet (which would also include the ICMP, UDP, or TCP headers). As shown by the output below, the CLI command show statistics analysis engine can be used to verify activity of the signature engines. show statistics analysis-engine Analysis Engine Statistics Number of seconds since service started = The rate of TCP connections tracked per second = 0 The rate of packets per second = 5 Copyright 2009 Global Knowledge Training LLC. All rights reserved. 8

9 The rate of bytes per second = (output omitted) ---- Inspection Stats Inspector active call create delete createpct callpct AtomicL3L4_IP AtomicL3L4_ICMP AtomicL3L4_TCP AtomicL3L4_UDP FloodHostICMP Layer2ARP MSRPC_TCP MSRPC_UDP ServiceDnsUdp ServiceGeneric ServiceHttp ServiceNtp ServiceRpcUDP ServiceRpcTCP ServiceSnmp ServiceTNS String SweepUDP SweepICMP SweepTCP SweepOtherTcp TrafficIcmp TrafficIcmpDDOS TrojanUdp Note that even though the documentation on CCO classifies the ATOMIC.IP as an engine, the transport layer protocols TCP, UDP, and ICMP are separated out statistically. Also note that the line Number of seconds since service started substantiates that the analysis engine is running. Troubleshooting the Generation of Alerts Once the determination has been made that the sensor is properly capturing packets and the analysis engine is running, there are a number of ways to confirm proper signature alert configuration and behavior. One such method using IDM is depicted below where a signature with SigID 3200 was chosen for editing. Note here that not only is the signature disabled (which means alerting will be as well), but also the signature is marked as Copyright 2009 Global Knowledge Training LLC. All rights reserved. 9

10 retired. This latter term means that the sensor will not dynamically allocate memory resources for the data structure necessary for scanning for this signature. When Edit is clicked, the bottom image results. In the above window, the Enabled and Retired Status are changed using the dropdown menu. The show events alert CLI command can be used on a directly connected terminal or using IP-based SSH terminal emulators to yield a live monitor. While this is a useful tool for troubleshooting the absence of ANY alerts, it would be practically infeasible in a live production network due to the volume and speed of displayed information. A sample output is shown next for a large ICMP packet. Copyright 2009 Global Knowledge Training LLC. All rights reserved. 10

11 show events alert evidsalert: eventid= severity=informational vendor=cisco originator: hostid: Sensor1 appname: sensorapp appinstanceid: 365 time: 2009/02/18 01:32: /02/18 01:32:43 UTC signature: description=large ICMP Traffic id=2151 version=s1 subsigid: 0 marscategory: DoS/Host interfacegroup: vs0 vlan: 0 participants: attacker: addr: locality=out target: addr: locality=out os: idsource=learned relevance=relevant type=linux riskratingvalue: attackrelevancerating=relevant targetvaluerating=medium 35 threatratingvalue: 35 interface: ge0_0 protocol: icmp The corresponding area of IDM to observe these alerts (after they occur, not in real-time) is under the Monitoring Tab, Events menu selection as seen below. Another configurable alert property is the Alert Summarization mode which will influence the frequency of observed firings of the signature. As shown on the next page in the IDM Signature Edit screen for SigID 1102, the default Summary Mode is Fire Once. A little known undocumented feature of this alerting mode is that the frequency of alerts in this mode cannot exceed once every two minutes; therefore, to observe the most frequent alerting behavior, Fire All should be chosen. Copyright 2009 Global Knowledge Training LLC. All rights reserved. 11

12 Troubleshooting Additional Signature Actions Besides the basic Produce Alert default setting for all Enabled (and also unretired ) signatures, additional sensor actions can be taken, if desired, by configuring these on an individual signature-by-signature basis. A comprehensive list of the possible signature actions is depicted below, taken from the Configuration --> Policies --> Signature Definitions section, Actions button. Of all the actions depicted in the Assign Actions dialog box below right, the Request Block Connection and Request Block Host options each require that the IPS sensor initiate either a SSH or a telnet session to the blocking device (Cisco ASA/PIX, router, or Catalyst 6500 switch). Once the login profile, the blocking device properties, and the blocking interface designation and Access Control List (ACL) assignments have properly been configured and applied to the sensor, it will immediately attempt to establish a session with the blocking device. The sensor will stay logged in for as long as the router/firewall/catalyst switch is defined as a blocking device in the IPS configuration. This facilitates the quick application or removal of the blocking access-list entry or shun command, the latter used by the PIX or ASA. An example of this behavior is given below by revisiting the troubleshooting scenario shown at the beginning of this paper. Copyright 2009 Global Knowledge Training LLC. All rights reserved. 12

13 Perimeter Router DMZ Subnet / /24 ASA/PIX Outside-PC DMZ-Srv NAT: Admin-PC The managed device in this example is the Perimeter Router, which will be logged into by the sensor using telnet. The traces below illustrate the both the translation of the IPS Command and Control interface private IP address and the occupied terminal line of the router using this translated address. ASA# show xlate -- (output omitted) -- Global Local PERIM#show users Line User Host(s) Idle Location * 0 con 0 admin idle 00:00: vty 0 idle 00:00: As shown above, the sensor s IP address of is being translated to , and this IP address is being used to log in the Perimeter Router terminal line vty0. Copyright 2009 Global Knowledge Training LLC. All rights reserved. 13

14 Summary By now you should understand troubleshooting the basic proper operation of the Cisco Intrusion Prevention System (IPS). We have intentionally omitted the discussion of auxiliary (although frequently used) monitoring tools such as IPS Event Viewer, the newer IPS Manager Express, and CiscoSecure MARS. If alerts can be seen on the CLI console of the sensor, but not by any of these tools, the problem usually lies in the mis-configuration of the monitoring tool along with the omission of adding the monitoring IP address as an allowed host on the IPS. Once the IPS sensor has been properly installed and the management and sensing interfaces have been initialized, the next task is to ensure that signature alerts are being observed. With the successful viewing of these alarms, the accompanying challenge frequently becomes tuning the appliance or module to match the alerting behavior to the environment. Further VPN troubleshooting techniques are addressed in the other two white papers in this series: Troubleshooting Part I Connectivity Through ASA or PIX Firewall and Troubleshooting Part II: Virtual Private Networks. Learn More Learn more about how you can improve productivity, enhance efficiency, and sharpen your competitive edge. Check out the following Global Knowledge courses: IINS (Introduction to IOS Network Security) SNRS (Securing Networks with Cisco Routers and Switches) SNAF (Securing Networks with ASA Fundamentals) SNAA (Securing Networks with ASA Advanced) MARS (Cisco Monitoring Analysis and Reporting System) CANAC (Cisco Appliance for Network Admission Control) For more information or to register, visit or call COURSES to speak with a sales representative. Our courses and enhanced, hands-on labs offer practical skills and tips that you can immediately put to use. Our expert instructors draw upon their experiences to help you understand key concepts and how to apply them to your specific work situation. Choose from our more than 700 courses, delivered through Classrooms, e-learning, and On-site sessions, to meet your IT and management training needs. About the Author Douglas B. McKillip, P.E., CCIE #1851 is the president and principal consultant of Innovative Integrators Incorporated, a Delaware Corporation actually based in Delaware. In addition to a BS and MS in Chemical Engineering from M.I.T., Doug also later obtained an M.S. from the University of Delaware in Computer and Information Science. After 15 years of experience at DuPont and a brief stint with the original startup company associated with the Raptor EagleTM Firewall, Doug began his now 15+ year career of teaching and consulting, specializing in Internet Security with hardware from Cisco Systems since Doug s hobbies include playing piano at his local church and physical activities such as running, cycling, swimming, golf, skiing, and windsurfing. He and his wife Karen reside in Wilmington, Delaware. Doug can be reached at innovativeint@mindspring.com. Copyright 2009 Global Knowledge Training LLC. All rights reserved. 14