USE OF PASSIVE NETWORK MAPPING TO ENHANCE SIGNATURE QUALITY OF MISUSE NETWORK INTRUSION DETECTION SYSTEMS

Transcription

1 USE OF PASSIVE NETWORK MAPPING TO ENHANCE SIGNATURE QUALITY OF MISUSE NETWORK INTRUSION DETECTION SYSTEMS Burak Dayıoğlu, Attila Özgit Dept. of Computer Engineering, Middle east Technical University, Ankara, Turkey Abstract. Misuse detection systems are known to be producing high rates of false positive alerts. High rates of false alerts adversely affect system usability and dynamic countermeasure generation. Network misuse detection systems are almost always assumed to be static systems, which do not have information about their working environment. Passive network mapping is suggested as a way of self-learning for network intrusion detection systems. The collected information may be used to eliminate known ambiguities of such systems and increase the quality of system output by eliminating some of the false alerts. A prototype implementation of the concept and the analysis of its performance is presented. 1. Introduction In order to protect sensitive information and systems, use of systematic approaches is a necessity. To increase organizational security, three different but closely coupled measures need to be in place: a. Security Policy and Procedures b. Technology c. Education, Training and Awareness All of these measures have dynamic characteristics; as the organization's values, culture and technology base changes in time, these measures should be evaluated and re-assessed again. Before getting involved in technical details, the first step is to build a security policy. The security policy sets principles that drive all security related decisions. It is the base for all rules and practices governing the use, management, protection and distribution of sentitive information. The second security measure, technology, should be used (i) to enforce a predefined security policy and (ii) to monitor system and data use to detect policy violations. Intrusion detection systems (IDS) monitor computer and network usage for signs of policy violations. The system generates alerts (e.g. messages, SMS notifications, pager messages etc.) whenever an event that does not comply the predefined security policy is detected.

2 ID systems are classified in two major categories according to the event source of the system. In host based systems, host audit trails, system call traces and application logs are used as the event source. On the other hand, in network based systems, raw network traffic is used as the input of the system. Another common categorization of ID systems is according to the detection strategy. Anomaly detection consists of first establishing the normal behavior profiles of objects (e.g. users, systems, programs) and observing the actual behavior of those objects to detect significant variations of actual happenings from the established normal. Misuse detection consists of first collecting detailed information about actual attacks and codifying them into scenarios. Attack scenarios are than turned into machine readable form and the system tries to match events with predefined attack scenarios. Because of the ease of development and deployment, network based misuse detection systems are dominating the market today (Jackson 1999). 2. Ambiguities For Network Based Misuse Detection Systems The input collection mechanism of network ID sensors is similar to that of the more conventional network sniffers. Sniffers are programs which allow eavesdropping on computer networks, most common use of which is to debug networking problems. The data collection mechanism of network based ID systems is similar to that of sniffers. Network ID systems, as sniffers, are passive network components which only monitor traffic and do not involve in any communication activity on the monitored network; this is essentially a requirement for the system to be protected from attackers. Regardless of the working model all network based ID systems require a reliable traffic monitoring component as raw network traffic is the only input of the system. One of the basic requirements for any network based ID system is to be able to monitor communications correctly. Monitoring communication correctly involves examining each packet and deciding; (i) whether or not the packet will ever reach its ultimate destination, (ii) whether or not the packet will be accepted by the receiving host, (iii) what will happen on the receiving host if the packet is accepted and (iv) what will be the end state of the receiving host if the packet is accepted. The decisions made for these four issues by the network ID system may be so important that wrong decisions for even one of them may adversely affect system output. Keeping state of communications as a passive observing host has even more difficulties. Attempting to keep state of a communication which utilizes a non-trivial protocol is difficult because of two issues: 1. A robust and correct passive analyzer implementation for the protocol may be too difficult. The complexity of building a correct passive protocol analyzer is parallel to the complexity of building a correct protocol implementation. Past research on TCP implementation differences and correctness revealed that even protocols which have been in use for many years can have problems (Comer 1994, Guha 1995, Gebis 1998, Paxson 1997, Ptacek 1998, Paxson 1999, Floyd 2001). Many of the analyzed TCP implementations have had implementation problems some of which were serious. The case should not be any different for passive monitoring systems. 2. As the complexity of the protocol increase, major differences in protocol implementations increase as well. With such behavioral differences in different

3 implementations of the protocol, the network ID system should not only be able to keep state according to the written protocol specification, but also be able to keep state according to different implementations (even the broken ones). Such a monitoring requires not only extensive knowledge of different implementations of the same protocol but also which hosts on the network has which implementation in order to keep state as it is being done on those hosts. All of the highlighted difficulties bring ambiguities into the decisions of the ID system. In their groundbreaking paper, Ptacek and Newsham had given a list of ambiguities for passive protocol analyzers (Ptacek and Newsham 1998): a. IP TTL field may not be large enough for the number of hops to the destination b. Packet may be too large for a downstream link to handle without fragmentation c. Destination may be configured to drop source-routed packets d. Destination may time partially received fragments out differently depending on its operating system e. Destination may reassemble overlapping fragments differently depending on its operating system f. Destination host may not accept TCP packets bearing certain options g. Destination may implement PAWS and silently drop packets with old timestamps h. Destination may resolve conflicting TCP segments differently depending on its operating system i. Destination may not check sequence numbers on RST messages Ptacek and Newsham suggested that, if certain environmental information is given to the network ID system via some secondary channel, these ambiguities may be resolved. While researching a similar issue, the authors have found another ambiguity in the way different operating systems handle T/TCP traffic (Braden 1994). While some operating systems support T/TCP some others still do not support this protocol. While operating systems which do not support T/TCP at all discard all data included in the first SYN packet of a TCP three way handshake, T/TCP supporting operating systems use this data. This different treatment of some data causes an ambiguity for the network ID system; the system cannot decide whether the data in the first packet of a TCP three way handshake will be used or not. Correct handling of network traffic depends on the resolution of these ambiguities or else systems may generate wrong output in terms of both false negative (i.e. attacks being hidden from the system) and false positive (i.e. legitimate events to be flagged as intrusive) errors. 3. Resolving Ambiguities By Collecting Environmental Information If the ambiguities discussed in section 2 were resolved, the system output would have been much more dependable with less decision errors. The largest volume of ambiguities would be resolved by supplying host operating system information to the network ID system. To provide such environmental information to an ID system, one can either choose to do this via human operator support or via some sort of a learning approach. In most cases, it can be

4 realistically expected that, a human operator can provide much more information than a system can collect on its own. Exact network topology information is such a piece of information; although it is possible to gather information about network topology by programmed systems, it might not be possible if other network components (e.g. router, switches etc.) do not cooperate via an appropriate network management protocol such as SNMP. Hosts and their interactions with other networked systems change over time so that information provided to the network ID system even a short while ago might become outdated and appropriate updates may become a necessity. If such updates are not applied, the system may still be having the ambiguities. Even more, the information available at the system may become wrong (e.g. a particular host's operating system has changed) resulting in a misleading wrong output. Thus, if environmental information is to be provided to the ID system, in any way, timely updates of the information is of primary importance. Although human operators can provide the most detailed and correct information there is an inherent problem with human support; human may make serious mistakes by either forgetting to update a system or by providing wrong information. In both cases, the system output would be less dependable. Because of the risks imposed by human intervention, it is preferred to utilize automated systems that learn about their environment. To implement such an automated learning strategy, there are two alternative ways; active network mapping and passive network mapping. In active network mapping, a process which is started periodically sends query packets to all possible endpoint addresses and examines the reply messages. The process is active, because the mapping party actively involves in network communications, sending and receiving packets. It is possible to construct a network inventory including platform and service software information among with network topology information to some degree (Fyodor 1998, Goldsmith 1998, Arkin 2001). However, such a module, being active inherently is dangerous if it is to be run on the network ID system. Active participation of network ID system to communications is generally deprecated because of the risks involved. In passive network mapping, the network ID system, as part of its input acquisition procedure, may examine the captured packets and deduce some reasoning about one or both endpoints of the communication. Passive network mapping is done via examining various fields of packets. For example, by examining source IP address and TCP port number of SYN-ACK packets, one can build a list of open TCP ports on a particular host. As such, examining initial TTL values and TCP receive window sizes may reveal host operating system brand and versions as the default values of these two variables are different on different operating systems (Spitzner 2000, Vision 2001, Arkin 2001, Malan 2000, Smart 2000). As the process is totally passive, it is more appropriate to run as part of a network ID system. As only captured packets are examined, the process is not only passive but also non-intrusive. As passive network mapping is totally bound on the existance of communication, if one resource is not ever involved in any communication it will be totally hidden from the inventory built by the network ID system. If no packets from one resource is seen on the network by the ID system, no information about that ID system will be available and the ID system will be clueless about that particular resource.

5 4. Implementation of the Learning Concept A proof of concept module has been developed for the Snort Lightweight Network Intrusion Detection System (Roesch 1998). The proof of concept plug-in implements passive operating system fingerprinting to detect host operating systems on the monitored network. The business logic employed is a direct implementation of that of "p0f" tool by Michal Zalewski. The fingerprinting plug-in, when added to Snort, fingerprints hosts on the monitored network upon detection of the TCP SYN or TCP SYN-ACK packet of a TCP three way handshake. It examines various fields of a TCP packet to fingerprint hosts. A full listing of the fields examined are given in the list below: a. TCP Window Size: Different TCP implementations announce different TCP receive window sizes b. IP TTL: Different IP implementations have different initial TTL values c. TCP Maximum Segment Size: Different TCP implementations have different default maximum segment sizes d. IP Don't Fragment Flag: Some implementations use IP DF flag on SYN packets while some others do not e. TCP Window Scaling: Whether the sending TCP supports dynamic window scaling or not (if yes, the value) f. TCP Selective Acknowledge Okay: Some implementations allow selective TCP acknowledgements, while some others do not g. No operation: Some implementations set no operation on TCP SYN's, some do not The original p0f version 1.7 fingerprint database is extended with data collected from the rich and heterogeneous network environment of METU. Sample fingerprint entries from the database are listed below: win9x:8192:128:1460:1:0:1:1:windows 9x linux:512:64:1460:0:0:0:0:linux 2.0.x aix:16384:64:512:0:-1:0:0:aix 4.2 solaris:8760:255:1460:1:-1:1:1:solaris 2.6 / 2.7 The collected host operating system information is stored in a table structure in memory which is periodically checkpointed to disk for use in stop-start conditions. The entries in this table are periodically aged and removed. This feature supports running multiple operating systems on a single host; the ID system detects operating system change in at most agingperiod duration. Currently, the fingerprinting plug-in supports more than ten operating systems. 5. Eliminating False Positive Alerts By Using Destination Specifiers The collected operating system information can be used to resolve the ambiguities discussed in Section 2. As the resolution of the ambiguities is straightforward given that certain environmental information is available, it is not discussed in this paper.

6 It is possible to extend the misuse specification language of network ID systems to incorporate destination specific features such as operating system information. By extending existing signatures with additional qualifiers, the signatures can be turned into more specific forms which would help eliminate some of the false positive alerts. Adding operating system qualifiers to existing misuse signatures is expected to turn weak signatures to more stronger ones. The ArachNIDS open misuse signature database contains such weak misuse signatures (Vision 2000). Some sample rules in the form of Snort rules are given below: (1) alert TCP $EXTERNAL any -> $INTERNAL any (msg: "IDS442/rpcstatdx-exploit"; flags: A+; content: "/bin c74604 /sh";) (2) alert UDP $EXTERNAL any -> $INTERNAL any (msg: "IDS346/shell- NOOP-Sparc-udp2"; content: " a61c c013 a61c c013 a61c c013 a61c c013 ";) (3) alert TCP $EXTERNAL any -> $INTERNAL any (msg: "IDS283/shellcodex86-setuid0"; flags: A+; content: " b017 cd80 ";) All three rules do simple packet payload analysis to find the occurence of a pattern in the payload. However, no source or destination IP address and port number is specified so that any communication containing the pattern will be flagged as intrusive. The pattern lengths make the situation even worse; 10 bytes in the first, 16 in the second and a mere 4 in the third. In their current form, a large binary file transfer would trigger these rules. Addition of operating system qualifiers will render the rules to become stronger through the addition of another condition to be evaluated before an actual alert is triggered. The same three sample rules, this time with the proposed affected operating system qualifiers are given below: (1) alert TCP $EXTERNAL any -> $INTERNAL any (msg: "IDS442/rpcstatdx-exploit"; flags: A+; content: "/bin c74604 /sh"; osaffected: linux) (2) alert UDP $EXTERNAL any -> $INTERNAL any (msg: "IDS346/shell- NOOP-Sparc-udp2"; content: " a61c c013 a61c c013 a61c c013 a61c c013 "; osaffected:solaris) (3) alert TCP $EXTERNAL any -> $INTERNAL any (msg: "IDS283/shellcodex86-setuid0"; flags: A+; content: " b017 cd80 "; osaffected: linux,bsd,sco) A processor plug-in implementing the osaffected qualifier has been developed. The plug-in implementing the osaffected qualifier works together with the passive operating system detection plug-in presented in Section 4. The dynamically gained information about host operating systems is later used by the actual misuse detection engine to decide whether the destination of a particular attack is vulnerable or not. If the attack target is immune to that particular attack, the alert is simply suppressed. For example, if a particular attack affecting hosts running the Microsoft Internet Information Server is directed towards a host running a flavor of the UNIX operating system, the alert is suppressed. This is due to the a-priori knowledge that, Internet Information Server software is only available on Microsoft operating systems.

7 It is accepted that, total suppression of alerts going towards non-vulnerable hosts is not an acceptable strategy for intrusion detection and tracking at some installations. Some organizations may prefer to track even the least significant anomalous activity on their networks because of the security sensitivity of the information they process. To support such institutions, a slightly modified strategy can be utilized. Instead of totally suppressing alerts, the system can dynamically change the priority of the alert generated decreasing priority of alerts which are triggered for hosts which are non-vulnerable to the particular attack. Dynamic de-prioritization may still help the human operator by listing alerts in decreasing importance. Although such a de-prioritization strategy is not implemented in the prototype code, this effect can be achieved with small modifications. 6. Performance Measurements of Prototype Extensions Performance measurements of the two plug-ins, one for passive operating system fingerprinting and one for the affected operating system qualifier, has been done in two dimensions; (i) execution time overhead and (ii) alert suppression rates. The measurements has been done over three different datasets each collected from different networks and recorded to files. Two of the datasets were daily traffic from university environments; dataset-a was collected from a departmental network while dataset-b was collected from a Turkish University's only wide-area connection point. The third dataset, dataset-c consists of traffic from seven identical web servers in a single subnet which provide some sort of registration services to a large user base for a limited time of the year. For the measurements, all Snort rules in the ArachNIDS database are examined and osaffected option with the list of affected operating systems are added with the support of the ArachNIDS maintainer, Max Vision Execution Time Overhead Execution speed is critical for network ID systems for them to be able to cope with highspeed network traffic. Developers of network ID systems spend much time in selecting the best algorithms, system designs and implementation strategies for the systems to be able to handle traffic flows over modern high-speed networks with lower packet loss rates. Because of the existance of such a hard execution speed constraint on the design and development of network ID systems, every possible functionality to be added to a network ID system should be weighted carefully for its execution time costs. To measure the execution time costs associated with the two plug-ins introduced in Section 4 and Section 5, the GNU profiler (gprof) is used. Profiling information provided by gprof is used to measure time spent in various functions. The UNIX time tool is used to measure total execution times with and without the plug-ins. Dataset-C, which consists of over 25,000,000 IP packets is used as the test data for execution time measurements. Running Snort on Dataset-C without the plug-ins disabled takes on a certain Pentium based computer system, while running Snort over the same data with plug-ins enabled takes seconds; which is than verified with profiler the profiler report. This result can be interpreted as the plug-ins brought 6% execution time overhead to the baseline Snort system.

8 6.2. Alert Suppression Rate Alert suppression measurements has been done over three datasets in three runs. In the first run, Snort is run over a dataset without the passive mapping plug-ins. In the second run, Snort is run over the same data with the passive mapping plug-ins enabled. The second run output includes not only alerts and event logs but also information about number of hosts fingerprinted and hosts that couldn't be fingerprinted. Before the third run, all missing fingerprint entries from the fingerprint database are filled in and the perfect fingerprint database for that particular data is formed. Then Snort is run over the data using this perfect fingerprint database with the plug-ins enabled, as the third pass. This pass is required to measure best-case performance where full-knowledge of host operating systems is available. It should be noted that, such a perfect fingerprint database is not realistic outside of the laboratory environments. Running the three passes over dataset-a provided the results in Table 1. The "# of Alerts" column shows the number of alerts triggered during a particular pass of Snort over the data. The second column enlists the percentage of total alerts suppressed because of the affectedos plug-in. The third and fourth columns are used to list number of identified and unidentified host operating systems in the given pass. The last column is the ratio of number of identified operating systems identified over the total number of operating systems available. Table 1. Dataset-A Results (Start Time: 08/03/01-17:50; End Time: 08/05/01-12:49; Duration: 43:01 Hours; Host Count: 36) # of Alerts % Suppressed OS Ident OS Unident % Ident Pass N/A N/A N/A N/A Pass % ,6% Pass ,9% % Running the three passes in Dataset-B, which is a larger network with a more heterogenous structure resulted with the figure in Table 2. Table 2. Dataset-B Results (Start Time: 08/03/01-11:35; End Time: 08/06/01-08:57; Duration: 69:22 Hours; Host Count: 68) # of Alerts % Suppressed OS Ident OS Unident % Ident Pass N/A N/A N/A N/A Pass % % Pass ,2% %

9 Dataset-C contains traffic of a homogenous network of seven web servers all of which are running the same software suite with the same data; they are almost identical except their network addresses. Running Snort over Dataset-C resulted with the data in Table 3. Table 3. Dataset-C Results (Start Time: 07/12/01-10:32; End Time: 07/31/01-09:55; Duration: 455:23 Hours; Host Count: 7) # of Alerts % Suppressed OS Ident OS Unident % Ident Pass N/A N/A N/A N/A Pass ,4% % Pass ,4% % Examining the alerts suppressed by the system and the rules that would have triggered those alerts an interested issue has been noted. In most cases, the subject rules were the relatively weak ones and the use of the plug-ins turned them into more stronger ones eliminating false positives. In fewer cases, the use of the plug-ins resulted in elimination of alerts that would have been triggered because of attacks going towards non-vulnerable hosts. Such alerts, if fired, cannot be classified as false positives but still as the destination is non-vulnerable to a particular attack, suppression of de-prioritization of the alert is a desired feature. In the Snort rule base used for testing purposes, which is extended with the use of the osaffected keyword, none of the "probe" type of rules is marked with affected operating system information. A probe rule is a detection rule, which is used to identify information gathering attempts of an attacker. Usually, such probes are applied to a target before crafting an actual attack to enumerate the vulnerabilities of the target host or network. By sending a probe, the attacker gains knowledge of (i) whether or not the target is having some feature and (ii) various attributes of it if the feature is available. For example an attacker might be sending probes to identify whether a particular host runs a web server or not and if a web server is available the brand and version number of it. However, even the host is not running a web server, the probes provide some knowledge to the attacker. At least, by the probes attacker has learned that the target is immune to web server attacks. As probe rules provide information to the attacker in any case, virtually every operating system is affected from probes and suppression of probe alerts through the addition of affected OS qualifiers is not possible and not done on the rule base used for testing. The alerts generated for the three test datasets include between 45% and %70 percent probe alerts none of which could be suppressed because of the issue discussed in the previous paragraphs. The average alert suppression rate of the system in the best-case is around 10% with a low variance. This low variance is believed to be a coincidence. Tests revealed that system

10 performance varies heavily depending on the diversity of the environment and the services available on the network. 7. Summary and Results In this study use of passive network mapping to collect environmental information and the use of this knowledge to eliminate false positive errors is examined. Two plug-ins, one for passive operating system fingerprinting and one for utilizing this knowledge by extending the Snort rule grammer has been introduced with performance measurement results. The collected information can be used not only to eliminate some of the false positive errors but also to resolve some of the ambiguities involved in the passive monitoring approach of network ID systems. Improving the prototype plug-ins and integrating them to the mainstream Snort source tree is being considered by the core Snort development team. Addition of the plug-ins to the Snort distribution and extending signature databases to include affected OS information for misuse signatures would increase the output quality of this successful open-source ID project. References Arkin O., (2001), ICMP Usage in Scanning or Understanding Some of the ICMP's Hazards v3.0, On line article at Braden R., (1994), T/TCP: TCP Extension for Transactions Functional Specification, Internet Engineering Task Force, Request for Comments, RFC Comer D., and and Lin J., (1994), Probing TCP Implementations, In Proceedings of the USENIX Summer Conference. Floyd S., and Paxson V., (2001), Difficulties in Simulating the Internet, IEEE/ACM Transactions on Networking. Fyodor, (1998), Remote OS Detection via TCP/IP Stack Fingerprinting, On-line article at Gebis J., (1998), RATSPIT: Remote Automatic TCP Stack Probing and Inspection Tool, CS261 Final Report, University of California at Berkeley. Goldsmith D., and Schiffman M., (1998), Firewalking: A Traceroute-like Analysis of IP Packet Responses to Determine Gateway Access Control Lists, Technical Report, Cambridge Technology Partners. Guha B. and Mukherjee B., (1996), Network Security via Reverse Engineering of TCP Code: Vulnerability Analysis and Proposed Solutions, In Proceedings of the IEEE Infocom Conference. Jackson K. A., (1999), Intrusion Detection System Product Survey, Los Alamos National Laboratory, LA-UR R. Malan et. al., (2000), Transport and Application Protocol Scrubbing, In Proceedings of the IEEE Infocom Conference, 2000 Paxson V., (1997), Automated Packet Trace Analysis of TCP Implementations, In Proceedings of the ACM SIGCOMM Conference.

11 Paxson V., et. al., (1999), Known TCP Implementation Problems, Internet Engineering Task Force Request for Comments, RFC Ptacek T., and Newsham T., (1998), Insertion, Evasion and Denial of Service: Eluding Network Intrusion Detection, Technical Report, Secure Networks Inc. Roesch M., (1998), Snort - Lightweight Intrusion Detection for Networks, In Proceedings of the 13th LISA Conference of USENIX Association. Spitzner L., (2000), Know Your Enemy: Passive Fingerprinting, Online article at Smart M., et. al., (2000), Defeating TCP/IP Stack Fingerprinting, In Proceedings of the 9th USENIX Security Symposium. Vision M., (2000), ArachNIDS Misuse Signature Database, On line resource at Vision M., (2001), Passive Host Fingerprinting, On-line article at