Collaborative Intrusion Prevention

Transcription

1 Collaborative Intrusion Prevention Simon P. Chung and Aloysius K. Mok Department of Computer Sciences University of Texas at Austin Austin, TX, 78712, USA phchung, Abstract Intrusion Prevention Systems (IPSs) have long been proposed as a defense against attacks that propagate too fast for any manual response to be useful. In an important class of IPSs, the host-based IPSs, honeypots are used to collect information about attacks. The collected information will then be analyzed to generate countermeasures against the observed attack. Unfortunately, these IPSs can be rendered useless by techniques that allow the honeypots in a network to be identified ([1, 9]). In particular, attacks can be designed to avoid targeting the identified honeypots. As a result, the IPSs will have no information about the attacks, and thus no countermeasure will ever be generated. The use of honeypots is also creating other practical issues which limit the usefulness/feasibility of many host-based IPSs. We propose to solve these problems by duplicating the detection and analysis capability on every protected system; i.e., turning every host into a honeypot. In this paper, we will first lay out the necessary features of any scheme for such large scale collaboration in intrusion prevention, then we will present a framework called Collaborative Intrusion Prevention (CIP) for realizing our idea of turning every host into a honeypot. 1 Introduction It is a well recognized fact that the manual generation and application of patches are too slow to protect vulnerable systems from attacks, with the most cited demonstration of this fact being the outbreak of the SQLSlammer worm. In response, many have proposed intrusion prevention systems (IPSs) as a defense against this threat from fast-propagating attacks. One important class of IPSs is the host-based IPSs 1 The research reported here is supported partially by a grant from the Office of Naval Research under contract number N By host-based IPSs we refer to all IPSs that analyze how the target host processes attack traffic. (e.g. [5, 8, 2, 10, 13, 7]). Many existing host-based IPSs employ honeypots to detect and collect information about attacks so that countermeasures against the detected attacks can be generated 2. However, there are a few drawbacks in these systems: 1. As shown in [1, 9], the addresses of honeypots can be known to the attackers, allowing them to avoid interaction with the honeypots. As a result, the IPSs will be unable to collect any information about the new attacks, and thus no countermeasures will be generated. Furthermore, the heavy-weighted analysis commonly performed in the honeypots also makes them easy target of DoS attacks. In short, the honeypots have become a single point of failure in the IPSs that use them to collect information about attacks. 2. The great variety of OSs/applications, along with the many different versions of the same software running on different protected hosts is also creating some practical difficulties; a large number of honeypots may be needed so that attacks against hosts running rare OSs/applications, or a specific version of an application/library will be covered. This may amount to a non-trivial cost to set up and manage all these honeypots. 3. The passive nature of honeypots also makes them unsuitable for studying attacks that require some user interaction (e.g. a successful attack on web browsers usually requires the victim to visit some contaminated web pages). Even though this problem may be 2 For many host-based IPSs, it is infeasible to deploy the component for collecting attack information in production systems; the detection/analysis process either requires a recompilation or extensive instrumentation/modification of the underlying system (e.g. address space layout randomization [13, 7], selective transactional emulator [10]), or incur an extremely high performance overhead (e.g. dynamic tain analysis [5, 8], close-to-full emulation [2].). In other word, the systems responsible for the collection of attack information are effectively honeypots, systems with no value other than being attacked and compromised

2 alleviated by techniques like [12], we believe it is not trivial to generalize this technique. 4. To some system administrators, honeypots are still hazardous components to be avoided. As a result, the use of honeypots may harm the deployability of hostbased IPSs. The worries concerning the use of honeypots may be further deepened by the analysis performed in the honeypots after they are attacked: in many host-based IPSs, buffered attack packets are replayed in the honeypot in order to observe how a vulnerable system behaves before it is compromised. In this paper, we propose the collaborative intrusion prevention (CIP) framework as a solution to the problems that spawned from the use of honeypots. Under the proposed framework, every production system will be equipped to detect and analyze attacks, and to generate and distribute countermeasures against them. In other word, every host under the collaborative intrusion prevention framework will play both the role of a protected system and that of the honeypot in traditional host-based IPSs. With every host being effectively a honeypot, we will have a large number of duplicates for the honeypot. As a result, the honeypot will no longer be a single point of failure in the IPS; the attacker can not evade the IPS by avoiding certain systems in the target network. Attempt to compromise any protected host may lead to the detection of the attack, and the production of countermeasure against the attack. Furthermore, each attempted attack will have the same chance of being detected. Thus, the more attack attempts are made, the higher the probability that a countermeasure will be made available to stop the attack. The threat of DoS against the honeypots also ceases to exist: a DoS that blinds our scheme will also make the target systems unavailable for being compromised. Finally, with every host being capable of detecting and generating countermeasures against attacks, the diversity in the software being run on the protected hosts is a much lesser issue. Even though rare applications are covered only by the few hosts running them, they are still not completely unprotected; on the contrary, in traditional IPSs, if there are no honeypots dedicated for these applications, attacks against them will go unchecked. 2 Desirable Features Before presenting the details about the proposed CIP framework, we will lay out a set of features necessary for any IPS that distributes the detection and countermeasure generation capabilities over all protected systems: 1. Low Cost: In order to have a large number of participating host (i.e. honeypots for detecting and analyzing attacks), the detection/analysis components executed on participating hosts should have minimal installation and running cost. Furthermore, each participant should be able to determine the amount of overhead incurred. 2. Egocentric Participants: In addition to having low installation and operating cost, a good incentive should also be provided to encourage participation. In particular, the cost incurred on participating hosts should produce direct improvement on its security status, in addition to the indirect benefit of receiving countermeasures generated by other hosts. Furthermore, the amount of contribution made by a host to the collaborative scheme as a whole should be proportional to the direct gain in security obtained by that individual host. 3. Simple Collaboration Scheme: The participating hosts should collaborate in a loosely-coupled manner that requires minimal, decentralized management. A loosely-coupled collaboration will not only reduce the cost of participation, but also improve the robustness of the scheme, while decentralized administration of the collaboration is necessary to avoid introducing a single point of failure in the scheme. 4. Fault/Traitor Tolerant: We accept that some (hopefully very few) hosts will be compromised before a countermeasure against a new attack is available. Thus, the collaborative scheme should be designed so that it won t be completely defenseless no matter how many hosts have been compromised; the remaining hosts should have a non-zero probability of outputting effective countermeasures to stop the attack. Furthermore, the scheme should also be robust against the malicious behavior of the compromised host. In particular, the countermeasures output by compromised hosts should not have significant adverse effect on the uncompromised ones, and hosts that output large volume of bogus countermeasures should be identified. 3 Collaborative Intrusion Prevention At the core of the CIP framework lies the randominspection-based intrusion detection proposed in [3]. By random-inspection-based intrusion detection, we refer to a class of intrusion detection techniques based upon the monitoring mechanism proposed in [3], and the IDS presented in [3] is only one instance in the class. As we argue in the next section, almost all random-inspection-based detection technique are suitable for our proposed framework. In addition to detecting attacks, the random-inspectionbased intrusion detection is also used in the CIP framework to patch all protected systems. In fact, the main focus of

3 the countermeasure generation process is to extract information that helps other hosts to detect the observed attack. Before we present the details of this countermeasure generation process, we will first have a brief review on randominspection-based intrusion detection. º½ ÖÓÙÒ Ê Ò ÓÑ ÁÒ Ô Ø ÓÒ Random inspection (proposed in [3]) is a mechanism for observing the behavior of user processes at random points during their execution. It is based on the hardware facility called the performance counter (which can be found on many CPUs) that generates a performance-counter interrupt every k instructions executed in the user space. To perform truly random inspection, we reset k to some random value after each interrupt. In the following discussion, we assume each thread has a separate performance-counter value (i.e. the number of instructions before the next performance counter interrupt); and this value is saved/restored at thread context switches. In a random-inspection-based IDS, we observe the behavior of the current process at every performance-counter interrupt to determine if it has been compromised. We call each of these occasions where we perform the intrusion detection an inspection point, and the average frequency at which inspections occur the inspection frequency. In the prototype system in [3], we simply check at each inspection point whether the currently executed instruction lies in code space or data space; any execution of data will be considered as an intrusion. Obviously, this prototype can only detect injected code attacks. However, we argue that other classes of attacks can also be detected with randominspection-based IDSs. For example, we are currently extending our prototype to detect existing-code attacks (a.k.a ret-to-libc attacks) by checking the stack contents at each inspection. Note that the non-deterministic manner in which inspections are performed means intrusion detection in a randominspection-based IDS is inherently probabilistic. Even though some attacks can be detected with certainty, there is generally a non-zero probability that the IDS will fail to detect an attack, especially when the IDS is operating under a low inspection frequency. As shown in [3], if inspections occur (on average) every k instructions executed, and the attacked process executes y instructions in an illegal state (a state that allows a positive detection should an inspection occurs), the probability that the attack will be detected is Ý È ½. What makes the random-inspection-based intrusion detection suitable for our CIP framework is its adjustable performance overhead. The overhead of performing random-inspection-based intrusion detection can be controlled through the inspection frequency; as shown in the experiments in [3], the lower the inspection frequency, the smaller the performance impact will be. The probabilistic intrusion detection mechanism also provides an elegant way for hosts to collaborate. By simply performing the randominspection-based detection, a host can improve the overall probability of detecting an attack; if n hosts are participating in the collaboration, the detection probability with all Ý hosts considered as a whole becomes ½ Ò (as opposed to the È given above). º¾ ÓÙÒØ ÖÑ ÙÖ Ò Ö Ø ÓÒ As mentioned before, the main function of countermeasures under our framework is to improve the protected hosts ability to detect imminent attacks with little extra performance overhead. This goal can be easily achieved by broadcasting the last inspection point before an attack is detected. In the future, whenever the execution in a protected host reaches this point, the host will perform randominspection-based intrusion detection at a higher frequency (as we ll argue in the next section, this will significantly improve the chance of stopping the attacks targeted by the countermeasure). If no intrusion is detected after a fixed number of inspections, the inspection frequency will return to its normal value. In summary, a host under the CIP framework works as follows: 1. Detection: Before any attacks detected or countermeasures received, the host performs random-inspectionbased intrusion detection at a low frequency ½. After each inspection, if no intrusion is detected, the identity of the inspection point, as well as the new random value in the performance counter will be recorded, and separate records will be kept for each thread in the system. 2. Countermeasure Generation: Once an intrusion is detected, the record for the previous inspection point and the saved performance counter value of the thread involved are retrieved. These are basically all the information needed for a countermeasure against the detected attack. However, the information retrieved may be refined to make the countermeasure more portable among hosts. For example, the virtual address of the last normal inspection point may be expressed as the name of the containing module and the offset from the beginning of the module. After all the necessary refinement, the countermeasure will be distributed to all other hosts. 3. Patching: Upon receiving the countermeasure from another host, the last normal inspection point identified will be extracted, and a breakpoint will be inserted at the corresponding instruction so that an exception will

4 be generated every time it is executed. A record will also be established to associate the marked instruction with the performance counter value, k, given in the countermeasure. 4. Stopping Attacks: When a thread executes any instruction marked in the patching stage, the execution of the thread will enter the alert mode. Randominspection-based intrusion detection will then be performed at a higher frequency ¾. The execution in alert mode lasts for ½ instructions (i.e. it ends after ¾ ¾ ½µ inspections), where k is the performance counter value associated with the instruction that triggers the alert mode execution. When the alert mode execution ends, the inspection frequency of the involved thread will return to ½. In addition to performing random inspection at higher frequency, the execution in the alert mode can also take measures to facilitate the recovery should any attack be detected. For example, modifications to critical resources can be delayed or redirected, and only be committed if no intrusion is detected during the alert mode execution (in [6], these are called cordoning-in-space and cordoning-in-time respectively). One important observation about the proposed countermeasure generation process is that it can be used with virtually any random-inspection-based detection technique. As long as the analysis performed at each inspection point is self-contained and does not depend on the results of previous inspections, the last normal inspection point and the saved performance counter value can be used as an effective countermeasure to enhance the defense on other systems. In other word, our framework is not tied to any particular analysis performed at the random inspection points, but is suitable for any detection system that uses random inspection to monitor processes. º ÀÓÛ Ó Ø ÓÙÒØ ÖÑ ÙÖ ËØÓÔ ØØ Now, let s consider the effectiveness of the countermeasures generated. Essentially, the countermeasures in our framework identify execution points reached before the actual attacks occur. Let s assume for the moment that all exploitations of a particular vulnerability will bring the victim process to the last normal inspection point given in the countermeasure before the control hijacking occurs. Furthermore, the way we determine the duration of the alert mode execution guarantees that the attacked thread will be executing in alert mode when the control hijacking occurs. Thus, it seems the high frequency inspection will very likely detect the attack concerned. However, note that we do not know how often the attacked thread will be in an illegal state (a state where the intrusion is detectable) while being in the alert mode. In the worst case, if the detection which produced the countermeasure occurs at the very first moment that the attack executes in an illegal state, and if the execution that follows quickly leaves this state, the alert mode execution will have virtually no chance of detecting the attack. However, if we assume the attacked process will remain in the illegal state during the first ½ ¾ (or more) instructions executed following the control hijacking, detection in the alert mode is guaranteed. From our analysis in [3], it seems not uncommon for attacks to have such an initial period in which they are very vulnerable to detection. Thus, the generated countermeasures should be effective in stopping attacks. Now let s revisit the assumption that all exploitations of the same vulnerability will bring the execution to the last normal inspection point. We believe this assumption should be valid most of the time, especially if the execution point concerned is close to the point of control hijacking. Even if the above assumption is not true, there should be very few execution paths that can lead to a successful exploitation of any given vulnerability. As a result, a few countermeasures, each with a last normal inspection point on a different execution path will be sufficient to stop all attacks targeting a particular vulnerability. 4 Evaluation After giving the details of the proposed framework, we will evaluate it against the list of desirable features given in Sect. II: 1. Low Cost: To participate under the CIP framework, a host needs three pieces of software: the randominspection-based IDS, the patching component, and the facility for broadcasting (and receiving) countermeasures. While the broadcasting facility is pretty standard among host-based IPSs, we will focus on the first two components. The IDS under our framework is basically a simple device driver to start up the random inspection. Due to the self-containedness requirement, we expect the code for the analysis at inspection points to be very simple also. Similarly, the patching component is a light-weight device driver that looks for the modules that need to be patched, inserts the breakpoints, and handles any resulting breakpoint exceptions. In terms of the running cost, it is dominated by the performance overhead incurred by the intrusion detection, and can be limited to any chosen level by keeping the inspection frequency sufficiently low. 2. Egocentric Participants: Under the CIP framework, the only altruistic behavior required of a host is the distri-

5 bution of the generated countermeasure, which only incurs a minimal cost. For the other activities performed on behalf of the collaboration, the participating hosts can benefit directly from them; both the intrusion detection and the countermeasure generation help the host better defend itself. In fact, the overhead incurred from the intrusion detection (which is the major running cost for participants) is directly related to the chance that a host will stop an attack targeting itself. 3. Simple Collaboration Scheme: Except for the distribution of generated countermeasures, hosts participating in the collaboration do not need to communicate with each other. All hosts can perform exactly the same task of random-inspection-based intrusion detection independently, and yet their contributions will add up to improve the chance of detecting and stopping an attack. In other word, the collaboration requires no management except for maintaining the mechanism for distributing countermeasures; there is no need to allocate/synchronize the work to/from different hosts. In fact, as far as the countermeasure generation process is concerned, there is no need for maintaining any membership information about the collaborating hosts. 4. Fault/Traitor Tolerant: The extremely loosely-coupled collaboration scheme means that losing a small portion of hosts will have very minor effect in the effectiveness on the whole. In fact, the loss of any number of hosts will not render the remaining hosts completely defenseless, only the probability of stopping an attack is affected. Furthermore, if the underlying detection mechanism has zero false positive rate, bogus countermeasures generated by a compromised host can do no more harm than incurring extra performance hit on other hosts. Of course, a large volume of such bogus countermeasures can still lead to a DoS on all the participating hosts. We will leave the defense against the misbehavior of compromised hosts as future work. 5 Related Work º½ ÔÔÐ Ø ÓÒ ÓÑÑÙÒ ØÝ Similar to the CIP framework, the Application Community scheme in [11] distributes the work of detection and countermeasure generation over a community of hosts running the same application. As opposed to our looselycoupled, minimal-management collaboration scheme, the Application Community employs a static division of labor; all functions executed by the protected application are identified, and each participating host is assigned to monitor a fixed set of functions. We believe this static work allocation in [11] makes the Application Community scheme vulnerable to very similar problems addressed in this work. First of all, if the attackers can identify the hosts responsible for monitoring the vulnerable functions exploited by their attack, they can evade the Application Community scheme just as they evade the traditional host-based IPSs. By measuring the time it takes different hosts to process requests that utilize different functions, it is not impossible for the attacker to identify a superset of all hosts monitoring the target function. Furthermore, since the work discovery is done on a per-application manner, the Application Community may also have poor scalability; multiple communities have to be set up and managed, and each host has to join multiple communities so that all its applications are protected. Finally, the Application Community appears to be a very altruistic scheme; with vulnerabilities usually found in the few rarely executed functions, we believe many hosts under the Application Community scheme are monitoring functions that will never be the target of any attack. In other word, very few hosts will directly benefit from the monitoring task performed for the Application Community scheme. On the contrary, the CIP framework proposed does not have any of the above problems, as we have shown in the previous section. º¾ Ä Á Ë»ÄÁ Ë Our work also borrows heavily from the LAIDS/LIDS framework in [4]. Both the CIP framework and that in [4] employs self-contained analysis to perform intrusion detection, so that the same mechanism can be used for both detection and patching. As a result of this similarity, the countermeasure generation algorithms of the two frameworks are almost the same. However, the detection used in [4] incurs a high, untunable performance overhead, which makes it impossible to duplicate the detection and countermeasure generation capability on production systems. The LAIDS/LIDS framework also provides no easy way of dividing the detection and countermeasure generation task so that the performance overhead can be amortized among multiple hosts. Thus, the major contribution of the CIP framework over the work in [4] lies in the introduction of the randominspection-based IDS, which allows hosts to perform intrusion detection at any chosen level of performance overhead, as well as an easy way of dividing and combining the task of intrusion prevention among hosts. 6 Conclusions In this work, we argue that the use of honeypots to collect information about attacks is creating a single point of failure in traditional host-based IPSs: once the honeypots are identified, attackers can make the IPSs blind to their

6 attack by avoiding interactions with the honeypot. The heavyweight analysis usually performed in these honeypots also make them easy target for DoS attacks. Furthermore, though dedicated honeypots are suitable for collecting information about attacks against critical network services, it appears infeasible to scale up the approach to cover the great variety of applications running in an entire network. To overcome the above difficulties, we proposed the Collaborative Intrusion Prevention (CIP) framework, under which every host is capable of detecting attacks and generating countermeasures against them. In other word, every host under our framework acts both as the honeypot and a protected system in traditional host-based IPSs. Furthermore, by using random-inspection-based intrusion detection techniques, the CIP framework allows hosts to collaborate in intrusion prevention with minimum management. Except for the distribution of countermeasures, there is no need for hosts to communicate with each other, nor is there the need to keep track of all the participating hosts. Each host under our framework performs exactly the same intrusion detection and countermeasure generation task independently, yet, the contribution from each host will automatically add up to improve the system s overall defense against new attacks. With this massive duplication of the honeypot and the simple collaboration scheme, there is no single point of failure in our system. The system will remain functional (though less effective) with the failure of any number of hosts. Finally, the CIP framework also provides very good incentive for hosts to participate; the performance overhead suffered from performing random-inspection-based detection will strengthen the defense of a participant, in addition to improving the security of all the other hosts. References [6] R. Hu and A. K. Mok. Detecting unknown massive mailing viruses using proactive methods. In Proceedings of 7th International Symposium on Recent Advances in Intrusion Detection (RAID), France, Sept [7] Z. Liang and R. Sekar. Fast and automated generation of attack signatures: A basis for building self-protecting servers. In Proceedings of the 12th ACM Conference on Computer and Communication Security (CCS 05), Virginia, Nov [8] J. Newsome, D. Brumley, D. Song, J. Chamcham, and X. Kovah. Vulnerability-specific execution filtering for exploit prevention on commodity software. In Proceedings of 13th Annual Network and Distributed System Security Symposium (NDSS 06), San Diego, Feb [9] M. Rajab, F. Monrose, and A. Terzis. Fast and evasive attacks: Highlighting the challenges ahead. In Proceedings of 9th International Symposium on Recent Advances in Intrusion Detection (RAID), Hamburg, Sept [10] S. Sidiroglou, M. Locasto, S. Boyd, and A. Keromytis. Build a reactive immune system for software services. In Proceedings of the USENIX Annual Technical Conference, 2005, Apr [11] S. Sidiroglou, M. Locasto, and A. Keromytis. Software selfhealing using collaborative application. In Proceedings of 13th Annual Network and Distributed System Security Symposium (NDSS 06), San Diego, Feb [12] Y. Wang, D. Beck, X. Jiang, R. Roussev, C. Verbowski, S. Chen, and S. King. Automated web patrol with strider honeymonkeys: Finding web sites that exploit browser vulnerabilities. In Proceedings of 13th Annual Network and Distributed System Security Symposium (NDSS 06), Feb [13] J. Xu, P. Ning, C. Kil, Y. Zhai, and C. Bookholt. Automatic diagnosis and response to memory corruption vulnerabilities. In Proceedings of the 12th ACM Conference on Computer and Communication Security (CCS 05), Virginia, Nov [1] J. Bethencourt, J. Franklin, and M. Vernon. Mapping internet sensors with probe response attacks. In Proceedings of The 13th USENIX Security Symposium, Aug [2] D. Brumley, J. Newsome, D. Song, H. Wang, and S. Jha. Towards Automatic Generation of Vulnerability-Based Signatures. In Proceedings of The 2006 IEEE Symposium on Security and Privacy, Oakland, May [3] S. P. Chung and A. K. Mok. On random-inspection-based intrusion dectection. In Proceedings of 8th International Symposium on Recent Advances in Intrusion Detection (RAID), Seattle, Sept [4] S. P. Chung and A. K. Mok. The laids/lids framework for systematic ips design. In Proceedings of the Fourth IEEE International Information Assurance Workshop (IWIA 2006), UK, Apr [5] M. Costa, J. Crowcroft, M. Castro, A. Rowstron, L. Zhou, L. Zhang, and P. Barham. Vigilante: End-to-end containment of internet worms. In Proceedings of 20th ACM Symposium on Operating Systems Principles, Brighton, Oct 2005.