Relative Entropy-based Filtering of Internet Worms by Inspecting TCP SYN Retry Packets

Transcription

1 Relative Entropy-based Filtering of Internet Worms by Inspecting TCP SYN Retry Packets Byungseung Kim and Saewoong Bahk School of Electrical Engineering and Computer Science Seoul National University, Seoul, Korea {kbs, Abstract Although many defense techniques against scanning worms have been developed, they have difficulty in ingress filtering if the incoming scanning traffic has insufficient intensity, which is usually the case. To make matters worse, legitimate Internet services behaving like worms and dynamic network environments undermines the efficacy of the techniques. In this paper, we propose a simple and efficient defense algorithm against Internet scanning worms that has high detection rate and low false positive rate. Our defense algorithm observes the protocol behavior of TCP SYN retries and applies a relative entropy scheme, which is used for measuring the distance between two distributions, to process the collected information. It builds up the black-list to isolate detected hosts from the Internet, and adjusts related parameters adaptively according to the observed traffic. Moreover, it acquires the simplicity and effectiveness at the ingress point only by inspecting SYN retry on a unidirectional link, which makes the defense mechanism easily applicable to a network. Against real-life traces, we investigate the performance of our algorithm and compare it with that of SNORT. The results manifest clearly that our algorithm outperforms the rate-based detection technique in terms of detection rate, detection speed and false positive rate. 1 Introduction Recently, worm epidemics have become a grave concern by demonstrating their formidable power to incapacitate various internet services and exhaust network resources. For instance, CodeRed, Blaster, Nimda, and SQL Slammer worms inflicted huge economic and social damages, and their mutations are still threatening the Internet environment. A distinct feature of the worms is their selfpropagation behavior that is enabled by fast and automatized scanning for possible victims. They can even spread globally in just a few minutes [5]. Most worm detection algorithms commonly watch the port-scanning behavior in which an infected host attempts to request far more new connections than a legitimate host would [2, 12]. Scanning worms target several specific service ports that are known to be vulnerable to bufferoverflow. In this process, they cause high rate of failed connections. These can be verified by gathering ICMP unreachable messages in the monitored network [9] or analyzing highly anomalous traffic relative to the common traffic distribution [3, 4, 15]. Moreover, since they frequently use uniformly distributed IP addresses as their target hosts in random IP scanning for finding vulnerable hosts, they expose some specific packet flows between the monitored network and the Internet. Due to these idiosyncrasies, scanning worms can be detected [1]. However, there are some complications in detecting scanning worms. First, it is not easy to determine the threshold over which the suspicious behavioral patterns described above are positively identified. Second, some internet services that show similar behaviors to the worms are likely to incur a lot of false positives. For instance, a P2P client often behaves like a scanning worm when searching for P2P servers. Third, most detection algorithms have difficulty in ingress filtering that is very important in protecting a target network because even an inner infection makes worms propagate all over the target network quickly by local scanning. Scanning worms does not target a single network but many networks in a randomly local and global manner. Therefore, the scan rate of a worm observed at the ingress point of a network is only a tiny fraction of that at the scanning source. In this paper, we propose an effective defense algorithm against intrusion of scanning worms, named Relative Entropy based Filtering (REF). It achieves high detection rate and low false positive rate even when the scanning traffic from worms is mingled with legitimate scanning-type flows such as P2P traffic. Moreover, our algorithm can be easily

2 implemented on the top of any existing IDS based network thanks to its simplicity. 2 Failed connection request A TCP-based scanning worm attempts connection via the 3-way handshake to some specific ports of vulnerable hosts before transmitting its malicious code to them. While it probes target IP addresses by using local and global random scans without prior knowledge about vulnerable hosts, it receives some negative responses from them whose examples are notification messages such as ICMP-T3 and RST for connection failure. Therefore, it can be an efficient way to inspect failed connection-requests in detecting scanning worms. 2.1 Investigation of failed connectionrequests from a real-life trace An intermediate router that receives a SYN packet from a disabled host which IP address is not allocated or network is down sends an ICMP-T3 message as a negative response to the sender, or silently drops the SYN packet. If an enabled host receives a packet destined to its unavailable service port, it also returns a negative response or nothing according to its network configuration, which is commonly tuned by OS s default setting. Then, if the sender receives negative responses or nothing within Retransmission Time Out (RTO), it attempts to retransmit the SYN packet immediately or after a given amount of back-off time. From a real-life Internet packet trace, we measured the ratios of the number of packets of each response type to SYN packets on port 80 (Web) and port 135 (W32.Blaster/Gaobot), which represent a normal service port and a scanned port, respectively. The trace used in our experiments have been collected on a bidirectional link of the boarder gateway between an AS network and the Internet. Table 1 summarizes specifications of our real-life traces. There are four kinds of responses to a SYN packet: SYN/ACK, RST, ICMP, and non-response. As shown in Fig. 1 (a), SYN/ACK packets account for the majority of the responses to SYN packets sent to port 80 by legitimate hosts. On the contrary, in Fig. 1 (b), no-responses account for 88.6% of total responses, whereas negative responses, RST and ICMP, account for 10%. Consequently, two statistics illustrated in Fig. 1 manifest that a considerable number of routers and hosts do not respond to SYN packets that they cannot process. In other words, it is hard to detect failedconnection requests at a gateway by observing negative responses. Table 1. A summary of the captured traces by TCPDUMP Trace IP Block Capturing Time Size/Avg. BW 03 Ajou /19 network 11:37, July 3, Gbyte/ /20 network 13:00, July 4, Mbps 04 Ajou /19 network 23:15, July 28, Gbyte/ /20 network 23:42, July 29, Mbps SYN/ACK (76.5%) Non-Response (18%) (a) Port 80 (Web) - responses Non-Response (88.6%) RST (4.05%) ICMP-T3 8 ( %) ICMP-T (1.47%) SYN/ACK 4175 (0.977%) RST (4.15%) ICMP-T11 ICMP-T (4.48%) 7500 (1.76%) (b) Port 1025 (W32.Blaster) - responses Figure 1. Relative ratios of each response type to outgoing SYN packets in 04 Ajou trace. 2.2 Efficient method to find failed connection-requests The retry for the failed connection-request is persistently attempted at least twice whatever returns from the destination. Most legitimate connections succeed in the first connection request [6]. Therefore, investigating retries can be a simple way to finding failed connection requests. We utilizes the 32-bit initial TCP sequence number (ISN) in TCP field for that. Its increment of 1 every 0.1 second gives uniqueness and randomness to the SYN packet [14], and all retry SYN packets for a connection request have the same ISN. Comparing ISNs of observed SYNs, we can find out all the failed-connection requests without maintaining the state table of 4-tuple. We use a hash-table to record and compare ISNs of observed SYNs.

3 3 Relative Entropy based Filtering As described in previous section, observing retry SYNs is an efficient approach to finding failed connectionrequests. So we use this approach for anomaly detection against scanning worms. In this section, we propose a relative entropy based packet filtering scheme that uses the information about retry SYNs. 3.1 Relative Entropy based Filtering Rank of destinations inducing retry SYNs First, we lay a theoretical basis to utilize the relative entropy scheme for detecting worms. The main difference between retry SYNs incurred by normal services and those incurred by worms comes from the dispersion degree of their DIPs. If a popular server such as WEB and P2P servers reaches a capacity limit or experiences some problems, it will reject connection requests from legitimate hosts. In this case, many retry SYNs will be generated towards the server. Therefore, their DIPs form a low degree of dispersion, and make a difference in the number of their inducing retry SYNs. On the contrary, DIPs of retry SYNs sent by scanning worms form a uniform distribution because they are generated randomly or sequentially towards vulnerable hosts. This phenomenon of retry SYNs can be modeled by Zipf s law, which is usually used to analyze naturally occurring phenomena [10]. It usually refers to the occurrence size of some event relative to its rank k as f(k : s, N) = 1/k s P Nn=1 1/n s, (1) where N is the number of elements, k is their rank, and s is the exponent characterizing the distribution. In our algorithm, letting N and k be the number of destinations inducing retry SYNs and their rank, respectively, f(k : s, N) will then be the fraction of retry SYNs towards to the k th heaviest server. To verify that the behavior of retry SYNs fits well with Zipf s law, we ranked the DIPs according to the number of sources sending retry SYNs to them. All the retry SYNs from a source to a destination were counted as a single retry SYN in order to exclude DoS attack and retry SYNs after the first retry. That is, all retry SYNs having the same source IP, destination IP, and destination port are counted as one retry SYN. Fig. 2 depicts the rank-distributions of hosts inducing SYN retries on normal and scanned ports. As shown in Fig. 2 (a), the rank-distributions on scanned ports have the uniform distribution, but those on normal ports except port 80 do not. In Fig. 2 (b), the rank distributions on normal ports except port 80 are fitted into the CDF of Zifp s law of which the parameter s is in [0.5, 1]. CDF P2P (port 8404) SMTP (port 25) Web (port 80) Worm (port 135) Worm (port 1025) Rank (a) Comparison of rank distributions for scanned and normal ports in 04 Ajou trace CDF SMTP (port 25) P2P (port 8404) Zipf's law (s=1) Zipf's law (s=0.5) Rank of destinations (b) Matching the rank distribution by Zipf s law Figure 2. Measured rank distributions. The distribution on Web port 80 seems to be formed by scanning worms. This is because retry SYNs from scanning worms are observed considerably more than those from legitimate Web clients. We found a lot of retry SYNs sent to port 80 of many DIPs which were not registered on a DNS server and did not provide any Web services. Thus we inferred that they were generated by random or local scanning of worms. We will analyze the traffic on port 80 in detail in Section Relative Entropy Likelihood Ratio There are some caveats in utilizing the rank-distribution and the DIP dispersion of retry SYNs. First, it is not straightforward to adjust the threshold and form an exact shape of the distribution for scanning detection on a port. So it is hard to go beyond telling whether scanning worms exist on the port by these methods. Second, it is impossible to block malicious traffic on a port although high entropy is measured because it can incur the rejection of many normal services. Therefore, we should be able to find packets of infected worms for dropping. To do so, we propose an algorithm utilizing the relative entropy scheme. The basic idea is to compare the measured rankdistribution with the normal and scanning rankdistributions, and to determine whether Internet worms target the monitored port. The purpose of comparison is to obtain the relative entropy which is a measure of the distance

4 between two distributions. Let D j = {d 1, d 2, d 3,..., d N } denote the j th sample set of destinations ranked by the number of their inducing SYN retries n(d k ). This sample set is obtained at the end of each sampling period where the number of observed retry SYNs reaches a given threshold, which we define as M j (= P N k=1 n(d k )) for the j th sampling period. Then we calculate the relative probability of having rank k destinations in a given sample set as p(d k ) = n(d k ) P Nn=1 n(d n ). (2) Let q norm (d k ) and q scan (d k ) represent the probability mass functions of the normal and scanning rank-distributions that are modeled by the Zipf s law and the uniform distribution as 1/k s q norm (d k ) = P and Nn=1 1/n s q scan(d k ) = 1 N, (3) respectively. If the assumed the distribution is q when the measured distribution is p, we can express the relative entropy D(p q) as D(p q) = NX k=1 p(d k ) log p(d k ) q(d k ) This entity measures the inefficiency of assuming that the distribution is q when the true distribution is p. [11]. The relative entropy is always non-negative, and zero if and only if p = q. If two distributions are similar, the relative entropy approaches zero. Now, we define the relative entropy likelihood ratio (RER) of the j th sample as (4) RER j = log D(p qnorm) D(p q scan). (5) If D(p q norm) > D(p qscan), the conjecture of p qscan is more reasonable than that of p q norm. Therefore, if RER j > 0, the measured distribution p is originated from scanning worms. Otherwise, it is originated from intermittent normal services. Essentially, our algorithm combines the rate-based approach with the relative entropy scheme. It counts the number of retry SYNs sent from the Internet to the target network. If the number of observed retry SYNs reaches M j in the j th sampling period, it forms the rank-distribution described above and we calculate RER j. RER renders our algorithm determine whether a measured rank-distribution is originated from legitimate services or not, even if the distribution does not fit the Zipf s law exactly. In obtaining the rank-distribution, most legitimate flows that are not incurring failed connection requests are excluded from sampling since they rarely generate retry SYNs. So the behavior of retry SYNs from scan traffic can be seen clearly, and our algorithm can detect scan traffic even if normal flows overwhelm scanning flows Build the black and white list Our defense system, REF, records infected hosts found by using the RER on the black-list and maintains the list to control the inflow of scanning from them. If a port is detected by RER analysis, it can be appropriate to drop or limit connection requests on it to prevent worms from propagating in the target network. However, if the detected port is a well-known service port such as port 80 (Web) and port 22 (SSH), even the rate-limiting can unintendedly cause a denial-of-service attack by itself. Legitimate servers commonly have n(d k ) > 1 because retry SYNs generated by legitimate clients head for them. On the contrary, scanning worms induce their target to have n(d k ) = 1 since each of them randomly generates its target IP on each scan attempt. Therefore, in a scanned period (RER j > 0), lower ranked destinations are created by infected hosts generating retry SYNs while higher ranked destinations by legitimate hosts. That is, if REF perceives the existence of scanning in a period, it detects a source inducing a ranked destination with n(d k ) = 1 as an infected host, and record it on the black-list. Other sources sending retry SYNs to a ranked destination with n(d k ) > 1 in the scanned period and all the observed sources in the normal period (RER j < 0), are regarded as legitimate ones and not recorded on the black-list. However, in case that just one legitimate source sends retry SYNs to a destination in the scanned period, it can be mis-detected as a scanning worm because its destination has n(d k ) = 1. Therefore, REF also maintains a white-list as a complement to the blacklist where it keeps high ranked DIPs as active servers in the target network. Then, even if a legitimate source induces a ranked-destination to have n(d k ) = 1 in a scanned period, it can avoid being misconceived as an infected host if its destination is on the whitelist. Specifically, the white-list updates the ranked destinations having n(d k ) > 1 in the normal and scanned periods. It operates well in managing some private or group servers such as P2P which may cause rank change so often. 4 Performance Evaluation In this section, we evaluate the performance of our defense system. For that, we executed REF and two portscan detection modules of SNORT against our real traces and compared their performances. In our experiment, we found many kind of worms in 04 Ajou trace, but two kind of worms, CodeRedII and Nimda, in 03 Ajou trace. Therefore, we mainly used 04 Ajou for the performance evaluation.

5 Table 2. Result of REF in 04 Ajou trace Port (Kind) Total SYN Retry SYN Worm Norm Black-list Higher/Lower rate FN (rate) FP (rate) 2745 (W32.Beagle) 13,917 8,329 1, , / (0.09) 0(0) 1025 (W32.Spybot) 7,990 4, / (0.16) 0(0) 6129 (W32.Mockbot) 49,413 25, / (0.16) 0(0) 135 (W32.Blaster) 49,413 29, / (0.007) 0(0) 139 (W32.Spybot) 3,471 1, / (0.063) 0(0) 1433 (W32.Spybot) 21,082 7, / (0.011) 0(0) 4899 (W32.Rahack) 40,404 9, / (0) 0(0) 1023 (W32.Sasser) 1, / (0) 0(0) 9493 (file-guri) 115,294 26, , / (0) 0(0) 8404 (v-share) 48,607 1, / (0) 0(0) 25 (SMTP) 164,566 74, , / (0) 0(0) 80 (Web) 103,647 11, , / (0.12) 212(0.03) RER RER port 25(SMTP) port 80(Web) port 8404 (v-share) port 9493 (file-guri) Time (sec) (a) Normal service ports port 1025 (W32.Spybot) port 135 (W32.Blaster/Gaobot) port 2745 (W32.Beagle) port 6129 (W32.Mockbot) Time (sec) (b) Scanned ports Figure 3. RER comparison for scanned and normal ports in 04 Ajou trace. 4.1 REF of Real Trace Before applying our defense system to the real-life traces, we sorted out all the service ports in the traces on the basis of the amount of SYN traffic and the port list in [7,8]. From 04 Ajou trace, we found seven normal ports, eight scanned ports, and five unknown ports, which collectively account for 90% of the total SYNs. We inspected traffic to the unknown ports denoted as unassigned in the port list. As a result, we could only conjecture that the unassigned ports 9548, 7581, and may be unknown P2P or streaming service ports from the fact that large size files were transported on those ports. As to the traffic on ports and 6667, we observed that one source persistently sent many SYN packets to one destination, which were rejected by the destination. Therefore, we infer that the traffic is DoS attack or a mis-configured transaction. Figs. 3 (a) and (b) show RERs on the normal and scanned ports in 04 Ajou trace. Negative RERs demonstrate that traffics on ports 25, 8404, and 9493 are legitimate, but positive RERs on scanned ports show that scanning worms persistently attempt to infiltrate into the target network through the scanned ports. These results manifest that RER is an effective measure in detecting scanned ports. In Fig. 3 (a), REF rarely runs RERs. This is not because the amount of retry SYNs on the ports is small, but because the whitelist excludes retry SYNs sent to legitimate servers from being registered. RER being positive on port 80, which is mainly utilized by Web services, suggests that many worms attempted to infect Web servers in our target network. Tables 2 summarizes the result of building the blacklist in 04 Ajou trace. It shows that REF not only detected most infected hosts on scanned ports but also achieved very low FP rates on normal ports. The FN rates on scanned ports 2745, 1025, and 6129 in 04 Ajou are slightly high. However, considering that the averages of 30% higher and 30% lower SYN arrival rates on them described in the Higher/Lower rate column are very low, REF shows a remarkable performance. Our REF accomplished zero false positive rate even if the volume of SYN packets on P2P ports such as 9493 and 8404 was much larger than that on scanned ports. This convinces us that our algorithm can be used practically. 4.2 Comparison with SNORT To compare the performance of the REF with that of SNORT, we executed portscan2 and sfportscan preprocessor modules of SNORT that are designed for detecting and tracking portscans [12, 13]. In SNORT, the portscan2 module counts new connection requests from each host for a given period. If the

6 Detection Rate REF spp_sfportscan spp_portscan2 and compared it with that of SNORT. The results manifest that our algorithm outperforms rate-based detection algorithms at the ingress point with respect to detection rate and false positive rate. References Scanned Port Number (a) Detection rates of REF and SNORT Figure 4. Comparison of REF and SNORT on the scanned ports in 04 Ajou trace. counter for a host exceeds the threshold, it regards the host as a scanner or worm. Because there is no reference values concerning the parameters of portscan2, we varied the parameters from extremely low to high and compared their results in order to find optimal values. As a result, the threshold of 3 and the period of 30 sec (i.e., 0.1 scan rate) were chosen as the optimal values at which we obtained the best detection of 1,772 (2.1%) and 772 (1.5%) false positives on ports 8404 and 9493, respectively. The sfportscan detects a port scan by tracking connections and negative responses that are kinds of error packets such as RST and ICMP packets in querying connections from the target host to other hosts. For simple comparison of detection rate, the sfportscan is tuned to High sensitivity option even if it incurred 3,993 and 784 FP on ports 9493 and 8404, respectively. Fig. 4 illustrates the detection rates of the three detection algorithms on scanned ports in 04 Ajou trace. The two SNORT modules show much lower detection rate for all the scanned ports than REF except for ports 4899 and This is because whereas the scan rates to ports 4899 and 1023 are high enough for a rate-based detection algorithm in detecting scanning worms, those to other ports are not. 5 Conclusion In this paper, we presented an effective defense algorithm named REF against scanning worms intruding into a target network. For that, our algorithm adopts a relative entropy scheme that uses the information about retry SYN packets. This contrasts with existing threshold-based detection algorithms that have difficulty in detecting scanning worms at the ingress point of the network because a small amount of scan traffic, even though it is originally very high, observed at the ingress point does not expose visible traffic anomaly. Moreover, REF has the ability in adjusting its parameters adaptively to the observed traffic. By using reallife traces, we evaluated the performance of our algorithm [1] C. Zou, L. Gao, W. Gong, and D. Towsley, Monitoring and Early Warning for Internet Worms, the 10th ACM CCS, [2] M. Williamson, Throttling Viruses: Restricting Propagation to Defeat Malicous Mobile Code, June, [3] J. Jung, V. Paxon, W. Berger, and H. BalakrishnanFast Portscan Detection Using Sequential Hypothesis Testing, IEEE Security and Privacy, [4] C. Leckie and R. Kotagiri, A Probabilisitic Approach to Detecting Network Scans, IEEE NOMS02. [5] S. Staniford, V. Paxson, and N. Weaver, How to 0wn the Internet in your Spare Time, the 11th USENIX Security Symposium, [6] H.Kim, J.Kim, I.Kang and S.Bahk, Preventing Session Table Explosion in Packet Inspection Computers, IEEE Transactions on Computers, Vol.54,No.2,Feb [7] doshelp, trojanports, trojanports.htm.8 [8] IANA, port list, port-numbers [9] G. Bakos and V. Berk, Early Detection of Internet Worm Activity by Metering ICMP Destination Unreachable Activity, SPIE, [10] George K. Zipf, Human Behaviour and the principle of Least-Effort, Addison-Wesley, [11] Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, John Wiley & Sons, [12] M. Roesch, Snort: Lightweight intrusion detection for networks, LISA-99, [13] D. Roelker, spp sfportscan, viewcvs.cgi/snort/doc/readme.sfportscan?rev=1.6. [14] J. Postel,transmission control protocol, RFC 793, Sep [15] A. Lakhina, M. Crovella, and C. Diot, Mining Anomalies Using Traffic Feature Distributions, SIG- COMM 05, August 21-26, 2005.