Detecting DDoS Attacks Based on Multi-stream Fused HMM in Source-End Network

Transcription

1 Detecting DDoS Attacks Based on Multi-stream Fused HMM in Source-End Network Jian Kang, Yuan Zhang, and Jiu-bin Ju Department of Computer Science & Technology, Jilin University, Changchun, , China Abstract. DDoS (Distributed Denial-of-Service) attacks detection system deployed in source-end network is superior in detection and preventionthanthatinvictimnetwork,becauseitcanperceiveandthrottle attacks before data flow to Internet. However, the current existed works in source-end network lead to a high false-positive rate and falsenegative rate for the reason that they are based on single-feature, and they couldn t synthesize multi-features simultaneously. This paper proposes a novel approach using Multi-stream Fused Hidden Markov Model (MF-HMM) on source-end DDoS detection for integrating multi-features simultaneously. The multi-features include the S-D-P feature, TCP header Flags, and IP header ID field. Through experiments, we compared our original approach based on multiple detection feature with other main algorithms (such as CUSUM and HMM) based on single-feature. The results present that our approach effectively reduces false-positive rate and false-negative rate, and improve the precision of detection. 1 Introduction Comparing with DDoS detection system in victim network, source-end DDoS detection not only can perceive and prevent from attacks early, but also enhance security and QoS of the whole network. However, the attack flow in sourceend network is so dispersive that the traditional detecting algorithm troubled in distinguishing attack flows and normal flows, and led to high false-positive rate and false-negative rate. Thus, the key problem is how to raise precision and sensitivity of source-end DDoS detection. The existed detection sysytemsarebasedonsingle-feature extracted from source-end network, so they could not synthesize multiple factors. Although the single-feature detection algorithm has been improved, it limited in precision rising it cannot describe complex diversification in source-end network. Therefore, this paper proposes a novel approach using Multi-stream Fused Hidden Markov Model (MF-HMM) on source-end DDoS detecting for integrating multi-features simultaneously. The multiple factors include the S-D-P feature, the Flags and the ID field contained in TCP/IP header. Experiments can help us compare MF-HMM with other models like CUSUM algorithm and HMM based on single observing feature. The results present that MF-HMM D. Pointcheval, Y. Mu, and K. Chen (Eds.): CANS 2006, LNCS 4301, pp , c Springer-Verlag Berlin Heidelberg 2006

2 Detecting DDoS Attacks Based on MF-HMM in Source-End Network 343 effectively reduce the false-positive rate and false-negative rate. The MF-HMM proposed in this paper can adapt to diversified network and raise the precision of detection. 2 Related Work Mirkovic et al. proposed D-WARD as a representative source-end DDoS detection system in [1]. In a normal TCP session, the flow from source to destination (which is defined as TCP sent to) is controlled by the reverse acknowledge flow (TCP received from). Under DDoS attacking, TCP sent to is far greater than TCP received from. D-WARD defines max tcprto as the max possible rate for TCP sent to/tcp received from under normal network environments. If the observed rate is higher than max tcprto in real time, it is determined as an attack. However, the false-positive rate and false-negative rate in D-WARD is high. Paper [2] extracted the same ratio with that in D-WORD as observing feature. But because of introducing a nonparametric change point detection method in statistics and improving D-WARD by nonparametric recursive CUSUM algorithm, the improved system is more advanced in detecting precision than D- WARD. Peng et al. in [3] considered the number of new source IP addresses appeared in data flow in unit time as observing feature. The abnormal increase of this number determines if attacks appeared. They used CUSUM algorithm to detect source-end DDoS attacks. However, high false-positive rate is led because they took only one feature into account. Zhou et al. in [4] used HMM to detect DDoS attacks. They use TCP Header Flags to describe TCP package as observing feature. They constructed the observing sequence with the weight sum of each bit of TCP Header Flags, and trained HMM by data packages under normal network. The trained HMM can be seen as criterion to judge if there are attacks. Therefore, existing researches on source-end DDoS detecting system are based on single-feature. Although there are improvements to the algorithms themselves, the insufficient detection information contained in single-feature constrains the enhancement of the detecting precision. 3 Multi-features Extraction Moore et al. in [5] presented a famous result: most DDoS attacks use TCP package (over 94%), then UDP package (2%) and ICMP package (2%). From the result, we can see the importance of detecting TCP packages in DDoS attacks. Thus, in this paper, extracting and detecting multi-features of TCP Flooding attacks are to be discussed. Analyzed characteristics and mechanisms of representative DDoS attacks, we defined the conception of S-D-P feature. Preparing for MF-HMM represented in Sect.4, we constructed multi-features including S- D-P feature, TCP Header Flags and ID field in IP Header.

3 344 J. Kang, Y. Zhang, and J.-b. Ju 3.1 TCP Header Flags We choose TCP Header Flags as one of the features describing TCP package in source-end network. In order to represent this feature in numerical value, we define different weights to different flags as [4]. Figure 1 presents the weights. Equation (1) is to calculate the observing feature value of TCP Header Flags. Fig. 1. Weights of different TCP header Flags O i =2 5 URG+2 4 ACK+2 3 PSH+2 2 RST+2 1 SYN+2 0 FIN. (1) Calculated through (1), we could get observation symbol set V={1, 2,...,63}. 3.2 IP Header Identification Field In DDoS detecting, ID field in IP header is significant for detection. In general, ID field is written by operating system (OS). Main DDoS attacks use IP Spoof strategy, and they fill in ID fields in different random algorithms. Those random algorithms can be learned in [6]. Thus, it is obvious the distinction between ID field written by spoof strategy and ID field written by OS. In IP header, the length of ID field is 16 bits, and the corresponding value range is In order to reduce this large set and assure light computation, mapping is needed. Our experiments show that ID fields distributed averagely, and Table.1 presents the way we mapped the values. Table 1. Mapping Rule of ID Fields ID range Mapped value According to Table.1, the observation symbol set V = {1, 2,...,67}. 3.3 S-D-P Feature When attacks appear in source-end work, IP addresses and port numbers will change obviously. Because attackers spoof source IP addresses to avoid detecting and tracing back. And in order to deplete victim s resources in a short

4 Detecting DDoS Attacks Based on MF-HMM in Source-End Network 345 time, they send large numbers of spoofed packages to one or more ports of the victim. Thus, research on IP addresses and port numbers is necessary to DDoS detection. We use three-tuple (IP source, IP destination, P ORT destination) to specify S-D-P feature of TCP/IP header. Here, IP source presents source IP addresses; IP destination presents destination IP addresses; PORT destination presents destination port number. If S-D-P feature is modeled by HMM, the observation symbol set would contain elements. That set is so huge that we reduce it through mapping. In general, IP addresses are divided into five classes:a, B, C, D, and E. Class D and Class E are so scarcely appeared that can be overlooked. We map IP Address Class A, Class B, and Class C to hexadecimal identifiers according to their binary codes in the first byte. In the same way, port number range can be divided into three parts: well-known port, registered port, and dynamic (private or ephemeral) port. We map them to hexadecimal identifiers. Table 2 presents the mapping. Table 2. Mapping of IP addresses and ports IP Address class First byte(binary) Identifier(hex) Class A 0 0X1 Class B 10 0X2 Class C 110 0X3 Port type Port range(decimal) Identifier(hex) Well-known port X1 Registered port X2 Dynamic/Private/Ephemeral ports X3 Through the mapping above, there are elements in the new observation symbol set. So, the new observation symbol set V = {1, 2,...,27}. 4 MF-HMM We use the Multi-stream Fused (MF-HMM) proposed by Zeng et al. in [7] to synthesize multi-features effective to precision of detecting.accordingtothe maximum entropy principle and the maximum mutual information (MMI) criterion, MF-HMM constructs a new structure linking multiple HMMs. MF-HMM is the generalization of Two-stream Fused HMM [8]. It is suitable for the recognition and detection with multiple features problem. Paper [7] pointed out the advantages of MF-HMM:

5 346 J. Kang, Y. Zhang, and J.-b. Ju 1. Every observing feature can be modeled by a component HMM, so the performance of every feature can be analyzed individually. And the analysis could be used for feature selection. 2. Compared with other existing models (for example, CHMM [9] and MHMM [10] e.g.), MF-HMM reaches a better balance between model complexity and performance. 3. Reliabilities of component HMM can be used to adjust the corresponding weights in final fusion. And if one component HMM fails due to some reason, the other HMM can still work. Thus, the final fusion performance can be robust. In our source-end network DDoS detection system, we use Multi-stream Fused HMM with three features which described in sect MF-HMM Overview HMM is the basis of MF-HMM. And paper [11] discussed HMM in detail. Let {O (i),i =1,..., n} represents n tightly coupled observing sequences. Assume that {O (i),i =1,..., n} can be modeled by n corresponding HMMs with hidden states {Q (i),i =1,..., n}. In MF-HMM, an optimal solution for p(o (1) ; O (2) ;...; O (n) )isgivenbyˆp(o (1) ; O (2) ;...; O (n) ) according to the maximum entropy principle and the maximum mutual information criterion. There are two steps in calculating ˆp(O (1) ; O (2) ;...; O (n) ). First, the i-th ˆp (i) (O (1) ; O (2) ;...; O (n) ) can be given through (2). ˆp (i) (O (1) ; O (2) ;...; O (n) ) = p(o (1) )p(o (2) )...p(o (n) ) p(q (i),o (1),...,O (i 1),O (i+1),...,o (n) ) p(q (i) )p(o (1) ) p(o (i 1) )p(o (i+1) ) p(o (n) ) = p(q (i) p(o (1),...,O (i 1),O (i+1),...,o (n) Q (i) ) And assuming (2) p(o (1),..., O (i 1),O (i+1),..., O (n) Q (i) )= n j i,j=1 p(o (j) Q (i) ) (3) It has a good record in recognizing and detecting DDoS attacks, though the conditional independence assumption is always violated in practice. The success is because of the small number of parameters to be estimated in assumption. Without this assumption, some complicated algorithms need more training data, and are more susceptible to local maximum during parameter estimation. So, the estimate of ˆp (i) (O (1) ; O (2) ;...; O (n) ) can be given by (4). ˆp (i) (O (1) ; O (2) ;...; O (n) )=p(o (i) ) n j i,j=1 p(o (j) Q (i) ) (4)

6 Detecting DDoS Attacks Based on MF-HMM in Source-End Network 347 There are different expressions to different i. To our Multi-stream Fused HMM, Equation (4) corresponds to (5), (6), (7). ˆp (1) (O (1) ; O (2) ; O (3) )=p(o (1) )p(o (2) Q (1) )p(o (3) Q (1) ) (5) ˆp (2) (O (1) ; O (2) ; O (3) )=p(o (2) )p(o (1) Q (2) )p(o (3) Q (2) ) (6) ˆp (3) (O (1) ; O (2) ; O (3) )=p(o (3) )p(o (1) Q (3) )p(o (2) Q (3) ) (7) Thus, the estimate of ˆp(O (1) ; O (2) ;...; O (n) ) can be calculated by (8). In practice, if those n component HMMs have different reliabilities, they may be combined by different weights for a better result. In our experiment, the weights of ID field, S-D-P feature, and TCP header flags in turn are 0.3, 0.33, and Here, ˆp(O (1) ; O (2) ;...; O (n) )= n λ (i) =1. i=1 n λ (i) ˆp (i) (O (1) ; O (2) ;...; O (n) ) (8) i=1 4.2 Learning Algorithm of MF-HMM There are three main steps in the learning algorithm of MF-HMM: 1. n component HMMs are trained independently by representative algorithm (Baum Welch Algorithm, Segmented K-Means Algorithm, or Hybrid Method EM Algorithm [11]) 2. The best hidden state sequences of the component HMMs are estimated by the Viterbi algorithm [11]. 3. Calculate the coupling parameters between the n HMMs, viz. ˆB (i,j) =argmaxp(o (j) ˆQ (i) ) i, j =1, 2,..., n, i j. (9) B (i,j) To our Multi-stream Fused HMM, step one is to do: ˆΠ (1), Â(1), ˆB (1) =arg max (log p(o (1) )) (10) Π (1),A (1),B (1) ˆΠ (2), Â(2), ˆB (2) =arg max (log p(o (2) )) (11) Π (2),A (2),B (2) ˆΠ (3), Â(3), ˆB (3) =arg max (log p(o (3) )) (12) Π (3),A (3),B (3) And then step two: ˆQ (1) =argmax Q (1) (log p(o (1),Q (1) (13)

7 348 J. Kang, Y. Zhang, and J.-b. Ju ˆQ (2) =argmax Q (2) (log p(o (2),Q (2) )) (14) ˆQ (3) =argmax Q (3) (log p(o (3),Q (3) )) (15) At last, step three is to estimate the coupling parameters between HMM1, HMM2, and HMM3: ˆB (1,2) =argmax B (1,2) p(o (2) ˆQ (1) ) (16) ˆB (1,3) =argmax B (1,3) p(o (3) ˆQ (1) ) (17) ˆB (2,1) =argmax B (2,1) p(o (1) ˆQ (2) ) (18) ˆB (2,3) =argmax B (2,3) p(o (3) ˆQ (2) ) (19) ˆB (3,1) =argmax B (3,1) p(o (1) ˆQ (3) ) (20) ˆB (3,2) =argmax B (3,2) p(o (2) ˆQ (3) ) (21) 5 DDoS Detection and Estimation 5.1 Assumption Based on MF-HMM Detection The distinction is obviously between the data package in normal state and that under attacking. That is to say, to MF-HMM trained with normal data stream, the output probability of normal data package sequences is more than that of data package sequences with attacks. Thus, we determine whether attacks or normal depended on output probability of MF-HMM. 5.2 Pretreatment to Detected Sequence In experiment, we construct detected sequence with the three features mentioned in sect.3 from detected data stream. Let the length of the detected sequence is L. Split the detected sequence with a k length splitting window, and the sequence can be divided into L/k subsequences. So the set of these subsequences is {X i }, here 1 i L/k.

8 Detecting DDoS Attacks Based on MF-HMM in Source-End Network Attack Determination Algorithm Input each subsequence X i to MF-HMM, calculate the output probability log ˆp(O (1) ; O (2) ; O (3) ). If that probability is smaller than the threshold of the output: ε, markx i as questionable subsequence. Calculated and marked all of the subsequences, we get the ratio δ through (22). numbers of questionable subsequences δ = (22) numbers of all subsequences At last, compare δ with the attack Threshold: ifδ > Threshold, it is determined that DDoS attacks are taking place; else, there is no attack. Figure 2 shows the process of attack detection and determination. B (1,2) B (2,1) B (1,3) B (3,1) B(2,3) B (3,2) Fig. 2. DDoS detection process based on MF-HMM with three features 6 Experiment In order to build MF-HMM based on normal source-end network and confirm ε and δ, we collected data for three months 10 times per day, and 1,000,000 data packages per time. To attack data stream, we used representative DDoS attack tool TFN2K, which deployed in several hosts. S-D-P feature, TCP header flags, and ID fields are used when building Multi-stream Fused HMM. We compared MF-HMM based on multiple features with other four detection algorithms based on single-feature: 1. MF-HMM based on three features is called TF-HMM; 2. HMM use S-D-P feature only is called SDP; 3. HMM use TCP header flags only is called TCP-flag; 4. HMM use ID field only is called ID-segment; 5. Detection based on nonparametric recursive CUSUM algorithm is called CUSUM.

9 350 J. Kang, Y. Zhang, and J.-b. Ju 6.1 Output Probabilities of TF-HMM in Normal State and Attacking State In experiment 1, we observe TF-HMM s output probability. To both the normal state and attacking state, sampling last for 300 seconds, and overlap the two into one time axis as presented in Fig.3. Fig. 3. logp in normal state and in attacking state in TF-HMM In Fig.3, the abscissa t represents time, with the unit of second. The ordinate logp represents the output probability log ˆp(O (1) ; O (2) ; O (3) ). We can see the obvious difference of logp in TF-HMM between normal state and attacking state. In normal state, the value of logp fluctuated in the range of ; while under the attacks, the peak value could reach to eight times of the normal value, even larger. Attacks were launched two times in Fig.3: the first time near 42 second lasted for about 30 seconds; the second time near 175 second lasted for about 50 seconds. 6.2 False-Positive Rate and False-Negative Rate Experiment In experiment 2, we compared the false-positive rate and the false-negative rate of 5 detection algorithms in different network environments. It means that there are data from 10 different groups the first 5 groups are captured from different network services and mutative stream intensity, without attacks; while the last 5 groups are experiments under attacking. These data were inputted into detection system, and the results were presented by table.3. From table.3, all of the first 5 groups showed us the false-positive. Especially, the 3rd group used CUSUM algorithm led a high false-positive rate to 65 times. The result from TF-HMM is closer to REAL than other algorithms

10 Detecting DDoS Attacks Based on MF-HMM in Source-End Network 351 Table 3. False-positive and false-negative of 5 detection algorithms No.1 No.2 No.3 No.4 No.5 No.6 No.7 No.8 No.9 No.10 SDP TCP-flag ID-segment CUSUM TF-HMM REAL based on single-feature. In the 6 th group, we launched attacks intensely, and all algorithms result to false-negative reports. The false-negative rate of CUSUM was the highest, while TF-HMM performed better than other algorithms. In the 10 th group, we increased both of the attack intension and normal data stream: false-positive rate of TF-HMM was lower than other algorithms. In addition, it is valuable to mention that, in the 9 th group, under the large normal data stream, we launched attacks for two times separately. The results showed us: TF-HMM could recognize attack accurately; in contrast, other algorithms could not. Thus, TF-HMM is sensitive in source-end detection, and adapt to the new DDoS attacks with high distribution and low attack intension. To sum up, in this 5 detection algorithms, CUSUM algorithm cannot learn the normal network state, so leads to high false-positive rate and false-negative rate. The other algorithms based on single-feature cannot reflect real condition of source-end network actually because of the limitation of single-feature, though they improved detection systems. TF-HMM based on three features synthesizes more detection information, enhances the precision of detection, and is better than the other. 6.3 Average Detection Rate Experiment We definite Detection Rate (DR) is the ratio of the number of attacks detected and the number of attacks real existed, viz. the percentage of the recognized attacks in the whole real attacks. In experiment 3, we varied attack intension, normal data stream intension, and sampling time. Thus, there are obvious differences between every two groups. From the 100 groups of data gained, we calculated their average DRs as presented in Table.4. Table 4. Average DRs of 5 Detection Algorithms CUSUM ID-segment S-D-P TCP-flag TF-HMM 48.64% 60.78% 68.93% 70.27% 91.12%

11 352 J. Kang, Y. Zhang, and J.-b. Ju The average DRs of the three algorithms based on single-feature are higher than CUSUM algorithm, so learning algorithm HMM can reflect the variety in source-end network better. However, using fewer detection information, these three algorithms have handicaps in increasing the precision of detection. In contrast, TF-HMM based on three features, with the average DR of 91.12% which is 1.87 times of CUSUM algorithm, utilizes more detection information, and increases the precision of detection to a satisfied result. 7 Conclusion DDoS detection attacks in source-end network could perceive attacks before it enter to Internet. Comparing with DDoS detection in victim, source-end DDoS detection is superior in recognizing DDoS attacks and tracing back the attack sources. However, because attack stream is thin in source-end network, a sensitive and accurate algorithm is needed. The existing works are based on single-feature, so could not synthesize multiple information. Although there is improvement on single-feature detection algorithm, it is hard to depict the subtle varieties in source-end network, and so leads to the limitation on improving detection precision. To the problems above, this paper proposes a novel DDoS detection algorithm synthesizing multiple features. The multi-features include S-D-P feature, TCP header Flags and IP header ID field, and MF-HMM is used in this algorithm. Experiments show us the results that MF-HMM perform better than other 4 algorithms based on single-feature, and effectively reduce the false-positive rate and false-negative rate. The MF-HMM proposed in this paper is effective to detect the new DDoS attacks with high distribution and low attack intension in source-end network. References 1. Jelena, Mirkovic.: D-WARD. Source-End Defense Against Distributed Denial-of- Service Attacks, (2003), CSD of UCLA, Jian Kang, Zhe Zhang, Jiu-bin Ju: Protect e-commerce against DDoS attacks with improved D-WARD detection system. IEEE International Conference on e- Technology, e-commerce and e-service, Hong Kong, April Tao Peng, Christopher Leckie, Kotagiri Ramamohanarao: Detecting Distributed Denial of Service Attacks Using Source IP Address Monitoring. Networking 2004, Athens, Greece, May Dongqing Zhou, Haifeng Zhang: A DDoS Attack Detection Method Based on Hidden Markov Model. Journal of Computer Research and Development, Vol.42, (2005) D.Moore, G.Voelker, S.Savage: Inferring internet denial-of-service activity. The 10th USENIX Security Symposium, Washington, Chang-Han Jong, Shiuh-Pyng Shieh: Detecting Distributed DoS/Scanning by Anomaly Distribution of Packet Fields. International Computer Symposium, 2002

12 Detecting DDoS Attacks Based on MF-HMM in Source-End Network Zeng Z, Tu J, Pianfetti: Audio-visual affect recognition through multi-stream fused HMM for HCI. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June Pan, H., Levinson, S., Huang, T.S., and Liang, Z.P.: A fused Hidden Markov Model With Application to Bimodal Speech Processing. IEEE Transaction on Signal Processing, Vol.52, No.3, (2004) Brand, M., Oliver, N.: Coupled hidden Markov models for complex action recognition. Computer Vision Pattern Recognition, (1997) Saul, L.k., Jordan, M.I.: Mixed memory Markov model: Decomposing complex stochastic processes as mixture of simpler ones. Machine Learning, Vol.37, (1999) Rabiner, L.R.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of IEEE, Vol.77, No.2, February 1989