A Multi-Stage Network Anomaly Detection Method for Improving Efficiency and Accuracy

Transcription

1 Journal o Inormation Security, 2012, 3, Published Online January 2012 ( A Multi-Stage Network Anomaly Detection Method or Improving Eiciency and Accuracy Yuji Waizumi, Hiroshi Tsunoda, Masashi Tsuji, Yoshiaki Nemoto Graduate School o Inormation Sciences (GSIS), Tohoku University, Miyagi, Japan wai@ecei.tohoku.ac.jp Received September 29, 2011; revised October 27, 2011; accepted November 10, 2011 ABSTRACT Because o an explosive growth o the intrusions, necessity o anomaly-based Intrusion Detection Systems (IDSs) which are capable o detecting novel attacks, is increasing. Among those systems, low-based detection systems which use a series o packets exchanged between two terminals as a unit o observation, have an advantage o being able to detect anomaly which is included in only some speciic sessions. However, in large-scale networks where a large number o communications takes place, analyzing every low is not practical. On the other hand, a timeslot-based detection systems need not to prepare a number o buers although it is diicult to speciy anomaly communications. In this paper, we propose a multi-stage anomaly detection system which is combination o timeslot-based and low-based detectors. The proposed system can reduce the number o lows which need to be subjected to low-based analysis but yet exhibits high detection accuracy. Through experiments using data set, we present the eectiveness o the proposed method. Keywords: Network Anomaly Detection; Timeslot-Based Analysis; Flow-Based Analysis; Multi-Stage Traic Analysis; Flow Reduction 1. Introduction In recent years, intrusions such as worms and denial o service attack have become a major threat to the Internet. In particular, novel intrusions such as novel worms and zero-day attacks are increasing and are responsible or a big damage to the Internet. For detecting intrusions, Network Intrusion Detection Systems (NIDSs) have gained attention. NIDSs are classiied into misuse detection system and anomaly detection system. In misuse detection systems such as Snort [1], intrusions are detected by matching signatures which are prepared manually in advance. They are highly popular in network security because they exhibit higher detection accuracy and generate ewer alse positives or known intrusions than anomaly detection systems. However, developing signatures is cumbersome and time-consuming task because they have to be made by security experts manually. Thereore, novel intrusions can cause a signiicant damage to the Internet beore signatures are developed. On the other hand, anomaly detection systems such as NIDES [2] and ADAM [3] can detect unknown intrusions. This is because these methods detect intrusions based on the deviation rom the normal behavior, and thus do not require a pre-hand knowledge o intrusions. However, these methods tend to generate more alse positives than signature base IDSs. Although a lot o researchers carried out to increase the detection accuracy, still higher detection accuracy is demanded. Thereore, we ocus our research on anomaly detection systems. In anomaly detection systems, network traic is analyzed using observation units such as timeslot and low. A timeslot-based detection has an advantage o being able to detect network anomaly states eectively. On the other hand, the low-based analysis is capable o examining each communication in a more detail orm. Our group has proposed a combination o timeslot-based and low-based detections and shown its eectiveness [4]. However, in a low-based analysis, a large number o buers have to be prepared. Analyzing all lows o network traic is not realistic, and the buer size can be vulnerability to Denial o Service (DoS) attacks because all low analysis can result in a buer overlow. In this paper, we propose a high accuracy multi-stage anomaly detection system which can reduce the number o lows necessary to be analyzed. The proposed system consists o two detection stages. The irst stage is a timeslot-based detector which picks up lows need to be analyzed by low-based detector in detail. It then inspects only these suspicious lows in the second stage, thus, computational load and buer size to analyze lows can

2 Y. WAIZUMI ET AL. 19 be reduced. The remainder o this paper is organized as ollows. Section 2 explains timeslot-based and low-based analyses, and mentions issues in a combination o these analyses. In Section 3, we proposed a multi-stage anomaly detection system. Evaluation o the proposed system is presented in Section 4. Finally, Section 5 concludes this paper. 2. Combination o Timeslot-Based and Flow-Based Analyses Anomaly detection systems generally analyze traic in observation units such as timeslots and lows. In this section, we explain these units or the intrusion detection and introduce a conventional method which combines the two detectors. Furthermore, issues in the conventional method are also presented Timeslot-Based Analysis Anomaly detection oten uses timeslot-based analysis [4-6]. In this method, the overall traic is separated into timeslots o ixed length and its eatures, which are numerical values representing the network state, are extracted rom traic in the timeslot. It has an advantage o low buer storage since this analysis releases buers ater each timeslot. However, it is diicult or this method to speciy anomalous communication lows Flow-Based Analysis A low is deined as a set o packets which have the same values or the ollowing three header ields. Protocol (TCP/UDP) Source/Destination address pair Source/Destination port pair A TCP low ends with FIN or RST lags and UDP lows are terminated by time-out T u. A low is oten used in anomaly detection [4,7,8]. A low-based analysis method can analyze each bidirectional communication in detail and can speciy each anomalous communication. However, in this analysis, buers must be prepared or every low. The number o buers to be prepared lineally increases with as increase in the number o lows. Thus, this method possesses a risk o buer overlow. Thereore, storage o buers is a bottleneck in the low-based analysis and vulnerability to DoS attacks A Conventional Combination Method Our research group has proposed a combined system using the timeslot-based and the low-based analyses in parallel [4]. Figure 1(a) shows the overview o the conventional system, which we term as a parallel system. Network traic is inputted to both the timeslot-based and the low-based detectors, and is analyzed by each detector. A combination o timeslot-based and low-based detectors can detect intrusions eectively by taking advantage o the merits possessed by both o these methods. Thereore, the combination system is highly accurate in anomaly detection and [4] shows the eectiveness o the parallel system through some experiments using DARPA data set [9]. However, it is still necessary to address the problem o large buer storage in the low-based analysis. For reducing the amount o data to be analyzed by low-based analysis, packet sampling [10-12] and setting short timeouts [13] have been proposed. However, by using the ormer, it is diicult to observe lows which consist o only ew packets, and thus there is a high chance o missing important packets during detection. Since novel worms tend to be ew packets in order to spread as ast as possible [14], such worms are diicult to be sampled. In the latter case, since long traic lows will be split up i its interval o arrival time o packets exceeds the low timeout, the short timeouts causes increasing the number o lows and declining eiciency and accuracy [11]. As a result, we consider that these approaches suer rom lack o inormation or detecting anomalous events and exhibit low detection accuracy. Since packet sampling and setting short timeout diminish data o each low without any regards or evaluating anomaly, it may result in lack o inormation needed to detect anomalous lows. For avoiding lack o inormation, not data o each low but the number o lows should be reduced with appropriate criteria. 3. A Multi-Stage Anomaly Detection System In this section, we propose a multi-stage network anomaly detection system. It uses ewer amount o buer, but yet detects intrusions with high accuracy Outline Figure 1(b) shows the overview o the proposed multistage (a) (b) Figure 1. Overview o parallel system and multi-stage system.

3 20 Y. WAIZUMI ET AL. system. The proposed system consists o timeslotbased detector in the irst stage and low-based detector in the second stage. The role o the irst stage is to exclude lows that can be judged as obvious anomalous or obvious normal. For such lows, detailed analysis by the low-based detector is not needed. By excluding such lows, the proposed method can achieve the both high eiciency and high accuracy in network anomaly detection. Considering the role o the irst stage, we utilize timeslotbased detector in it. In the timeslot-based detector, the anomaly level o a time slot basically depends on the number o anomalous lows. Thereore, i timeslots have extremely high or extremely low degree o anomaly, we can consider lows included in such timeslots as obvious anomalous or normal, respectively. Only lows included in the remaining timeslots need to be analyzed by the low-based analysis. Thereore the proposed system can reduce the number o lows that need to be analyzed by the low-based analysis. The architecture o the proposed system is illustrated in Figure 2. Network traic is input into both the timeslotbased detector and the dump module. The timeslot-based detector analyzes the network traic or each timeslot. Based on the anomaly level o the timeslot, it classiies each slot into three levels which are anomalous, suspicious and normal. Normal slots are excluded rom urther analysis. For anomalous slots, this detector generates alerts to administrators to inorm that anomalous traic is detected. For suspicious slots, inormation needed to aggregate packets to assemble lows, which we term suspicious inormation, is sent to the dump module. This inormation includes start and end times o the suspicious slot, source/destination IP addresses and source/destina- tion port numbers. The role o the dump module is to pick up packets based on the suspicious inormation, and to assemble them into suspicious lows. These lows are sent to the low-based detector. The low-based detector analyzes them and generates alarms i any anomaly is detected. As a result, it is not necessary to analyze lows in normal and anomalous slots. Thereore, the proposed system can reduce the number o lows that need to be stored in a buer. In Sections 3.2 and 3.3, we describe the timeslot-based detector and the low-based detector respectively Timeslot-Based Detector Outline Figure 3 shows the slot classiication carried out by the timeslot-based detector. Firstly, each slot is classiied into either anomalous slot candidate or normal slot by a threshold ( Th ac ). Then the anomaly score o a slot exceeds the threshold ( Th ac ), the slot is regarded as an anomalous slot candidate. Next, the anomalous slot candidates are classiied into anomalous slots or suspicious slots by another threshold ( Th as ). When the anomaly score o an anomalous slot candidate exceeds the threshold (Th as ), this slot is judged as an anomalous slot. For anomalous slots, the timeslot-based detector outputs alerts. For suspicious slots, this detector generates suspicious inormation. For normal slots, this detector outputs nothing. In the end, only the suspicious slots whose anomaly scores in the range rom to are sent to second stage or a detail analysis. It is eective to detect anomaly based on packet header and payload individually [4]. Thereore, the timeslot-based detector has two modules, header-based detection module and payloadbased detection module. Figure 2. Detailed architecture o the proposed system.

4 Y. WAIZUMI ET AL. 21 Figure 3. Slot classiication using two thresholds in timeslotbased detector Header-Based Detection Module The header-based detection module analyzes network traic based on eatures mainly extracted rom the header o each packet included in each timeslot. The eatures (37 types) extracted are as ollows: 1) For all traic. a) Each number o packets o TCP/ UDP/ ICMP (3 types). b) The number o TCP bytes. c) The number o port varieties o TCP. d) Each number o TCP lags (5 types). e) The number o DNS packets. ) The number o ragment packets. g) The number o IP address varieties (each byte, 4 types). 2) For port #21, #22, #23, #25, #80 and #110. a) Each number o TCP lags (3 types). 3) For port #80. a) The maximum number o bytes between delimiter (space/line eed). b) A ratio o the number o message headers per the number o request lines. c) A ratio o the number o packets per the number o clients. These eatures become elements o a eature vector. This detector calculates a projection distances rom a eature vector to the irst principal components using Principal Component Analysis (PCA) and deines the projection distance as the anomaly score o the low rom which the eature vector is extracted. Based on the anomaly score, slots are labeled as anomalous slots, suspicious slots, or normal slots Payload-Based Detection Module The payload-based detection module analyzes network traic by eatures extracted rom payloads o a packet. In the payload-based detection module, packet payloads are divided into 8-bit codes. Appearance and transition probabilities o each code are calculated using training data. Then, we assume that the code sequences in packet payloads o each port generate with Markov process. Hence, the appearance probability o a code sequence x1, x2,, xn in a packet payload is obtained rom Equation (1) 1, 2,, n n n 1 P x x x P x P x x P x x where n is the length o a packet. Anomaly score H or each packet is deined as the measure o inormation o code sequences and are obtained rom Equation (2) because anomalous code sequences occur ew many times. The anomaly o a slot is deined as the maximum value o in that slot (1) H log P x, x,, xn (2) Based on H, slots are labeled as anomalous slots, suspicious slots, or normal slots Output o Timeslot-Based Detector Table 1 summarizes the inal outputs o the timeslot-based detector. When any one o the header-based and the payload-based detection modules label a slot as an anomalous slot, the timeslot-based detector outputs an alert. I header-based or payload-based detection modules label a slot as a suspicious slot, suspicious inormation is generated and sent to the dump module. When one detector labels a slot as an anomalous slot and the other labels a slot as a suspicious slot, the timeslot-based detector outputs both o alert and suspicious inormation. When both header-based and payload-based detection modules label a slot as a normal slot, the timeslot-based detector outputs nothing Flow-Based Detector Outline The timeslot-based detector creates suspicious inormation when a slot is classiied a suspicious slot. The lowbased detector analyzes suspicious lows which are assembled by the dump module using suspicious inormation. This detector classiies suspicious lows into anomalous lows or normal lows. Similar to the timeslot-based detector, the low-based detector consists o two analysis modules: header-based and payload-based detection modules. When any o the header-based and the payload-based detection modules classiies a low as anomalous, the low-based detector outputs an alert. PDM Table 1. Output o the timeslot-based detector. HDM Anomalous Suspicious Anomalous Suspicious Normal Alert Alert & SI Alert Alert & SI SI SI Normal Alert SI - HDM: Header-based Detection Module; PDM: Payload-based Detection Module; SI: Suspicious Inormation.

5 22 Y. WAIZUMI ET AL Header-Based Detection Module The header-based detection module detects anomalous lows using eatures extracted rom packet headers o a low. It analyzes the ollowing eatures (TCP: 19 types, UDP: 7 tyes): 1) For both o TCP and UDP lows. a) The number o packets. b) Inverse o the number o lows which have same port number. c) The number o ragment packets. 2) For TCP lows (only sending packets rom clients). a) Each number o TCP lags (8 types). b) Each number o packets with only a TCP lag (8 types). 3) For UDP lows. a) Each number o sending/receiving packets or clients (2 types). b) Each number o sending/receiving packets or clients (2 types). The projection distance rom a eature vector to the irst principle component is calculated by PCA using these eatures. The projection distances are then used as anomaly scores o lows. I some plural lows which have same source/destination IP addresses are observed in a short period o time, this module regards them as related lows and treats them as one set. Because some attacks, such as scan, consist o plural lows rom single IP address, assembling the related lows can promotes the eiciency o detection. When a low is observed, lows which have the identical IP addresses with and meet the conditions below are regarded as the related lows o t T t t (3) i where denotes a related low, i t and t i are the inish time o low and i, respectively. A parameter T is used to evaluate whether two lows and i are observed in a short period o time. Denoting the anomaly score o and i as a and a i, respectively, the anomaly score o the set o the related lows A can be expressed as: F N 1 A a a (4) F i i 0 where F denotes the set o the related lows and N is the maximum size o buer to store the related lows or evaluating A F. Even i each low o an intrusion has small anomaly score, we can detect such intrusion by evaluating A F. Moreover, i an alert or F is generated, this module does not generate more alerts or successive lows that have same source/destination IP addresses as F during t T. Thus, only one alert is generated or a single intrusion and we can avoid receiving redundant alerts Payload-Based Detection Module For each TCP low, a eature vector consists o 512 eatures which are the 256 codes rom client to server and the 256 codes rom server to client. For others, such as UDP low, the eature vector consists o 256 eatures which are the 256 codes rom client to server. Projection distances rom the eature vector rom the irst principle component are calculated by PCA using these eatures. The low-based detector deines the projection distances as anomaly scores o lows. This analysis is carried out or all TCP lows, port #20, #21, #23, #25 and #80 respectively. 4. Evaluation 4.1. Experimental Environment We use the 1999 DARPA o-line IDS evaluation data set [9] to investigate the number o lows reduced by the timeslot-based detector and to evaluate o detection accuracy. The ollowing inormation o intrusions is given in this data set. Intrusion instants. IP addresses and port numbers o intruders. IP addresses and port numbers o victims. Types o intrusions. This data set consists o network traic o 5 weeks. Week 1 and week 3 (10 days) traic are attack-ree while week 4 and week 5 (10 days) traic include some intrusions. Data o week 1 and week 3 are used or training the detectors o the proposed system, and intrusions included in week 4 and week 5 are the detection targets. In week 4 and week 5, nearly 700,000 TCP and UDP lows are included. Table 2 shows the values o parameters T t, T, u T and N, which are same as the parallel system [4]. A payload-based detection module in timeslot-based detector targets port #21, #25 and #80 on which the payloadbased detection module is able to train eectively as the data set. According to [15], and thresholds o low-based detector are set such that the number o alse positives does Table 2. Values o parameters set in evaluation. Parameter Description Value T t Timeslot Length 60 [sec] T u T N Timeout or Terminating UDP lows Available period o a base low i 600 [sec] 600 [sec] Maximum number o 10

6 Y. WAIZUMI ET AL. 23 not exceed 10 or each day (100 in 10 days). Thresholds o detection module in the proposed system are set as shown in Table 3. The value o Th ac is set as explained in the next subsection Flow Reduction Perormance By using suspicious low ratio per all lows in week 4 and week 5 ( Rs ), and the ratio o detectable intrusions in anomalous slot candidates per all intrusions ( Rac ), we evaluate low reduction eect o the proposed system. A low value o Rs is preerred as it implies that the number o lows which will be analyzed in the low-based detection modules are reduced. High R ac means that many intrusions included in the anomalous slots can be detected by the proposed method. Figure 4 shows changes in Rs with respect to Rac. This graph indicates that the bottom right portion o the line shows a high perormance. Figure 4 shows that Rs is about 40 percent when Rac is about 90 percent. That is to say, it is possible to detect 90 percent intrusions by analyzing merely 40 percent o the total lows. This indicates the proposed system can eectively reduce lows to be analyzed. Note that remained 10 percent intrusions become alse negatives in the proposed method. However, ater investtigating these intrusions, we ind that most o these intrusions cannot detect even by the low-based detector. Since the behavior o these intrusions is almost same as normal communication, it is diicult to detect such intrusions by the already existed timeslot-based and lowbased detector. As a result, we conclude that there can be ew additional alse negatives caused by the proposed method. Thereore, Thac is adjusted so that Rac becomes 90 percent in the next section Intrusion Detection Perormance In this section, the proposed system is evaluated in terms o detection accuracy. Table 4 indicates the number o detected intrusions, the total number o intrusion and detection rate or existing IDSs and the proposed system. As shown in Table 4, the proposed system has higher detection rate than other IDSs except NETAD. The proposed method achieves higher detection rate than the parallel system because the number o alse positives are reduced by the proposed method. When lows are generated by mistaken operation o a user (e.g., access to closed service), these lows have a tendency to be detected by the low-based detector and cause alse positives. In case that some lows are included in normal slots in the proposed method, however, the low-based detector does not need to analyze and detect such lows. This results in reduction o alse positives. Table 3. A limitation number o alse positives or each day set in evaluation or each detector. Timeslot Flow Header Payload Header Payload Table 4. A comparison with other IDSs. IDS Detection rate (detected attacks/detectable attacks) Expert-1 [16] 50.3% (85/169) Expert-2 [17] 46.8% (81/173) Dmine [18] 40.2% (41/102) Forensics [19] 55.6% (15/27) NETAD [20] 71.4% (132/185) Parallel system [4] 60.8% (104/171) Proposed system 68.4% (117/171) Figure 4. Changes in R with respect to s R. ac NETAD indicates higher detection ratio than the proposed system. However, NETAD has some drawbacks in practical situations. It detects intrusions using the appearance number o IP addresses. This causes many alse positives in networks which provide services to any users. Moreover, intruders can easily evade the NETAD s detection because the IP addresses o intruders can be regarded as normal i they can access victims normally beorehand. In this regard, the proposed system can be applied to any networks because it does not use eatures such as IP addresses. Next, NETAD analyzes traic using only the irst ew portion o payloads included in each packet. Thereore, NETAD is not able to detect intrusions which have anomalies the latter hal o packet payloads. On the other hand, the proposed system analyzes all payloads, and can detect these intrusions. Consequently, the proposed system will work well in practical situations.

7 24 Y. WAIZUMI ET AL. 5. Conclusions and Future Works In this paper, we proposed a multi-stage anomaly detection system which is combination o timeslot-based and low-based detectors. To obtain high intrusion detection accuracy, a detection system should analyze each observed low in detail. This low-based analysis, however, needs high computational cost and large buer to store lows. The computational cost and buer size can be vulnerability or DoS attacks. To avoid this potential risk, the proposed system reduces the computational cost and buer size by adopting timeslot-based detection modules, which can work with lower computational cost and smaller buer size, at the stage o prior to low-based detector. In the detection experiment, we demonstrated that the proposed system can reduce the number o lows which needs to be analyzed at the low-based detection modules to 40 percent with high detection accuracy compared with existing intrusion detection systems. Thus, the proposed system can avoid the risk which arises rom the computational cost and buer size with high detection accuracy. But, some lows were classiied as non attack lows at the irst detection stage. This is a potential drawback o the proposed system and a uture work. REFERENCES [1] M. Roesch, Snort-Lightweight Intrusion Detection or Networks, LISA 99 Proceedings o the 13th USENIX Conerence on System Administration, USENIX Association, Berkeley, 7-12 November [2] D. Anderson, T. F. Lunt, H. Javits, A. Tamaru and A. Baldes, Detecting Unusual Program Behavior Using the Statistical Component o the Nextgeneration Intrusion Detection Expert System (NIDES), Computer Science Laboratory SRI-CSL 95-06, May [3] R. Sekar, M. Bendre, D. Dhurjati and P. Bollineni, A Fast Automaton-Based Method or Detecting Anomalous Program Behaviors, Proceedings o the 2001 IEEE Symposium on Security and Privacy, Oakland, [4] Y. Sato, Y. Waizumi and Y. Nemoto, Improving Accuracy o Network-Based Anomaly Detection Using Multiple Detection Modules, Proceedings o IEICE Technical Report, NS , 2004, pp [5] P. Barord, J. Kline, D. Plonka and A. Ron, A Signal Analysis o Network Traic Anomalies, Proceedings o ACM SIGCOMM Internet Measurement Workshop (IMW) 2002, Marseille, November 2002, pp doi: / [6] T. Oikawa, Y. Waizumi, K. Ohta, N. Kato and Y. Nemoto, Network Anomaly Detection Using Statistical Clustering Method, Proceedings o IEICE Technical Report, NS , IN , CS , Oct, 2002 pp [7] Y. Waizumi, D. Kudo, N.Kato and Y. Nemoto, A New Network Anomaly Detection Technique Based on Per- Flow and Per-Service Statistics, Proceedings o International Conerence on Computational Intelligence and Security, Xi an, December 2005, pp [8] A. Lakhina, M. Crovella and C. Diot, Characterization o Network-Wide Anomalies in Traic Flows, Proceedings o the ACM/SIGCOMM Internet Measurement Conerence, Taormina, October 2004, pp [9] DARPA Intrusion Detection Evaluation, MIT Lincoln Labortory, Lincoln, [10] Inmon Corporation, Flow Accuracy and Billing, [11] N. Duield, C. Lund and M. Thorup, Properties and Prediction o Flow Statistics rom Sampled Packet Streams, Proceedings o ACM SIGCOMM Internet Measurement Workshop (IMW), Marseille, 6-8 November doi: / [12] N. Duield, C. Lund and M. Thorup, Flow Sampling under Hard Resource Constraints, Proceedings o ACM SIGMETRICS, New York, June [13] NeFlow, w/index.shtml. [14] P. Akritidis, K. Anagnostakis and E. P. Markatos, Eicient Content-Based Detection o Zero-Day Worms, Proceedings o the International Conerence on Communications (ICC 2005), Seoul, May [15] R. Lippmann, J. W. Haines, D. J. Fried, J. Korba and K. Das, The 1999 DARPA O-Line Intrusion Detection Evaluation, Computer Networks, Vol. 34, No. 4, 2000, pp doi: /s (00) [16] P. Neumann and P. Porras, Experience with EMERALD to DATE, Proceedings o 1st USENIX Workshop on Intrusion Detection and Network Monitoring, Santa Clara, 9-12 April 1999, pp [17] G. Vigna, S. T. Eckmann and R. A. Kemmerer, The STAT Tool Suite, Proceedings o the 2000 DARPA Inormation Survivability Conerence and Exposition (DIS- CEX), Hilton Head, January [18] S. Jajodia, D. Barbara, B. Speegle and N. Wu, Audit Data Analysis and Mining (ADAM), [19] M. Tyson, P. Berry, N. Willams, D. Moran and D. Blei, DERBI: Diagnosis, Explanation and Recovery rom computer Break-Ins, [20] M. Mahoney, Network Traic Anomaly Detection Based on Packet Bytes, Proceedings o ACM-SAC, Melbourne, 9-12 March 2003, pp