RUSMA MULYADI. Advisor: Dr. Daniel Zeng

Transcription

1 Evaluating Classification Algorithms for Intrusion Detection Systems RUSMA MULYADI Advisor: Dr. Daniel Zeng A Master Project Report Submitted to the Department of Management Information Systems In Partial Fulfillment of the Requirements For the Degree of Master of Science With a Major in Management Information Systems In the Eller College of Business The University of Arizona

2 TABLE OF CONTENT Evaluating Classification Algorithms for Intrusion Detection Systems... 1 Table of Content... 2 ABSTRACT INTRODUCTION LITERATURE REVIEW Practical Application Data Mining in Intrusion Detection MADAM ID of Columbia University Knowledge Discovery and Data Mining (KDD) Cup Related work EXPERIMENT Data Preprocessing Data mining tools Weka SVM Light Voting program Result Dataset Dataset Dataset 3 and the Voting Mechanism Evaluation CONCLUSION Appendix B: LIST OF FEATURES [14] Appendix C: CONFUSION MATRIX REFERENCES Page 2 of 42

3 ABSTRACT Based on the belief that different classification algorithms have their unique detection capabilities in identifying different attack categories, this paper proposes a classification model that allows us to benefit from the distinctive strength of different classification algorithms. As a preliminary experiment, this paper evaluates the detection ability of four classification algorithms (Neural Network, Naïve Bayes, RIPPER, and Support Vector Machines) using the KDD Cup 1999 data and utilizing Weka and SVM light programs. The results of these algorithms are then compared among themselves and also with the result of our initial voting mechanism that bases its final prediction only on the majority votes. Furthermore, they are also evaluated against the winning entry of the KDD Cup This experiment shows that the Naïve Bayes algorithm outperforms the other participating classification algorithms, including the result of our initial voting mechanism. We believe that this is caused by the current majority votes bias embedded in our initial voting mechanism. Keywords Intrusion detection system (IDS), data mining, misuse detection, anomaly detection, Neural Network, Support Vector Machines, Naïve Bayes, voting. Page 3 of 42

4 1 INTRODUCTION The CERT Coordination Center (CERT/CC), a security emergencies and incidents coordination center, formed by the Defense Advanced Research Projects Agency (DARPA) at the Software Engineering Institute (SEI) of Carnegie Mellon University, has received approximately 199,094 incident reports in the last three years ( st Quarter of 2003) which accounts for 88.5% of the total reports (225,049) they received from its inception in 1988 [1]. Along with this fact, CERT/CC reports that the persistent advancement of the attack techniques over the last few years due to the expertise of experienced intruders has produced complicated but easy to use intruder tools [2]. Consequently, achieving total computer network security in today s Internet environment requires a constant effort in securing, monitoring, testing, and continuous improvement of the network security itself [3]. Furthermore, one of the popular mechanisms for monitoring computer network security is the use of intrusion detection systems. The most common intrusion detection systems (IDSs) products available today are misuse detection based, in which intrusions are identified based on predetermined signatures, sets of rules that represent particular attacks. To cope with the rapid emergence of new vulnerabilities and exploits, signature updates are released periodically by the IDSs vendors [4]. Even though these IDSs can be considered effective and efficient in detecting known attacks, their inability to detect unknown attacks and the large number of false positives they generate remain their major constraints. The main reason for this last constraint is the difficulty in developing unique signatures, hence creating a huge amount of alarms that require further analysis, from the security analyst, Page 4 of 42

5 to identify whether they alert for real intrusions. For this reason, the data mining approach, an alternative that is very close to the anomaly detection strategy, has been receiving increased attention from the IDS researchers. Stolfo and Lee [5] introduce a systematic approach that utilize data mining techniques to build intrusion detection models. Data mining involves various types of algorithms, including the classification algorithm that categorizes data into predetermined categories. With regard to intrusion detection systems, the classification algorithm categorizes the audit data into either the normal or the intrusion class; therefore, it is one of the critical factors for successful implementation of data mining strategy for intrusion detection systems. The Fifth International Conference on Knowledge Discovery and Data Mining (KDD) held a competition with the task to build intrusion detection classifiers in Among the four attack categories included in the competition dataset, it appears the KDD Cup 1999 winner managed to achieve a high prediction rate for both the probe and denial of service (dos) attack categories, but not for the other two attack classes, i.e., the unauthorized access to local superuser (root) privileges (u2r) and the unauthorized access from remote systems (r2l) attack classes. Hence, it can be concluded that one of the major challenges of the data mining approach is identifying the classification model that can well fit and is applicable in the field of intrusion detection system. As one solution to this problem, we propose a classification model that allows us to benefit from the unique strength of different classification algorithms. Since the different categories of intrusions have unique characteristics, we will select algorithms that are known to have the capabilities of detecting the four main intrusion types, (1) Page 5 of 42

6 denial of service, (2) unauthorized access from remote systems, (3) unauthorized access to local superuser (root) privileges, and (4) probing. We will also assign different weight to each algorithm depending on its ability to detect the different intrusion types and use a voting mechanism to determine the ultimate prediction, based on the calculated majority weighted score. For the success of this voting mechanism, the preliminary selection of the participating algorithms is very critical to prevent the voting mechanism from falling into false majorities. This problem might occur when most of the selected algorithms fail or falsely predict an intrusion. This approach is primarily aimed to reduce the number of false alarms generated by both the current anomaly and misuse based intrusion detection systems. In this preliminary experiment of the model, we first evaluated the intrusion detection capability of four different algorithms, i.e., Neural Network, Naïve Bayes, RIPPER, and Support Vector Machines and then used a voting mechanism to identify the final prediction. The voting mechanism implemented in this experiment base its ultimate prediction on the majority votes; the weighting logic has not been embedded in the voting mechanism yet due to the exploratory nature of this experiment. The result of this experiment would help us determine the algorithms that can be incorporated in the future implementation of our proposed approach. This rest of this report is organized as follows. Chapter 2 discusses the literature review of the intrusion detection concepts, including a brief description of several commercial intrusion detection systems and previous research in the field. Chapter 3 details the experiment conducted as part of this report. In addition, the result is discussed Page 6 of 42

7 and evaluated, based on those of the previous research. Finally, Chapter 4 consists of the discussion of future work and conclusion. 2 LITERATURE REVIEW Anderson (1980) defines intrusion attempt or threat as the potential possibility of a deliberate unauthorized attempt to access information, manipulate information, or render a system unreliable or unusable (as cited by Sundaram [6]). Examples of intrusion include denial of service, probing, and malicious use of computer systems. In her book Intrusion Detection, Bace [7] defines intrusion detection as a mechanism for monitoring breaches of information technology (IT) security policy. She elaborates the three key components of an intrusion detection system as (1) the information or data source, (2) the analysis engine, and (3) the response component. As the audit data are collected from their sources the operating system audit log and the network traffic log, they are passed through an analysis engine for detection of possible intrusions. The response component then proceeds with the appropriate action such as logging the intrusion events for later reporting and automatic blocking of the intrusions based on the outcome of the analysis engine. With relation to the above analysis engine component, there are two major intrusion detection strategies presented by Anderson (1980), Denning and Neumann (1985), Denning (1987), and Sebring et al. (1988) (as cited by Axelsson [19]). The first approach, anomaly detection, detects attacks by identifying deviations of a system s behavior from its normal condition. The second approach, signature detection, determines intrusions based on predetermined signatures, a collection of rules representing the pattern of a particular attack. Page 7 of 42

8 The Cisco Secure Intrusion Detection Systems book [3] describes the advantages and disadvantages of each strategy in great detail. The main benefit of the anomaly detection approach is its ability to detect unknown or new attacks because it flags any deviations from normal behavior as intrusions. In addition, it hides the training data used in building the normal profile from the attackers and, thus, they do not exactly know which activities trigger a certain intrusion alarm. However, this training process can be very time and resource consuming and is sometimes insufficient because the changes in people s activities complicate the decision of appropriate training level. According to the Cisco Secure Intrusion Detection Systems book [3], the signature detection approach is very effective and efficient for locating known attacks. Since each signature can be easily associated with a certain attack and is based on known intrusions, the false positive alarms rate is much lower than that of the anomaly detection approach. Moreover, this approach is easier to understand. While most of today s intrusion detection systems are based on the signature detection strategy, as described in the next section of this chapter, its inability to detect unknown attacks remains its major constraint. Due to the various drawbacks in the two analysis approaches above, the search for a more efficient and effective analysis engine to develop a more powerful intrusion detection system is still evolving, thereby introducing various alternative intrusion detection strategies. To illustrate, Halme and Kahn (1988) and Lunt (1988) present the idea of combining the two intrusion detection strategies (as cited in [19]). Also, Stolfo and Lee [13] discuss MADAM ID (Mining Audit Data for Automated Models for Page 8 of 42

9 Intrusion Detection), a novel framework that applies data mining techniques to build intrusion detection models. 2.1 Practical Application The popular commercial intrusion detection systems currently available in the industry are briefly described in this section based on the white papers and the product datasheets published in their websites. Cisco s intrusion detection system product, known as Cisco Secure IDS, utilizes the mix of simple pattern matching, stateful pattern matching, protocol decode-based signatures and heuristic-based signatures [8]. While the simple pattern matching focuses the intrusion analysis only on a single packet, the stateful pattern matching inspects multiple packets within the state of a connection stream. The protocol decode-based signatures, extending the stateful pattern matching technique, examine the elements of the protocols for violations from the RFCs. The heuristic-based signatures perform algorithmic or statistical evaluation on the network traffic. Among the above techniques, the protocol decode-based signatures occupy most of Cisco s signatures. The Sourcefire Network Sensor, a component of the Sourcefire Intrusion Management System (IMS), is based on the open source rule-based network intrusion detection technology Snort. Some preprocessors are employed to perform complex stateful protocol analysis and normalization, detecting protocol anomalies through a variety of methods [9]. The Dragon Network Sensor, a component of the Dragon 6.0 Intrusion Detection solution produced by Enterasys Networks, bases its detection strategy on signaturebased pattern matching, protocol monitoring, and anomaly detection techniques [10]. Page 9 of 42

10 The Internet Security Systems RealSecure Network Sensor and Gigabit Network Sensor utilize a combination of 7-layer protocol analysis and attack pattern matching detection technologies to interpret network activity [11]. The Network Flight Recorder s network intrusion detection system employs advanced signature and stateful protocol analysis for many network-based protocols [12]. 2.2 Data Mining in Intrusion Detection Data mining is defined by Fayyad et al. (1996) as the process of (automatically) extracting useful models from large stores of data (as cited by Stolfo and Lee [5]). Stolfo and Lee [5] identify the three types of algorithms that are important for mining audit data as: the classification algorithms, the link analysis algorithms, and the sequence analysis algorithms. The classification algorithms, categorizing the audit data into several predetermined classes (for instance, the normal class or the attack class), output classifiers such as decision trees or rules. The link analysis algorithms identify correlations among the features extracted from the audit data and generate association rules. The sequence analysis algorithms, representing the sequential patterns of the audit events, present the basis for expanding intrusion detection models to include temporal statistical measures MADAM ID of Columbia University In their article [13], Lee and Stolfo present MADAM ID (Mining Audit Data for Automated Models for Intrusion Detection), a novel framework that applies data mining techniques to build intrusion detection models. In this framework, the raw audit data, after being preprocessed into connection records, are passed through data mining and Page 10 of 42

11 classification programs for features extraction and detection models construction, respectively. The classification algorithm used in this framework is RIPPER. To enhance the accuracy, efficiency, and scalability of the framework, the authors utilize metalearning (as cited from Chan and Stolfo, 1993), a mechanism for inductively learning the correlation of predictions by a number of (base) classifiers, to merge information from the various detection models. To evaluate their framework, Lee and Stolfo employed the algorithms and tools of MADAM ID to process both the tcpdump and the Solaris BSM audit data, obtained from the 1998 DARPA Intrusion Detection Evaluation Program. The resulting models from the two different audit data were compared for their ability to detect various types of attacks. Since both models appeared to provide consistent results, the authors concluded that the overall performance was not improved by combining the two models Knowledge Discovery and Data Mining (KDD) Cup 1999 The task of the Third International Knowledge Discovery and Data Mining Tools Competition, held as part of The Fifth International Conference on Knowledge Discovery and Data Mining, was to build a predictive model (classifier) capable of distinguishing between bad connections, called intrusions or attacks, and good normal connections [14]. Prof. Sal Stolfo of Columbia University and Prof. Wenke Lee of North Carolina State University made available the training and test data for the competition. The data is originated from the dataset used in the 1998 DARPA Intrusion Detection Evaluation Program. This initial dataset is organized and administered by the MIT Lincoln Labs that Page 11 of 42

12 set up a LAN simulating a typical U.S. Air Force LAN peppered with intrusions to obtain nine weeks of raw tcpdump data. All attacks in the training and test datasets fall into one of four intrusions categories. First, the denial of service (dos) attack is characterized by an explicit attempt by an attacker to prevent legitimate users of a service from using that service. Examples include apache2, back, land, mailbomb, SYN flood, ping of death, process table, smurf, syslogd, teardrop, udpstorm. Second, the unauthorized access from a remote machine (r2l) attack usually involves an attacker gaining network access to a local user s account by exploiting existing vulnerabilities. Examples include dictionary, ftp_write, guest, imap, named, phf, sendmail, xlock, xsnoop. Third, the unauthorized access to local super user (root) privileges (u2r) attack involves an attacker gaining super user (root) access to a system by exploiting existing vulnerabilities using a normal user account. Examples include eject, ffbconfig, fdformat, loadmodule, perl, ps, xterm. Last, the probing (surveillance) attack is characterized by an attacker scanning a computer network to collect useful information for future attacks. Examples include ipsweep, msan, nmap, saint, satan. These attack examples are explained further in Appendix A. Evaluating the 24 entries submitted for the competition using the cost matrix in Table 6, the mean cost value of KDD Cup 1999 winning entry is calculated as while those of the non-winning entries range from to [29]. Since there is no significant average cost difference between entries with adjacent ranks, all the best seventeen entries can be considered as performing well and the worst seven entries as inferior. Page 12 of 42

13 The KDD Cup 1999 winning entry, submitted by Pfahringer [15], utilizes the C5 boosted trees for constructing the final predictor of an ensemble of 50x10 C5 decision trees. The author describes the process itself as cost-sensitive bagged boosting. First, fifty samples were drawn from the initial 5 million odd examples set, each sample always included 4000 examples of the probe attack class, of the normal class, of the dos attack class and all examples of the two smallest classes, i.e., the u2r and r2l attacks. Moreover, duplicate entries were removed from the original data set. Second, an ensemble of ten C5 decision trees was induced using both C5 s error-cost and boosting options for each sample. Last, final predictions were calculated from the 50 single predictions by minimizing the conditional risk. It appears that the KDD Cup 1999 data has then become a widely used and accepted dataset for evaluating data mining classification algorithms in the intrusion detection field. Mukkamala et al. [16] evaluate the performance of neural networks (from the MATLAB package) and support vector machines (from the freeware SVM light program) for application in intrusion detection systems. Applying these algorithms on the KDD Cup1999 data, they extracted the features that model users system behavior, and employed them to construct intrusion detection classifiers. Mukkamala et al. conclude that although the training and testing time for the support vector machines are significantly shorter than those of the neural networks, both algorithms perform very well as indicated by their high accuracy rate (>99%). In addition, they identify the major shortcoming of the support vector machines as the fact that it can only perform binary classification. Page 13 of 42

14 In another article [17], Mukkamala et al. examine the importance of input features selection to improve the performance of intrusion detection system, based on the argument that the less input features used, the less processing resources required and most likely the more accurate the outcomes are. They utilized two ranking techniques the performance based input ranking method and the performance metrics method to rate the significance of input features for each attack categories (normal, probe, denial of service/dos, unauthorized access to local superuser (root) privileges/u2r, and unauthorized access from remote systems/r2l) in the KDD Cup 1999 data. In the performance based technique, Mukkamala et al. removed one input feature from the data at a time, used the resultant dataset to train and test the classifiers, compared the performance of the classifiers to its initial performance with all input features, and ranked the importance of each input feature based on the comparison result. The authors measured three criteria the overall accuracy of classification among the five attack categories, the training time, and the testing time with regard to the performance metrics method. Mukkamala et al. report that the two ranking methods perform consistently as they generate a large number of overlapping features for most of the attack categories except for the normal and the u2r categories. In addition, they conclude that there are no significant performance differences among using the important features of each attack category, using the union of all important features, using the union of important and secondary features for each attack category, and using all the 41 features provided by KDD Cup 1999 data. Page 14 of 42

15 Agarwal and Joshi [18] propose PNrule, a two-stage process of rule-induction framework to solve the intrusion detection problem and use the KDD Cup 1999 data to evaluate their proposed model. The authors claim that there is an inevitable problem of small disjuncts with the general to specific classification learning techniques, because they tend to simultaneously pursue the optimization of both recall (total coverage of the target class) and precision (accuracy with respect to false positive) objectives. Agarwal and Joshi believe that overcoming each objective independently will lead to a model with better generalization capability. Agarwal and Joshi compared PNrule s overall performance to those of the 23 KDD Cup 1999 participants and evaluated PNrule s performance for binary classification of the four attack classes (probe, denial of service/ dos, user to super-user/u2r, and remote to local/r2l) based on those of RIPPER and C4.5rules. The authors find that PNrule s overall performance outperformed those of the KDD Cup 1999 participants in both the accuracy and the misclassification cost, particularly in the r2l attack category whose size and misclassification cost are very small and high respectively. In addition, PNrule significantly improves the counts of both recall and precision for the probe and r2l attack classes; it performs comparably to two other algorithms for the dos and u2r attack categories Related work In this section, various research in the area of intrusion detection, not necessarily directly related to the topic of this paper, are presented to serve as information source for our future research reference. To illustrate, they include the base-rate fallacy phenomenon of intrusion detection system, the cost sensitive intrusion detection models, the decentralized misuse detection for distributed attacks, the state transition analysis Page 15 of 42

16 technique for intrusion detection, and the different data modeling methods of intrusion detection systems. Among the numerous quality aspects of intrusion detection systems effectiveness, efficiency, ease of use, security, interoperability, transparency, and collaboration, Axelsson [19] discusses the base-rate fallacy issue constraining the effectiveness of an intrusion detection system and explains that it is the ability of an intrusion detection system to reduce false alarms, not its ability to correctly detect intrusions that limits its effectiveness. Based on the cited studies of intrusion detector performance in his article, Axelsson concludes that the performance of the anomaly-based methods are far from meeting the effectiveness standard, i.e. a false alarm rate less than 1/100,000 per event. Although the studies appear to show that the performance of current signature intrusion detection systems are quite close to the required effectiveness standard, the author still questions the validity of this generalization. Wenke et. al. [20] present the issues related to the construction of cost-sensitive intrusion detection models by examining the major IDS cost factors to build a total IDS cost model. They formulate the cumulative IDS cost model as the sum of the operational costs (the cost of processing and analyzing the streams of intrusion events) and the consequential costs (the costs incurred as a consequence of prediction, i.e.: false negative, false positive, true positive, true negative, and misclassified hit). Wenke et. al. proposes the building of multiple intrusion detection models with each using features sets from different cost levels to reduce the operational cost. In this model, the high cost models are examined only when evaluation of the low cost models does not achieve sufficient Page 16 of 42

17 prediction accuracy. To reduce the consequential costs, the authors suggest a costsensitive decision module that bases its response decision on the trade-off between the damage costs and the response costs. Wenke et. al., successfully tested the effectiveness of their proposed models by applying them to the 1998 DARPA Intrusion Detection Evaluation Program data, claim that their major contribution of the study of cost-sensitive modeling for intrusion detection is the development of a cost factor analysis framework and the construction of cost-sensitive intrusion detection models. Ning et al. [21] explore the abstraction-based misuse detection in distributed environments. They develop a framework to detect distributed attacks, based on the relationship among the distributed events involved in the attacks, and construct a decentralized detection approach by decomposing the attacks signatures into a set of detection tasks, coordinated by a distributed algorithm, to be executed in different intrusion detection systems. The authors aim to facilitate dynamic event abstraction and allow detection of unknown attacks that share the core features of the existing signature. Ning et al. [22] constructed a prototype called Coordinated Attack Response and Detection Systems (CARDS), composed of three independent but cooperative components signature manager, monitor and directory service - to assess the viability of the proposed ideas. The signature manager creates specific signatures for all affected systems of a generic signature; the monitors receive detection tasks from the signature managers and cooperate with each other to perform the tasks; and, the directory service acts as the information source for the other components. Although there are research Page 17 of 42

18 issues that need to be addressed, the authors conclude that their abstraction-based approach is feasible. Vigna et al. [23] discuss the application of the state transition analysis approach, a proven effective method for host-based intrusion detection, to detect network intrusions based on the state transition diagrams and the network hypergraphs that model the intrusions and the target network respectively. They then implemented the approach in a prototype called NetSTAT that consists of: the network fact base, the state transition scenario database, a collection of general purpose probes, and the analyzer. Participating in the MIT Lincoln Laboratory s off-line intrusion detection system evaluation and the Air Force Research Laboratory (AFRL), Vigna et al. utilized NetSTAT prototype to identify intrusion signatures and it performed very well. Michael and Ghosh [24] present two techniques to fully automate the generation of state-based characterizations a program s normal behavior; they are the state-based anomaly detector and the two string transducer (with and without confidence-based). The authors utilized the Sun s BSM audit data obtained from various sources Lincoln Laboratories, Johns Hopkins University, and their own data to evaluate the performance of their proposed algorithms. They conclude that although the n-gram matching technique performs slightly better than the two proposed algorithms, it requires a longer learning period. Warrender et al. [25] evaluate the ability of four different techniques sequence time-delay embedding (stide), stide with frequency threshold (t-stide), a data mining technique ( RIPPER, and Hidden Markov Models/HMMs) to accurately model normal behavior and identify attacks. The authors obtained the data from various sources; for Page 18 of 42

19 example, the live normal data, the different types of programs, the programs with size and complexity variation, and the different types of intrusions; and include only the programs that run with privilege. Warrender et al. s experiment showed that although the Hidden Markov Models provided an outstanding accuracy rate at high computational costs, it did not introduce much improvement when compared to the performance of the much simpler sequence time-delay embedding (stide) method. Thus, the authors conclude that simple modeling method is more than sufficient to handle system call data, unless an extraordinary accuracy rate is required with significant computational resources available. 3 EXPERIMENT In this experiment, we examine the intrusion detection ability of four data mining classification algorithms neural network, naïve bayes, RIPPER, and support vector machines using the KDD (Knowledge Discovery and Data Mining) Cup 1999 data [14]. Besides comparing the performances of these algorithms among themselves, they are evaluated against that of the KDD Cup winner. Furthermore, the results from the different algorithms are run through a voting program to determine the combined result. 3.1 Data The three datasets used in this experiment are obtained from the KDD Cup 1999 data, each connection record consists of a set of 41 features as detailed in Appendix B. Whereas the first two datasets have 10,000 records in each of its training and test data, the last one has 50,000 records. Page 19 of 42

20 Both the training and test data of the first dataset are acquired solely from a ten percent subset of the KDD Cup 1999 training dataset. On the other hand, the training and test data in the second and third dataset are obtained from a ten percent subset of the KDD Cup 1999 training and test datasets respectively. KDD Cup % training data: 4,940,200* KDD Cup % test data (with corrected label): 311,029 Dataset 1: Both training and test data are portions of 10% of KDD Cup 1999 training data Training data 10,000 Test data 10,000 Dataset 2: Training and test data are respectively portions of 10% of KDD Cup 1999 training and test data Training data 10,000 Test data 10,000 Dataset 3: Training and test data are respectively portions of 10% of KDD Cup 1999 training and test data Training data 50,000 Test data 50,000 *One record is discarded Table 1: The three datasets The main consideration when selecting the datasets is to preserve the distribution of the different attack categories and the data sequence, hence sustaining a good representation of the original dataset. For this reason, each dataset is selected as a consecutive cluster of the entire data that maintains certain distribution of the attack categories. The table below shows the distribution of attack categories for the ten percent subset of KDD Cup 1999 data and the three datasets of this experiment. KDD Cup % Train KDD Cup % Test Dataset 1 Training Dataset 1 Test Normal Attacks by category Probe DoS U2R R2L Total 19.69% 0.83% 79.24% 0.01% 0.23% 80.31% 19.48% 1.34% 73.90% 0.07% 5.21% 80.52% % % % % % % % % % % % % Page 20 of 42

21 Dataset 2 Training Dataset 2 Test Dataset 3 Training Dataset 3 Test % % % % Normal Attacks by category Probe DoS U2R R2L Total % % % % % % % % % % % % % % % % % % % % Table 2: The attack category distribution of the 10% KDD Cup data and the three datasets The following table lists the various attacks included in each dataset classified by their attack categories. Attacks types by category Probe DoS U2R R2L Dataset 1 Training Dataset 1 Test Dataset 2 Training Dataset 2 Test Dataset 3 Training Dataset 3 Test ipsweep satan portsweep nmap ipsweep satan satan portsweep ipsweep portsweep ipsweep satan portsweep mscan ipsweep nmap satan teardrop smurf neptune teardrop smurf teardrop neptune pod teardrop neptune back pod smurf back processtable udpstorm smurf neptune pod Table 3: The attacks in the three dataset Rootkit loadmodule buffer_overflow Loadmodule buffer_overflow httptunnel Rootkit loadmodule buffer_overflow rootkit rootkit buffer_overflow loadmodule loadmodule buffer_overflow multihop perl rootkit xterm ps multihop warezclient warezmaster sendmail snmpgetattack guess_passwd httptunnel multihop, warezclient warezclient spy multihop spy warezclient warezmaster imap multihop guess_passwd sendmail xlock xsnoop snmpgetattack Page 21 of 42

22 3.2 Preprocessing The preprocessing program, written in perl, comprises 3 sub processes: 1. The data validity checking, to ensure that each record of the training and test data consists of 41 features. One invalid record is found and discarded from the training data. 2. The attack categorization, to classify individual attack types into five categories using the categorization awk script [29] provided by Charles Elkan, KDD Cup 1999 organizer, into: 0: Normal 1: Probe 2: Denial of Service (DoS) 3: User-to-root (U2R) 4: Remote-to-local (R2L) 3. The input data preparation, to ensure that the input data format meets the requirement of the data mining tools. 3.3 Data mining tools Weka Weka is a collection of machine learning algorithms developed by Witten et al. [26]. Initially, three Weka classes, i.e. Neural Network, Naïve Bayes, JRIP (Weka implementation of Repeated Incremental Pruning to Produce Error Reduction / RIPPER), are used. Due to the performance constraint of Weka s original Neural Network class when dealing with large dataset, a faster Weka compatible implementation of neural network algorithm developed by Klautau [27] is later used. This experiment utilizes Weka version and accepts most of the default configuration built in Weka classes. Page 22 of 42

23 Moreover, the Naïve Bayes class is run twice, the first one uses the normal distribution for numeric attributes and the second one uses the kernel density estimator for numeric attributes. The last one is referred as NaïveBayes_Kernel in the next sections SVM Light SVM light, an implementation of support vector machines (SVMs) by Joachims [28] used in Mukkamala et al. [16], is utilized in this experiment. During the training process, the kernel option is set to radial basis function kernel type and the default regularization parameter is set to c= Since the support vector machines algorithm can only perform binary classification and this experiment requires classification of the attacks by category, five SVMs are built: the first SVM classifies the dataset into normal or abnormal connections; the last four SVMs identify the attack category of the abnormal connections Voting program The third dataset s results are run through a voting program, written in Perl, to determine the final prediction based on the majority votes. Page 23 of 42

24 3.4 Result Evaluating the overall accuracy among the classification algorithms, Neural Network appears to maintain the highest accuracy across the three datasets while consuming the most processing resources. Dataset 1 Dataset 2 Dataset 3 Neural Network 19.56% 95.35% 91.89% SVM* 18.79% 85.38% 75.12% NaïveBayes 18.87% 85.88% 87.80% NaïveBayes_Kernel 19.08% 92.94% 88.04% JRIP 17.86% 91.92% 91.85% *The accuracy obtained on binary classification (normal or abnormal connection) Table 4: Overall Accuracy of the Classification Algorithms Dataset 1 Dataset 2 Dataset 3 Build Test Build Test Build Test model model model model model model Neural Network SVM >= * NaïveBayes NaïveBayes_Kernel JRIP *This is a CPU run-time estimate; the actual CPU run-time obtained is negative Table 5: Overall CPU run-time performance (in seconds) To be able to compare the result of this experiment with the winning entries of KDD Cup 1999, the same cost matrix [29] as illustrated in Table 6 is used together with the resulting confusion matrices of each algorithm on Dataset 3 to calculate the average misclassification cost per test example. normal probe DOS U2R R2L normal Probe DOS U2R R2L Table 6: KDD Cup 1999 Cost Matrix Page 24 of 42

25 3.4.1 Dataset 1 The overall accuracy rate for this dataset is relatively low across the different classification algorithms, ranging from 17.86% to 19.56%. Further analysis of the detection rate by attack types and the attack category distribution of the dataset reveal that most of the connections that are correctly predicted by the classification algorithms are normal connections because most of the attack types introduced in the test data are new attack types that are not included in the training data as shown in Table 3. Based on the results in Table 7, all classification algorithms perform very well in determining normal connections as indicated by the high accuracy rate that ranges from 93.99% to 100%. In addition, only the probe attack category can be identified with a high accuracy rate, i.e % by both Neural Network and NaïveBayes. Although the attack types introduced in training data (ipsweep, satan) and test data (portsweep, nmap) are different, all attack types within the probe attack category usually involve making connections to numerous hosts and ports in a given period of time and hence, can be identified by the classification algorithms through the time-based traffic and hostbased traffic features embedded in the 41 features provided in KDD Cup 1999 data. Consequently, this finding supports the result shown by Lee and Stolfo [13]. Furthermore, none of the classification algorithms can identify any attacks included in the dos attack categories. The reason is because the attack types included in the test data (Neptune) is different from the ones in the training data (teardrop, smurf) and each utilizes different protocol, i.e., Neptune uses TCP, teardrop uses UDP, and smurf uses ICMP. Page 25 of 42

26 In addition, only the Neural Network can at least identify one of the 36 u2r attacks included in the test data. Even though both the training and test data share two similar attacks types (loadmodule and the buffer_overflow), it appears that the number of u2r attacks in the training data is not sufficient (in this case 38.9% of its number in the test data) for training the classification algorithm. Also, both Neural Network and Naive Bayes show a similar low detection performance in relation to r2l attack category, roughly 12% detection rate. Not only the attack types are different in both training and test data, but the number of r2l attacks in the training data is also only 37% of its number in the test data. Normal Attack Category Probe DOS U2R R2L Test data dataset Number of correct prediction Neural Network/ NN SVM NaïveBayes NaïveBayes_Kernel JRIP Detection rate (% of correct prediction) Neural Network/ NN SVM NaïveBayes NaïveBayes_Kernel JRIP Table 7: Dataset 1 Detection Rate (in % of correct prediction) Dataset 2 The overall accuracy rate for this dataset is relatively high across the different classification algorithms and the highest among the three datasets, ranging from 85.38% to 95.35%. This is because 77.71% of total test data is normal data and it is included as part of this experiment only to demonstrate how the different distribution of dataset can affect the overall accuracy performance of the classification algorithms. As also shown Page 26 of 42

27 in Table 8, all of the classification algorithms encounter difficulties in determining u2r attack types, except the Naïve Bayes which can successfully identify three of the six u2r attacks included in the test data. Normal Attack Category Probe DOS U2R R2L Test data dataset Number of correct prediction Neural Network/ NN SVM NaïveBayes NaïveBayes_Kernel JRIP Detection rate (% of correct prediction) Neural Network/ NN SVM NaïveBayes NaïveBayes_Kernel JRIP Table 8: Dataset 2 Detection Rate (in % of correct prediction) Dataset 3 and the Voting Mechanism The overall accuracy rate for this dataset across the different algorithms ranges 75.12% to 91.89%. This accuracy interval is slightly lower when compared to that of the second dataset. Among the three datasets, this third dataset is the closest representation of the KDD Cup % data for three reasons. First, both the training and test data are acquired from the KDD Cup 1999 training and test data. Second, the attack category distribution of this data set is reasonably close to that of the KDD Cup 1999 data. Last, it has the richest set of attacks among the three datasets with 14 and 27 attack types included in the training and test data respectively in addition to the total of 50,000 connection records included in each the training and test data. The results of this dataset are run through a voting mechanism that bases only on the majority votes to determine the final prediction. Due to the extensive processing Page 27 of 42

28 resource required by Neural Network, we evaluate two voting models by including and excluding Neural Network in the voting mechanism. Based on the results in Table 9, all classification algorithms perform very well in determining the normal connections and the dos attack category as indicated by the high accuracy rate that ranges from 77.61% to 99.99% and 98.84% to 100% respectively. Among the participating classification algorithms, only the Naïve Bayes shows a consistent detection performance across the different attack categories. Even though it shows the lowest detection rate with relation to identifying normal connections (only 77.61%), it presents the best overall performance among classification algorithms with regard to the four attack categories. While evaluating the two voting models, the one that includes Neural Network result appears to offer a slightly better detection rate. However, the performances of both voting methods are still inferior to that of the Naïve Bayes because of the bias within this preliminary voting mechanism that favors the majority votes regardless of the detection capability of each algorithm. Further examination of the information in Table 9 below show that Naïve Bayes is the best predictor for probe, u2r, and r2l attack categories while SVM is the best predictor, with slightly better detection rate among the others, for the normal connections and the dos attack category. Page 28 of 42

29 Normal Attack Category Probe DOS U2R R2L Test data dataset Number of correct prediction Neural Network/ NN SVM NaïveBayes NaïveBayes_Kernel JRIP Voting without NN Voting with NN Detection rate (% of correct prediction) Neural Network/ NN SVM NaïveBayes NaïveBayes_Kernel JRIP Voting without NN Voting with NN Table 9: Dataset 3 Detection Rate (in % of correct prediction) Using the cost matrix in Table 6 and the confusion matrices in Page 29 of 42

30 Appendix C, the average misclassification cost per test example for each classification algorithm except SVM is calculated. As shown in Table 10, Naïve Bayes achieves the lowest average cost per test example, which is consistent with its best detection rate among all participating algorithms as illustrated in Table 9 above. Dataset 3 Neural Network/ NN SVM* NaïveBayes NaïveBayes_Kernel JRIP Voting without NN Voting with NN * Data not available Table 10: Average Cost per Test Example 3.5 Evaluation The comparison between the best classification algorithm participant in this experiment, i.e. Naïve Bayes and the winning entry of KDD Cup 1999 is presented in this section. In term of the average misclassification cost, Naïve Bayes achieves a cost of , which is slightly higher than the cost of achieved by KDD Cup 1999 winning entry. When comparing the Naïve Bayes confusion matrix (Table 12) with the one produced by the KDD Cup 1999 winning entry (Table 13), it appears that Naïve Bayes offers higher accuracy rate in detecting dos, u2r, and r2l attack categories than the KDD Cup 1999 winning entry. The main drawback of the Naïve Bayes is in the normal connection detection capability where it can only attain an accuracy rate of 77.61% compared to the 99.50% of the KDD Cup 1999 winning entry. With regard to the probe attack category, both the Naïve Bayes and the KDD Cup 1999 winning entry present comparable results, roughly 83% accuracy rate. Page 30 of 42

31 One possible argument for the higher accuracy produced by Naïve Bayes on dos, u2r, and r2l is the attack category distribution of Dataset 3 that is not an exact match of that of the KDD Cup 1999 Data as shown in Table 11. Distribution of Attacks by category DoS U2R R2L Dataset 3 - Training 64.92% 0.04% 2.05% Dataset 3 - Test 63.24% 0.07% 6.14% KDD Cup % Train 79.24% 0.01% 0.23% KDD Cup % Test 73.90% 0.07% 5.21% Table 11: Comparison of Attack Category Distribution predicted actual % correct % % % % % % correct 96.74% 91.29% 87.94% 2.92% 54.09% Table 12: Dataset 3 Naïve Bayes Confusion Matrix predicted actual % correct % % % % % % correct 74.60% 64.80% 99.90% 71.40% 98.80% Table 13: KDD Cup 1999 Winning Entry Confusion Matrix 4 CONCLUSION Most of the classification algorithms perform very well in detecting the normal, probe and dos connection categories, but not the r2l and u2r connection categories because only a small percentage of the whole dataset is usually categorized into r2l and u2r attack categories. For the same reason, the overall accuracy rate is always relatively high regardless of the ability of the classification algorithm in detecting either the r2l or Page 31 of 42

32 u2r attack category; therefore, it is rather misleading to decide on the performance of a classification algorithm solely based on the overall accuracy rate, i.e. the overall percentage of correct predictions. Based on our experiment result, Naïve Bayes outperforms all the other participating classification algorithms, including the result of our preliminary voting mechanism, particularly in the r2l and u2r attack categories. To overcome the current bias embedded in the voting mechanism, our future voting mechanism would need to assign greater weight on the Naïve Bayes prediction, especially with regard to the r2l and u2r attack categories. Further work is still needed to construct a robust formula for the voting mechanism. With regard to the relatively low detection rate of both the u2r and r2l attack categories, there are still needs for searching and evaluating other classification algorithms that can provide better detection rate. In addition, the domain experts may also need to be consulted to extract additional input features that are useful for detecting these two attack categories. Although this experiment shows that the detection rates for the other three connection categories (normal, probe, and dos) are relatively high among all the participating algorithms, further examination is needed to determine the most costeffective classification algorithms. The areas for future research include implementing the customized voting mechanism that embeds the different weights placed on each participating classification algorithm and incorporates a cost-sensitive model. Moreover, the biggest challenge will be to build and implement a framework that combines the anomaly detection approach and the data mining techniques, thereby establishing an intrusion detection system that Page 32 of 42