THE BAYESIAN RECEIVER OPERATING CHARACTERISTIC CURVE AN EFFECTIVE APPROACH TO EVALUATE THE IDS PERFORMANCE

Transcription

1 БСУ Международна конференция - 2 THE BAYESIAN RECEIVER OPERATING CHARACTERISTIC CURVE AN EFFECTIVE APPROACH TO EVALUATE THE IDS PERFORMANCE Evgeniya Nikolova, Veselina Jecheva Burgas Free University Abstract: The aim of intrusion etection systems is to protect the computer networks from attacks or unauthorize access to their valuable resources. For that purpose they have to perform real-time processing of current system activity an make a ecision whether the pattern is normal or intrusive. The present paper is an extension to our previous work, which presente an anomaly-base IDS moel, base on some ecoing algorithms. It introuces the evaluation of the propose methoology, using the Bayesian Receiver Operating Characteristics Curve. Key wors: atabase, www programming, ODBC, JDBC atabase connections.. Introuction Information security is an important issue in the contemporary ubiquitous network environments. Intrusion etection systems (IDS) monitor the events occurring in a network or a single host an analyse them in orer to etect attempts to security violations. The two main categories of IDS principles are signature etection (misuse etection) an anomaly etection []. Signature etection attempts to ientify events that misuse a system an is performe by creating moels of known intrusions. Current events are compare against preliminarily escribe intrusion moels to make a etection ecision. This metho is goo in iscovering known attacks, but cannot etect new intrusions with high egree of probability. Anomalybase IDS, on the contrary, create a moel of normal system use an look for activity that oes not conform [6]. Any significant eviation is labele as an attack, since it oes not fit the efine moel. The concept of IDS quality is critical, but not clearly efine yet. There are many ctors to consier when evaluating IDSs: spee, cost, effectiveness, ease-of-use, scalability, interoperability, etc. Since the main task of IDS is to recognize whether an intrusion attempt is present or absent, the etector s performance coul be consiere as the most important feature of any IDS [5]. Classification accuracy IDSs eals with such funamental problems as how to compare two or more IDSs, how to evaluate the performance of IDS, an how to etermine the best configuration of the IDS [2]. Regarless of the unerlying methoology, an IDS performance can be treate as a binary classification problem an therefore escribe by its receiver operating characteristic 248

2 БСУ Международна конференция - 2 (ROC) curve [4]. It is a graph of the etection probability versus lse alarm rate. It takes into account both lse positives, when the IDS flags normal activity as anomalous an lse negatives, when the IDS reports an attack as normal activity in error. The ROC curve coul be applie in orer to analyse the traeoff between the two types of errors. A methoology with perfect iscrimination has a ROC plot that passes through the upper left corner, consequently the closer the ROC plot is to the upper left corner, the higher the overall accuracy of the test []. In the case of intrusion etection however, the set of normal activity ata contains many more patterns than the set, which contains the intrusion ata. Uner this setting, many classifiers ten to concentrate on the large classes an isregar the ones with small number of patterns. As a result, IDSs with ROC curves achieving goo operating points still prouce a large amount of lse alarms in real environments. The present paper consiers etection effectiveness as a general metric to compare IDSs. More etails about the applie intrusion etection methoology, which is base on the Junction Tree algorithm an the conucte simulation experiments an their results, are presente in our previous works [7, 8]. The IDSs performance evaluation is base on the Bayesian Receiver Operating Characteristic Curve [3]. B-ROC provies a better way to evaluate an compare classifiers in the case of class imbalances, compare to the classical ROC curves. B-ROCs can be use for comparing classifiers without any assumptions of misclassification costs. 2. The Bayesian Receiver Operating Characteristic Curve Let I enotes whether a given observation x was generate by an intrusion (represente by I=) or not (enote as NI=). Also let A enotes whether the output of an IDS is an alarm (enote by A=) or not (enote by NA=). An IDS can then be efine as an algorithm that receives a continuous ata stream X={x,x 2,...,} an classifies each input x j as being either a normal event or an attack, i.e. IDS: X A, NA. The system can be in one of two states or conitions: either with an intrusion present (I) or with no intrusion present (NI). The prior probability of an intrusion is calle p=p[i=]. The IDS reports either an intrusion alarm (A) or no alarm (NA). The parameters of the IDS s ROC curve are: the probability of an alarm given an intrusion, the etection probability, p =P(A I)= β (or the probability of no alarm given an intrusion, P(NA I)=β), an the probability of an alarm given no intrusion, the lse alarm probability, p =P(A NI)=α. Thus, α an β are the probabilities of the two types of reporting errors. An IDS s receiver operating characteristic (ROC) curve escribes the relationship between the two operating parameters of the IDS, its probability of etection, p = β, an its lse alarm probability, p =α. That is, the ROC curve isplays the β provie by the IDS at a given α. It also isplays the α provie by the IDS at a given β. The ROC curve thus summarizes the performance of the IDS. A plot lying above an to the left of another plot inicates greater observe accuracy. If on a graph containing ROC curves, a single curve lies outsie of every other curve, then it ominates the others completely. It is possible for an ROC curve to ominate partially, meaning over a certain region of the ROC space but not over the entire graph. Area uner the ROC curve (AUC) is a measure of the overall performance of a iagnostic test an is interprete as the average value of sensitivity for all possible values of specificity. It can take on any value between an, since both the x an y axes have values ranging from to. The closer AUC is to, the better the performance of the test, an a test with an AUC value of is one that is perfectly accurate. 249

3 БСУ Международна конференция - 2,2 ROC curve for p 3,8,6,4,2,2,4,6,8,2 synthetic senmail Figure. ROC curves of synthetic senmail an name In Figure, although the AUC of the two tests have the same value of,94, the two curves ominate over a select region of the ROC graph. This is a typical problem with the comparison of classifiers by using ROC. Since in general case it is ifficult to etermine misclassification costs, we can get a better comparison of two classifiers without them, using the Bayesian receiver operating characteristic (B-ROC) curve. The Bayesian etection rate or Positive Preictive Value (PPV) is the posterior probability of intrusion given that the IDS fire an alarm, i.e. the probability that a pattern is intrusive when restricte to those patterns which test positive. The PPV coul be compute as follows: name PPV pp. p. p ( p). p, where p is a priori probability, p - the probability of etection, p - the probability of lse alarm. Its value is maximize when the lse alarm rate of etector goes to zero, even if the etection rate also tens to zero. Therefore we exchange the PPV with the Negative Preictive Value (NPV): NPV p. p. p ( p). p p. It represents the patterns with negative test, which are not anomalous, i.e. are results of normal system activity. The NPV assesses the reliability of negative test performance. The B-ROC curve is a metho of graphically emonstrating the relationship between an PPV, pwhere we use for x-axis -PPV the Bayesian lse alarm rate. This curve is a well efine continuous an represents non-ecreasing function. B-ROC curves get a better comparison between the two classifiers without the assumption of any misclassification costs as might be seen in Figure 2: 25

4 sensitivity БСУ Международна конференция - 2 B-ROC curve for p 3,2,8,6,4,2,2,4,6,8 synthetic senmail name Figure 2. B-ROC curves of synthetic senmail an name 3. The B-ROC curve for comparison of classifiers In a series of previous works the authors examine the application of some ecoing algorithms an techniques in anomaly-base IDS, since this recognition problem was consiere as a ecoing problem. The propose methoology consists of two stages the first contains the HMM creation an its ajustment using the graient metho, an the secon one inclues the intrusion recognition itself. The authors applie the well-known techniques as the Bahl-Cocke-Jelinek-Raviv (BCJR or the MAP) ecoing algorithm, the max-log-map algorithm an the junction tree algorithm (JTA) uring the recognition stage. The obtaine results are presente an iscusse in [9]. Since the intrusion etection is a binary classification (normal or intrusive activity), many statistical methos for the results evaluation were applie FPR, FNR, sensitivity, specificity, accuracy, ROC curves, etc. A ROC curve is a frequently applie non-parametric approach for binary classification metho evaluation. In a ROC curve each p value can be plotte against its corresponing p value to create the iagram for the examine processes for IDS base on BCJR algorithm (see Figure 3). The points in the upper left corner of the ROC space, which is prouces by the propose methoology for the processes synthetic ftp, name an xlock are (,9;,97), (,5;,98), (,3;,99) respectively.,2 ROC curve,8,6,4,2 synthetic ftp xlock name -,2,2,4,6,8,2 -specificity Figure 3. The ROC curve for IDS base on BCJR algorithm In orer to evaluate the effectiveness of IDS base on the Max-log MAP algorithm the PPV, NPV an accuracy were calculate for the processes name, synthetic ftp an xlock, which results are presente in Table : 25

5 sensitivety БСУ Международна конференция - 2 synthetic ftp name xlock PPV=4,9% PPV=7,5% PPV=7,9% NPV=99,5% NPV=99,4% NPV=99,4% Accuracy=87,8% Accuracy=89,8% Accuracy=9,% Table. The PPV, NPV an accuracy for the examine processes Consiering the obtaine preictive values, we see in Table that all PPV are between 4,9% an 8,7%, while all NPV are between 99,3% an 99,5%. Since the positive preictive values refer to the chance that a positive test result will be correct, the obtaine results show that the propose metho correctly classifies the patterns with high egree of probability. On the other han, negative preictive value is concerne only with negative test results. From the Table we see that the propose methoology prouces results with excellent negative preictive values. The both preictive values epen on the prevalence of the intrusions, since they epen on the number of true positives an lse negatives an true negatives an lse positives, respectively. Since the accuracy values for all processes belong to the interval (87%, 92%), we can conclue that the propose methoology prouces precise an reliable etection results. Figure 4 contains the ROC curves for the examine processes, obtaine by JTA. The points in upper left corner of the ROC space for the processes synthetic senmail, synthetic ftp, name an xlock are (,6;,94), (,5;,99), (,55;,95) an (,25;,85) respectively. As the point (, ) enotes the perfect etection, the propose methoology prouces reliable an qualitative results while istinguishing the normal activity from abnormal one. ROC curve - JTA synthetic senmail synthetic ftp name xlock specificity Figure 4. ROC curve for IDS base on JTA It is clear from figures 3 an 4 that the comparison between results, obtaine for the ifferent processes when the corresponing ecoing algorithm was applie, is not a trivial task, since the algorithms prouce results with very small eviations. In orer to compare the effectiveness of IDS base on these algorithms we plot the B- ROCs, which results are presente in Figures 5-7: 252

6 БСУ Международна конференция - 2,2,8,6,4,2,2,4,6,8 BCJR JTA Max-log MAP Figure 5. B-ROC curves for synthetic ftp,2,8,6,4,2,2,4,6,8 BCJR JTA Max-log MAP Figure 6. B-ROC curves for name,2,8,6,4,2,2,4,6,8 BCJR Max-log MAP JTA Figure 7. B-ROC curves for xlock 253

7 БСУ Международна конференция - 2 It coul be seen from figures 5-7 that the methoology, base on BCJR ecoing algorithm, outperforms the two other methoologies for the three examine processes. The methoology, base on JTA, prouces balance results for the processes name an synthetic ftp, an prouces better classification results compare to the methoology, base on max-log- MAP algorithm. At last, the methoology, base on max-log-map algorithm, prouces better classification results, compare to those, prouce by JTA for the process xlock. We shoul mention that regarless of the ct, that methoology, base on BCJR ecoing algorithm, outperforms the two other methoologies for the examine processes, the two other methoologies prouce reliable an qualitative results while istinguishing the normal activity from abnormal one. Conclusion We believe that the B-ROC provies a better way to evaluate an compare classifiers in the case of uncertain values of p. When comparing two classifiers, there are cases in which by using the B-ROC, we o not nee cost values in orer to ecie which classifier woul be better for given values of p. Note also that B-ROCs consier parameters that are irectly relate to exact quantities that the operator of a classifier can measure. References:. Axelsson S., Intrusion Detection Systems: A Taxonomy an Survey, Technical Report No 99-5, Dept. of Computer Engineering, Chalmers University of Technology, Sween, March Carenas A. A., J. S. Baras, K. Seamon, A framework for the evaluation of intrusion etection systems, Proceeings of the 26 IEEE Symposium on Security an Privacy, May 26, Berkeley/Oaklan, CA, ISBN: , pp Carenas A. A., J. S. Baras, B-ROC curve for the assessment of classifiers over imbalance ata sets, Egan J.P., Signal etection theory an ROC-analysis, Acaemic Press, Gaffney J.E., J.W. Ulvila, Evaluation of Intrusion Detectors: A Decision Theory Approach, The IEEE Symposium on Security an Privacy, 2, pp Lee W., S. J. Stolfo, A Framework for Constructing Features an Moels for Intrusion Detection Systems, In ACM Transactions on Information an System Security, Vol. 3, No. 4, 2, pp Nikolova E., V. Jecheva, Anomaly Base Intrusion Detection Base on the Junction Tree Algorithm, Journal of Information Assurance an Security, Dynamic Publishers Inc., Vol. 2, Issue 3, 27, pp Nikolova E., V. Jecheva, Some Evaluations of the Effectiveness of Anomaly Base Intrusion Detection Systems Base on the Junction Tree Algorithm, Proceeings of the 5 th CITSA 28, Orlano, Floria, June 29 th - July 2 n, 28, vol., pp Nikolova E., V. Jecheva, Chapter: The Decoing Algorithms as Techniques for Creation the Anomaly Base Intrusion Detection Systems, Engineering the Computer Science an IT, IN-TECH, Vienna, Austria, 29, ISBN Zweig M.H., G. Campbell, Receiver-operating characteristic (ROC) plots: a funamental evaluation tool in clinical meicine, Clinical Chemistry, Vol.39, Num.4, (993), pp