Modifying ROC Curves to Incorporate Predicted Probabilities

Transcription

1 Moifying ROC Curves to Incorporate Preicte Probabilities Cèsar Ferri DSIC, Universitat Politècnica e València Peter Flach Department of Computer Science, University of Bristol José Hernánez-Orallo DSIC, Universitat Politècnica e València Athmane Sena DSIC, Universitat Politècnica e València Abstract The area uner the ROC curve (AUC) is becoming a popular measure for the evaluation of classifiers, even more than other more classical measures, such as error/accuracy, logloss/entropy or precision. The AUC measure is specifically aequate to evaluate in two-class problems how well a moel ranks a set of accoring to the probability assigne to the positive class. One shortcoming of AUC is that it ignores the probability values, an it only takes the orer into account. On the other han, logloss or MSE are alternative measures, but they only consier how well the probabilities are calibrate, an not its orer. In this paper we introuce a new probabilistic version of AUC, calle pauc. This measure evaluates ranking performance, but also takes the magnitue of the probabilities into account. Seconly, we present a metho for visualising a proc curve such that the area uner this curve correspons to pauc.. Introuction Receiver Operating Characteristic (ROC) analysis (Provost & Fawcett, 998; Swets, Dawes & Monahan, 2; Provost & Fawcett, 2) has been proven very useful for evaluating given classifiers in cases when the cost matrix or the final class istribution is not known when the classifiers were constructe. ROC analysis provies tools to select a set of classifiers that woul behave optimally an reject sub-optimal classifiers. In orer to o this, the convex hull of all the classifiers is constructe, giving a curve (a convex polygon). Aitionally, the Area Uner the ROC Curve, AUC, can be use as a simple measure that can be use to estimate Appearing in Proceeings of the ICML 25 workshop on ROC Analysis in Machine Learning, Bonn, Germany, 25. Copyright 25 by the author(s)/owner(s). CFERRI@DSIC.UPV.ES PETER.FLACH@BRISTOL.AC.UK JORALLO@DSIC.UPV.ES ASENAD@DSIC.UPV.ES how well a classifier acts as a ranker, i.e., a classifier that sorts the accoring to their probability of being positive. ROC analysis an the AUC measure have been use extensively in the area of meical ecision making (McClish, 987; Hanley & McNeil, 982; Mossman & Somoza, 99; Zweig & Campbell, 993), in the fiel of knowlege iscovery, ata mining, pattern recognition (Aams, & Han, 999) an science in general (Swets, Dawes & Monahan, 2). There are various ways to construct a ROC curve an calculate AUC. One of them is base in sorting the on their preicte probabilities of being positive, an accumulate the percentage of positives an negatives appearing from left to right ( an ). For instance, given five, with the following probabilities an classes: (.9 +,.6,.55 +,.2,. ). We just sort the probabilities an inicate with a + or if they are actual positives or negatives. In Figure we show how to raw the ROC curve from the previous values. We just place them on a single axis (going from to, as shown, or alternative, going from to if we use complementary probabilities). Then, going from left to right we move a segment up or right in the ROC space. The size of the segment is proportional to the number of positives or negatives, respectively. From this ROC curve we can compute the AUC. The following metho for calculating AUC is from (Wu an Flach, 25): for each positive on the sorte list, accumulate how many negatives follow it. Alternatively, we can accumulate for each negative how many positives precee it. For instance, the AUC of the above example is 5/6=.833, as shown in Figure.

2 .9 ative.6.55 Positive.2. P + Figure. ROC curve an AUC base on ranking. AUC =.8333 The problem of AUC is that it ignores the probability values, because it only takes the orer into account. For instance, consier the following case: (.5 +,.5 ). The moel oes not clearly istinguish between the positive an the negative (their probabilities are almost the same), but the AUC is. A small variation of the given probabilities coul give the following situation: (.5-,.499+), where the area is, as seen in Figure 2. Positive.5.5 AUC = ative P+ Positive ative AUC = Figure 2. Small changes in probabilities can give rise to large changes in AUC. One possible solution to this problem is the use of confience intervals for ROC curves (Macskassy & Provost, 24). However, it is not clear how these confience bans can affect the AUC value when the probabilities are non-uniformly istribute from to. Another option is to use alternative measures, such as MSE or RMS (Square Error), logloss or cross-entropy or Kullback-Leibler Distance; for many authors, this is a better groune measure for evaluating the quality of estimate probabilities. The MSE an logloss are alternative measures, but they only consier how well the probabilities are calibrate, an not to its orer. The Brier score consiers both separability an calibration. However, all these measures ignore ranking performance. The outline of the paper is as follows. In Section 2 we efine a probabilistic version of AUC, pauc, an iscuss how to raw proc curves whose area is pauc. This relies on a smoothing parameter, an in Section 3 we iscuss ways to compute. Section 4 conclues. This paper has similar aims to (Wu an Flach, 25), the approaches are however ifferent an have been evelope inepenently. P+ 2. pauc an proc curves It is well-known that AUC is equivalent to the Wilcoxon sum of ranks test, which estimates the probability that a ranomly chosen positive is ranke before a ranomly chosen negative. The iea is to aapt this to obtain an estimate of the score ifference between a ranomly chosen pair of positive an negative. An unbiase estimator is obtaine by the ifference between the average scores of positives an the average scores of negatives. As this is a quantity between - an +, it is actually a probabilistic version of the Gini coefficient, which is equal to the area below the curve above the iagonal. We can transform this into an AUC-like measure by aing an iviing by 2. This gives the following efinitions: pgini = pauc = x P(x) Pos y Θ P( P( x) P( Pos + = 2 P( x) y + Pos 2 Θ P( Here, P(x) an P( are the preicte probability of positive instance x an negative instance y to belong to the positive class, as before, an Pos an are the total numbers of positive an negative. Thus, pauc can be interprete as the mean of the average positive score assigne to positives an the average negative score (preicte probability to belong to the negative class) assigne to negatives. There are many where AUC > pauc but also cases (more rarel where pauc > AUC. For instance, (.65 +,.55 +,.45,.35 ) has AUC= an pauc=.6. An ( +,., + ) has AUC=.5 an pauc=.7. The main question is whether pauc has a representation in the form of a curve. It woul be useful to be able to have a corresponing curve from which the user can choose the threshol, accoring to the skew (cost matrix an class istribution) at application time. For instance, accoring to Figure 2, it is apparently the same to choose threshol.99 than to choose.55 for any skew (slope) greater than (45 ). In this work we present a metho for visualising a proc curve, base on a representation of segments (with a uniform or normal istribution) of constant with or eviation, such that the area uner this curve correspons to pauc. The key iea is to consier probabilities as intervals, i.e., to consier that a moel giving probability.8 to an example means that the probability is aroun.8. The simplest approximation to this iea is to consier segments of a fixe with. For instance, Figure 3 illustrates how a curve an an area can be obtaine with

3 this iea from the ata: (.9 +,.6,.55 +,,2,. ). As we can see on the right curve, there is a iagonal segment which correspons to an area (aroun part of the secon an thir point) where we have a mixture of positive an negative influence. One awkwar thing about this metho is that some segments overrun the interval [,]. There is no intuitive meaning on negative probabilities or probabilities greater than. However, these segments just try to moel their area of influence in a symmetrical way. Obviously, we coul have avoie exceeing the interval [,] but this woul have complicate the mathematical an also the graphical interpretation of the final curves. Aitionally, other possibilities coul be to use segments aroun p/( p), going then from to, with proportionally ajuste segments, where we avoi the problem of exceeing the intervals, but the segments woul be asymmetrical. Positive ative P+ Figure 3. Drawing a proc curve with probability segments. With our approach, we just use the segments to show some kin of probability or smoothing reflecting the iea that two ifferent test with closer probabilities coul be wrongly orere. The main question is which with to use. In orer to analyse this, we have implemente a program that raws the curve an computes its area for ifferent values of. Figure 4 shows how the curve an the area evolves epening on the value of for the example {.9 +,.8,.6 +,.3,.2 }. As we can see from the curves, the first curve (=) is logically equivalent to the original ROC curve. As long as increases, the area iminishes because the positive an negative influence regions overlap. The last curve (=8) is almost the iagonal, with.5. As we can see, can be greater than an can excee the interval [,]. Aitionally, an more interestingly, there is a such that = pauc, in particular, =.596. Summing up, in this case, it seems that: If = = AUC If =.596 = pauc If.5 The results in the next section generalise this. Figure 4. proc curves for ifferent values of. 2. Some Theoretical Results In what follows, we try to generalise the previous three cases. THEOREM : For any set of labelle by probabilities an true labels, if = = AUC Proof: Trivial, since originally the AUC is compute with each point with no with, as seen in Figure. THEOREM 2: For any set of labelle by probabilities an true labels, if.5 Proof: If then the overlapping between the segments is maximal or, in other wors, the percentage of the regions with no overlapping is neglectable, i.e.,. Consequently, we have that for every region in the ensity space we have that the same percentage of positives an negatives, an this means that the curve has always slope.5, given a iagonal from (,) to (,) in the ROC space, so making an Area =.5. Before stating the last theorem (there exists a such that = pauc), we have to show that is continuous, PROPOSITION 3: is continuous, i. for all ε> then there exists a γ> such that for all x <γ implies Area(x) < ε. Proof: This is easy to see. If two have the same probability with opposite class, then with =, they are consiere to overlap. If increases, the overlapping area

4 increases continuously with the increment of. If two of ifferent probability with opposite class, there is a =u (>) such that they start to overlap. This overlapping, at =u is, so it is continuous on the left. From that point the overlapping increases continuously from, so it is continuous on the right. Nonetheless, it is important to see that is not necessarily increasing neither ecreasing in all the omain. For instance, { +,.5,.49 +, } has AUC=.75, pauc=.745. For =, as we have seen, =.75. For =, =.5, as we have seen as well. However, we can see that for =.2, =.874. Consequently, it increases initially but then ecreases. In cases with more, this can have more local maxima an minima. Now that we know that is continuous, we nee to analyse the relationship with AUC. First, we give efinitions for AUC, pauc an without using ranks: AUC = g( y Pos ) with if x > y g( =.5 if x = y if x < y with g pauc =.5 + ( y 2 Pos ) with ( = x y Area ( ) = g ( Pos g( 2 ( c) x y ( = with c = 2 2 ( c) x y with c = 2 if if if x y x y < an x > y x y < an x y THEOREM 4: For any set of labelle by probabilities an true labels, there exists a real number such that = pauc. Proof: We consier four mutually exclusive an exhaustive cases. Case a) Let us consier that AUC pauc an pauc.5. Necessarily, since Area() = AUC an lim =.5, an is continuous, there exists a such that = pauc. Case b) Let us consier that AUC pauc an pauc.5. Necessarily, since Area() = AUC an lim =.5, an is continuous, there exists a such that = pauc. Case c) Let us consier that AUC pauc an pauc.5. Since pauc.5, the term: Pos ( must be negative. This implies that the average overlapping is negative. This clearly means that there are correctly orere pairs (positives before negatives) which are closer than other inverte pairs (negatives before positives). Otherwise, pauc couln t be lower than.5 an lower than AUC. This means that with increasing values of we can further ecrease the, which at = is equal to AUC. We can ecrease this until pauc (at least) since g ( grows quicker than (. Case ) Let us consier that AUC pauc an pauc.5. Since pauc.5, the term: Pos ( must be positive. This implies that the average overlapping is positive. This clearly means that there are inverte pairs (negatives before positives) which are closer than other correctly orere pairs (positives before negatives). Otherwise, pauc couln t be greater than.5 an greater than AUC. This means that with increasing values of we can further increase the, which at = is equal to AUC. We can increase this until pauc (at least) since g ( grows quicker than (. We have shown that there exists a such that = pauc. However, is this unique? The answer is no as we can see in the following example. Example: Consier the ranking { +, +,.6,.6,.5 +,.49999,.45 +,.45 +,, }. The AUC of this ranking is.68. The pauc is approximately.67. Logically Area()=.68. For =.5, we have that =.66 approx. Consequently, there exists a between an.5 which makes = pauc. But for =.4 we have that =.732. Consequently, from =.5 with =.66 an with =.4 with =.732, there must be an intermeiate such that = pauc. Since this is clearly ifferent from the previous one, this problem shows two values of such that = pauc.

5 simplicity, it seems that the interpretation of a probability such as.8, as a probability aroun.8 oes not have its best interpretation with a uniform istribution with crisp intervals. A more natural option woul be to consier a normal (Gaussian) istribution centere on the probability, where the factor to gauge is also calle, but represents the ouble of the stanar eviation. =, Area=.68 =.5, Area=.66 pauc=.67 pauc=.67 =.4, Area=.73 Figure 5. Oscillating area for increasing values of gives two ifferent where = pauc. The entire evolution of is shown in Figure 6.,8,7,6 PAUC,5,4,3 pauc,2,,9,38,57,76,95,5,34,53,72,9 2, 2,29 2,48 2,67 2,86 3,6 3,25 3,44 3,63 3,82 4, 4,2 4,39 4,58 4,77 4,97 Figure 6. Oscillating area for increasing values of gives two ifferent where = pauc. 2.2 Interpretation of After the previous results, let us stop for analysing the interpretation of. pauc can be seen as a smoothing of AUC. The use of segments also acts as a smoothing, where the probability is istribute over an area (uniforml instea to a single point (which is quite unlikel. This has a natural interpretation of. However, the value of to be use is, however, woul be a big question. Theorem 4 ensures that we can always construct a proc curve that expresses a given pauc, which is a moifie ROC curve. This curve is closer to the original ROC curve for small an more ifferent for big. The pauc value can be seen as a way to etermine the for the an thus fining an aequate egree of smoothing. The, on the other han, is useful to fin a graphical interpretation to pauc. The values of can be interprete as points where both egrees of smoothing match. As we have seen, there can be more than one with = pauc. Any of these has a meaning, however, in what follows we choose the canonical as the smallest one. This is justifie because the smallest is, the curve is closer to the original AUC curve. 2.3 Using normally istribute segments Even though in the previous sections we have worke with uniform segments (rectangles), because of their Figure 7. Using normally istribute segments to obtain a proc curve. Here, is shown to be twice the stanar eviation. Nonetheless, it is expecte to have similar results (both theoretically an practicall. Aitionally some of the curves are expecte to be smoother. 3. How is calculate to have = pauc It oesn t seem easy to fin an analytical way to compute the minimum such that = pauc. In the following subsection we sketch how it coul be for a simple example. 3. Computing analytically In the last sections we have shown that there is at least one value for where the value of the area is the pauc. Can this value be foun analytically? We have researche in this irection without positive results. The behaviour of the proc curve is very epenent on the problem configuration (number an probabilities), an we believe that there is not a irect way to compute the value for given a set of positive an negative. Let us illustrate the complication of this computation with a very simple setting. Consier a positive example with value a, an a negative example b, where a>b. Clearly, AUC= an pauc=(a b+)/2. The proc curve will have the following shape:

6 x x curve. Nonetheless, some similarities remain, such as the concavity at the left-bottom part of the smoothe curves, an the top-right segment where the true positive rate is. If =pauc, using the proc curve, we have x 2 /2= pauc= (a-b+)/2. From the segment representation we have x=(b a+)/. Simplifying these expressions we get a b =. For instance, if we b a + have a=.6, an b=.4, then =.89. However, it is ifficult to obtain irect formulas for more complicate cases. 3.2 Computing numerically. Examples Instea of an analytical solution, we propose a numerical solution. The propose solution is an iterative metho that fins the value numerically. It is base on increasing from to in small ε since the proper value is foun. This is mae from left to right to fin the smallest possible such that =pauc. The ε is chosen sufficiently small to avoi passing through the correct. Also, a small ε also ensures that the ifference between the pauc an the is usually small (<.). Below, we show some with the ROC curve an the corresponing proc curve. Example { } AUC:.833 pauc:.658 Uniform proc Curve for =.65 an for =: Normal proc Curve for =.74 an for =: In this case, the smoothing is strong ( is relatively high) an the proc curve is significantly ifferent to the ROC Example 2 { } AUC:.8 pauc:.6 Uniform proc Curve for =.7 an for =: Normal proc Curve for =.8 an for =: In this case, we can see a more general situation with more points. We can see that the proc curve acts as a kin of smoothing of the ROC curve. 3.3 General interpretation of the proc curves From the previous we can get an iea of what proc curves mean. They smooth the ROC curves, especially when ifferences between probabilities of consecutive of ifferent class are small. The value is foun globally, so it can give high overlap in some areas but small overlap in others. So, especially when is not too high, some parts of the ROC curve might seem unaffecte (this is especially the case because we use a uniform istribution an not normal istribution). The greater the number of, the ifference of probabilities between consecutive of ifferent classes usually becomes smaller an hence the proc curve has a closer (but smoother) corresponence to the original ROC curve. Summing up, as long as the number of example is greater both curves ten to be more similar, because both are softer. The ifferences are more important especially when the ROC curve is not very reliable: few an small ifferences in probabilities. A secon issue is the use of the new curves for

7 establishing probability threshols accoring to some application skew (cost matrix + class istribution). In classical ROC analysis, given the skew at application time, we get the slope of an iso-accuracy line, which gives us the threshol (or the threshol interval) which woul give the best accuracy for that skew. With proc curves, we can o the same thing, an, interestingly, we get ifferent results. We will just reconsier example seen before: { }, because we can see that ifferences are strong an also because we see some awkwar results. Using the original ROC curve, we see that slopes greater than 55 (approx.) cut the curve at point % an 5%. This means that we woul choose a probability between.9 an.8 as a threshol. For slopes lower than 55 (approx.), the cutpoint is at 33% an %. This means that we woul choose a probability threshol between.6 to.3. On the other han, if we use the proc curves, we have a ifferent picture: The first picture shows the cut point with a slope of 6º. In this case, we have =9% an =9% (which is very ifferent from =% an =5% as in the ROC curve). Using the original probabilities, this woul mean that the threshol shoul be between.9 an.8 as before. Similarly, we can compute these values for the other two skews, as shown in the following table: Skew / Threshol in (slope) in ROC curve ROC curve / in proc curve Threshol in proc using the original prob. 6º % / 5% [.9,.8] 9% / 9% [.9,.8] 45º 33% / % [.6,.3] 4% / 58% [.8,.6] 3º 33% / % [.6,.3] 76% / 95% [.3,.2] From the previous table we see that the / are much more graual than for the original ROC curve. This is usually the case when we have few. However, note that using smoothe probabilities (the segments or the Gaussian function) means that we can get values outsie the interval [, ], which are ifficult to interpret as a threshol. In general, if we consier with more points, it is less likely to get threshols outsie [, ] an the use of proc to establish the threshol coul be more reliable. However, it woul be necessary to perform an empirical analysis on this. 4. Conclusions Several works have emonstrate that accuracy has many rawbacks as an evaluation measure for classification moels. The Area Uner the ROC curve (AUC) has been promote as a more suitable alternative for evaluating classifiers. However, this measure is only concentrate in how the moel ranks the, without taking into account the probabilities associate to the. We have illustrate this behaviour by means of a simple example where we illustrate how a slight variation of the confiences prouces an abrupt change in the AUC. Having in min these limitations, a new measure, namely probabilistic Area Uner the ROC curve (pauc) has been efine. In this case both aspects (ranking an probabilities) are consiere. Although pauc avois many of the rawbacks of the AUC measure, it apparently loses one of the strongest points of the AUC measure: the possibility of representing the performance of the evaluate moel, i.e. the ROC curves. In this work, we introuce a representation of the proc curves. The basic iea of rawing proc curves is base on the original ROC curves, but in this case we consier probabilities as intervals. These segments are set to a with, an varying this value we are able to fin a curve whose area is exactly the pauc of the moel. For constructing the intervals we have stuie two ifferent approaches: uniform or normal istribution. Both approaches obtain similar results. We have also presente some theoretical results of the propose representation. proc curves help to know ifferent etails about the performance of the evaluate moel with respect to classical ROC curves o. Although proc curves have a similar shape to classical ROC curves, they are usually smoother. For a high number of, both curves ten to be more similar, because both are softer. In general, we can fin the main ifferences when the ROC curve is not very reliable: few an small ifferences in probabilities. We have also use the curves to obtain cut points. However, obtaining a goo probability threshol from here has shown to present many caveats. This suggests to stuy asymmetrical segments, where we never excee the interval [..]. Another option woul be a normalisation of the probabilities after applying the segments of size or the Gaussian function. Currently we are working on an experimental evaluation on whether pauc is a goo selection measure (when we o not know the skew), an also showing experimentally whether there is a metho to establish the threshol from proc curves (when we know the skew) that can be better

8 than the threshol obtaine by ROC curves. Finally, although the approach is ifferent, we woul like to connect this work with other previous works on confience bans for ROC curves, as well as comparing to other measures, theoretically an experimentally. Zweig, M.H. an Campbell, G. (993) Receiveroperating characteristic (ROC) plots: a funamental evaluation tool in clinical meicine, Clin. Chem, 993; 39: Acknowlegements We woul like to thank Shaomin Wu for some iscussions aroun the iea of incorporating probabilities into the AUC measure. This work has been partially supporte by the EU (FEDER) an the Spanish MEC, uner grant TIN C4-2 an Generalitat Valencian uner grant GV4A-389. References Aams, N.M. an Han, D.J. (999) Comparing classifiers when the misallocation costs are uncertain Pattern Recognition, Vol. 32 (7) (999) pp Hanley, J.A. an McNeil, B.J. (982) The meaning an use of the area uner a receiver operating characteristic (ROC) curve Raiology. 982: 43: McClish, D.K. (987) Comparing the areas uner more than two inepenent ROC curves Me. Decis. Making, 987; 7: Macskassy, S.A. an Provost, F.J. (24) Confience Bans for ROC Curves: Methos an an Empirical Stuy ROCAI 24: 6-7, 24. Mossman, D. an Somoza, E. (99) ROC curves, test accuracy, an the escription of iagnostic tests J. Neuropsychiatr. Clin. Neurosci. 99, 3: Provost, F. an Fawcett, T. (997) Analysis an visualization of classifier performance: Comparison uner imprecise class an cost istribution Proc. of The Thir International Conference on Knowlege Discovery an Data Mining (KDD-97), pp , Menlo Park, CA: AAAI Press. Provost, F. an Fawcett, T. (2) Robust Classification for Imprecise Environments Machine Learning 42, 23-23, 2 Swets, J., Dawes, R., an Monahan, J. (2) Better ecisions through science Scientific American, October 2, Wu, S. an Flach, P.A. (25) Score an Weighte AUC Metrics for Classifier Evaluation an Selection, 2 n Workshop on ROC Analysis in Machine learning, ROCML 5.