Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates?

Transcription

1 Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to compare the relative performance among competing models?

2 Underfitting and Overfitting Overfitting Underfitting: when model is too simple, both training and test errors are large

3 Notes on Overfitting Overfitting results in decision trees that are more complex than necessary Training error no longer provides a good estimate of how well the tree will perform on previously unseen records Need new ways for estimating errors

4 Estimating Generalization Errors Re-substitution errors: error on training (Σ e(t) ) Generalization errors: error on testing (Σ e (t)) Methods for estimating generalization errors: Optimistic approach: e (t) = e(t) Pessimistic approach: For each leaf node: e (t) = (e(t)+0.5) Total errors: e (T) = e(t) + N 0.5 (N: number of leaf nodes) For a tree with 30 leaf nodes and 10 errors on training (out of 1000 instances): Training error = 10/1000 = 1% Generalization error = ( )/1000 = 2.5% Reduced error pruning (REP): uses validation data set to estimate generalization error

5 Occam s Razor Given two models of similar generalization errors, one should prefer the simpler model over the more complex model For complex models, there is a greater chance that it was fitted accidentally by errors in data Therefore, one should include model complexity when evaluating a model

6 Minimum Description Length (MDL) X y X 1 1 X 2 0 X 3 0 X 4 1 X n 1 A Yes 0 A? No B? B 1 B 2 C? 1 C 1 C B X y X 1? X 2? X 3? X 4? X n? Cost(Model,Data) = Cost(Data Model) + Cost(Model) Cost is the number of bits needed for encoding. Search for the least costly model. Cost(Data Model) encodes the misclassification errors. Cost(Model) uses node encoding (number of children) plus splitting condition encoding.

7 How to Address Overfitting Post-pruning Grow decision tree to its entirety Trim the nodes of the decision tree in a bottom-up fashion If generalization error improves after trimming, replace sub-tree by a leaf node. Class label of leaf node is determined from majority class of instances in the sub-tree Can use MDL for post-pruning

8 Example of Post-Pruning Training Error (Before splitting) = 10/30 Class = Yes 20 Class = No 10 Error = 10/30 A? Pessimistic error = ( )/30 = 10.5/30 Training Error (After splitting) = 9/30 Pessimistic error (After splitting) = ( )/30 = 11/30 PRUNE! A1 A2 A3 A4 Class = Yes 8 Class = Yes 3 Class = Yes 4 Class = Yes 5 Class = No 4 Class = No 4 Class = No 1 Class = No 1

9 Examples of Post-pruning Optimistic error? Case 1: Don t prune for both cases Pessimistic error? C0: 11 C1: 3 C0: 2 C1: 4 Don t prune case 1, prune case 2 Reduced error pruning? Case 2: Depends on validation set C0: 14 C1: 3 C0: 2 C1: 2

10 Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to compare the relative performance among competing models?

11 Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to compare the relative performance among competing models?

12 Model Evaluation and Selection Evaluation metrics: How can we measure accuracy? Other metrics to consider? Use test set of class-labeled tuples instead of training set when assessing accuracy Methods for estimating a classifier s accuracy: Holdout method, random subsampling Cross-validation Bootstrap Comparing classifiers: Confidence intervals Cost-benefit analysis and ROC Curves 12

13 Metrics for Performance Evaluation Focus on the predictive capability of a model Rather than how fast it takes to classify or build models, scalability, etc. Confusion Matrix: PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes a b Class=No c d a: TP (true positive) b: FN (false negative) c: FP (false positive) d: TN (true negative)

14 Metrics for Performance Evaluation PREDICTED CLASS Class=Yes Class=No ACTUAL CLASS Class=Yes Class=No a (TP) c (FP) b (FN) d (TN) Most widely-used metric: Accuracy = a a + b + + d c + d = TP TP + TN + TN + FP + FN

15 Limitation of Accuracy Consider a 2-class problem Number of Class 0 examples = 9990 Number of Class 1 examples = 10 If model predicts everything to be class 0, accuracy is 9990/10000 = 99.9 % Accuracy is misleading because model does not detect any class 1 example

16 Cost Matrix PREDICTED CLASS C(i j) Class=Yes Class=No ACTUAL CLASS Class=Yes C(Yes Yes) C(No Yes) Class=No C(Yes No) C(No No) C(i j): Cost of misclassifying class j example as class i

17 Computing Cost of Classification Cost Matrix ACTUAL CLASS PREDICTED CLASS C(i j) Model M 1 PREDICTED CLASS Model M 2 PREDICTED CLASS ACTUAL CLASS ACTUAL CLASS Accuracy = 80% Cost = 3910 Accuracy = 90% Cost = 4255

18 Cost vs Accuracy Count ACTUAL CLASS PREDICTED CLASS Class=Yes Class=No Class=Yes a b Class=No c d Accuracy is proportional to cost if 1. C(Yes No)=C(No Yes) = q 2. C(Yes Yes)=C(No No) = p N = a + b + c + d Accuracy = (a + d)/n Cost ACTUAL CLASS PREDICTED CLASS Class=Yes Class=No Class=Yes p q Class=No q p Cost = p (a + d) + q (b + c) = p (a + d) + q (N a d) = q N (q p)(a + d) = N [q (q-p) Accuracy]

19 Classifier Evaluation Metrics: Confusion Matrix Confusion Matrix: Actual class\predicted class C 1 C 1 C 1 True Positives (TP) False Negatives (FN) C 1 False Positives (FP) True Negatives (TN) Example of Confusion Matrix: Actual class\predicted class buy_computer = yes buy_computer = no Total buy_computer = yes buy_computer = no Total Given m classes, an entry, CM i,j in a confusion matrix indicates # of tuples in class i that were labeled by the classifier as class j May have extra rows/columns to provide totals 19

20 Classifier Evaluation Metrics: Accuracy, Error Rate, Sensitivity and Specificity A\P C C C TP FN P C FP TN N P N All Classifier Accuracy, or recognition rate: percentage of test set tuples that are correctly classified Accuracy = (TP + TN)/All Error rate: 1 accuracy, or Error rate = (FP + FN)/All Class Imbalance Problem: One class may be rare, e.g. fraud, or HIV-positive Significant majority of the negative classand minority of the positive class Sensitivity: True Positive recognition rate Sensitivity = TP/P Specificity: True Negative recognition rate Specificity = TN/N 20

21 Cost-Sensitive Measures Precision (p) = a a Recall (r) = a + b F - measure (F) = a + c 2rp r + p = 2a 2a + b + c Precision is biased towards C(Yes Yes) & C(Yes No) Recall is biased towards C(Yes Yes) & C(No Yes) F-measure is biased towards all except C(No No) Weighted Accuracy = w a w d 4 w c + w d 1 w a w b

22 Classifier Evaluation Metrics: Precision and Recall, and F-measures Precision: exactness what % of tuples that the classifier labeled as positive are actually positive Recall: completeness what % of positive tuples did the classifier label as positive? Perfect score is 1.0 Inverse relationship between precision & recall F measure (F 1 or F-score): harmonic mean of precision and recall, F ß : weighted measure of precision and recall assigns ß times as much weight to recall as to precision 22

23 Classifier Evaluation Metrics: Example Actual Class\Predicted class cancer = yes cancer = no Total Recognition(%) cancer = yes (sensitivity cancer = no (specificity) Total (accuracy) Precision = 90/230 = 39.13% Recall = 90/300 = 30.00% 23

24 Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to compare the relative performance among competing models?

25 Methods for Performance Evaluation How to obtain a reliable estimate of performance? Performance of a model may depend on other factors besides the learning algorithm: Class distribution Cost of misclassification Size of training and test sets

26 Learning Curve Learning curve shows how accuracy changes with varying sample size Requires a sampling schedule for creating learning curve: Arithmetic sampling (Langley, et al) Geometric sampling (Provost et al) Effect of small sample size: - Bias in the estimate - Variance of estimate

27 Methods of Estimation Holdout Reserve 2/3 for training and 1/3 for testing Random subsampling Repeated holdout Cross validation Partition data into k disjoint subsets k-fold: train on k-1 partitions, test on the remaining one Leave-one-out: k=n Stratified sampling oversampling vs undersampling Bootstrap Sampling with replacement

28 Evaluating Classifier Accuracy: Holdout & Cross-Validation Methods Holdout method Given data is randomly partitioned into two independent sets Training set (e.g., 2/3) for model construction Test set (e.g., 1/3) for accuracy estimation Random sampling: a variation of holdout Repeat holdout k times, accuracy = avg. of the accuracies obtained Cross-validation (k-fold, where k = 10 is most popular) Randomly partition the data into k mutually exclusive subsets, each approximately equal size At i-th iteration, use D i as test set and others as training set Leave-one-out: k folds where k = # of tuples, for small sized data *Stratified cross-validation*: folds are stratified so that class dist. in each fold is approx. the same as that in the initial data 28

29 Evaluating Classifier Accuracy: Bootstrap Bootstrap Works well with small data sets Samples the given training tuples uniformly with replacement i.e., each time a tuple is selected, it is equally likely to be selected again and re-added to the training set Several bootstrap methods, and a common one is.632 boostrap A data set with d tuples is sampled d times, with replacement, resulting in a training set of d samples. The data tuples that did not make it into the training set end up forming the test set. About 63.2% of the original data end up in the bootstrap, and the remaining 36.8% form the test set (since (1 1/d) d e -1 = 0.368) Repeat the sampling procedure k times, overall accuracy of the model: 29

30 Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to compare the relative performance among competing models?

31 ROC (Receiver Operating Characteristic) Developed in 1950s for signal detection theory to analyze noisy signals Characterize the trade-off between positive hits and false alarms ROC curve plots TP (on the y-axis) against FP (on the x-axis) Performance of each classifier represented as a point on the ROC curve changing the threshold of algorithm, sample distribution or cost matrix changes the location of the point

32 Model Selection: ROC Curves ROC (Receiver Operating Characteristics) curves: for visual comparison of classification models Originated from signal detection theory Shows the trade-off between the true positive rate and the false positive rate The area under the ROC curve is a measure of the accuracy of the model Rank the test tuples in decreasing order: the one that is most likely to belong to the positive class appears at the top of the list The closer to the diagonal line (i.e., the closer the area is to 0.5), the Vertical axis represents the true positive rate Horizontal axis rep. the false positive rate The plot also shows a diagonal line A model with perfect accuracy will have an area of

33 ROC Curve - 1-dimensional data set containing 2 classes (positive and negative) - any points located at x > t is classified as positive At threshold t: TP=0.5, FN=0.5, FP=0.12, FN=0.88

34 ROC Curve (TP,FP): (0,0): declare everything to be negative class (1,1): declare everything to be positive class (1,0): ideal Diagonal line: Random guessing Below diagonal line: prediction is opposite of the true class

35 Using ROC for Model Comparison No model consistently outperform the other M 1 is better for small FPR M 2 is better for large FPR Area Under the ROC curve Ideal: Area = 1 Random guess: Area = 0.5

36 How to Construct an ROC curve Instance P(+ A) True Class Use classifier that produces posterior probability for each test instance P(+ A) Sort the instances according to P(+ A) in decreasing order Apply threshold at each unique value of P(+ A) Count the number of TP, FP, TN, FN at each threshold TP rate, TPR = TP/(TP+FN) FP rate, FPR = FP/(FP + TN)

37 How to construct an ROC curve Class Threshold >= TP FP TN FN TPR FPR ROC Curve:

38 Test of Significance Given two models: Model M1: accuracy = 85%, tested on 30 instances Model M2: accuracy = 75%, tested on 5000 instances Can we say M1 is better than M2? How much confidence can we place on accuracy of M1 and M2? Can the difference in performance measure be explained as a result of random fluctuations in the test set?

39 Confidence Interval for Accuracy Prediction can be regarded as a Bernoulli trial A Bernoulli trial has 2 possible outcomes Possible outcomes for prediction: correct or wrong Collection of Bernoulli trials has a Binomial distribution: x Bin(N, p) x: number of correct predictions e.g: Toss a fair coin 50 times, how many heads would turn up? Expected number of heads = N p = = 25 Given x (# of correct predictions) or equivalently, acc=x/n, and N (# of test instances), Can we predict p (true accuracy of model)?

40 Confidence Interval for Accuracy For large test sets (N > 30), P acc has a normal distribution with mean p and variance p(1-p)/n ( Z < < Z α / 2 1 α / 2 = 1 α acc p p(1 p) / N ) Area = 1 - α Z α/2 Z 1- α /2 Confidence Interval for p: N acc + Z ± Z + 4 N α / 2 α / 2 p = 2 2( N + Z ) α / 2 acc 4 N acc 2

41 Confidence Interval for Accuracy Consider a model that produces an accuracy of 80% when evaluated on 100 test instances: N=100, acc = 0.8 Let 1-α = 0.95 (95% confidence) From probability table, Z α/2 = α Z N p(lower) p(upper)

42 Comparing Performance of 2 Models Given two models, say M1 and M2, which is better? M1 is tested on D1 (size=n1), found error rate = e 1 M2 is tested on D2 (size=n2), found error rate = e 2 Assume D1 and D2 are independent If n1 and n2 are sufficiently large, then e e 1 2 ~ ~ N N ( µ, σ ) 1 ( µ, σ ) Approximate: ˆ σ = i e (1 e ) i i n i

43 Comparing Performance of 2 Models To test if performance difference is statistically significant: d = e1 e2 d ~ N(d t,σ t ) where d t is the true difference Since D1 and D2 are independent, their variance adds up: 2 σ t = σ + σ ˆ σ + ˆ σ 1 2 e1(1 e1) e2(1 e2) = + n1 n2 1 2 At (1-α) confidence level, d t = d ± Z α / 2 σ ˆ t

44 An Illustrative Example Given: M1: n1 = 30, e1 = 0.15 M2: n2 = 5000, e2 = 0.25 d = e2 e1 = 0.1 (2-sided test) 0.15(1 0.15) 0.25(1 0.25) ˆ = + = σ d At 95% confidence level, Z α/2 =1.96 d t = ± = ± => Interval contains 0 => difference may not be statistically significant

45 Comparing Performance of 2 Algorithms Each learning algorithm may produce k models: L1 may produce M11, M12,, M1k L2 may produce M21, M22,, M2k If models are generated on the same test sets D1,D2,, Dk (e.g., via cross-validation) For each set: compute d j = e 1j e 2j d j has mean d t and variance σ t k Estimate: 2 ( d d) 2 j= 1 j = ˆ σ d t t = d k( k ± t 1) 1 α, k 1 ˆ σ t

46 Estimating Confidence Intervals: Classifier Models M 1 vs. M 2 Suppose we have 2 classifiers, M 1 and M 2, which one is better? Use 10-fold cross-validation to obtain and These mean error rates are just estimates of error on the true population of future data cases What if the difference between the 2 error rates is just attributed to chance? Use a test of statistical significance Obtain confidence limits for our error estimates 46

47 Estimating Confidence Intervals: Null Hypothesis Perform 10-fold cross-validation Assume samples follow a t distribution with k 1 degrees of freedom (here, k=10) Use t-test (or Student s t-test) Null Hypothesis: M 1 & M 2 are the same If we can reject null hypothesis, then we conclude that the difference between M 1 & M 2 is statistically significant Chose model with lower error rate 47

48 Estimating Confidence Intervals: t-test If only 1 test set available: pairwise comparison For i th round of 10-fold cross-validation, the same cross partitioning is used to obtain err(m 1 ) i and err(m 2 ) i Average over 10 rounds to get t-test computes t-statistic with k-1 degrees of freedom: where an d If two test sets available: use non-paired t-test where where k 1 & k 2 are # of cross-validation samples used for M 1 & M 2, resp. 48

49 Estimating Confidence Intervals: Table for t-distribution Symmetric Significance level, e.g., sig = 0.05 or 5% means M 1 & M 2 are significantly different for 95% of population Confidence limit, z = sig/2 49

50 Estimating Confidence Intervals: Statistical Significance Are M 1 & M 2 significantly different? Compute t. Select significance level (e.g. sig = 5%) Consult table for t-distribution: Find t value corresponding to k-1 degrees of freedom (here, 9) t-distribution is symmetric: typically upper % points of distribution shown look up value for confidence limit z=sig/2 (here, 0.025) If t > z or t < -z, then t value lies in rejection region: Reject null hypothesis that mean error rates of M 1 & M 2 are same Conclude: statistically significant difference between M 1 & M 2 Otherwise, conclude that any difference is chance 50

51 Issues Affecting Model Selection Accuracy classifier accuracy: predicting class label Speed time to construct the model (training time) time to use the model (classification/prediction time) Robustness: handling noise and missing values Scalability: efficiency in disk-resident databases Interpretability understanding and insight provided by the model Other measures, e.g., goodness of rules, such as decision tree size or compactness of classification rules 51