Controlling False Alarms with Support Vector Machines

Transcription

1 Controlling False Alarms with Support Vector Machines Mark Davenport Clayton Scott Rice University dsp.rice.edu Richard Baraniuk

2 The Classification Problem Given some training data find a classifier that generalizes

3 Conventional Classification Signal / Pattern: Label: Classifier: Probability of error: Goal:

4 A Practical Approach Support Vector Machines (SVMs) offer a practical, nonparametric method for learning from data General idea: use kernel trick hyperplane classifiers maximize the margin [Cortes, Vapnik (1995)]

5 The Kernel Trick Problem: Linear classifiers perform poorly

6 The Kernel Trick Problem: Linear classifiers perform poorly Solution: Map data into feature space

7 The Kernel Trick Problem: Linear classifiers perform poorly Solution: Map data into feature space

8 The Kernel Trick Problem: Linear classifiers perform poorly Solution: Map data into feature space

9 Maximum Margin Principle Problem: Many classifiers to choose from

10 Maximum Margin Principle Problem: Many classifiers to choose from Solution: Pick one that maximizes margin

11 -SVM [Schölkopf et. al. (2000)]

12 What s Wrong? Sometimes false alarms are more/less important than misses False alarm: object detected, but not present Miss: object present, but not detected

13 What Else? Class frequencies are often not represented in the training data minimizing P E can ignore smaller class Prior probabilities are usually unknown 100 training samples 50 have leukemia 50 do not 50% of population has leukemia

14 Neyman-Pearson Classification Solution: Recast the problem False alarm: Miss: Goal:

15 Simple Approach Bias-shifting ad-hoc, but oft-used

16 Simple Approach Bias-shifting ad-hoc, but oft-used

17 Cost-Sensitive SVMs We need to explicitly treat the classes differently during training 2 -SVM Equivalent to the 2C-SVM How to pick + and -? [Chew, Bogner (2001)] [Davenport (2005)]

18 Controlling False Alarms: 2 -SVM Perform grid search over parameters - +

19 Controlling False Alarms: 2 -SVM Perform grid search over parameters Estimate false alarm and miss rates

20 Controlling False Alarms: 2 -SVM Perform grid search over parameters Estimate false alarm and miss rates Set

21 Controlling False Alarms: 2 -SVM Perform grid search over parameters Estimate false alarm and miss rates Set

22 Performance Evaluation We need a scalar measure of performance we want to evaluate our ability to achieve a specific point on the ROC

23 Performance Evaluation Theorem: [Scott (2005)]

24 Experimental Results Use Gaussian kernel Performance averaged over 100 permutations 4 benchmark datasets We report thyroid heart cancer banana P F P M E BS SVM BS SVM BS SVM BS SVM mean P F, P M median E 2 -SVM clear winner BS: 2 -SVM: = 0.1 bias-shifting our approach

25 Error Estimation True False Alarm Rate Cross-Validation P F P F - + Filtered Cross-Validation - + CV has high variance Filtering reduces the variance and yields a better error estimate P F - +

26 Filtering Results Filtering provides strong performance gains Shape of the filter doesn t seem to matter Gaussian window Uniform (boxcar) filter Median filter thyroid heart cancer banana GS: FGS: P F P M E GS FGS GS FGS GS FGS GS FGS = 0.1 grid search filtered grid search

27 Coordinate Descent Technique for reducing training time Eliminates full grid search

28 Coordinate Descent Technique for reducing training time Eliminates full grid search

29 Coordinate Descent Technique for reducing training time Eliminates full grid search

30 Coordinate Descent Technique for reducing training time Eliminates full grid search

31 Coordinate Descent Technique for reducing training time Eliminates full grid search

32 Coordinate Descent Technique for reducing training time Eliminates full grid search

33 Coordinate Descent Technique for reducing training time Eliminates full grid search

34 Coordinate Descent Results Training time is almost as fast as bias-shifting Performance comparable to full grid search Many more techniques for fast search possible thyroid heart cancer banana CD: FGS: P F P M E CD FGS CD FGS CD FGS CD FGS = 0.1 coordinate descent filtered grid search

35 Conclusion 2 -SVM consistently outperforms bias-shifting at controlling false alarms Simple techniques improve performance more accurate error estimation through filtering faster training through coordinate descent Applications: anomaly detection with minimum volume sets minimax classification Code available at

36 References C. Scott and R. Nowak, A Neyman-Pearson approach to statistical learning, IEEE Transactions on Information Theory, B. Schölkopf, A. J. Smola, R. Williams, and P. Bartlett, New support vector algorithms, Neural Computation, H. G. Chew, R. E. Bogner, and C. C. Lim, Dual- support vector machine with error rate and training size biasing, ICASSP C. C. Chang and C. J. Lin, Training support vector classifiers: Theory and algorithms, Neural Computation, M. Davenport, The 2 -SVM: A cost-sensitive extension of the -SVM, C. Scott, Performance measures for Neyman-Pearson classification,