Similarity-Binning Averaging: A Generalisation of Binning Calibration

Transcription

1 10 th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL 2009) Similarity-Binning Averaging: A Generalisation of Binning Calibration Antonio Bella, Cèsar Ferri, José Hernández-Orallo and María José Ramírez-Quintana Universitat Politècnica de València, Spain

2 Introduction Traditional Calibration Methods Calibration by Multivariate Similarity-Binning Averaging Experimental Results Conclusions and Future Work 2

3 10 th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL 2009) Introduction Universitat Politècnica de València, Spain

4 4 Training Data Supplier Product Quantity Price Delivered on time? S1 P NO S2 P NO S1 P YES S1 P YES S1 P YES S2 P NO S2 P YES S1 P NO S1 P YES S1 P YES S2 P YES S2 P YES S1 P YES S2 P NO Quantity P1 Product YES (4.0) Customer Product Quality Price S1 P S2 P P3 Supplier YES (2.0) NO (3.0) NO (2.0) YES (3.0) New Data Data Mining Model P2 <=75 >75 S1 S2 Delivered on time? YES NO Prob. (Yes)

5 5

6 A classifier is calibrated if, for a sample of examples with predicted probability p, the expected proportion of positives is near to p. Uncalibrated Model Calibrated Model Predicted Probability Predicted Probability Proportion of Positives Proportion of Positives

7 10 th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL 2009) Traditional Calibration Methods Universitat Politècnica de València, Spain

8 Binning averaging method. Pair-adjacent violators algorithm (PAV). Platt s method. 8

9 Based in ordering instances. Only binary problems (directly). Problem attributes are only used for calculating estimated probability. Estimated probability (of the positive class) is only used for ordering instances. All examples in a bin have the same calibrated probability. 9

10 Probability calibration by similarity (k-most similar instances). Applicable to multiclass problems. Use estimated probabilities (of all the classes) and, also, the problem attributes for computing similarity between instances. More information can improve the calibrated probability. Each example has a calibrated probability. 10

11 10 th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL 2009) Calibration by Multivariate Similarity-Binning Averaging Universitat Politècnica de València, Spain

12 Training Dataset Validation Dataset (VD) X 11, X 12 X 1n, Y 1 X 21, X 22 X 2n, Y 2 X m1, X m2 X mn, Y m X 11, X 12 X 1n, Y 1 X 21, X 22 X 2n, Y 2 X r1, X r2 X rn, Y r New Instance (I) X I1, X I2 X In Calibration Stage Classification Technique Probabilistic Classification Model M New Instance with Estimated Probabilities (IP) X I1, X I2 X In, p(i,1), p(i,2) p(i,c) Model Generation Stage 12 X 11, X 12 X 1n, p(1,1), p(1,2) p(1,c), Y 1 X 21, X 22 X 2n, p(2,1), p(2,2) p(2,c), Y 2 X r1, X r2 X rn, p(r,1), p(r,2) p(r,c), Y r Validation Dataset with Estimated Probabilities (VDP) Probability Estimation Stage k most similar (SB) p*(i,1), p*(i,2) p*(i,c) Calibrated Probabilities

13 Typical learning process. A classication technique is applied to a training dataset to learn a probabilistic classication model (M). Training Dataset X 11, X 12 X 1n, Y 1 X 21, X 22 X 2n, Y 2 X m1, X m2 X mn, Y m This stage may not exist if the model is given beforehand (a hand-made model or an old model). Classification Technique M 13 Probabilistic Classification Model

14 The trained model M gives the estimated probabilities associated with a dataset. This dataset can be the same used for training, or an additional validation dataset VD. Validation Dataset (VD) X 11, X 12 X 1n, Y 1 X 21, X 22 X 2n, Y 2 X r1, X r2 X rn, Y r M The estimated probability for each class is joined as new attribute, creating a new dataset VDP. 14 X 11, X 12 X 1n, p(1,1), p(1,2) p(1,c), Y 1 X 21, X 22 X 2n, p(2,1), p(2,2) p(2,c), Y 2 X r1, X r2 X rn, p(r,1), p(r,2) p(r,c), Y r Validation Dataset with Estimated Probabilities (VDP)

15 To calibrate a new instance I: 1. Obtain estimated probabilities from the classication model M. 2. Add these probabilities to the instance creating a new instance (IP). 3. Select the k-most similar instances to this new instance from the dataset VDP. 4. The calibrated probability of this instance I for each class is the predicted class probability of the k-most similar instances using all attributes. New Instance with Estimated Probabilities (IP) VDP New Instance (I) X I1, X I2 X In M X I1, X I2 X In, p(i,1), p(i,2) p(i,c) k most similar (SB) p*(i,1), p*(i,2) p*(i,c) 15 Calibrated Probabilities

16 10 th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL 2009) Experimental Results Universitat Politècnica de València, Spain

17 20 binary datasets from the UCI repository 2 different settings: o Training and test sets (75% / 25%) o Training, validation and test sets (56% / 19% / 25%) Classification techniques (WEKA): o Naïve Bayes, J48, IBk (k=10) and Logistic Regression Baseline methods: o o Class: classification techniques without calibration 10-NN: 10 most similar instances with the original attributes 17

18 Calibration methods: o o o o Binning averaging (10 bins) PAV algorithm Platt s method Similarity-Binning Averaging (SBA) (k=10) Calibration measures: o o Calibration by overlapping bins (CalBin) Pure calibration measure Mean Squared Error (MSE) Hybrid measure Brier score decomposition Calibration loss and refinement loss 18

19 Dataset ClassT 10-NNT BinT PAVT PlattT SBAT BinV PAVV PlattV SBAV AVG

20 Dataset ClassT 10-NNT BinT PAVT PlattT SBAT BinV PAVV PlattV SBAV AVG

21 10-NNT BinT PAVT PlattT SBAT BinV PAVV PlattV SBAV CalBin = ClassT = = 10-NNT = = BinT PAVT PlattT = SBAT (col. wins, ties =, row wins ) = BinV PAVV PlattV 10-NNT BinT PAVT PlattT SBAT BinV PAVV PlattV SBAV MSE = = ClassT = 10-NNT = BinT = = PAVT = PlattT = SBAT BinV 21 = PAVV PlattV

22 10 th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL 2009) Conclusions and Future Work Universitat Politècnica de València, Spain

23 New calibration method. Binning by constructing the bins using similarity to select the k-most similar instances (estimated probabilities and problem attributes). Experimental results show a significant increase in calibration for both measures considered, over three traditional calibration techniques. Can be applied to multiclass problems. 23

24 24

25 10 th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL 2009) Thanks for your attention! Antonio Bella Universitat Politècnica de València, Spain