Model Enhancement in Data Mining: Calibration, ROC Analysis, Model Combination and Mimetic Models

Transcription

1 Model Ehacemet i Miig: Calibratio, ROC Aalysis, Model Combiatio ad Mimetic Models José Herádez-Orallo Dpto. de Sistemas Iformáticos y Computació, Uiversidad Politécica de Valecia, jorallo@dsic.upv.es Rome, 18th May

2 Outlie Motivatio Miig Models. Features ad evaluatio Model Calibratio Cotext (ROC) Aalysis Model Combiatio Model Simplificatio Mimetic Models Geeral idea Specific implemetatios for calibratio, adaptatio, combiatio, simplificatio ad revisio. Coclusios 2

3 Motivatio A (simplistic) view of Miig: (Iformatio) Miig Models (Kowledge) Models are the product of data miig Strategic decisios are supported o the iferred models. 3

4 Motivatio EXAMPLE: BANK AGENT Must I grat a mortgage to this customer? Historical : cid Credit-p (years) Credit-a (euros) Salary (euros) Ow House Defaulter accouts Returscredit yes 2 o yes 0 yes yes 1 o o 0 yes o 0 o... Miig Patter / Model: If Defaulter-accouts > 0 the Returs-credit = o If Defaulter-accouts = 0 ad [(Salary > 2.500) or (credit-p > 10)] the Returs-credit = yes 4

5 Motivatio A model provides support i decisio makig. Are we still makig decisios i a uscietific way? This questio is raised by: Swets, J.A., Dawes, R.M., & Moaha, J. (2000). Better decisios through sciece Scietific America, 283,

6 Motivatio Model quality ca be assessed accordig to may issues. (SEMANTICS) Is the model accurate? Is it always equally reliable? Does it optimise some utility fuctio? Is it cosistet with previous kowledge? (SYNTAX) Is the model itelligible? Ca it be icluded ito SW applicatios? 6

7 Motivatio I may cases, a model ca be good o may of the previous issues, but ot all of them. This usually happes whe a existig model has to be adapted to ew cotexts or simply because the cotext has chaged. With the mortgage example, a model which has worked well before the bak crisis might work much worse ow. Also frequetly, i order to get better accuracy, we have to make models more complex or we re forced to use some techiques which are ot itelligible (eural etworks, support vector machies, ). 7

8 Motivatio Optios for chagig a model: Retraiig: a simple idea, but gatherig sufficiet recet data is ot always possible. Throwig away a validated model i order to be substituted by a brad ew oe may be risky. Model revisio is also a optio if we wat to avoid the re-traiig the models. However, ot may kids of models ca be revised. Model ehacemet is a optio which tries to preserve the sytax ad/or the sematics while improvig o some other issues. 8

9 Motivatio Further reasos for model ehacemet: I orgaisatios, data miig models must coexist with had-made models (busiess rules, protocols, kow-how, egieered processes, etc.). May had-made models ca also be ehaced if we have some past or operatioal data available. We ca do the KDD process without the modellig stage Iitial data itegratio warehouse preparatio Miable view models kowledge Modellig Evaluatio Deploymet decisios Revisio 9

10 Motivatio Example: We have a had-made model i a isurace compay about the fees for a car isurace. If age > 25 ad car_brad is ot Ferrari the bous If drivig_licece_years > 3 the bous If geder = female the bous If car_brad = FIAT the bous We have a lot of data about customers, costs ad accidets Why do t we aalyse the previous model to refie or improve its behaviour accordig to curret data? 10

11 Motivatio If data miig ca be see as follows: (Iformatio) Miig Models (Kowledge) Model ehacemet ca be see as follows: Old Models (Kowledge) (Iformatio) Model Ehacemet New Models (Kowledge) 11

12 Motivatio Defiitio: Model ehacemet is ay techique that improves the results of a model i oe or more quality features without sigificatly affectig the others. Model ehacemet icludes: Calibratio Cotext (ROC) aalysis Model combiatio Model simplificatio (e.g. pruig) Mimetic techique 12

13 Miig Models: Features ad Evaluatio We will just preset ideas for predictive models: Classificatio Regressio I both cases, we have several measures of overall error. Classificatio: % misclassified istaces Regressio: Mea squared error But error distributio is also relevat. Error ca be differet at differet subsets of data (classes, rages, etc.). If the error is ot uiformly distributed, that meas that we ca locally improve the model. 13

14 Miig Models: Features ad Evaluatio Example of badly distributed classificatio model: Actual Predicted Mortgage problem Grat Good customer 700 Bad customer 800 Do ot grat The global error is 1,000 but errors gratig mortgages to bad customers are by far much more frequet tha errors ot gratig mortgages to good customers. 14

15 Miig Models: Features ad Evaluatio Example of badly distributed regressio model: Actual customer expediture Predicted customer expediture The global error is small, but i some areas the error is usually positive while i others the error is egative. 15

16 Miig Models: Features ad Evaluatio Aother issue is the cofidece or probability of a predictio: Example of a Classificatio model: Customer X will be a good customer with probability 0.9. Customer Y will be a good customer with probability 0.6. Our error is greater if customer X is fially a bad customer. Example of a Regressio model: Estimated expediture for customer X will be i the iterval [7..9] with cofidece 99%. This is sometimes more useful tha a predictio such as

17 Miig Models: Features ad Evaluatio Aother issue is pairwise or multiwise rakig: Example of a Classificatio model: Our model predicts that customer X is better customer tha customer Y. There is o eed to give probabilities for rakig istaces. If I sed a promotio to X istead of Y ad fially it is the case that Y is better customer tha X, we have made a rakig error. Example of a Regressio model: Our model predicts that customer X is goig to make more expediture tha customer Y. Agai, if it is the other way roud, we have made a rakig error. 17

18 Miig Models: Features ad Evaluatio Models (both leared or had-made) are hece complex etities ad ca be aalysed ad improved i may ways. Calibratio: cofidece ad probability estimatio is improved. Cotext (ROC) aalysis: rakig ability is improved ad behaviour is optimised to a specific cotext. Model combiatio: combiig predictios from several models improves the overall result. Model simplificatio (e.g. pruig): removig very specific parts usually provide a more geeral model which is also simpler. Mimetic techique: a geeral techique that ca be used as a substitute for revisio ad also for the previous techiques. 18

19 Calibratio Probability calibratio i classificatio. cid 105 Credit-p (years) No No No Yes Yes Yes Yes No No No No Yes 10 Credit-a (euros) Actual Prob_ M1 Prob_ M2 Pred_ M1 Pred_ M2 value Yes No Yes 0.3 No No No Same umber of errors (same accuracy: 4/6 = 0.67) But, accordig to probability estimatio, M 1 is much better tha model M 2. 19

20 Calibratio Probability calibratio i classificatio. A model is well calibrated if the probabilities are close to 1 whe the model is sure of a istace beig i the class ad 0 whe it is sure of the istace ot beig i the class. Values should be closer to 0.5 whe the model has more ucertaity. A more precise defiitio: A model is well calibrated if we take a example with estimated probability p meas that the true probability of this example of beig of the class is p. A more operative defiitio: A model is well calibrated if we take a set of examples with average estimated probability p meas that the percetage of examples of beig of the class is p. 20

21 Calibratio Probability calibratio i classificatio. Reliability diagrams: 1 1 Proportio of TRUE cases Proportio of TRUE cases Average predicted probability 1 0 Average predicted probability 1 The examples for which the probability is i betwee 0.3 ad 0.4 (mea 0.345) are 4% of class TRUE ad 96% of class FALSE. 21 If the model were well calibrated, it should be 34.5% of class TRUE ad 65.5% of class FALSE.

22 Calibratio Probability calibratio i classificatio. Techiques to calibrate probabilities: Apply a trasformatio fuctio to the probabilities without affectig the model. The model is ehaced wrt. probability estimatio. Some popular techiques: Platt scalig [Platt 2000]: a sigmoid fuctio is applied to the probabilities. The parameters of this sigmoid fuctios are estimated accordig to the probability deviatio over a validatio dataset. Isotoic regressio: a fuctio is iferred i such way that it maps origial estimated probabilities with calibrated predictios. 22

23 Calibratio Calibratio i regressio. Errors are ot regularly distributed for the output values predicho 3 2 predicho real real There are several DM techiques which produce calibrated models (liear regressio) but other methods (local regressio, o-liear regressio, eural etworks, decisio trees) ad had-made models usually geerate ucalibrated models. 23

24 Cotext (ROC) Aalysis Rakig i classificatio. cid Credit-p (years) Credit-a (euros) Actual Prob_ M1 Prob_ M2 Pred_ M1 Pred_ M2 value No No No No Yes Yes Yes Yes Yes No No No No No Yes Yes No No No No Yes No Yes Yes If the threshold betwee Yes ad No is placed at Prob=0.5, both models have the same umber of errors (same accuracy: 0.66). If the threshold is set at Prob=0.38, model 2 is able to classify all cases correctly. There is o threshold for which model 1 is able to do that well. Model 2 is a better raker tha model 1 is. 24

25 Cotext (ROC) Aalysis Cotext adaptatio by chagig the threshold: cid 105 Credit-p (years) No No No Yes Yes Yes Yes No No No No Yes 10 Credit-a (euros) Actual Prob_ M1 Prob_ M2 Pred_ M1 Pred_ M2 value Yes No Yes 0.3 No No No If we have a cotext where we wat to restrict credit, we would put the threshold higher. Otherwise, we ca put the threshold lower. We ca adapt to differet cotexts. A good classifier is the oe which ca perform well i a wide rage of cotexts. 25

26 26 ROC Curve of a Soft Classifier: Example: Cotext (ROC) Aalysis Actual Class Predicted Class p p p p p p p p p p p p p p p p p p p p p p p... Tom Fawcett

27 Cotext (ROC) Aalysis Cotext A ROC Aalysis of several soft classifiers: I this zoe the best classifier is ists I this zoe the best classifier is ists2 Cotext B Robert Holte We must preserve the classifiers that have at least oe best zoe. 27

28 Model Combiatio Methods Model Combiatio Methods. If we have (or ca geerate) multiple ad heterogeeous models we ca combie them through votig or other fusio methods. Much better results (i terms of error) tha sigle models whe the umber ad variety of classifiers is high. Differet topologies: simple, stackig, cascadig, a 1 a 2 a m Decisio Tree a 1 a Neural C 2 2 a Net m Combied Predictio Fusio C 1 a 1 C a 1 2 Decisio a Tree m a 1 a 2 a m Neural Net C 2 Decisio Tree Combied Predictio a 1 a 2 a m SVM C Simple Combiatio a 1 a 2 a m SVM C Stackig 28

29 Model Combiatio Methods Model Combiatio Methods. Mai Drawbacks: Comprehesibility: the model is defied i terms of a combiatio of models, where the attributes are ot the origial attributes. Computatioal costs: huge amouts of memory ad time are required to obtai ad store the set of models (esemble). Throughput: the applicatio of the combied model is slow. 29

30 Model Simplificatio Some models ca be very accurate but also very complex. Had-made models are usually patched very frequetly. With time, a had-made model is a collectio of patches ad exceptios. Leart models ca suffer the problem of overfittig. Whe cotext o reality chages, some specific rules are o loger valid, while the geeral rules still are. Complex models are more difficult to uderstad, to apply, to implemet, to itegrate with other kowledge or to modify. 30

31 Model Simplificatio Example: If age > 25 ad car_brad is ot Ferrari the bous If drivig_licece_years > 3 the bous If geder = female the bous If car_brad = FIAT the bous If age > 25 the bous If drivig_licece_years > 3 the bous If geder = female the bous The ew model will probably have a similar accuracy tha the old oe but it is simpler: easy to spread ad apply. Give a leart or had-made model, it is iterestig to have some techiques that allow us to simplify them, while maitaiig its performace. 31

32 Model Simplificatio Techiques are usually very depedet o the model represetatio: Decisio tree models. Techiques: pruig, rule coversio, Rule-based models. Techiques: rule mergig, rule elimiatio, etc. Algebraic models. Techiques: variable elimiatio, order reductio. I some applicatios ot oly reducig the umber of rules or the sytactical complexity of the model is importat but the umber of attributes which are required to make a decisio. If we do t eed to ask the car_brad, the calculatio of the fee becomes easier. 32

33 Mimetic Models Goal: to obtai a simple, comprehesible model that is similar to a give model. Mai Simple Idea: Treat the origial model as a oracle. Try to mimic the oracle. Several ways to mimic the oracle: Query Learig: askig the oracle to capture its sematics. Extesioal (classical) learig: geerate a iveted dataset from the oracle. This secod (simpler) optio is called: Simple Mimetic Models 33

34 Mimetic Models Graphically (whe mimickig a DM model): Traiig T First Learig Algorithm First Model (Oracle) Ω Complex, large, icomprehesible, but highly accurate model First Stage Sematically similar? Simple comprehesible model Ulabelled Distributio First Model (Oracle) Labelled Traiig + Labelled Secod Stage Secod Model (Mimetic) Ω R T + R Secod Learig Algorithm µ 34

35 Mimetic Models Graphically (whe mimickig a had-made model): Operatioal T Give Model (Oracle) Ω Complex, large, icomprehesible, but highly accurate model Sematically similar? Simple comprehesible model Distributio Ulabelled Give Model (Oracle) Labelled Operatioal + Labelled Secod Model (Mimetic) Ω R T + R Learig Algorithm µ 35

36 Mimetic Models Graphically (if we have ulabelled data available): Give Model (Oracle) Ω Complex, large, icomprehesible, but highly accurate model Sematically similar? Simple comprehesible model Ulabelled Give Model (Oracle) Labelled Secod Model (Mimetic) Ω R Learig Algorithm µ 36

37 Mimetic Models Example: If age > 25 ad car_brad is ot Ferrari the bous If drivig_licece_years > 3 the bous If geder = female the bous If car_brad = FIAT the bous Mimetic techique If age > 24.2 the bous If drivig_licece_years > 1.95 ad geder = female the bous If uiversity_studet = TRUE the bous The ew model captures the origial sematics but it differs o the expressio ad rules (model re-egieerig) 37

38 Mimetic Models Questios: Ca this arragemet be implemeted easily? Yes, it ca be doe i a simple data miig package such as Clemetie. How do we geerate the iveted dataset? How large the iveted dataset should be? How much similar/accurate the mimetic model ca be? Ca these mimetic models be short ad hece comprehesible? There are several experimetal ad theoretical results o this.. 38

39 Mimetic Models What kid of dataset? Ulabelled: It will be labelled by the oracle. Quite frequetly we have lots of ulabelled data. We have lots of uclassified s, ucategorised webpages, etc. I this case we ca use all these data, such as i semisupervised learig. If we do ot have data available. We ca geerate it radomly: Uder what distributio? Uiform Distributio: easier to be geerated.» For omial values, just a uiform selectio of the see values.» For umerical values, the max ad mi values must be determied to costruct a uiform iterval. Traiig Distributio (estimatio of the prior distributio). 39

40 Mimetic Models Should we add ay other labelled data? Two optios: If we do t have extra labelled data, the iveted dataset is solely composed of: The ulabelled or radom data labelled by the oracle. The sematics of the ew model coverges to the sematics of the oracle If we use extra labelled data, the iveted dataset is composed of: the available labelled dataset. + the radom data labelled by the oracle. We are able to adapt / correct / complete the sematics of the oracle. 40

41 Mimetic Models Which method should be use to geerate the mimetic model? Ay comprehesible model learig techique is a reasoable choice: Decisio tree learig Rule learig Algebraic models Fuzzy models 41

42 Mimetic Models Does the mimetic model preserves the sematics of the origial model? Several works i the literature cofirm that. For istace, we will show some results with the followig settig: 16 datasets from the UCI repository. 10x10-fold cross-validatio (the 1/10 part is reserved for the validatio of the secod stage). J4.8, Baggig ad Boostig from WEKA were used for compariso. 42

43 Mimetic Models Studyig the Method for Geeratig the Iveted set: Best results whe the prior distributio is used ad the traiig set is preserved (joitly with the radom dataset). 43

44 Mimetic Models Studyig the Size of the Iveted set: Better results the larger the iveted dataset is (although for some datasets there is a saturatio poit) 44

45 Mimetic Models Ca we use the mimetic techique for model ehacemet? Model simplificatio Calibratio Cotext (ROC) aalysis Model combiatio Ad also for: Revisio The mimetic techique ca be see as a way to cosolidate the model ehacemets. 45

46 Mimetic Models Mimetic models for model simplificatio: Cosider a complex model. Usig decisio trees as modellig techique, we ca: Fid a trade-off betwee fidelity ad simplicity, usig the pruig level ad the dataset size as parameters. The larger the dataset, the larger the fidelity, but also larger Ulabelled Give Model (Oracle) the umber of rules. R Small Labelled Learig Algorithm µ Simple Mimetic Model Ω R Large Labelled Learig Algorithm µ 46 Complex Mimetic Model

47 Mimetic Models Mimetic models for model simplificatio: Size ca be reduced sigificatly without losig the good accuracy of the method. 47

48 Mimetic Models Mimetic models for model simplificatio: Similar results as boostig with a sigle tree. 48

49 Mimetic Models Mimetic models for calibratio: We take a ucalibrated model, calibrate it ad the lear a ew model usig the calibrated probabilities to geerate proportios of elemets. Ulabelled Give Model (Oracle) Calibrated Labelled Calibrated Mimetic Model Ω Calibratio Techique R Learig Algorithm µ 49

50 Mimetic Models Mimetic models for cotext adaptatio: If the cotext requires more bias towards a class, we ca make the iveted dataset with a higher proportio of that class. That meas that the mimetic model is biased towards the ew cotext ad its class proportio is adapted. But the rules are also modified accordigly. Ulabelled Give Model (Oracle) Labelled Biased Labelled Adapted Mimetic Model Ω R R Learig Algorithm µ WITH THRESHOLD= 0.5: 60% class TRUE 40% class FALSE oversamplig WITH THRESHOLD= 0.87: 20% class TRUE 80% class FALSE WITH THRESHOLD= % class TRUE 80% class FALSE 50

51 Mimetic Models Mimetic models for model combiatio: We have see example of a combied model take as a oracle. Here we are iterested i the costructio of a uified model from several diverse models. Example: we have two differet isurace policies i two compaies that are beig merged. Ca we make a uified model? Ulabelled Give Model 1 (Oracle 1) Ω Labelled R 1 Joit Labelled Itegrated Mimetic Model Give Model 2 (Oracle 2) Ω R 2 R 1 Learig Algorithm µ 51

52 Mimetic Models Mimetic models for revisio: Give a old model which begis to behave worse o a ew situatio (chage), we ca reuse part of the sematics of the old model while icludig ew iformatio about the ew situatio i order to make it cope with the chages. Ulabelled Old Model (Oracle) New Labelled T New + Labelled New Model (Mimetic) Ω R T + R Learig Algorithm µ 52

53 Coclusios We have just give some clues o how importat it is to deal with models ad uderstad them. Models ca be trasformed, ad improved o may issues. They ca be adapted to chagig cotexts of applicatio. Techiques are applicable to data miig models comig from a traiig stage (learig) but they ca also be applied to hadmade models. This provides may tools for kowledge reegieerig i orgaisatios. Classical model ehacemet techiques iclude calibratio, cotext adaptatio, model combiatio ad model simplificatio. A geeral techique, called mimetic models, ca be used for model ehacemet ad trasformatio, ad also for model revisio. 53