Enhancements to basic decision tree induction, C4.5

Transcription

1 Ehacemets to basic decisio tree iductio, C4.5 1

2 This is a decisio tree for credit risk assessmet It classifies all examples of the table correctly ID3 selects a property to test at the curret ode of the tree ad uses this test to partitio the set of examples The algorithm the recursively costructs a sub tree for each partitio This cotiuous util all members of the partitio are i the same class That class becomes a leaf ode of the tree 2

3 The credit history loa table has followig iformatio p(risk is high)=6/14 p(risk is moderate)=3/14 p(risk is low)=5/14 gai(icome)=i(credit table)-e(icome) gai(icome)= gai(icome)=0.967 bits gai(credit history)=0.266 gai(debt)=0.581 gai(collateral)=

4 Overfitig Validatio Cross Validatio, Cofusio Matrix, LIFT, ROC Curves Reduced-Error Pruig C4.5 From Trees to Rules Cotigecy table Overfitig The ID3 algorithm grows each brach of the tree just deeply eough to perfectly classify the traiig examples Difficulties may be preset: Whe there is oise i the data Whe the umber of traiig examples is too small to produce a represetative sample of the true target fuctio The ID3 algorithm ca produce trees that overfit the traiig examples 4

5 We will say that a hypothesis overfits the traiig examples - if some other hypothesis that fits the traiig examples less well actually performs better over the etire distributio of istaces (icluded istaces beyod traiig set) Overfittig Cosider error of hypothesis h over Traiig data: error trai (h) Etire distributio D of data: error D (h) Hypothesis hîh overfits traiig data if there is a alterative hypothesis h ÎH such that error trai (h) < error trai (h ) ad error D (h) > error D (h ) 5

6 Overfittig How ca it be possible for a tree h to fit the traiig examples better tha h, but to perform more poorly over subsequet examples Oe way this ca occur whe the traiig examples cotai radom errors or oise 6

7 Traiig Examples Day Outlook Temp. Humidity Wid Play Teis D1 Suy Hot High Weak No D2 Suy Hot High Strog No D3 Overcast Hot High Weak Yes D4 Rai Mild High Weak Yes D5 Rai Cool Normal Weak Yes D6 Rai Cool Normal Strog No D7 Overcast Cool Normal Weak Yes D8 Suy Mild High Weak No D9 Suy Cold Normal Weak Yes D10 Rai Mild Normal Strog Yes D11 Suy Mild Normal Strog Yes D12 Overcast Mild High Strog Yes D13 Overcast Hot Normal Weak Yes D14 Rai Mild High Strog No Decisio Tree for PlayTeis Outlook Suy Overcast Rai Humidity Yes Wid High Normal Strog Weak No Yes No Yes 7

8 Cosider of addig the followig positive traiig example, icorrectly labaled as egative Outlook=Suy, Temperature=Hot, Humidty=Normal, Wid=Strog, PlayTeis=No The additio of this icorrect example will ow cause ID3 to costruct a more complex tree Because the ew example is labeled as a egative example, ID3 will search for further refiemets to the tree As log as the ew erroeous example differs i some attributes, ID3 will succeed i fidig a tree ID3 will output a decisio tree (h) that is more complex the the origial tree (h ) Give the ew decisio tree a simple cosequece of fittig oisy traiig examples, h will outperform h o the test set 8

9 Avoid Overfittig How ca we avoid overfittig? Stop growig whe data split ot statistically sigificat Grow full tree the post-prue How to select ``best'' tree: Measure performace over traiig data Measure performace over separate validatio data set Pruig Remove the least reliable braches 9

10 Chages ad additios to ID3 i C4.5 Icludes a module called C4.5RULES, that ca geerate a set of rules from ay decisio tree It uses pruig heuristic to simplify decisio trees i a attempt to produce results Easier to uderstad Less depedet o a particular traiig set used The origial test selectio heuristic has also bee chaged C4.5 History ID3, CHAID 1960s C4.5 iovatios (Quila): permit umeric attributes deal sesibly with missig values pruig to deal with for oisy data C4.5 - oe of best-kow ad most widely-used learig algorithms Last research versio: C4.8, implemeted i Weka as J4.8 (Java) Commercial successor: C5.0 (available from Rulequest) 10

11 Idustrial-stregth algorithms For a algorithm to be useful i a wide rage of real-world applicatios it must: Permit umeric attributes Allow missig values Be robust i the presece of oise Be able to approximate arbitrary cocept descriptios (at least i priciple) Basic schemes eed to be exteded to fulfill these requiremets Numeric attributes Stadard method: biary splits E.g. temp < 45 Ulike omial attributes, every attribute has may possible split poits Solutio is straightforward extesio: Evaluate ifo gai (or other measure) for every possible split poit of attribute Choose best split poit Ifo gai for best split poit is ifo gai for attribute Computatioally more demadig 11

12 Weather data omial values Outlook Temperature Humidity Widy Play Suy Hot High False No Suy Hot High True No Overcast Hot High False Yes Raiy Mild Normal False Yes If outlook = suy ad humidity = high the play = o If outlook = raiy ad widy = true the play = o If outlook = overcast the play = yes If humidity = ormal the play = yes If oe of the above the play = yes Weather data - umeric Outlook Temperature Humidity Widy Play Suy False No Suy True No Overcast False Yes Raiy False Yes If outlook = suy ad humidity > 83 the play = o If outlook = raiy ad widy = true the play = o If outlook = overcast the play = yes If humidity < 85 the play = yes If oe of the above the play = yes 12

13 Cotiuous Valued Attributes Create a discrete attribute to test cotiuous Temperature = C (Temperature > C) = {true, false} Where to set the threshold? Temperatur 15 0 C 18 0 C 19 0 C 22 0 C 24 0 C 27 0 C PlayTeis No No Yes Yes Yes No (see paper by [Fayyad, Irai 1993] Example Split o temperature attribute: Yes No Yes Yes Yes No No Yes Yes Yes No Yes Yes No E.g. temperature < 71.5: yes/4, o/2 temperature ³ 71.5: yes/5, o/3 Ifo([4,2],[5,3])= 6/14 ifo([4,2]) + 8/14 ifo([5,3]) = bits Place split poits halfway betwee values Ca evaluate all split poits i oe pass! 13

14 Avoid repeated sortig! Sort istaces by the values of the umeric attribute Time complexity for sortig: O ( log ) Sort order for childre ca be derived from sort order for paret Time complexity of derivatio: O () Drawback: eed to create ad store a array of sorted idices for each umeric attribute Split Iformatio C4.5, a successor of ID3 uses a extesio to iformatio gai kow as gai ratio Overcomes the bias of Iformatio gai Applies a kid of ormalizatio to iformatio gai usig a split iformatio value 14

15 The split iformatio value represets the potetial iformatio geerated by splittig the traiig data set D ito v partitios, correspodig to v outcomes o attribute A SplitIfo does t cosider the classificatios Joha Gamper ad Moua Kacimi The gai ratio is defied as The attribute with the maximum gai ratio is selected as the splittig attribute 15

16 Other Ehacemets Allow for cotiuous-valued attributes Dyamically defie ew discrete-valued attributes that partitio the cotiuous attribute value ito a discrete set of itervals Hadle missig attribute values Assig the most commo value of the attribute Assig probability to each of the possible values Attribute costructio Create ew attributes based o existig oes that are sparsely represeted This reduces fragmetatio, repetitio, ad replicatio 16

17 Covertig a Tree to Rules Outlook Suy Overcast Rai Humidity Yes Wid High No Normal Yes Strog Weak R 1 : If (Outlook=Suy) Ù (Humidity=High) The PlayTeis=No R 2 : If (Outlook=Suy) Ù (Humidity=Normal) The PlayTeis=Yes R 3 : If (Outlook=Overcast) The PlayTeis=Yes R 4 : If (Outlook=Rai) Ù (Wid=Strog) The PlayTeis=No R 5 : If (Outlook=Rai) Ù (Wid=Weak) The PlayTeis=Yes No Yes It is ot satisfactory to form a rule set by eumeratig all paths of the tree... 17

18 Quila strategies of C4.5 Derive a iitial rule set by eumeratig paths from the root to the leaves Geeralize the rules by possible deletig coditios deemed to be uecessary Group the rules ito subsets accordig to the target classes they cover Delete ay rules that do ot appear to cotribute to overall performace o that class Order the set of rules for the target classes, ad chose a default class to which cases will be assiged The resultat set of rules will probably ot have the same coverage as the decisio tree Its accuracy should be equivalet Rules are much easier to uderstad Rules ca be tued by had by a expert 18

19 From Trees to Rules Oce a idetificatio tree is costructed, it is a simple matter to cocert it ito a set of equivalet rules Example from Artificial Itelligece, P.H. Wisto 1992 A ID3 tree cosistet with the data Hair Color Blod Lotio Used No Yes Red Emily Brow Alex Pete Joh Sarah Aie Daa Katie Subured Not Subured 19

20 Correspodig rules If the perso s hair is blode ad the perso uses lotio the othig happes If the perso s hair color is blode ad the perso uses o lotio the the perso turs red If the perso s hair color is red the the perso turs red If the perso s hair color is brow the othig happes Uecessary Rule Atecedets should be elimiated If the perso s hair is blode ad the perso uses lotio the othig happes Are both atecedets are really ecessary? Droppig the first atecedets produce a rule with the same results If the the perso uses lotio the othig happes To make such reasoig easier, it is ofte helpful to costruct a cotigecy table it shows the degree to which a result is cotiget o a property 20

21 I the followig cotigecy table oe sees the umber of lotio users who are blode ad ot blode ad are subured or ot Kowledge about whether a perso is blode has o bearig whether it gets subured Perso is blode (uses lotio) Perso is ot blode (uses lotio) Not subured Subured Check for lotio for the same rule Not subured Subured Perso uses lotio 2 0 Perso uses o lotio 0 2 Has a bearig o the result 21

22 Uecessary Rules should be Elimiated If the perso uses lotio the othig happes If the perso s hair color is blode ad the perso uses o lotio the the perso turs red If the perso s hair color is red the the perso turs red If the perso s hair color is brow the othig happes Note that two rules have a cosequet that idicate that a perso will tur red, ad two that idicate that othig happes Oe ca replace either the two of them by a default rule 22

23 Default rule If the perso uses lotio the othig happes If the perso s hair color is brow the othig happes If o other rule applies the the perso turs red What is CART? Classificatio Ad Regressio Trees Developed by Breima, Friedma, Olshe, Stoe i early 80 s. Itroduced tree-based modelig ito the statistical maistream Rigorous approach ivolvig cross-validatio to select the optimal tree Oe of may tree-based modelig techiques. CART -- the classic CHAID C5.0 Software package variats (SAS, S-Plus, R ) Note: the rpart package i R is freely available 23

24 Our philosophy i data aalysis is to look at the data from a umber of differet viewpoits. Tree structured regressio offers a iterestig alterative for lookig at regressio type problems. It has sometimes give clues to data structure ot apparet from a liear regressio aalysis. Like ay tool, its greatest beefit lies i its itelliget ad sesible applicatio. -- Breima, Friedma, Olshe, Stoe Idea: Recursive Partitioig Take all of your data. Cosider all possible values of all variables. Select the variable/value (X=t 1 ) that produces the greatest separatio i the target. (X=t 1 ) is called a split. If X< t 1 the sed the data to the left ; otherwise, sed data poit to the right. Now repeat same process o these two odes You get a tree Note: CART oly uses biary splits. 24

25 Gii Idex The Gii Idex (used i CART) measures the impurity of a data partitio D m: the umber of classes p i : the probability that a tuple i D belogs to class Ci The Gii Idex cosiders a biary split for each attribute A, say D1 ad D2. The Gii idex of D give that partitioig is: A weighted sum of the impurity of each partitio 25

26 The reductio i impurity is give by The attribute that maximizes the reductio i impurity is chose as the splittig attribute Biary Split: Cotiuous- Valued Attributes D: a data partitio Cosider attribute A with cotiuous values To determie the best biary split o A What to examie? Examie each possible split poit The midpoit betwee each pair of (sorted) adjacet values is take as a possible splitpoit 26

27 How to examie? For each split-poit, compute the weighted sum of the impurity of each of the two resultig partitios (D1: A<= split-poit, D2: A > split-poit) The split-poit that gives the miimum Gii idex for attribute A is selected as its splittig subset Biary Split: Discrete-Valued Attributes D: a data partitio Cosider attribute A with v outcomes {a 1...,a v } To determie the best biary split o A Examie the partitios resultig from all possible subsets of {a 1...,a v } Each subset S A is a biary test of attribute A of the form A S A? 2 v possible subsets. We exclude the power set ad the empty set, the we have 2 v -2 subsets 27

28 How to examie? For each subset, compute the weighted sum of the impurity of each of the two resultig partitios The subset that gives the miimum Gii idex for attribute A is selected as its splittig subset 28

29 29

30 Comparig Attribute Selectio Measures Iformatio Gai Biased towards multivalued attributes Gai Ratio Teds to prefer ubalaced splits i which oe partitio is much smaller tha the other Gii Idex Biased towards multivalued attributes Has difficulties whe the umber of classes is large Teds to favor tests that result i equal-sized partitios ad purity i both partitios Attributes with Cost Cosider: Medical diagosis : blood test costs 1000 secs Robotics: width_from_oe_feet has cost 23 secs. How to lear a cosistet tree with low expected cost? Replace Gai by : Gai 2 (A)/Cost(A) [Ta, Schimmer 1990] 2 Gai(A) -1/(Cost(A)+1) w w Î[0,1] [Nuez 1988] 30