Our second algorithm. Comp 135 Machine Learning Computer Science Tufts University. Decision Trees. Decision Trees. Decision Trees.

Transcription

1 Comp 135 Machie Learig Computer Sciece Tufts Uiversity Fall 2017 Roi Khardo Some of these slides were adapted from previous slides by Carla Brodley Our secod algorithm Let s look at a simple dataset for motivatio: Outlook Temp Humidity Widy Play Suy Hot High False o Suy Hot High True o Overcast Hot High False es Raiy Mild High False es Raiy Cool ormal False es Raiy Cool ormal True o Overcast Cool ormal True es Suy Mild High False o Suy Cool ormal False es Raiy Mild ormal False es Suy Mild ormal True es Overcast Mild High True es Overcast Hot ormal False es Raiy Mild High True o Class: Play teis Attributes: Outlook Temp Humidity Widy DTs give a differet way to idetify regios i istace space: Recursively split o values of features to defie regios that have sigle label What does this look like with umerical features? (with threshold ode tests, for example temp>23) Decisio trees ca represet ay discrete fuctio of discrete attributes! Why? Give traiig set, ca we build a tree that agrees with the data? (yes; easy; why?) What is a good decisio tree? Give traiig set, how ca we build a good tree? 1

2 Which attribute should we choose for root of tree? A umerical example: [Pos,eg] before ad after split [50,50]à [35,15] + [15,35] [50, 0] + [0,50] [10,30] + [30,10] + [10,10] [25,25] + [25,25] Cotiuig to Split Assume we picked Outlook for the root. The we must cotiue splittig each brach Util? Decisio Tree Learig Algorithm Fial Decisio Tree If data has a pure class Make leaf ode with that class Otherwise Pick feature to split o Divide data ito sub-datasets s j accordig to the feature s values Recursively build a tree for each subset Several selectio criteria have bee proposed ad used. Iformatio gai is commoly used (C4.5, J48) We eed to lear about etropy Etropy(p 1,...,p )= X p i log 1 p i = X p i log p i For 2 classes p 1 = p, p 2 =1 p ad this simplifies to Etropy(p) = X p log p (1 p) log (1 p) 2

3 ow cosider a split S! S 1,...,S k where the S i are subsets of S ad may iclude examples from multiple classes Gai(Split) = Et(S) X j S j S Et(S j) Example: calculatig Gai Outlook = Suy: etropy(2/5,3/5) = 2 / 5log(2 / 5) 3 / 5log(3 / 5) = bits Outlook = Overcast: etropy(1,0) = 1log(1) 0log(0) = 0 bits Outlook = Raiy: ote: defied as 0 etropy(3/5,2/5) = 3 / 5log(3 / 5) 2 / 5log(2 / 5) = bits Expected iformatio for attribute: ifo([3,2],[4,0],[3,2]) = (5/14) (4 /14) 0 + (5/14) = bits Cotiuig to Split gai(outlook ) = ifo([9,5]) ifo([2,3],[4,0],[3,2]) = = bits gai(outlook ) = bits gai(temperature ) = bits gai(humidity ) = bits gai(widy ) = bits gai(temperature ) = bits gai(humidity ) = bits gai(widy ) = bits Decisio Tree Learig Algorithm If data has a pure class Make leaf ode with that class Otherwise Pick feature to split o Divide data ito sub-datasets s j accordig to the feature s values Recursively build a tree for each subset Improved Heuristic for Wide Splits Gai(Split) = Et(S) Heuristic for wide splits SplitIfo = X j GaiRatio = X j S j S log S S j Gai SplitIfo S j S Et(S j) 3

4 Other Criteria / Tasks The Gii Criterio = 4p(1 p) The [KM] Criterio = 2 p p(1 p) What do these looks like? Criterio for Regressio = 1` v = 1` `X (v i v) 2 i=1 `X i=1 v i Real Valued Attributes aïve treatmet makes a very wide split with possibly oe example per brach. Is this good? Alterative picks threshold t ad tests (feature >= t) to get a biary split. How ca we pick t? Missig Attribute Values The Bad ews Commo i real data We ca hadle this i a way that works across algorithms (that is, also for k). How? But we ca do better with a solutio tailored for decisio trees. How? This does ot quite work Accuracy Size of tree (umber of odes) O traiig data O test data Overfittig i DT Overfittig i DT Why Does this happe? Mi # poits at leaf for split to be legal Few examples at lower levels i tree Quatities calculated ot reliable i this case Eve worse with oisy data Ad whe features ot sufficietly rich Solutios? Stop growig tree if o iformatio Pruig: grow full tree ad the test whether some parts should be removed. How? ote that full tree always looks better o traiig data! so just usig accuracy o traiig data will ot work 4

5 Overfittig i DT Solutio 1 (C4.5, J48): uses a cofidece iterval based o class ratio at leaf ad umber of examples i the traiig set. This is ot fully justified but works well i practice. Pruig i C4.5 / J48 Stadard ormal distributio Solutio 2: use a validatio set. Kow as reduced error pruig (REP) Z Pruig i C4.5 / J48 REP Example p is true error ad f is observed error Algorithm uses p(1 p) f p, ad some reasoig to claim that (actual formula used by C4.5 is more complex): r 1 with cofidece 1, p apple f + 4 Z The error rate at each ode is replaced with the upper boud The the best pruig ca be chose Decisio Errors red color black Keep 3 Traffic desity Prue 5 >75 <=75 l h m >70 <=70 Red sox 4, 2 1, 9 t f >74 <=74 other <=78 >78 full moo y Cereal Rice crispies Chex All Bra red Decisio Errors Keep 11 REP Example color black Traffic desity Prue >75 <=75 14 l h m 3, 8 >70 <=70 4, 2 1, 9 Red sox t >74 <=74 2, 8 2, 1 f 2, 8 other <=78 >78 full moo y Cereal Rice crispies Chex All Bra DT Recap DT divide the example space through recursive splits of feature values Recursive learig algorithm relies of good choice of root attribute IG ad other criteria are used for choice Several variats, improvemets ad geeralizatios Overfittig is a sigificat issue: solved by pruig or other methods 5