Data Mining. Decision Tree. Hamid Beigy. Sharif University of Technology. Fall 1396

Transcription

1 Data Mining Decision Tree Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

2 Table of contents 1 Introduction 2 Decision tree classification 3 Building decision trees 4 ID3 Algorithm Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

3 Introduction 1 The decision tree is a classic and natural model of learning. 2 It is closely related to the notion of divide and conquer. 3 A decision tree partitions the instance space into axis-parallel regions, labeled with class value 4 Why decsion trees? Interpretable, popular in medical applications because they mimic the way a doctor thinks Can model discrete outcomes nicely Can be very powerful, can be as complex as you need them C4.5 and CART decision trees are very popular. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

4 n not be trained incrementally. Decision tree classification D5, ID5R are samples of incremental induction of decision tre ing 1 Structure of decsion trees Each internal node tests an attribute Each branch corresponds to attribute value Each leaf node assigns a classification. E. Utgoff, Incremental Induction of Decision Trees, Machine Learning, V 6, Decision Tree for PlayTennis [9+,5-] Outlook? Sunny Overcast Rain [2+,3-] Humidity? Yes Wind? [4+,-] High Normal Strong Light [3+,2-] No Yes [+,3-] [2+,-] No Yes [+,2-] [3+,-] Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

5 e idea of binary classification trees is not unlike that of the histogram - partition the featur Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24 Decision surface Histogram Classifier Linear Classifier 1 Tree Classifier (a) (b) (c) Figure 9.2: (a) Histogram classifier ; (b) Linear classifier; (c)tree classifier. 3 Binary Classification Trees

6 poor generalization characteristics, and then prune this tree, to avoid overfitting. Building decision trees Growing Trees The growing process is based on recursively subdividing the feature space. Usually the subdivisions are splits of existing regions into two smaller regions (i.e., binary splits). For simplicity, the Until now we have been referring to trees, but did not made clear how do trees relate to splits are perpendicular to one of the feature axis. An example of such construction is depicted partitions. It turns out that any decision tree can be associated with a partition of the input in Figure 9.3. space X and vice-versa. In particular, a Recursive Dyadic Partition (RDP) can be associated with a (binary) tree. In fact, this is the most e cient way of describing a RDP. In Figure 9.4 we illustrate the procedure. Each leaf of the tree corresponds to a cell of the partition. The nodes in the tree correspond to the various partition cells that are generated in the construction and so on... of the tree. The orientation of the dyadic split alternates between the levels of the tree (for the example of Figure 9.4, at the root level the split is done in the horizontal axis, at the level below that (the level of nodes 2 and 3) the split is done in the vertical axis, and so on...). The tree is called dyadic because the splits of cells are always at the midpoint along one coordinate axis, and consequently the Figure sidelengths 9.3: Growing of all cells a recursive are dyadic binary (i.e., tree powers (X of =[, 2). 1] 2 ). 1 Decsion trees recursively subdivide the feature space. 2 The test variable specifies the division Often the splitting process is based on the training data, and is designed to separate 1 data 1 with di erent labels as much as possible. In such constructions, 1 the splits, 4 and hence the treestructure itself, are data dependent. This causes major di culties for the analysis (and tunning) of these methods. Alternatively, the splitting and subdivision could be taken independent from 4 5 the training data. The latter approach is the one we are going to investigate in detail, since it is more amenable to analysis, and we will consider Dyadic Decision Trees and Recursive Dyadic Partitions (depicted in Figure 9.4) in particular Figure 9.4: Example of Recursive Dyadic Partition (RDP) growing (X =[, 1] 2 ). In the following we are going to consider the 2-dimensional case, but all the results can be Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

7 Building decision trees (example) 1! Training Examples for Concept PlayTennis Training examples for PlayTennis An Illustrative Example Day Outlook Temperature Humidity Wind PlayTennis? 1 Sunny Hot High Light No 2 Sunny Hot High Strong No 3 Overcast Hot High Light Yes 4 Rain Mild High Light Yes 5 Rain Cool Normal Light Yes 6 Rain Cool Normal Strong No 7 Overcast Cool Normal Strong Yes 8 Sunny Mild High Light No 9 Sunny Cool Normal Light Yes 1 Rain Mild Normal Light Yes 11 Sunny Mild Normal Strong Yes 12 Overcast Mild High Strong Yes 13 Overcast Hot Normal Light Yes 14 Rain Mild High Strong No! ID3 Build-DT using Gain( )! How Will ID3 Construct A Decision Tree? Hamid Beigy (Sharif University of Technology) Data Machine Mining Learning Fall / 24

8 Building decision trees (cont.) How to build a decision tree? 1 Start at the top of the tree. 2 Grow it by splitting attributes one by one. 3 Assign leaf nodes. 4 When we get to the bottom, prune the tree to prevent overfitting. How choose a test variable for an internal node? Choosing different measures result in different algorithms. We describe ID Gini index Entropy.3.2 Misclassification error p Node impurity measures for two-class classification, as a function of the Hamid Beigy (Sharif University proportion of Technology) p in class 2. Cross-entropy Data Mining has been scaled to pass through Fall / 24

9 One-against-one classification ID3 uses information gain to choose a test variable for an internal node. The information gain of D relative to attribute A is the expected reduction in entropy due to splitting on A. Gain(D, A) = H(D) [ ] Dv D H(D v ) v values(a) where D v is {x D : x.a = v}, the set of examples in D where attribute A has value v Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

10 ID3 Algorihm 1! Training Examples for Concept PlayTennis An Illustrative Example Day Outlook Temperature Humidity Wind PlayTennis? 1 Sunny Hot High Light No 2 Sunny Hot High Strong No 3 Overcast Hot High Light Yes 4 Rain Mild High Light Yes 5 Rain Cool Normal Light Yes 6 Rain Cool Normal Strong No 7 Overcast Cool Normal Strong Yes 8 Sunny Mild High Light No 9 Sunny Cool Normal Light Yes 1 Rain Mild Normal Light Yes 11 Sunny Mild Normal Strong Yes 12 Overcast Mild High Strong Yes 13 Overcast Hot Normal Light Yes 14 Rain Mild High Strong No! ID3 Build-DT using Gain( )! How Will ID3 Construct A Decision Tree? H(D) = (9/14) log(9/14) (5/14) log(5/14) =.94bits H(D, Humidity = High) = (3/7) log(3/7) (4/7) log(4/7) =.985bits H(D, Humidity = Normal) = (6/7) log(6/7) (1/7) log(1/7) =.592bits Machine Learning Gain(D, Humidity) =.94 (7/14).985 (7/14).592 =.151bits Gain(D, Wind) =.94 (8/14) (6/14) 1. =.48bits Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

11 ID3 Algorihm An Illustrative Example onstructing A Decision Tree 1! Training Examples for Concept PlayTennis for PlayTennis using ID3 [ 2 ] Day Outlook Temperature Humidity Wind PlayTennis? 1 Sunny Hot High Light No 2 Sunny Hot High Strong No 3 Overcast Hot High Light Yes Temperature Humidity Wind PlayTennis? 4 Rain Mild High Light Yes Hot High Light No 5 Rain Cool Normal Light Yes Hot High Strong No t Hot 6 High Rain Cool Yes Normal Strong No Mild 7 HighOvercast LightCool Yes Normal Strong Yes Cool 8 Normal Sunny Light Mild Yes High Light No Cool 9 Normal Sunny Strong Cool No Normal Light Yes t Cool 1 Normal Rain Strong Mild Yes Normal Light Yes Mild 11 HighSunny LightMild No Normal Strong Yes Cool 12 Normal Overcast LightMild Yes High Strong Yes Mild 13 Normal Overcast LightHot Yes Normal Light Yes Mild 14 Normal Rain Strong Mild Yes High Strong No t Mild High Strong Yes t Hot Normal Light Yes Mild! ID3 Build-DT High using Gain( ) Strong No! How Will ID3 Construct A Decision Tree? [9+, 5-] Gain(D, Humidity) =.151bits Outlook.151 bits Attribute bits Gain(D, Wind) =.48bits ) = Gain(D,.29 bits Temperature) =.29bits 246 bitsgain(d, Outlook) =.246bits ttribute (Root of Subtree) Machine Learning Sunny [2+, 3-] Overcast [4+, -] Rain [3+, 2-] Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

12 - Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

13 ID3 Algorihm An Illustrative Example 1! Training Examples for Concept PlayTennis Day Outlook Temperature Humidity Wind PlayTennis? 1 Sunny Hot High Light No 2 Sunny Hot High Strong No 3 Overcast Hot High Light Yes 4 Rain Mild High Light Yes 5 Rain Cool Normal Light Yes 6 Rain Cool Normal Strong No 7 Overcast Cool Normal Strong Yes 8 Sunny Mild Incremental High Learning Light of Decision No Trees 9 Sunny Cool Normal Light Yes 1 Rain Mild 27 Normal Light Yes 11 Sunny Mild! ID3 can Normal not be trained incrementally. Strong Yes 12 Overcast Mild! ID4, ID5, High ID5R are samples Strong of incremental induction Yes of decision trees 13 Overcast Hot Normal Light Yes 14 Rain Mild! Reading High Strong No! ID3 Build-DT using Gain( )! How Will ID3 Construct A Decision Tree? Gain(D Sunny, Humidity) =.97bits Gain(D Sunny, Wind) =.2bits Gain(D Sunny, Temperature) =.57bits " P.E. Utgoff, Incremental Induction of Decision Trees, Machine Learning, Vol. 4, pp ,1989. [2+,3-] Machine Learning [9+,5-] High Humidity? Outlook? Sunny Overcast Rain Normal Yes [4+,-] Strong Wind? Light [3+,2-] No Yes [+,3-] [2+,-] No Yes [+,2-] [3+,-] Machine Learning Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

14 Inductive Bias in ID3 Types of Biases 1 Preference (search) bias Put priority on choosing hypothesis. 2 Language bias Put restriction on the set of hypotheses considered Which Bias is better? 1 Preference bias is more desirable. 2 Because, the learner works within a complete space that is assured to contain the unknown concept. Inductive Bias of ID3 1 Shorter trees are preferred over longer trees. 2 Occams razor : Prefer the simplest hypothesis that fits the data. 3 Trees that place high information gain attributes close to the root are preferred over those that do not. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

15 Overfitting in ID3 How can we avoid over-fitting? 1 Prevention Stop training (growing) before it reaches the point that overfits. Select attributes that are relevant (i.e., will be useful in the decision tree) Requires some predictive measure of relevance 2 Avoidance Allow to over-fit, then improve the generalization capability of the tree. Holding out a validation set (test set) 3 Detection and Recovery Letting the problem happen, detecting when it does, recovering afterward Build model, remove (prune) elements that contribute to overfitting How to select Best tree? 1 Training and validation set Use a separate set of examples (distinct from the training set) for test. 2 Statistical test Use all data for training, but apply the statistical test to estimate the over-fitting. 3 Define the measure of complexity Halting the grow when this measure is minimized. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

16 Pruning algorithms Reduced-Error Pruning. oss-validation Approach aining and Validation Sets, node) btree rooted at node Reduced-Error Pruning error af (with majority label of associated examples) ed-error-pruning (D) D train (training / growing ), D validation (validation / pruning ) tree T using ID3 on D train y on D validation decreases DO n-leaf node candidate in T p[candidate] Prune (T, candidate) uracy[candidate] Test (Temp[candidate], D validation ) p with best value of Accuracy (best increase; greedy) ed) T The effect of reduced error pruning on error Accuracy Acc true error training error Number of nodes in tree Size Machine Lea On training data.7.65 On test data.6 Post-pruned tree.55 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

17 Pruning algorithms Prunning algorithms 1 Reduced Error Pruning 2 Pessimistic Error Pruning 3 Minimum Error Pruning 4 Critical Value Pruning 5 Cost-Complexity Pruning Reading 1 F. Esposito, D. Malerba, and G. Semeraro, A Comparative Analysis of Methods for Pruning Decision Trees, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 19, No. 5, pp , May S. R. Safavian and D. Landgrebe, A Survey of Decision Tree Classifier Methodology, IEEE Trans on Systems, Man, and Cybernetics, Vol. 21, No. 3, pp , Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

18 Continuous Valued Attributes Example Two methods for handling continuous attributes 1 Discretization (e.g., histogramming) Break real-valued attributes into ranges in advance Example high = {Temp > 35C} med = {1C < Temp 35C} low = {Temp 1C} 2 Using thresholds for splitting nodes A a produces subsets A a and A > a. Information gain is calculated the same way as for discrete splits How to find the split with highest Gain Example length label Thresholds Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

19 Test =?, > Missing Data Training: evaluate Gain (D, A) where for some x D, a value priority (or cost too high) Testing: classify a new example x without knowing the value! Solutions: Incorporating a Guess into Calculation of G Problem: What If Some Examples Missing Values of A? Consider dataset, a value for A is not given Day Outlook Temperature Humidity Wind PlayTennis? 1 Sunny Hot High Light No 2 Sunny Hot High Strong No 3 Overcast Hot High Light Yes 4 Rain Mild High Light Yes 5 Rain Cool Normal Light Yes 6 Rain Cool Normal Strong No 7 Overcast Cool Normal Strong Yes 8 Sunny Mild??? Light No 9 Sunny Cool Normal Light Yes 1 Rain Mild Normal Light Yes 11 Sunny Mild Normal Strong Yes 12 Overcast Mild High Strong Yes 13 Overcast Hot Normal Light Yes 14 Rain Mild High Strong No the value of A tion of Gain(D, A) nis? What is the decision tree? [9+, 5-] Outlook Machine Learning [2 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

20 Attributes with Many Values Problem: If attribute has many values such as Date, Gain() will select it (why?) One Approach: Use GainRatio instead of Gain Gain(D, A) = H(D) GainRatio(D, A) = SplitInformation(D, A) = v values(a) [ ] Dv D H(D v ) Gain(D, A) SplitInformation(D, A) [ Dv D log D ] v D v values(a) SplitInformation: directly proportional to values(a), i.e., penalizes attributes with more values. Preference bias (for lower branch factor) expressed via GainRatio(.) Alternative attribute selection : Gini Index Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

21 Handling Attributes With Different Costs Problem: In some learning tasks the instance attributes may have associated costs. Solutions 1 ExtendedID3 2 TanandSchlimmer 3 Nunez Gain(S, A) Cost(A) Gain 2 (S, A) Cost(A) where w [, 1] is a constant. 2 Gain(S,A) 1 (Cost(A) + 1) w Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

22 et of X reaching node m. Regression Tree e, the goodness of a split is measured by timated value. In regression tree, the goodness of a split is measured by the mean square error from the estimated value. A 1 True F 1 (A-A 1 )] False F[11+, 2 (A-A2-] 1 )] References 1 L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees, Belmont, CA; Wadsworth International Group, (1984). 2 D. Malerba, F. Esposito, M. Ceci, and A. Appice, Top-Down Induction of Model Trees H. Friedman, with Regression and R. Splitting A. Olshen, Nodes, IEEE Trans. and on Pattern C. J. Analysis Stone, and Machine Cla Intelligence, Vol. 25, No. 5, pp , May 24. es, Belmont, CA; Wadsworth International Gro Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

23 Other types of decision trees Decision Trees ariate Trees univariate trees, the test at each internal node just uses only one of input 26 ttributes.! Univariate Trees ivariate Univariate " Trees In univariate trees trees, the test at each internal node just uses only one of input multivariate In attributes. univariate trees, the trees, test the at test each atinternal each internal node can nodeuse justall uses input onlyattributes. one of input or! example: Multivariate attributes. Consider Treesa data set with numerical attributes. # The " test In can multivariate be made trees, using the the test weighted at each linear internal combination node can use of all some input input attributes. attributes. Multivariate trees or example " For X=(x example: Consider a data set with numerical attributes. In multivariate 1,x 2 ) be the input attributes. Let f (X)=w # The test can trees, be made the using test the at weighted each internal linear combination node +w can 1 x of some use 1 +w 2 x all 2 can be used for input input attributes. st at an internal node. Such as f (x) >. attributes. " For example X=(x 1,x 2 ) be the input attributes. Let f (X)=w +w 1 x 1 +w 2 x 2 can be used for test at an internal node. Such as f (x) >. (w +w 1 x 1 +w 2 x 2 )> (w +w 1 x 1 +w 2 x 2 )> Decision Trees True True False False YES YES [11+, NO2-] [11+, NO2-] ing! Reading. E. Brodley " C. E. Brodley and P. E. Utgoff, Multivariate Vol. 19, References and P. E. Utgoff, Multivariate Decision Trees, Machine Learning, Vol. 19, p , pp , C. E. Brodley and P. E. Utgoff, Multivariate Decision Trees, Machine Learning, Vol. 19, pp , Machine Learning Machine Learning Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

24 4, ID5, ID5R are samples of incremental induction of decision trees Incremental decision trees eading " P.E. Utgoff, Incremental Induction of Decision Trees, Machine Learning, Vol. 4 ID3 can not be trained incrementally. 186,1989. ID4, ID5, ID5R are samples of incremental induction of decision trees [9+,5-] Outlook? Sunny Overcast Rain [2+,3-] Humidity? Yes Wind? [4+,-] High Normal Strong Light [3+,2-] No Yes [+,3-] [2+,-] No Yes [+,2-] [3+,-] References Machine Learning 1 P.E. Utgoff, Incremental Induction of Decision Trees, Machine Learning, Vol. 4, pp ,1989. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24

25 References 1 P.E. Utgoff, Incremental Induction of Decision Trees, Machine Learning, Vol. 4, pp , D. Malerba, F. Esposito, M. Ceci, and A. Appice, Top-Down Induction of Model Trees with Regression and Splitting Nodes, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 25, No. 5, pp , May F. Esposito, D. Malerba, and G. Semeraro, A Comparative Analysis of Methods for Pruning Decision Trees, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 19, No. 5, pp , May S. R. Safavian and D. Landgrebe, A Survey of Decision Tree Classifier Methodology, IEEE Trans on Systems, Man, and Cybernetics, Vol. 21, No. 3, pp S. K. Murthy, Automatic Construction of Decision Trees from Data: A Multi- Disciplinary Survey, Data Mining and Knowledge Discovery, Vol. 2, pp Hamid Beigy (Sharif University of Technology) Data Mining Fall / 24