Lecture 2 :: Decision Trees Learning

Transcription

1 Lecture 2 :: Decision Trees Learning 1 / 62

2 Designing a learning system What to learn? Learning setting. Learning mechanism. Evaluation. 2 / 62

3 Prediction task Figure 1: Prediction task :: Supervised learning 3 / 62

4 Lecture outline Motivation Decision trees in general Classification and regression trees (CART & ID3) Building tree Binary questions, i.e. binary splits Best split selection Learning mechanism :: ID3 algorithm Over-fitting, pruning More than one decision tree 4 / 62

5 Motivation :: Medical diagnosis example Real world objects heart attack patients admitted to the hospital. Features age and medical symptoms indicating the patient s conditions (got by measurements), like blood pressure. Output values high risk patients (G) and not high risk patients (F). Goal Identification of G- and F- patients on the basis of measurements. 5 / 62

6 Motivation :: Medical diagnosis example (2) Like a doctor, based on your experience, try to formulate criteria to identify the patients: 1 1st question: Is the minimum systolic blood pressure over the initial 24 hour period greater than 91? 2 If "no", then the patient is not high risk patient. 3 If "yes", 2nd question: Is age greater than 62.5? 4 If "no", then the patient is not high risk patient. 5 If "yes", 3rd question:... 6 / 62

7 Motivation :: Medical diagnosis example (3) Figure 2: Medical diagnosis problem: classification rules (Credits: Leo Breiman et al., 1984) 7 / 62

8 Motivation :: Medical diagnosis example (4) Like a machine learning expert, design a classifier based on the training data... 8 / 62

9 Decision trees in general A tree is a graph G = (V, E) in which any two vertices are connected by exactly one simple path. Figure 3: A tree in graph theory: any two vertices are connected by exactly one simple path. 9 / 62

10 Decision trees in general (2) A tree is called a rooted tree if one vertex has been designated the root, in which case the edges have a natural orientation, towards or away from the root. Figure 4: A rooted tree 10 / 62

11 Decision trees in general (3) A decision tree G DT = (V DT, E DT ) is a rooted tree where V DT is composed of internal decision nodes and terminal leaf nodes. Figure 5: A decision tree 11 / 62

12 Decision trees in general (4) Decision tree learner building corresponds to building decision tree GDT D = (V DT D, E DT D ) based on a data set D = { x, y : x X, y Y }. A decision node t VDT D is a subset of D and the root node t = D. Each leaf node is designated by an output value. Decision tree learner building GDT D = (V DT D, E DT D ) corresponds to repeated splits of subsets of D into descendant subsets, beginning with D itself. Splits (decisions) are generated by questions. 12 / 62

13 Figure 6: A decision tree learner 13 / 62

14 Decision trees in general :: Geometrical point of view Tree-based methods partition the feature space into a set of regions, and then fit a simple model in each region, for ex. majority vote for classification, constant value for regression. 14 / 62

15 Classification trees :: Geometrical point of view (2) Figure 7: Decision trees: a geometrical point of view (Credits: Alpaydin, 2004) 15 / 62

16 Regression trees :: Geometrical point of view Figure 8: Regression trees: a geometrical point of view 16 / 62

17 Historical excursion Classification trees: Y is a categorical output feature. Regression trees: Y is a numerical output feature. CART: ClAssification and Regression trees ID3 C4.5: classification trees 17 / 62

18 Historical excursion (2) Figure 9: Two decision tree conceptions 1 1 Automatic Interaction Detection(AID) 18 / 62

19 Once the decision tree learner is built, an unseen instance is classified by starting at the root node and moving down the tree branch corresponding to the feature values asked in questions. 19 / 62

20 Milestone #1 To learn a decision tree, one has to 1 define question types, 2 generate (best) splits, 3 design/apply a learning mechanism, 4 face troubles with over-fitting. 20 / 62

21 Notation Features Attr = {A 1, A 2,..., A M }, Values(A i ) is a set of all possible values for feature A i. Instances (i.e. feature vectors) x = x 1, x 2,..., x M, Output Values Y. D A i v = { x, y D x i = v}. 21 / 62

22 CART :: Question type definition Let s define a set of binary questions Q as follows: Each split depends on the value of only a single feature, i.e. each question tests a feature. Then each branch corresponds to feature value. For each categorical feature A m having values Values(A m ) = {b 1, b 2,..., b L }, Q includes all the questions of the form Is x m R? as R 2 Values(Am). For each numerical feature A m, Q includes all questions of the form Is x m k? for all k ranging over (, + ). 22 / 62

23 CART :: Question type definition (2) I.e. we restrict attention to recursive binary partitions, i.e. the feature speace is partitioned into a set of rectangles. 23 / 62

24 ID3 :: Question type definition Let s define a set of questions Q as follows: Each split depends on the value of only a single feature, i.e. each question tests a feature. Then each branch corresponds to feature value. For each categorical feature A m having values Values(A m ) = {b 1, b 2,..., b L }, Q includes all the questions of the form Is x m = b j? as j = 1,..., L. For each categorical feature A m having values Values(A m ) = {b 1, b 2,..., b L }, Q includes all the questions of the form Is x m R? as R 2 Values(Am). For each numerical feature A m, Q includes all questions of the form Is x m k? for all k ranging over (, + ). 24 / 62

25 ID3 :: Question type definition (2) Figure 10: ID3: Non-binary splits 25 / 62

26 Milestone #2 To learn a decision tree, one has to 1 define question types, 2 generate (best) splits, 3 design/apply a learning mechanism, 4 face troubles with over-fitting. 26 / 62

27 Best split generation :: Classification trees We say a data set is pure (or homogenous) if it contains only a single class. If a data set contains several classes, then we say that the data set is impure (or heterogenous). Decision tree classifier attempts to divide the M-dimensional feature space into homogenous regions. The goal of adding new nodes to a tree is to split up the instance space so as to minimize the impurity of the training set. : 5, : 5 : 9, : 1 heterogenous homogenous high level of impurity low degree of impurity 27 / 62

28 Best split generation :: Classification trees (2) 1. Define the node proportions p(j t), j = 1,..., c, to be the proportion of the instances x, x, j t, so that c j=1 p(j t) = Define a measure i(t) of the impurity of t as a nonnegative function Φ of the p(1 t), p(2 t),..., p(c t), i(t) = Φ(p(1 t), p(2 t),..., p(c t)), (1) such that Φ( 1 c, 1 c,..., 1 c ) = max, i.e. the node impurity is largest when all examples are equally mixed together in it. Φ(1, 0,..., 0) = 0, Φ(0, 1,..., 0) = 0,..., Φ(0, 0,..., 1) = 0, i.e. the node impurity is smallest when the node contains instances of only one class 3. Define the goodness of split s to be the decrease in impurity i(s, t) = i(t) p L i(t L ) p R i(t R ). p L is a proportin of examples in t going to t L, similarly with p R. 28 / 62

29 Best split generation :: Classification trees (3) Figure 11: Split in decision tree (Credits: Leo Breiman et al., 1984) 4. Define a candidate set S of splits at each node: the set Q of questions generates a set S of splits of every node t. Find split s with the largest decrease in impurity: i(s, t) = max s S i(s, t). 29 / 62

30 Best split generation :: Classification trees (4) 5. Discuss four splitting criteria: Misclassification Error i(t) MisclassificationError, Information Gain i(t) InformationGain, Gini Index i(t) GiniIndex. 30 / 62

31 Best split generation :: Classification trees (5) i(t) MisclassificationError = 1 max j=1,...,c p(j t) (2) : 0, : 6 : 1, : 5 : 2, : 4 : 3, : 3 M. E = = 0, = 0, = 0, 5 31 / 62

32 Best split generation :: Classification trees (6) c i(t) InformationGain = p(j t) log(p(j t). (3) j=1 i(s Ak, t) InformationGain = i(t) v Values(A k ) p tv i(t v ) = H(t) v Values(A k ) t v H(tv ) (4) t where t v is the subset of t for which attribute A k has value v. Gain(s, t) = i(s, t) InformationGain (5) 32 / 62

33 Best split generation :: Classification trees (7) The information gain favors features with many values over those with few values (many values, many branches, impurity can be much less). Example Let s consider the feature Date. It has so many values that it is bound to separate the training examples into very small subsets. As it has a high information gain relative to the training data, it is probably selected for the root node. But prediction behind the training examples is poor. Let s incorporate an alternative measure Gain ratio that takes into account the number of values of a feature. 33 / 62

34 Best split generation :: Classification trees (8) where GainRatio(s Ak, t) = Gain(s Ak, t) SplitInformation(s Ak, t), (6) SplitInformation(s Ak, t) = v Values(A k ) t v t log t v 2 t, (7) SplitInformation(s Ak, t) is the entropy of t with respect to the split s Ak, i.e. to the values of feature A k. 34 / 62

35 Best split generation :: Classification trees (9) i(t) GiniIndex = 1 j p 2 (j t). (8) 35 / 62

36 Best split generation :: Classification trees (10) : 0 : 1 : 2 : 3 : 6 : 5 : 4 : 3 Gini Entropy M.E For two classes (c = 2), if p is the proportion in the class "1", the measures are: Misclassification error: 1 max(p, 1 p) Entropy: p logp (1 p) log(1 p) Gini: 2p (1 p) 36 / 62

37 Best split generation :: Classification trees (11) Figure 12: Selection the best split :: Misclassification vs. Entropy vs. Gini See 37 / 62

38 Milestone #3 To learn a decision tree, one has to 1 define question types, 2 generate (best) splits: classification trees, regression trees, 3 design/apply a learning mechanism, 4 face troubles with over-fitting. 38 / 62

39 Best split generation :: Regression trees i(t) LeastSquaredError = 1 t (y i f (x i )) 2, (9) x i t K f (x i ) = ĉ k δ(x i R k ), (10) k=1 ĉ k = 1 R k x i R k y i. (11) I.e. i(t) LeastSquaredError = 1 t (y i ĉ k ) 2 (12) x i t 39 / 62

40 ĉ k as an average of y i for all training instances x i falling into t (i.e. into R k ) minimizes i(t) LeastSquaredError. The proof is based on seeing that the number which minimizes i (y i a) 2 is a = 1 N i y i. 40 / 62

41 Best split generation :: Regression trees (2) 2. Define the goodness of split s to be. i(s, t) = i(t) i(t L ) i(t R ) (13) 3. Define a candidate set S of splits at each node: the set Q of questions generates a set S of splits of every node t. Find split s : i(s, t) = min s S i(s, t). 41 / 62

42 Milestone #4 To learn a decision tree, one has to 1 define question types, 2 generate (best) splits: classification trees, regression trees, 3 design/apply a learning mechanism, 4 face troubles with over-fitting. 42 / 62

43 Learning mechanism :: ID3 algorithm(d, Y, Attr) Create a Root node for the tree. If x, x, y D : y = 1, return single node tree Root with label = +. If x, x, y D : y = 1, return single node tree Root with label = -. If Attr =, return a single node tree Root with label = most common value of Y in D, i.e. n = max v v Y, x,y D δ(v, Y (x)). Otherwise A := Splitting Criterion(Attr) Root := A v Values(A) Add a new tree branch below Root, corresponding to the test A(x) = v If D v = then below this new branch add a leaf node with label = n = max v δ(v, Y (x)); else below this new branch v Values(Y ), x,y D add the subtree ID3(D v, Y, Attr {A}) Return(Root). 43 / 62

44 Learning mechanism :: ID3 algorithm :: Comments ID3 is a recursive partitioning algorithm (divide & conquer), performs top-down tree construction. ID3 performs a simple-to-complex, hill-climbing search through H guided by a particular splitting criterion. ID3 maintains only a single current hypothesis. So ID3 for example is not able to determine any other decision trees consistent with training data. ID3 does not employ backtracking. 44 / 62

45 Milestone #5 To learn a decision tree, one has to 1 define question types, 2 generate (best) splits: classification trees, regression trees, 3 design/apply a learning mechanism, 4 face troubles with over-fitting. 45 / 62

46 Over-Fitting The divide and conquer algorithm partitions the data until every leaf contains examples of a single value, or until further partitioning is impossible because two examples have the same values for each feature but belong to different classes. Consequently, if there are no noisy examples, the decision tree will correctly classify all training examples. Over-fitting can be avoided by a stopping criterion that prevents some sets of training examples from being subdivided, or by removing some of the structure of the decision tree after it has been produced. 46 / 62

47 Over-Fitting When a node t was reached such that no significant decrease in impurity was possible (i.e. max s i(s, t) < β), then t was not split and became a terminal node. The class of this terminal node is determined as follows: majority vote for classification and average value for regression. But... If β is set too low, then there is too much splitting and the tree is too large. This tree might overfit the data. Increasing β: there maybe nodes t such that max s i(s, t) < β is small. But the descendants nodes t L and t R of t may have splits with large decreases in impurity. By declaring t terminal, one looses the good splits on t L or t R. This tree might not capture the important structure. Preferred strategy: Grow a large tree T 0, stopping the splitting process when only some minimum node size (say 5) is reached. Then prune T 0 using some pruning criteria. 47 / 62

48 Pruning :: C4.5 :: Reduced error pruning Figure 13: Reduced Error Pruning subtrees Ti already pruned, let Tf be the subtree corresponding to the most frequent value of A in the data, let L be a leaf labelled with the most frequent class in the data. 48 / 62

49 Pruning :: C4.5 :: Reduced error pruning (2) Split the data into training D and validation D v sets. Build a tree T to fit the data D. Evaluate impact of pruning each possible node on validation set D v : let E T be the sample error rate of T on D v, let E T be the sample error rate of T f f on D v, let E L be the sample error rate of L on D v. Let U CF (E T, D v ), U CF (E T f, D v ), U CF (E L, D v ) be the three corresponding estimated error rates (via confidence intervals). Depending on whichever is lower, C4.5 leaves T unchanged, replaces T by the leaf L, or replaces T by its subtree T f. 49 / 62

50 Milestone #6 To learn a decision tree, one has to 1 define question types, 2 generate (best) splits: classification trees, regression trees, 3 design/apply a learning mechanism, 4 face troubles with over-fitting. Going from ID3 to C / 62

51 ID3 :: Incorporating continuous-valued features ID3 is originally designed with two restrictions: 1 Discrete-valued predicted feature. 2 Discrete-valued features tested in the decision tree nodes. Let s extend it for the continuous-valued features. For a continuous-valued feature A, define a boolean-valued feature A c so that if A(x) c then A c (x) = 1 else A c (x) = / 62

52 ID3 :: Incorporating continuous-valued features (3) How to select the best value for the threshold c? Criterion-example Choose such c that produces the greatest information gain. Temperature EnjoySport No No Yes Yes Yes No c 1 = , c 2 = Gain(D, Temperature c1 ) =?, Gain(D, Temperature c2 ) =? 52 / 62

53 ID3 :: Handling training examples with missing feature values Consider the situation in which Gain(D, A) is to be calculated at node t in the decision tree. Suppose that x, c(x) is one of the training examples in D and that the value A(x) is unknown. Possible solutions Assign the value that is most common among training examples at node t. Alternatively, assign the most common value among examples at node t that have the classification c(x). 53 / 62

54 Milestone #7 To learn a decision tree, one has to 1 define question types, 2 generate (best) splits: classification trees, regression trees, 3 design/apply a learning mechanism, 4 face troubles with over-fitting. Going from ID3 to C Having a bag of decision trees / 62

55 More than one decision tree Some techniques, often called ensemble methods, construct more than one decision tree, e.g. Boosted Trees. 55 / 62

56 More than one decision tree :: Boosted trees In general, The more data, the better :: an approach of resampling Same learning mechanism, different classifiers h 1, h 2,..., h n. Boosting is a procedure that combines the outputs of many weak classifiers to produce a powerfull "committee". A weak classifier has an error rate only slightly better than random guessing. Schapire s strategy: Change the distribution over the training examples in each iteration, feed the resulting sample into the weak classifier, and then combine the resulting hypotheses into a voting ensemble, which, in the end, would have a boosted classification accuracy. 56 / 62

57 More than one decision tree :: Boosted trees (2) AdaBoost.M1 algorithm An iterative algorithm that selects h t given the performance obtained by weak learners h 1 h t 1 At each step t modifies training data distribution in order to favour hard-to-classify examples according to previous weak classifiers, trains a new weak classifier h t, selects the new weight α t of h t by optimizing a final classifier stops when impossible to find a weak classifier being better than chance, outputs a final classifier being the weighted sum of all weak classifiers. 57 / 62

58 More than one decision tree :: Boosted trees (3) Figure 14: AdaBoost (Credits: Hastie et al, 2009, pp. 338) 58 / 62

59 More than one decision tree :: Boosted trees (4) AdaBoost.M1 algorithm D 1 (i) = 1 n given D t and h t (i.e. update D t ): D t+1 (i) = D t(i) Z t.{ e αt if y i = h t (x i ) e αt if y i h t (x i ) = D t(i) Z t. exp( α t y i h t (x i )), (14) where Z t is normalization constant Z t = i D t(i) exp( α t y i h t (x i )) and α t = 1 1 ɛt 2 ln( ɛ t ) and ɛ t = Pr Dt [h t (x i ) y i ] = i:h t(x i ) y i D t (i) α t measures the importance that is assigned to h t. If ɛ t 1 2 (what is assumed without loss of generality), then α t 0; α t gets larger as ɛ t gets smaller. h final (x) = sgn( t α th t (x)): the final hypothesis is a weighted majority of the T hypotheses. 59 / 62

60 Final remarks Decision tree based classifiers are readily interpretable by humans. Instability of trees: learned tree structure is sensitive to the training data, so that a small change to the data can result in a very different set of splits. 60 / 62

61 For more details refer to Hastie, T. et al. The Elements of Statistical Learning. Springer, 2009, Section 9.2; Chapter 10 pp , 61 / 62

62 References Breiman Leo, Friedman Jerome H., Olshen Richard A., Stone Charles J. Classification and Regression Trees. Chapman & Hall/CRC, Boosting: A sample example: http: // Hunt, E. B. Concept Learning: An Information Processing Problem, Wiley Morgan, J. N., Sonquist, J. A. Problems in the analysis of survey data, and a proposal. Journal of the American Statistical Association 58, pp Quinlan, J. R. Discovering rules from large collections of examples: A case study, in D. Michie, ed., Expert Systems in the Micro Electronic Age. Edinburgh University Press Quinlan, J. R. C4.5: Programs for Machine Learning, Morgan Kaufmann, San Mateo, California / 62