Decision Tree CE-717 : Machine Learning Sharif University of Technology

Transcription

1 Decision Tree CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Some slides have been adapted from: Prof. Tom Mitchell

2 Decision tree Approximating functions of usually discrete domain The learned function is represented by a decision tree Application: Database mining 2

3 PlayTennis example: Training samples 3

4 A PlayTennis example Feature 1: Humidity Feature 2: Wind Feature 3: Outlook Feature 4: Temprature Output: Play Tennis Example: x=[low, weak, rain, hot] y=no 4

5 PlayTennis example: decision tree 5

6 Decision tree structure Internal node: a test on some attributes Branch: one of the possible results of the test Leaf: predicting y 6

7 Decision tree learning: Function approximation problem Problem Setting: Set of possible instances X Unknown target function f: X Y (Y is discrete valued) Set of function hypotheses H = * X Y + is a DT where tree sorts each x to a leaf which assigns y Input: Training examples *(x i, y i )+ of unknown target function f Output: Hypothesis H that best approximates target function f 7

8 Decision trees hypothesis space Suppose attributes are Boolean Disjunction of conjunctions Which trees to show the following functions? y = x 1 and x 3 y = x 1 or x 3 y = (x 1 and x 3 ) or(x 2 and x 4 ) 8

9 Decision tree as a rule base Decision tree = a set of rules Disjunctions of conjunctions of constraints on the attribute values of instances Each path from root to a leaf = conjunction of attribute tests All of the leafs with y = i are considered to find rule for y = i 9

10 How to construct basic decision tree? Learning an optimal decision tree is NP-Complete Instead, we use a greedy search based on a heuristic Which attribute at the root? The attribute A that is the best classifier Measure: how well splits the set into homogeneous subsets (having same value of target) We prefer decisions leading to a simple, compact tree with few nodes How to form descendant? Descendant is created for each possible value of A Training examples are sorted to descendant nodes 10

11 Basic decision tree learning algorithm (ID3) ID3(Examples, Attributes) 1) Create a root node r 2) If all examples have the same label or Attributes = *+ return r with proper label 3) A the best classifier attribute on Examples 4) For each value of A add a new branch below r and sort examples accordingly 5) Recursively grows the tree under each branch 11 Top down, Greedy, No backtrack

12 ID3 ID3 (Examples, Target_Attribute, Attributes) Create a root node for the tree If all examples are positive, return the single-node tree Root, with label = + If all examples are negative, return the single-node tree Root, with label = - If number of predicting attributes is empty then return Root, with label = most common value of the target attribute in the examples else A = The Attribute that best classifies examples. Testing attribute for Root = A. for each possible value, v i, of A Add a new tree branch below Root, corresponding to the test A =. Let Examples(v i ) be the subset of examples that have the value for A if Examples(v i ) is empty then below this new branch add a leaf node with label = most common target value in the examples else below this new branch add subtree ID3 (Examples(v i ), Target_Attribute, Attributes {A}) return Root 12

13 Which attribute is the best? 13

14 Which attribute is the best? A variety of heuristics for picking a good test Information gain: originated with ID3 (Quinlan,1979). Gini impurity 14

15 Entropy H X = P x i logp(x i ) x i X Amount of uncertainty associated with a specific probability distribution 15

16 Entropy Information theory: H X : expected number of bits needed to encode a randomly drawn value of X (under most efficient code) Most efficient code assigns log P(X = i) bits to encode X = i expected number of bits to code one random X is H(X) 16

17 Entropy for a Boolean variable H(X) Entropy as a measure of impurity P(X = 1) 1 H X = 0.5 log log = 1 H X = 1 log log 2 0 = 0 17

18 Conditional entropy H Y X = P X = i, Y = j logp Y = j X = i i j H Y X = P X = i P Y = j X = i logp Y = j X = i i j probability of following i-th branch for X Entropy of Y for samples sorted to i-th branch 18

19 Mutual Information I X, Y = H X H X Y = H Y H Y X The expected reduction in entropy caused by partitioning examples according to this attribute H Y : Entropy of Y before you split H Y X : Entropy after split 19

20 Information Gain Gain S, A = I S A, Y = H S Y H S Y A Gain S, A H S Y v Values(A) S v S H S v Y Y: target variable S: samples A: variable used to sort samples 20

21 Information Gain: Example 21

22 How to find the best attribute classifier? Information gain as our criteria for a good split attribute that maximizes information gain at each node Choose j-th attribute: j = argmax IG X i i = argmax i H Y H Y X i 22

23 Example 23

24 Decision tree algorithm The algorithm either reaches homogenous nodes or runs out of attributes Guaranteed to find a tree consistent with any conflict-free training set Conflict free training set: identical feature vectors always assigned the same class But not necessarily find the simplest tree. 24

25 How partition instance space? Decision tree Partition the instance space into axis-parallel regions, labeled with class value 25 Source: Duda & Hurt s Book

26 Bias in decision tree induction Information-gain gives a bias for trees with minimal depth. Implements a search (preference) bias instead of a language (restriction) bias. 26

27 Which Tree Should We Output? ID3: heuristic search through space of DTs Stops at smallest acceptable tree Occam s razor: prefer the simplest hypothesis that fits the data Ockham ( ) Principle of Parsimony: One should not increase, beyond what is necessary, the number of entities required to explain anything. 27

28 Why prefer short hypotheses? (Occam s Razor) Fewer short hypotheses than long ones a short hypothesis that fits the data is less likely to be a statistical coincidence highly probable that a sufficiently complex hypothesis will fit the data 28

29 Over-fitting problem ID3 usually perfectly classifies training data It tries to memorize every training data Poor decisions when very little data and may not reflect reliable trends A node that should be pure but had a single exception? Noise in the training data: the tree is erroneously fitting. For many (non relevant) attributes, the algorithm will continue to split nodes leads to over-fitting! 29

30 Over-fitting problem: an example Consider adding a (noisy) training example: Outlook Temp Humidity Sunny Hot Normal How does the tree change? Wind Strong PlayTennis No 30

31 General definition of overfitting Hypothesis H Training error: error train () Expected error: error true () overfits training data if there is a H such that error train < error train ( ) error true > error true ( ) 31

32 Over-fitting in decision tree learning 32

33 A question? How can it be made smaller and simpler? When should a node be declared a leaf? If a leaf node is impure, how should the category label be assigned? Pruning? 33

34 Avoiding overfitting Stop growing when the data split is not statistically significant. Grow full tree and then prune it. More successful than stop growing in practice. How to select best tree: Measure performance over separate validation set MDL: minimize size tree + size(missclassifications(tree)) 34

35 Reduced-error pruning Split data into train and validation set Build tree using training set Do until further pruning is harmful: Evaluate impact on validation set when pruning sub-tree rooted at each node Temporarily remove sub-tree rooted at node Replace it with a leaf labeled with the current majority class at that node Measure and record error on validation set Greedily remove the one that most improves validation set accuracy (if any). Produces smallest version of the most accurate sub-tree. 35

36 Rule post-pruning Convert tree to equivalent set of rules Prune each rule independently of others Sort final rules into desired sequence for use Perhaps most frequently used method (e.g., C4.5) 36

37 Continuous attributes Tests on continuous variables as boolean? Either use threshold to turn into binary or discretize Its possible to compute information gain for all possible thresholds (there are a finite number of training samples) Harder if we wish to assign more than two values (can be done recursively) 37

38 More issues on decision trees Better splitting criteria Information gain prefers features with many values. Missing feature values Features with costs Misclassification costs Incremental learning (ID4, ID5) Regression trees 38

39 Ranking classifiers Top 8 are all based on various extensions of decision trees Rich Caruana & Alexandru Niculescu-Mizil, An Empirical Comparison of Supervised Learning 39 Algorithms, ICML 2006

40 Decision tree advantages Simple to understand and interpret Requires little data preparation Can handle both numerical and categorical data Performs well with large data in a short time Robust Performs well even if its assumptions are somewhat violated 40