Classification: Decision Trees

Transcription

1 Classification: Decision Trees IST557 Data Mining: Techniques and Applications Jessie Li, Penn State University 1

2 Decision Tree Example Will a pa)ent have high-risk based on the ini)al 24-hour observa)on? A high-risk pa)ent usually cannot survive for more than 30 days. 2

3 Decision Tree Example Business marke)ng: predict whether a person will buy a computer? age income student credit_rating buys_computer 21 high no fair no 18 high no excellent no 35 high no fair no 45 medium no fair yes 46 low yes fair yes 50 low yes excellent no 32 low yes excellent yes 33 medium no fair no 21 low yes fair yes 50 medium yes fair yes 22 medium yes excellent yes 34 medium no excellent no 33 high yes fair yes 55 medium no excellent no 3

4 Basic Concept age? <=40 >40 student? credit rating? no yes excellent fair no yes no yes Tree is constructed by repeated splifng feature space into two subsets Defini)ons: node, terminal node (leaf node), parent node, child node. The union of the regions occupied by two child nodes is the region occupied by their parent node. Every leaf node is assigned with a class. A query is associated with class of the leaf node it lands it. 4 features label 4

5 5

6 2 x x1 6

7 The Three Elements to Grow a Tree The construction of a tree involves the following three elements: 1. The selection of the splits 2. The decisions when to declare a node terminal or to continue splitting it 3. The assignment of each terminal node to a class 7

8 The Three Elements The construction of a tree involves the following three elements: 1. The selection of the splits 2. The decisions when to declare a node terminal or to continue splitting it 3. The assignment of each terminal node to a class 8

9 Standard Set of Questions The input vector X = (X 1, X 2,, X p ) contains features of both categorical and ordered (numerical) types. Each split depends on the value of only a unique variable. In the example below, the split lines are parallel to coordinates 9

10 Standard Set of Questions The ques)ons in the splifng node are in the form of {Is X j apple c?} for all real-valued c. How many candidate ques)ons? Infinite? 10

11 Standard Set of Questions The ques)ons in the splifng node are in the form of {Is X j apple c?} for all real-valued c. Since the training data set is finite, there are only finitely many dis)nct splits that can be generated. 11

12 Standard Set of Questions Let a[ribute A be a con)nuous-valued a[ribute Must determine the best split point c for A Sort the value A in increasing order Typically, the midpoint between each pair of adjacent values is considered as a possible split point c Split: A c A > c (a i +a i+1 )/2 is the midpoint between the values of a i and a i+1 12

13 Standard Set of Questions How about categorical features? If Xj is categorical, taking values,say in {1,2,, M}, the ques)ons are in the form {Is X j 2 A}, wherea is a subset of {1, 2,...,M}. How many candidate ques)ons for one variable? Exponen)al number of subsets: 2 M Heuris)c efficient algorithms: see this link 13

14 Goodness of Split The goodness of split is measured by an impurity func)on defined for each node. Intui)vely, we want each leaf node to be pure, that is, one class dominates. VS. 14

15 The Impurity Function Leh region x: 8; o: 2 (0.8, 0.2) Right region x: 2; o: 12 (0.14, 0.86) All region x: 10; o: 14 (0.42, 0.58) Figure and slides are from h[p://sites.stat.psu.edu/~jiali/course/stat557/ 15

16 Goodness of a split: i(t)-i(t L )-i(t R )? Leh region x: 8; o: 2 i(t L ) = Φ(0.8, 0.2) All region x: 10; o: 14 i(t) = Φ (0.42, 0.58) Right region x: 2; o: 12 i(t R ) = Φ (0.14, 0.86) 16

17 Leh region x: 8; o: 2 i(t L ) = Φ(0.8, 0.2) All region x: 10; o: 14 i(t) = Φ (0.42, 0.58) Right region x: 2; o: 12 i(t R ) = Φ (0.14, 0.86) 17

18 Leh region x: 8; o: 2 i(t L ) = Φ(0.8, 0.2) I(t L ) = i(t L ) * 10 All region x: 10; o: 14 i(t) = Φ(0.42, 0.58) * 24 Right region x: 2; o: 12 i(t R ) = Φ(0.14, 0.86) I(t R ) = i(t R ) * 14 18

19 All region x: 10; o: 14 i(t) = Φ(0.42, 0.58) I(t) = i(t) * 24 Leh region x: 8; o: 2 i(t L ) = Φ(0.8, 0.2) I(t L ) = i(t L ) * 10 Right region x: 2; o: 12 i(t R ) = Φ(0.14, 0.86) I(t R ) = i(t R ) * ΔI(s,t) = Φ (0.42, 0.58)*24 - Φ(0.8, 0.2)*10 Φ(0.14, 0.86)*14 = 24*(Φ(0.42, 0.58) - Φ (0.8, 0.2)*(10/24) - Φ(0.14, 0.86)*(14/24))

20 The Impurity Function: Entropy Algorithms using Entropy: ID3 (Intera)ve Dichotomiser 3) C4.5 (successor of ID3) 20

21 The Impurity Function: Gini index Algorithms using Gini index: CART: Classifica)on And Regression Tree Breiman, L., J. Friedman, R. Olshen, and C. Stone. Classifica/on and Regression Trees. Boca Raton, FL: CRC Press,

22 Murphy, Figure 16.3, ginidemo Error rate Gini Entropy Consider 400 cases in each class One split op)on: (300, 100) and (100, 300) Another split op)on: (200, 400) and (200, 0) Error rate treats them the same Entropy and Gini index prefers la[er Node impurity measures for binary classifica)on. Entropy has been rescaled. 22

23 The Three Elements The construction of a tree involves the following three elements: 1. The selection of the splits 2. The decisions when to declare a node terminal or to continue splitting it 3. The assignment of each terminal node to a class 23

24 Stopping Criteria 1. The samples in a node have the same labels 2. The goodness of split is lower than a threshold 3. The number of samples in every leaf is below a threshold 4. The number of leaf nodes reaches a limit 5. Tree has exceeded the maximum desired depth 24

25 Stopping Criteria 25

26 In this example, the first split is always not good. Using stopping criteria based on threshold, it will will not split. Anima)on 26

27 The Three Elements The construction of a tree involves the following three elements: 1. The selection of the splits 2. The decisions when to declare a node terminal or to continue splitting it 3. The assignment of each terminal node to a class 27

28 Class Assignment Rule 28

29 The Three Elements to Grow a Tree The construction of a tree involves the following three elements: 1. The selection of the splits Gini index (CART) Entropy (ID3, C4.5) Misclassification error 2. The decisions when to declare a node terminal or to continue splitting it 3. The assignment of each terminal node to a class 29

30 Example: Iris Flower Data Iris setosa Iris versicolor Iris virginica 4 features Sepal length Sepal width Petal length Petal width h[p://en.wikipedia.org/wiki/iris_flower_data_set 30

31 Murphy, Figure 16.4(a) setosa versicolor virginica Sepal width Sepal length Iris data. We only show the first two features, sepal length and sepal width 31

32 4.5 4 setosa versicolor virginica Murphy, Figure 16.5(a), dtreedemoiris Sepal width Sepal length Full decision tree 32

33 4.5 4 Murphy, Figure 16.4(b) setosa versicolor virginica Sepal width Sepal length 33

34 Example: Diabetes data Diabetes data 768 samples 8 quantitative variables 2 classes R script: h[ps://onlinecourses.science.psu.edu/stat857/node/133 CART in matlab: h[p:// Decision tree in R: h[p://cran.r-project.org/web/packages/rpart/rpart.pdf 34

35 The Three Elements to Grow a Tree The construction of a tree involves the following three elements: 1. The selection of the splits Gini index (CART) Entropy (ID3, C4.5) Misclassification error 2. The decisions when to declare a node terminal or to continue splitting it grow to a full tree and then prune it 3. The assignment of each terminal node to a class 35

36 Bias and Variance (high) Bias High training error Under-fit Simple model (high) Variance High tes)ng error Over-fit Complex model Examples: Examples: An over-pruned A full tree tree A high-degree One-degree linear polynomial linear regression 36

37 Bias and Variance (high) Bias High training error Under-fit Simple model (high) Variance High tes)ng error Over-fit Complex model Examples: Examples: An over-pruned A full tree tree A high-degree One-degree linear polynomial linear regression 37

38 Pruning a Tree To prevent overfitting, we can stop growing the tree, but the stopping criteria tends to be myopic The standard approach is therefore to grow a full tree, and then to perform pruning We can use any of the previous stopping criteria with very low threshold to grow a full tree (a very large tree) 38

39 How to Evaluate a Tree (or generally a classification model)? All the labeled data Train Test Randomly select a por)on of data as training (e.g., 80%) Remaining data as tes)ng (e.g., 20%) Then, compare different trees (or models) 39

40 How to Evaluate a Tree (or generally a classification model)? Random par))on could be biased. Cross valida)on is preferred. Example: 5-fold cross valida)on Final result to report: Average accuracy over 5 rounds Standard devia)on of accuracy over 5 rounds 40

41 Parameter Tuning by Random Partition of Train/Test All the labeled data Train Test 1. Randomly choose a por)on as train and the remaining as test 1. Tree starts with one (terminal) node (m=1) 2. Train a tree with m terminal nodes 3. Test it and get the error rate 4. Grow the tree with one more split (m ß m + 1) 5. If the tree is not full yet, go to 2 2. Pick m* terminal nodes which give us the lowest error 41

42 Parameter Tuning by Cross-Validation Random par))on could be biased. Cross valida)on is preferred. 5-fold cross valida)on 1. Use k-fold cross-valida)on on training data 2. In each valida)on, (k-1)/k of training data is used as train, 1/k of training data is used as test 1. Tree starts with one (terminal) node (m=1) 2. Train a tree with m terminal nodes 3. Test it and get the error rate 4. Grow the tree with one more split (m ß m + 1) 5. If the tree is not full yet, go to 2 3. Pick m* terminal nodes which give us the lowest error 42

43 Evaluating a Tree with the Best Parameter Randomly pick 80% as train, 20% as test (if using 5-fold cross valida)on, need to do this process 5 )mes) 80% train 20% test Use m* terminal nodes to train a model on all training data Pick m* terminal nodes Note: there are 2 layers of train/test spli3ng!!! 43

44 Murphy, Figure 16.5, dtreedemoiris Cost (misclassification error) Cross validation Training set Min + 1 std. err. Best choice Number of terminal nodes 44

45 Murphy, Figure 16.6, dtreedemoiris 45

46 Murphy, Figure 16.6, dtreedemoiris pruned decision tree versicolor setosa virginica 3.5 y x 46

47 Example: Diabetes data Pick the minimum crossvalida)on error Nsplit = 5 # of terminal nodes = 6 R script: h[ps://onlinecourses.science.psu.edu/stat857/node/133 47

48 Pros of Trees Easy to interpret Handle mixed discrete and continuous inputs Insensitive to monotone transformations of the inputs Because the split points are based on ranking the data points Perform automatic variable selection Relatively robust to outliers Scale well to large data sets Can be modified to handle missing features 48

49 Missing Values Certain variables are missing in some training samples Often occurs in gene-expression microarray data Suppose each variable has 5% chance being missing independently. Then for a training sample with 50 variables, the probability of missing some variables is as high as 92.3% A sample to be classified may have missing variables Find surrogate splits Suppose the best split for node t is s which involves a question on X m. Find another split s on a variable X j, j m, which is most similar to s in a certain sense. Similarly, the second best surrogate split, the third, and so on, can be found Advantage over a generative model: not modeling the entire joint distribution of inputs Disadvantage: entirely ad-hoc Code missing as a new value for categorical variables 49

50 Cons of Trees Do not predict very accurately compared to other kinds of model Due to the greedy nature of the tree construction algorithm Trees are unstable Small changes to the input data can have large effects on the structure of the tree Due to the hierarchical nature of the tree-growing process Solution: bagging and boosting 50

51 Summary Decision tree CART: Classification And Regression Tree Grow a tree Selection of splits Stopping criteria Class assignments in leaf nodes Prune a tree Grow to a full tree Use cross-validation to prune 51