Example of DT Apply Model Example Learn Model Hunt s Alg. Measures of Node Impurity DT Examples and Characteristics. Classification.

Transcription

1 lassification-decision Trees, Slide 1/56 Classification Decision Trees Huiping Cao

2 lassification-decision Trees, Slide 2/56 Examples of a Decision Tree Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Refund: categorical Marital Status: categorical Taxable Income: continuous Cheat: class

3 10 lassification-decision Trees, Slide 3/56 Examples Example of a Decision of a Decision Tree (cont.) Tree Tid Refund Marital Status Taxable Income Cheat Splitting Attributes 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No Yes Refund No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Training Data Model: Decision Tree CS479/579: Data Mining Slide-4

4 10 lassification-decision Trees, Slide 4/56 Examples Example of a Decision of a Decision Tree (cont.) Tree Tid Refund Marital Status Taxable Income Cheat Splitting Attributes 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No Yes Refund No 4 Yes Married 120K No 5 No Divorced 95K Yes MarSt 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Training Data Model: Decision Tree CS479/579: Data Mining Slide-5

5 10 lassification-decision Trees, Slide 5/56 Examples Example of a Decision of a Decision Tree (cont.) Tree Tid Refund Marital Status Taxable Income Cheat Splitting Attributes 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No Yes Refund No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No MarSt Single, Divorced Married 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Training Data Model: Decision Tree CS479/579: Data Mining Slide-6

6 10 lassification-decision Trees, Slide 6/56 Examples of a Decision Tree (cont.) Example of a Decision Tree Tid Refund Marital Status Taxable Income Cheat Splitting Attributes 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No Yes Refund No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No MarSt Single, Divorced TaxInc Married 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Training Data Model: Decision Tree CS479/579: Data Mining Slide-7

7 10 lassification-decision Trees, Slide 7/56 Examples Example of a Decision of a Decision Tree (cont.) Tree Tid Refund Marital Status Taxable Income Cheat Splitting Attributes 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Refund Yes No MarSt Single, Divorced TaxInc < 80K 80K YES Married Training Data Model: Decision Tree CS479/579: Data Mining Slide-6

8 10 lassification-decision Trees, Slide 8/56 Examples of a Decision Tree (cont.) Example of a Decision Tree Tid Refund Marital Status Taxable Income Cheat Splitting Attributes 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Refund Yes No MarSt Single, Divorced TaxInc < 80K 80K YES Married Training Data Model: Decision Tree CS479/579: Data Mining Slide-7

9 10 Examples of a Decision Tree (cont.) Another Example of Decision Tree Tid Refund Marital Status Taxable Income 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No Cheat 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Married MarSt Yes Single, Divorced Refund No TaxInc < 80K 80K YES There could be more than one tree that fits the same data! lassification-decision Trees, Slide 9/56 CS479/579: Data Mining Slide-8

10 10 10 Example of DT Apply Model Example Learn Model Hunt s Alg. Measures of Node Impurity DT Examples and Characteristics Decision Tree Classification Task Decision Tree Classification Task Tid Attrib1 Attrib2 Attrib3 Class 1 Yes Large 125K No 2 No Medium 100K No 3 No Small 70K No 4 Yes Medium 120K No 5 No Large 95K Yes 6 No Medium 60K No 7 Yes Large 220K No 8 No Small 85K Yes 9 No Medium 75K No 10 No Small 90K Yes Training Set Tid Attrib1 Attrib2 Attrib3 Class 11 No Small 55K? 12 Yes Medium 80K? 13 Yes Large 110K? 14 No Small 95K? 15 No Large 67K? Test Set lassification-decision Trees, Slide 10/56 Induction Deduction Tree Induction algorithm Learn Model Apply Model Model Decision Tree

11 lassification-decision Trees, Slide 11/56 CS479/579: Data Mining Slide Example: Apply Model to Test Data Apply Model to Test Data Start from the root of tree. Test Data Refund Marital Status Taxable Income Cheat Yes Refund No No Married 80K? Single, Divorced MarSt Married TaxInc < 80K 80K YES

12 lassification-decision Trees, Slide 12/56 10 Example: Apply Model to Test Data (cont.) Apply Model to Test Data Test Data Refund Marital Status Taxable Income Cheat Yes Refund No No Married 80K? Single, Divorced MarSt Married TaxInc < 80K 80K YES

13 lassification-decision Trees, Slide 13/56 10 Example: Apply Apply Model Model to to Test Data (cont.) Test Data Refund Marital Status Taxable Income Cheat Yes Refund No No Married 80K? Single, Divorced MarSt Married TaxInc < 80K 80K YES

14 lassification-decision Trees, Slide 14/56 10 Example: Apply Apply Model Model to to Test Data (cont.) Test Data Refund Marital Status Taxable Income Cheat Yes Refund No No Married 80K? Single, Divorced MarSt Married TaxInc < 80K 80K YES

15 lassification-decision Trees, Slide 15/56 10 Example: Apply Apply Model Model to to Test Data (cont.) Test Data Refund Marital Status Taxable Income Cheat Yes Refund No No Married 80K? Single, Divorced MarSt Married TaxInc < 80K 80K YES

16 lassification-decision Trees, Slide 16/56 CS479/579: Data Mining Slide Example: Apply Model to Test Data (cont.) Apply Model to Test Data Test Data Refund Marital Status Taxable Income Cheat Yes Refund No No Married 80K? Single, Divorced MarSt Married Assign Cheat to No TaxInc < 80K 80K YES

17 10 10 Decision Tree Classification Task Example of DT Apply Model Example Learn Model Hunt s Alg. Measures of Node Impurity DT Examples and Characteristics Decision Tree Classification Task Tid Attrib1 Attrib2 Attrib3 Class 1 Yes Large 125K No 2 No Medium 100K No 3 No Small 70K No 4 Yes Medium 120K No 5 No Large 95K Yes 6 No Medium 60K No 7 Yes Large 220K No 8 No Small 85K Yes 9 No Medium 75K No 10 No Small 90K Yes Training Set Tid Attrib1 Attrib2 Attrib3 Class 11 No Small 55K? 12 Yes Medium 80K? 13 Yes Large 110K? 14 No Small 95K? 15 No Large 67K? Test Set lassification-decision Trees, Slide 17/56 Induction Deduction Tree Induction algorithm Learn Model Apply Model Model Decision Tree

18 lassification-decision Trees, Slide 18/56 Decision Tree Induction How many trees? Exponential in the number of attributes Many Algorithms: reasonably accurate, suboptimal, reasonable amount of time Hunt s Algorithm (basis of many others) CART (Classification and Regression Trees), a book by Breiman et al. ID3, C4.5 by Quinlan

19 lassification-decision Trees, Slide 19/56 Hunt s algorithm Let D t be the set of training records that reach a node t y = {y 1, y 2,, y c } are class labels General procedure If D t contains records that belong to the same class y t, then t is a leaf node labeled as y t If D t contains records that belong to more than one class Use an attribute test to split the data into smaller subsets Recursively apply the procedure to each subset

20 lassification-decision Trees, Slide 20/56 Decision Tree Induction Algorithms Design Issues How should the training records be split? How to specify the attribute test condition? How to determine the best split? Greedy strategy: split the records based on an attribute test that optimizes certain criterion When to stop splitting Naive: (1) all the records have identical attribute values; or (2) all the records belong to the same class Is there any better way? Early stop?

21 lassification-decision Trees, Slide 21/56 Specify the Attribute Test Condition? Depends on attribute types Nominal Ordinal Continuous Depends on number of ways to split 2-way split Multi-way split

22 lassification-decision Trees, Slide 22/56 Splitting Based on Nominal Attributes itting Based Splitting on Nominal Based on Attributes Nominal Attributes lti-way split: Multi-way Use as many split: Use partitions as many as distinct partitions as distinct es. Multi-way values. split: Use as many partitions as distinct values CarType CarType Family Sports Luxury Family Sports Luxury ary split: Divides Binary split: values Divides into two values Binary split: Divides valuessubsets. into two subsets. into two subsets. Need to find optimal Need to partitioning. find optimal Need to partitioning. find optimal partitioning., } CarType {Sports, {Family} Luxury} CarType OR {Family} {Family, Luxury} CarType {Family, {Sports} Luxury} OR CarType {Sports} CS479/579: Data Mining Slide-25 CS479/579: Data Mining Slide-25

23 ting Splitting Based Based on onordinal Attributes ased on Splitting Ordinal Attributes Based on Ordinal Attributes way split: Use as many partitions as distinct. it: Use as Multi-way many partitions split: Use Use as distinct as as many many partitions partitions as distinct as distinct values values. Size Small Small Size Large Medium Large split: Divides values into two subsets. ivides values Binary into split: two Divides subsets. values into two subsets. Need to find optimal partitioning. Need to find optimal partitioning. Binary split: Divides values into two subsets. Need to find optimal partitioning and preserve the order among attribute Need to find optimal partitioning. values. Size Size {Medium, Size OR {Large} Medium OR {Large} {Small, Medium} Large} Small {Large} Medium Size Large {Medium, {Small} Large} OR Size Size {Medium, {Small} Large} {Small} Size {Small, Size Size Large} {Medium} {Small, {Small, What about this split? Large} Large} {Medium} CS479/579: Data Mining Slide-26 CS479/579: Data Mining Slide-26 CS479/579: Data Mining Slide-26 is split? about this split? lassification-decision Trees, Slide 23/56 {Medium}

24 lassification-decision Trees, Slide 24/56 Splitting Based on Continuous Attributes Discretization to form an ordinal categorical attribute Static discretize once at the beginning Dynamic ranges can be found by equal interval bucketing, equal frequency bucketing (percentiles), or clustering. Binary decision: (A < v) or (A v) Consider all possible splits and find the best cut Can be more compute intensive

25 lassification-decision Trees, Slide 25/56 Splitting Based on Continuous Attributes Example Yes Taxable Income >80K? No <10K Taxable Income? >=80K [10K,25K) [25K,50K) [50K,80K) (i) Binary split (ii) Multi-way split

26 lassification-decision Trees, Slide 26/56 Determine the best split Before Splitting: 10How records to of determine class 0 the Best Split Before Splitting: 10 records of class 0, 10 records of class 1 10 records of class 1 Own Car? Car Type? Student ID? Yes No Family C0: 6 C1: 4 C0: 4 C1: 6 C0: 1 C1: 3 Sports C0: 8 C1: 0 Luxury c 1 c 10 C0: 1 C1: 7 Which test condition is the best? Which test condition is the best? C0: 1 C1: 0... C0: 1 C1: 0 c 11 C0: 0 C1: 1 c C0: 0 C1: 1

27 lassification-decision Trees, Slide 27/56 Determine the Best Split Node Impurity Splitting criterion Splitting attribute Splitting point or splitting subset Ideally, the resulting partitions at each branch are as pure as possible. Need a measure of node impurity The smaller the degree of impurity, the more skewed the class distribution. The BETTER. Node with class distribution (0,1) has zero impurity. Node with class distribution (0.5,0.5) has highest impurity.

28 lassification-decision Trees, Slide 28/56 Measures of Node Impurity Gini Index c 1 Gini(t) = 1 [p(i t)] 2 i=0 p(i t): the probability that an instance t in D t belongs to class C i, estimated by n i D, where n t i is the number of instances with class label C i. Entropy c 1 Entropy(t) = p(i t)log 2 p(i t) i=0 Log uses base 2: information is encoded in bits. Classification error Classification error(t) = 1 max i [p(i t)]

29 lassification-decision Trees, Slide 29/56 Determine the Best Split Information Gain Before splitting: C0 How N00 to Find the Best Split M0 = I (N) C1 N01 Before Splitting: C0 C1 N00 N01 M0 A? B? Yes No Yes No Node N1 Node N2 Node N3 Node N4 C0 C1 N10 N11 C0 C1 N20 N21 C0 C1 N30 N31 C0 C1 N40 N41 M1 M2 M3 M4 M12 Gain = M0 M12 vs M0 M34 CS479/579: Data Mining Slide-33 Gain: = I (parent) k N(v j ) j=1 N I (v j) Gain = M0 M12 vs M0 M34 M34

30 lassification-decision Trees, Slide 30/56 Measures of Node Impurity Gini c 1 Gini(t) = 1 [p(i t)] 2 i=0 p(i t): the relative frequency of class i at node t. Maximum? (1 1 n c ) when records are equally distributed among all classes, implying least interesting information Minimum? (0.0) when all records belong to one class, implying most interesting information First used in CART, which allows only binary splitting

31 lassification-decision Trees, Slide 31/56 Measures of Node Impurity Gini C1 0 C2 6 C1 1 C2 5 C1 2 C2 4 C1 3 C2 3 Gini? 1 ( 0 6 )2 ( 6 6 )2 = 0 1 ( 1 6 )2 ( 5 6 )2 = = ( 2 6 )2 ( 4 6 )2 = =

32 lassification-decision Trees, Slide 32/56 two partitions Example of DT Apply Model Example Learn Model Hunt s Alg. Measures of Node Impurity DT Examples and Characteristics Weighing partitions: Binary Attributes: Computing Gini Index r and Purer Partitions are sought for. Yes Node N1 B? No Node N2 CS479/579: Data Mining Slide-37 Parent C1 6 C2 6 Gini = (2/6) 2 Parent N1 N2 N1 N2 Gini(Children) C C1 5 1 = 7/12 * C C /12 * (4/6) 2 Gini(Parent) Gini=0.333 = 0.5 = Gini(N1) = 1 (5/7) 2 (2/7) 2 Gini(N2) = 1 (1/5) 2 (4/5) 2 Gini(Children) = 7/12 Gini(N1) + 5/12 Gini(N2) Gain = Gini(Parent) Gini(Children)

33 Categorical Attributes: Computing Gini Index Categorical Attributes: Computing Gini Index For each distinct value, gather counts counts for for each each class class in the in the dataset dataset Use the count matrix to make decisions Multi-way split Two-way split (find best partition of values) CarType Family Sports Luxury C C Gini CarType {Sports, Luxury} {Family} C1 3 1 C2 2 4 Gini CarType {Sports} {Family, Luxury} C1 2 2 C2 1 5 Gini Calculation example? Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 # lassification-decision Trees, Slide 33/56

34 lassification-decision Trees, Slide 34/56 Categorical Attributes: Computing Gini Index Gini(family) = 1-(1/25)-(16/25)=(8/25) Gini(sports) = 1-(4/9)-(1/9)=(4/9) Gini(luxury) = 1/2 Gini(all) = (5/10)*(8/25)+(3/10)*(4/9)+(2/10)*(1/2) =(4/25)+(2/15)+(1/10) = ( )/150 = 59/150 = 0.393

35 lassification-decision Trees, Slide 35/56 Continuous Attributes: Computing Gini Index Use Binary Decisions based on one value Several Choices for the splitting value Number of possible splitting values = Number of distinct values Each splitting value has a count matrix associated with it Class counts in each of the partitions, A < v and A v Simple method to choose best v For each v, scan the database to gather count matrix and compute its Gini index Computationally Inefficient! Repetition of work.

36 lassification-decision Trees, Slide 36/56 Continuous Attributes: Computing Gini Index Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes

37 lassification-decision Trees, Slide 37/56 Continuous Attributes: Computing Gini Index Continuous Attributes: Computing Gini Index... For efficient computation: for each for each attribute, attribute, Sort the attribute on values Sort the attribute on values Linearly Linearly scan scan these these values, values, each each time time updating update the the count count matrix matrix and computing gini index and compute gini index Choose the split position that has the least gini index Choose the split position that has the least Gini index Cheat No No No Yes Yes Yes No No No No Taxable Income Sorted Values Split Positions <= > <= > <= > <= > <= > <= > <= > <= > <= > <= > <= > Yes No Gini Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 #

38 lassification-decision Trees, Slide 38/56 Entropy c 1 Entropy(t) = p(i t)log 2 p(i t) i=0 Measures homogeneity of a node. Maximum (log(n c )) when records are equally distributed among all classes implying least information Minimum (0.0) when all records belong to one class, implying most information

39 lassification-decision Trees, Slide 39/56 Examples for Computing Entropy C1 0 C2 6 Entropy? c 1 Entropy(t) = p(i t)log 2 p(i t) C1 1 C2 5 C1 2 C2 4 i=0 C1 3 C2 3 0log0 1log1 = 0 (prob. is 0 means it does not happen, let 0log0 = 0) 1 6 log(1 6 ) 5 6 log(5 6 ) = log(2 6 ) 4 6 log(4 6 ) = 0.92

40 lassification-decision Trees, Slide 40/56 Splitting Based on Information Gain Information Gain = Entropy(p) Used in ID3 k j=1 N(v j ) N Entropy(v j) Disadvantage: Tends to prefer splits that result in large number of partitions, each being small but pure. Consider partition on ID? Entropy(children) = 0

41 lassification-decision Trees, Slide 41/56 Splitting Based on Gain Ratio Gain Ratio = Split Info = Information Gain Split Info k j=1 ( n j n log(n j n )) Parent node p is split into k partitions n j is the number of records in partition j Higher entropy partitioning (large number of small partitions) is penalized! Used in C4.5, successor of ID3 Designed to overcome the disadvantage of Information Gain

42 lassification-decision Trees, Slide 42/56 Classification Error Error(t) = 1 max i p(i t) Measures misclassification error made by a node. Maximum (1 1/n c ) when records are equally distributed among all classes, implying least interesting information Minimum (0.0) when all records belong to one class, implying most interesting information

43 lassification-decision Trees, Slide 43/56 Examples for Computing Classification Error C1 0 C2 6 C1 1 C2 5 Classification Error? 1 max(0, 1) = 0 Error(t) = 1 max i p(i t) C1 2 C2 4 C1 3 C2 3 1 max(1/6, 5/6) = 1 5/6 = 1/6 1 max(2/6, 4/6) = 1 4/6 = 1/3 0.5

44 Comparison among Splitting Criteria Comparison among Splitting Criteria Example of DT Apply Model Example Learn Model Hunt s Alg. Measures of Node Impurity DT Examples and Characteristics For Fora a2-class problem: lassification-decision Trees, Slide 44/56 CS479/579: Data Mining Slide-47

45 lassification-decision Trees, Slide 45/56 Stopping Criteria for Tree Induction Stop expanding a node when all the records belong to the same class Stop expanding a node when all the records have similar attribute values Early termination: use threshold

46 lassification-decision Trees, Slide 46/56 Examples ID3: Use Entropy, Information gain C4.5: Use Entropy, Normalized Information Gain (Gain Ratio) Download the software from: CART: Uses Gini Index, only binary splits Which measure is the best? Time complexity of DT induction increases exponentially with tree height shallower trees Shallow trees tend to have a large number of leaves and higher error rates

47 lassification-decision Trees, Slide 47/56 Example: C4.5 Simple depth-first construction. Uses Information gain Sorts continuous attributes at each node. Needs entire data to fit in memory. Unsuitable for large datasets. Download the software from:

48 DT Example (1) RID age income student credit rating buys computer 1 <= 30 high no fair no 2 <= 30 high no excellent no high no fair yes 4 > 40 medium no fair yes 5 > 40 low yes fair yes 6 > 40 low yes excellent no low yes excellent yes 8 <= 30 medium no fair no 9 <= 30 low yes fair yes 10 > 40 medium yes fair yes 11 <= 30 medium yes excellent yes medium no excellent yes high yes fair yes 14 > 40 medium no excellent no Class label attribute: buys computer Preprocess age attribute: <= 30: young 31 40: middle aged > 40: senior lassification-decision Trees, Slide 48/56

49 lassification-decision Trees, Slide 49/56 DT Example (2) RID age income student credit rating buys computer 1 young high no fair no 2 young high no excellent no 3 middle aged high no fair yes 4 senior medium no fair yes 5 senior low yes fair yes 6 senior low yes excellent no 7 middle aged low yes excellent yes 8 young medium no fair no 9 young low yes fair yes 10 senior medium yes fair yes 11 young medium yes excellent yes 12 middle aged medium no excellent yes 13 middle aged high yes fair yes 14 senior medium no excellent no

50 lassification-decision Trees, Slide 50/56 DT Example (3) Information gain based on Gini Index Number of classes: C 1 : for yes, 9 C 2 : for no, 5 Gini(D) = 1 ( 9 14 )2 ( 5 14 )2 = Consider using attribute income as splitting attribute: D 1 : income is {low, medium} Gini(D 1 ) = 1 ( 6 10 )2 ( 4 10 )2 = D 2 : income is {high} Gini(D 2 ) = 1 ( 2 4 )2 ( 2 4 )2 = 1 2 Gini(children) = = = = Gain = Gini(D) Gini(children) = 0.027

51 lassification-decision Trees, Slide 51/56 DT Example (4) Information gain based on Entropy Entropy(D) = 9 14 log 2( 9 14 ) 5 14 log 2( 5 14 ) = 0.94 Compute the expected entropy for each attribute. Start with attribute age. Consider multi-split (i.e., 3-split). D young : 2 yes, 3 no; Calculation? Avg Entropy = Gain(age) = = Compute gain for other attributes: Gain(income) = 0.029, Gain(student), etc.

52 lassification-decision Trees, Slide 52/56 DT Example (5) Gain ratio Consider attribute income. low: 4 medium: 6 high: 4 Split Info(D) = 4 14 log 2( 4 14 ) 6 14 log 2( 6 14 ) 4 14 log 2( 4 14 ) = Gain ratio(income) = = 0.031

53 lassification-decision Trees, Slide 53/56 Decision Tree R packages > install.packages("tree") > library(tree) > require(tree) > help(tree) > install.packages("rpart") > library(rpart) > require(rpart) > help(rpart)

54 lassification-decision Trees, Slide 54/56 Decision Tree Based Classification Advantages Inexpensive to construct Extremely fast at classifying unknown records Easy to interpret for small-sized trees Accuracy is comparable to other classification techniques for many simple data sets

55 lassification-decision Trees, Slide 55/56 Decision Boundary Decision boundary: border line between two neighboring regions of different classes Decision boundary is parallel to axes because test condition involves a single attribute at-a-time. Oblique decision tree Test condition may involve multiple attributes x + y < 1 More expressive representation Finding optimal test condition is computationally expensive Constructive induction Purpose: partition the data into homogeneous, non-rectangular regions. Composite attribute

56 lassification-decision Trees, Slide 56/56 Example of DT Apply Model Example Learn Model Hunt s Alg. Measures of Node Impurity DT Examples and Characteristics Tree Replication Tree Replication P Q R S 0 Q S 0 0 1