9/6/14. Our first learning algorithm. Comp 135 Introduction to Machine Learning and Data Mining. knn Algorithm. knn Algorithm (simple form)

Transcription

1 Comp 135 Introduction to Machine Learning and Data Mining Our first learning algorithm How would you classify the next example? Fall 2014 Professor: Roni Khardon Computer Science Tufts University o o o o o ooo o? x o x x xx x Some of the slides were adapted from previous slides by Carla Brodley k Algorithm (simple form) k Algorithm At training time do nothing. Store examples. When given new example: find k nearest neighbors Predict L= majority vote of their labels Theoretical basis + intuition: in the limit, when the dataset is dense, this should pick up all important regions Very flexible classifier: no prior commitment to the shape, density, or distribution of regions k: problems and extensions k: problems and extensions oisy labels in training data Expensive test time/application: Complexity of finding the neighbors How to choose k? The effect of close neighbors and far neighbors Completely dependent on the distance metric and representation E.g., Euclidean distance: May require ormalization of features Treats all dimensions equally Sensitive to high dimension/ irrelevant features 1

2 k: problems and extensions Can we predict real values instead of discrete labels (i.e., solve the regression problem)? Our second algorithm Let s look at a simple dataset for motivation: Outlook Temp Humidity Windy Play Sunny Hot High False o Sunny Hot High True o Overcast Hot High False es Rainy Mild High False es Rainy Cool ormal False es Rainy Cool ormal True o Overcast Cool ormal True es Sunny Mild High False o Sunny Cool ormal False es Rainy Mild ormal False es Sunny Mild ormal True es Overcast Mild High True es Overcast Hot ormal False es Rainy Mild High True o Class: Play tennis Attributes: Outlook Temp Humidity Windy Give a different way to identify regions in instance space: Recursively split on values of features to define regions that have single label What does this look like in Euclidean space? Decision trees can represent any discrete function of the attributes! Why? Given training set, can we build a tree that agrees with the data? (yes; easy; why?) What is a good decision tree? Given training set, how can we build a good tree? Let s focus on a simpler question: which attribute should we choose for root of tree? Example: [Pos,eg] before and after split [50,50]! [35,15] + [15,35] [50, 0] + [0,50] [10,30] + [30,10] + [10,10] [25,25] + [25,25] 2

3 Several selection criteria have been proposed and used. Information gain is commonly used (C4.5, J48) We need to learn about entropy Entropy(p 1,...,p n )= X p i log 1 p i = X p i log p i For 2 classes p 1 = p, p 2 =1 p and this simplifies to Entropy(p) = X p log p (1 p) log (1 p) ow consider a split S! S 1,...,S k where the S i are subsets of S and may include examples from multiple classes Gain(Split) = Ent(S) X j S j S Ent(S j) Gain(Split) = Ent(S) Heuristic for wide splits SplitInfo = X j GainRatio = X j S j S log S S j Gain SplitInfo S j S Ent(S j) Outlook = Sunny: info([2,3] ) = entropy(2/5,3/5) = 2 / 5log(2 / 5) 3/ 5log(3/ 5) = 0.971bits Outlook = Overcast: info([4,0] ) = entropy(1,0) = 1log(1) 0log(0) = 0 bits Outlook = Rainy: info([3,2] ) = entropy(3/5,2/5) = 3/ 5log(3/ 5) 2 / 5log(2 / 5) = 0.971bits Expected information for attribute: ote: defined as 0 info([3,2],[4,0],[3,2]) = (5/14) (4 /14) 0 + (5/14) = bits 3

4 Continuing to Split gain(outlook ) = info([9,5]) info([2,3],[4,0],[3,2]) = = bits gain(outlook ) = bits gain(temperature ) = bits gain(humidity ) = bits gain(windy ) = bits gain(temperature ) = bits gain(humidity ) = bits gain(windy ) = bits Final Decision Tree Decision Tree Learning Algorithm If data has a pure class Make leaf node with that class Otherwise Pick feature to split on Divide data into sub-datasets s j according to the feature s values Recursively build a tree for each subset Other Criteria / Tasks Real Valued Attributes The Gini Criterion = 4p(1 p) The [KM] Criterion = 2 p p(1 p) Criterion for Regression = 1` v = 1` `X (v i v) 2 i=1 `X i=1 v i aïve treatment makes a very side split with possibly one example per branch. Is this good? Alternative picks threshold t and tests (feature >= t) to get a binary split. How can we pick t? 4

5 The Bad ews This does not quite work Overfitting in DT Why Does this happen? Accuracy Size of tree (number of nodes) On training data On test data Few examples at lower levels in tree Quantities calculated not reliable in this case Even worse with noisy data And when features not sufficiently rich Overfitting in DT REP Example Solution 1 (C4.5, J48): uses a confidence interval based on class ratio at leaf and number of examples in the training set. This is not fully justified but works well in practice. Solution 2: use a validation set. Known as reduced error pruning (REP) Sock red Traffic density >75 <=75 l h m blue >70 <=70 Red sox t >74 <=74 f other <=78 >78 full moon y Cereal Rice crispies Chex n All Bran REP Example REP Example red Traffic density >75 <=75 l h m Sock Decision Errors 25, 1 Keep >70 <=70 7 >74 <=74 blue Prune 4 Pats win t f 2, 4 2, 5 other <=78 >78 full moon y Cereal Rice crispies Chex n All Bran red Decision Errors Keep 11 Traffic density Prune >75 <=75 14 l h m 3, 8 >70 <=70 4, 2 1, 9 Red sox t >74 <=74 2, 8 2, 1 f 2, 8 other <=78 >78 full moon y Cereal Rice crispies Chex n All Bran 5

6 Missing Attribute Values Common in real data We can handle this in a way that works across algorithms (that is, also for k). How? But we can do better with a solution tailored for decision trees. How? Recap We already know two learning algorithms And many variants or improvements over their basic forms Given new application: How do we know if one of these (say k) is doing well? And which one is doing better? 6