Decision Trees: Representation
|
|
|
- Gloria Jennings
- 5 years ago
- Views:
Transcription
1 Decision Trees: Representation Machine Learning Fall 2018 Some slides from Tom Mitchell, Dan Roth and others 1
2 Key issues in machine learning Modeling How to formulate your problem as a machine learning problem? How to represent data? Which algorithms to use? What learning protocols? Representation Good hypothesis spaces and good features Algorithms What is a good learning algorithm? What is success? Generalization vs overfitting The computational question: How long will learning take? 2
3 Coming up (the rest of the semester) Different hypothesis spaces and learning algorithms Decision trees and the ID3 algorithm Linear classifiers Perceptron SVM Logistic regression Combining multiple classifiers Boosting, bagging Non-linear classifiers Nearest neighbors 3
4 Coming up (the rest of the semester) Different hypothesis spaces and learning algorithms Decision trees and the ID3 algorithm Linear classifiers Important issues to consider Perceptron SVM 1. What do these hypotheses represent? Logistic regression 2. Implicit assumptions and tradeoffs Combining multiple classifiers Boosting, bagging 3. Generalization? Non-linear classifiers Nearest 4. neighbors How do we learn? 4
5 This lecture: Learning Decision Trees 1. Representation: What are decision trees? 2. Algorithm: Learning decision trees The ID3 algorithm: A greedy heuristic 3. Some extensions 5
6 This lecture: Learning Decision Trees 1. Representation: What are decision trees? 2. Algorithm: Learning decision trees The ID3 algorithm: A greedy heuristic 3. Some extensions 6
7 Representing data Data can be represented as a big table, with columns denoting different attributes Name Label Claire Cardie - Peter Bartlett + Eric Baum + Haym Hirsh + Shai Ben-David + Michael I. Jordan - 7
8 Representing data Data can be represented as a big table, with columns denoting different attributes Name Name has punctuation? Second character of first name Length of first name>5? Male/ Female? Claire Cardie No l Yes Female - Peter Bartlett No e No Male + Eric Baum No r No Male + Haym Hirsh No a No Male + Label Shai Ben- David Michael I. Jordan Yes h No Male + Yes i Yes Male - 8
9 Representing data Data can be represented as a big table, with columns denoting different attributes Name Name has punctuation? Second character of first name Length of first name>5? Male/ Female? Claire With Cardiethese four No attributes, how lmany unique rows Yes are possible? Female = 2704 Peter Bartlett No e No Male + Eric Baum If there are 100 No attributes, all binary, r how many Nounique rows Male are possible? Haym Hirsh No a No Male + Label Shai Ben- David Michael I. Jordan Yes h No Male - Yes i Yes Male + 9
10 Representing data Data can be represented as a big table, with columns denoting different attributes Name Name has punctuation? Second character of first name Length of first name>5? Male/ Female? Claire With Cardiethese four No attributes, how lmany unique rows Yes are possible? Female = 208 Peter Bartlett No e No Male + Eric Baum If there are 100 No attributes, all binary, r how many Nounique rows Male are possible? Haym Hirsh No a No Male + Label Shai Ben- David Michael I. Jordan Yes h No Male - Yes i Yes Male + 10
11 Representing data Data can be represented as a big table, with columns denoting different attributes Name Name has punctuation? Second character of first name Length of first name>5? Male/ Female? Claire With Cardiethese four No attributes, how lmany unique rows Yes are possible? Female = 208 Peter Bartlett No e No Male + Eric Baum If there are 100 No attributes, all binary, r how many Nounique rows Male are possible? Haym Hirsh No a No Male + Label Shai Ben- David Michael I. Jordan Yes h No Male - Yes i Yes Male + 11
12 Representing data Data can be represented as a big table, with columns denoting different attributes Name Name has punctuation? Second character of first name Length of first name>5? Male/ Female? Claire With Cardiethese four No attributes, how lmany unique rows Yes are possible? Female = 208 Peter Bartlett No e No Male + Eric Baum If there are 100 No attributes, all binary, r how many Nounique rows Male are possible? - (100 times) = 2 )** Haym Hirsh No a No Male + Label Shai Ben- David Michael I. Jordan Yes h No Male - Yes i Yes Male + 12
13 Representing data Data can be represented as a big table, with columns denoting different attributes Name Name has punctuation? Second character of first name Length of first name>5? Male/ Female? Label Claire With Cardiethese four No attributes, how lmany unique rows Yes are possible? Female = 208 Peter Bartlett No e No Male + Eric Baum If there are 100 No attributes, all binary, r how many Nounique rows Male are possible? - (100 times) = 2 )** Haym Hirsh No a No Male + Shai Ben- If we wanted Yes to store all possible h rows, this No number is Male too large. - David We need to figure out how to represent data in a better, more efficient way Michael I. Yes i Yes Male + Jordan 13
14 What are decision trees? A hierarchical data structure that represents data using a divide-and-conquer strategy Can be used as hypothesis class for non-parametric classification or regression General idea: Given a collection of examples, learn a decision tree that represents it 14
15 What are decision trees? Decision trees are a family of classifiers for instances that are represented by collections of attributes (i.e. features) Nodes are tests for feature values There is one branch for every value that the feature can take Leaves of the tree specify the class labels 15
16 Let s build a decision tree for classifying shapes Label=C Label=B Label=A 16
17 Let s build a decision tree for classifying shapes Label=C Label=B Label=A Before building a decision tree: What is the label for a red triangle? And why? 17
18 Let s build a decision tree for classifying shapes Label=C Label=B Label=A What are some attributes of the examples? 18
19 Let s build a decision tree for classifying shapes What are some attributes of the examples? Color, Shape Label=C Label=B Label=A 19
20 Let s build a decision tree for classifying shapes What are some attributes of the examples? Color, Shape Label=C Label=B Label=A Color? 20
21 Let s build a decision tree for classifying shapes What are some attributes of the examples? Color, Shape Label=C Label=B Label=A Color? Blue Red Green 21
22 Let s build a decision tree for classifying shapes What are some attributes of the examples? Color, Shape Label=C Label=B Label=A Color? Blue Red Green B 22
23 Let s build a decision tree for classifying shapes What are some attributes of the examples? Color, Shape Label=C Label=B Label=A Blue Shape? triangle square B A Color? Green Red B circle C 23
24 Let s build a decision tree for classifying shapes What are some attributes of the examples? Color, Shape Label=C Label=B Label=A triangle B Color? Blue Red Green Shape? B Shape? circle square square A C B circle A 24
25 Let s build a decision tree for classifying shapes 1. How do we learn a decision tree? Coming up soon 2. How to use a decision tree for prediction? What is the label for a red triangle? Label=C Just follow a path Label=B from the root to a leaf Label=A What are some attributes of the examples? What about a green triangle? Color, Shape B triangle Blue Shape? square A circle Color? C Red B square B Green Shape? circle A 25
26 Let s build a decision tree for classifying shapes 1. How do we learn a decision tree? Coming up soon 2. How to use a decision tree for prediction? What is the label for a red triangle? Label=C Just follow a path Label=B from the root to a leaf Label=A What are some attributes of the examples? What about a green triangle? Color, Shape B triangle Blue Shape? square A circle Color? C Red B square B Green Shape? circle A 26
27 Expressivity of Decision trees What Boolean functions can decision trees represent? Any Boolean function Every path from the tree to a root is a rule The full tree is equivalent to the conjunction of all the rules (Color=blue AND Shape=triangle ) Label=B) AND (Color=blue AND Shape=square ) Label=A) AND (Color=blue AND Shape=circle ) Label=C) AND. Any Boolean function can be represented as a decision tree. 27
28 Expressivity of Decision trees What Boolean functions can decision trees represent? Any Boolean function Every path from the tree to a root is a rule The full tree is equivalent to the conjunction of all the rules (Color=blue AND Shape=triangle ) Label=B) AND (Color=blue AND Shape=square ) Label=A) AND (Color=blue AND Shape=circle ) Label=C) AND. Any Boolean function can be represented as a decision tree. 28
29 Expressivity of Decision trees What Boolean functions can decision trees represent? Any Boolean function Every path from the tree to a root is a rule The full tree is equivalent to the conjunction of all the rules (Color=blue AND Shape=triangle ) Label=B) AND (Color=blue AND Shape=square ) Label=A) AND (Color=blue AND Shape=circle ) Label=C) AND. Any Boolean function can be represented as a decision tree. 29
30 Decision Trees Outputs are discrete categories But real valued outputs are also possible (regression trees) Well studied methods for handling noisy data (noise in the label or in the features) and for handling missing attributes Pruning trees helps with noise More on this later 30
31 Numeric attributes and decision boundaries We have seen instances represented as attribute-value pairs (color=blue, second letter=e, etc.) Values have been categorical How do we deal with numeric feature values? (eg length =?) Discretize them or use thresholds on the numeric values This example divides the feature space into axis parallel rectangles 31
32 Numeric attributes and decision boundaries We have seen instances represented as attribute-value pairs (color=blue, second letter=e, etc.) Values have been categorical How do we deal with numeric feature values? (eg length =?) Discretize them or use thresholds on the numeric values This example divides the feature space into axis parallel rectangles 32
33 Numeric attributes and decision boundaries We have seen instances represented as attribute-value pairs (color=blue, second letter=e, etc.) Values have been categorical How do we deal with numeric feature values? (eg length =?) Discretize them or use thresholds on the numeric values This example divides the feature space into axis parallel rectangles Y X 33
34 Numeric attributes and decision boundaries We have seen instances represented as attribute-value pairs (color=blue, second letter=e, etc.) Values have been categorical How do we deal with numeric feature values? (eg length =?) Discretize them or use thresholds on the numeric values This example divides the feature space into axis parallel rectangles Y X no Y>7 no yes X<3 no yes Y<5 no yes X < 1 yes
35 Numeric attributes and decision boundaries We have seen instances represented as attribute-value pairs (color=blue, second letter=e, etc.) Values have been categorical How do we deal with numeric feature values? (eg length =?) Discretize them or use thresholds on the numeric values This example divides the feature space into axis parallel rectangles Decision Y boundaries can be non-linear X no Y>7 no yes X<3 no yes Y<5 no yes X < 1 yes
36 Summary: Decision trees Decision trees can represent any Boolean function A way to represent lot of data A natural representation (think 20 questions) Predicting with a decision tree is easy Clearly, given a dataset, there are many decision trees that can represent it. [Exercise: Why?] Learning a good representation from data is the next question 36
37 Summary: Decision trees Decision trees can represent any Boolean function A way to represent lot of data A natural representation (think 20 questions) Predicting with a decision tree is easy Clearly, given a dataset, there are many decision trees that can represent it. [Exercise: Why?] Learning a good representation from data is the next question 37
38 Exercises 1. Write down the decision tree for the shapes data if the root node was Shape instead of Color. 2. Will the two trees make the same predictions for unseen shapes/color combinations? Label=C Label=B Label=A 3. Show that multiple structurally different decision trees can represent the same Boolean function of two or more variables. 38
Decision Trees: Representa:on
Decision Trees: Representa:on Machine Learning Fall 2017 Some slides from Tom Mitchell, Dan Roth and others 1 Key issues in machine learning Modeling How to formulate your problem as a machine learning
Supervised Learning: The Setup. Spring 2018
Supervised Learning: The Setup Spring 2018 1 Homework 0 will be released today through Canvas Due: Jan. 19 (next Friday) midnight 2 Last lecture We saw What is learning? Learning as generalization The
Machine Learning. Decision Trees. Manfred Huber
Machine Learning Decision Trees Manfred Huber 2015 1 Decision Trees Classifiers covered so far have been Non-parametric (KNN) Probabilistic with independence (Naïve Bayes) Linear in features (Logistic
Decision Trees: Discussion
Decision Trees: Discussion Machine Learning Spring 2018 The slides are mainly from Vivek Srikumar 1 This lecture: Learning Decision Trees 1. Representation: What are decision trees? 2. Algorithm: Learning
Ensemble Methods, Decision Trees
CS 1675: Intro to Machine Learning Ensemble Methods, Decision Trees Prof. Adriana Kovashka University of Pittsburgh November 13, 2018 Plan for This Lecture Ensemble methods: introduction Boosting Algorithm
Classification Algorithms in Data Mining
August 9th, 2016 Suhas Mallesh Yash Thakkar Ashok Choudhary CIS660 Data Mining and Big Data Processing -Dr. Sunnie S. Chung Classification Algorithms in Data Mining Deciding on the classification algorithms
Decision Trees Oct
Decision Trees Oct - 7-2009 Previously We learned two different classifiers Perceptron: LTU KNN: complex decision boundary If you are a novice in this field, given a classification application, are these
Decision Tree CE-717 : Machine Learning Sharif University of Technology
Decision Tree CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Some slides have been adapted from: Prof. Tom Mitchell Decision tree Approximating functions of usually discrete
PROBLEM 4
PROBLEM 2 PROBLEM 4 PROBLEM 5 PROBLEM 6 PROBLEM 7 PROBLEM 8 PROBLEM 9 PROBLEM 10 PROBLEM 11 PROBLEM 12 PROBLEM 13 PROBLEM 14 PROBLEM 16 PROBLEM 17 PROBLEM 22 PROBLEM 23 PROBLEM 24 PROBLEM 25
Nearest Neighbor Classification. Machine Learning Fall 2017
Nearest Neighbor Classification Machine Learning Fall 2017 1 This lecture K-nearest neighbor classification The basic algorithm Different distance measures Some practical aspects Voronoi Diagrams and Decision
CS6375: Machine Learning Gautam Kunapuli. Mid-Term Review
Gautam Kunapuli Machine Learning Data is identically and independently distributed Goal is to learn a function that maps to Data is generated using an unknown function Learn a hypothesis that minimizes
What is Learning? CS 343: Artificial Intelligence Machine Learning. Raymond J. Mooney. Problem Solving / Planning / Control.
What is Learning? CS 343: Artificial Intelligence Machine Learning Herbert Simon: Learning is any process by which a system improves performance from experience. What is the task? Classification Problem
Lecture 7: Decision Trees
Lecture 7: Decision Trees Instructor: Outline 1 Geometric Perspective of Classification 2 Decision Trees Geometric Perspective of Classification Perspective of Classification Algorithmic Geometric Probabilistic...
Bias-Variance Analysis of Ensemble Learning
Bias-Variance Analysis of Ensemble Learning Thomas G. Dietterich Department of Computer Science Oregon State University Corvallis, Oregon 97331 http://www.cs.orst.edu/~tgd Outline Bias-Variance Decomposition
CS 229 Midterm Review
CS 229 Midterm Review Course Staff Fall 2018 11/2/2018 Outline Today: SVMs Kernels Tree Ensembles EM Algorithm / Mixture Models [ Focus on building intuition, less so on solving specific problems. Ask
Business Club. Decision Trees
Business Club Decision Trees Business Club Analytics Team December 2017 Index 1. Motivation- A Case Study 2. The Trees a. What is a decision tree b. Representation 3. Regression v/s Classification 4. Building
Classification with Decision Tree Induction
Classification with Decision Tree Induction This algorithm makes Classification Decision for a test sample with the help of tree like structure (Similar to Binary Tree OR k-ary tree) Nodes in the tree
Random Forest A. Fornaser
Random Forest A. Fornaser alberto.fornaser@unitn.it Sources Lecture 15: decision trees, information theory and random forests, Dr. Richard E. Turner Trees and Random Forests, Adele Cutler, Utah State University
Machine Learning. Decision Trees. Le Song /15-781, Spring Lecture 6, September 6, 2012 Based on slides from Eric Xing, CMU
Machine Learning 10-701/15-781, Spring 2008 Decision Trees Le Song Lecture 6, September 6, 2012 Based on slides from Eric Xing, CMU Reading: Chap. 1.6, CB & Chap 3, TM Learning non-linear functions f:
Decision Tree Learning
Decision Tree Learning 1 Simple example of object classification Instances Size Color Shape C(x) x1 small red circle positive x2 large red circle positive x3 small red triangle negative x4 large blue circle
Evaluating Classifiers
Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts
A Systematic Overview of Data Mining Algorithms
A Systematic Overview of Data Mining Algorithms 1 Data Mining Algorithm A well-defined procedure that takes data as input and produces output as models or patterns well-defined: precisely encoded as a
Data Preprocessing. Supervised Learning
Supervised Learning Regression Given the value of an input X, the output Y belongs to the set of real values R. The goal is to predict output accurately for a new input. The predictions or outputs y are
Supervised vs unsupervised clustering
Classification Supervised vs unsupervised clustering Cluster analysis: Classes are not known a- priori. Classification: Classes are defined a-priori Sometimes called supervised clustering Extract useful
Decision trees. Decision trees are useful to a large degree because of their simplicity and interpretability
Decision trees A decision tree is a method for classification/regression that aims to ask a few relatively simple questions about an input and then predicts the associated output Decision trees are useful
Symbolic AI. Andre Freitas. Photo by Vasilyev Alexandr
Symbolic AI Andre Freitas Photo by Vasilyev Alexandr Acknowledgements These slides were based on the slides of: Peter A. Flach, Rule induction tutorial, IDA Spring School 2001. Anoop & Hector, Inductive
Machine Learning Techniques
Machine Learning Techniques ( 機器學習技法 ) Lecture 9: Decision Tree Hsuan-Tien Lin ( 林軒田 ) htlin@csie.ntu.edu.tw Department of Computer Science & Information Engineering National Taiwan University ( 國立台灣大學資訊工程系
Network Traffic Measurements and Analysis
DEIB - Politecnico di Milano Fall, 2017 Sources Hastie, Tibshirani, Friedman: The Elements of Statistical Learning James, Witten, Hastie, Tibshirani: An Introduction to Statistical Learning Andrew Ng:
Supervised Learning. Decision trees Artificial neural nets K-nearest neighbor Support vectors Linear regression Logistic regression...
Supervised Learning Decision trees Artificial neural nets K-nearest neighbor Support vectors Linear regression Logistic regression... Supervised Learning y=f(x): true function (usually not known) D: training
Text Categorization. Foundations of Statistic Natural Language Processing The MIT Press1999
Text Categorization Foundations of Statistic Natural Language Processing The MIT Press1999 Outline Introduction Decision Trees Maximum Entropy Modeling (optional) Perceptrons K Nearest Neighbor Classification
CS Machine Learning
CS 60050 Machine Learning Decision Tree Classifier Slides taken from course materials of Tan, Steinbach, Kumar 10 10 Illustrating Classification Task Tid Attrib1 Attrib2 Attrib3 Class 1 Yes Large 125K
Tree-based methods for classification and regression
Tree-based methods for classification and regression Ryan Tibshirani Data Mining: 36-462/36-662 April 11 2013 Optional reading: ISL 8.1, ESL 9.2 1 Tree-based methods Tree-based based methods for predicting
Decision Trees. Petr Pošík. Czech Technical University in Prague Faculty of Electrical Engineering Dept. of Cybernetics
Decision Trees Petr Pošík Czech Technical University in Prague Faculty of Electrical Engineering Dept. of Cybernetics This lecture is largely based on the book Artificial Intelligence: A Modern Approach,
Supervised Learning (contd) Linear Separation. Mausam (based on slides by UW-AI faculty)
Supervised Learning (contd) Linear Separation Mausam (based on slides by UW-AI faculty) Images as Vectors Binary handwritten characters Treat an image as a highdimensional vector (e.g., by reading pixel
Evaluating Classifiers
Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts
Machine Learning. Chao Lan
Machine Learning Chao Lan Machine Learning Prediction Models Regression Model - linear regression (least square, ridge regression, Lasso) Classification Model - naive Bayes, logistic regression, Gaussian
Large Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Machine Learning Basics - 1 U Kang Seoul National University U Kang 1 In This Lecture Overview of Machine Learning Capacity, overfitting, and underfitting
Machine Learning Techniques for Data Mining
Machine Learning Techniques for Data Mining Eibe Frank University of Waikato New Zealand 10/25/2000 1 PART VII Moving on: Engineering the input and output 10/25/2000 2 Applying a learner is not all Already
Classification: Feature Vectors
Classification: Feature Vectors Hello, Do you want free printr cartriges? Why pay more when you can get them ABSOLUTELY FREE! Just # free YOUR_NAME MISSPELLED FROM_FRIEND... : : : : 2 0 2 0 PIXEL 7,12
Slides for Data Mining by I. H. Witten and E. Frank
Slides for Data Mining by I. H. Witten and E. Frank 7 Engineering the input and output Attribute selection Scheme-independent, scheme-specific Attribute discretization Unsupervised, supervised, error-
CSC411/2515 Tutorial: K-NN and Decision Tree
CSC411/2515 Tutorial: K-NN and Decision Tree Mengye Ren csc{411,2515}ta@cs.toronto.edu September 25, 2016 Cross-validation K-nearest-neighbours Decision Trees Review: Motivation for Validation Framework:
DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University January 24 2019 Logistics HW 1 is due on Friday 01/25 Project proposal: due Feb 21 1 page description
Applying Supervised Learning
Applying Supervised Learning When to Consider Supervised Learning A supervised learning algorithm takes a known set of input data (the training set) and known responses to the data (output), and trains
Nearest Neighbor Predictors
Nearest Neighbor Predictors September 2, 2018 Perhaps the simplest machine learning prediction method, from a conceptual point of view, and perhaps also the most unusual, is the nearest-neighbor method,
Decision Tree Learning
Decision Tree Learning Debapriyo Majumdar Data Mining Fall 2014 Indian Statistical Institute Kolkata August 25, 2014 Example: Age, Income and Owning a flat Monthly income (thousand rupees) 250 200 150
Data Mining Practical Machine Learning Tools and Techniques
Decision trees Extending previous approach: Data Mining Practical Machine Learning Tools and Techniques Slides for Chapter 6 of Data Mining by I. H. Witten and E. Frank to permit numeric s: straightforward
All lecture slides will be available at CSC2515_Winter15.html
CSC2515 Fall 2015 Introduc3on to Machine Learning Lecture 9: Support Vector Machines All lecture slides will be available at http://www.cs.toronto.edu/~urtasun/courses/csc2515/ CSC2515_Winter15.html Many
Contents. Preface to the Second Edition
Preface to the Second Edition v 1 Introduction 1 1.1 What Is Data Mining?....................... 4 1.2 Motivating Challenges....................... 5 1.3 The Origins of Data Mining....................
Lecture 25: Review I
Lecture 25: Review I Reading: Up to chapter 5 in ISLR. STATS 202: Data mining and analysis Jonathan Taylor 1 / 18 Unsupervised learning In unsupervised learning, all the variables are on equal standing,
Supervised Learning: K-Nearest Neighbors and Decision Trees
Supervised Learning: K-Nearest Neighbors and Decision Trees Piyush Rai CS5350/6350: Machine Learning August 25, 2011 (CS5350/6350) K-NN and DT August 25, 2011 1 / 20 Supervised Learning Given training
CS273 Midterm Exam Introduction to Machine Learning: Winter 2015 Tuesday February 10th, 2014
CS273 Midterm Eam Introduction to Machine Learning: Winter 2015 Tuesday February 10th, 2014 Your name: Your UCINetID (e.g., myname@uci.edu): Your seat (row and number): Total time is 80 minutes. READ THE
Announcements. CS 188: Artificial Intelligence Spring Classification: Feature Vectors. Classification: Weights. Learning: Binary Perceptron
CS 188: Artificial Intelligence Spring 2010 Lecture 24: Perceptrons and More! 4/20/2010 Announcements W7 due Thursday [that s your last written for the semester!] Project 5 out Thursday Contest running
Lecture 5: Decision Trees (Part II)
Lecture 5: Decision Trees (Part II) Dealing with noise in the data Overfitting Pruning Dealing with missing attribute values Dealing with attributes with multiple values Integrating costs into node choice
15-780: Graduate Artificial Intelligence. Decision trees
15-780: Graduate Artificial Intelligence Decision trees Graphical models So far we discussed models that capture joint probability distributions These have many uses, and can also be used to determine
Decision Trees Dr. G. Bharadwaja Kumar VIT Chennai
Decision Trees Decision Tree Decision Trees (DTs) are a nonparametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target
9/6/14. Our first learning algorithm. Comp 135 Introduction to Machine Learning and Data Mining. knn Algorithm. knn Algorithm (simple form)
Comp 135 Introduction to Machine Learning and Data Mining Our first learning algorithm How would you classify the next example? Fall 2014 Professor: Roni Khardon Computer Science Tufts University o o o
Nearest neighbor classification DSE 220
Nearest neighbor classification DSE 220 Decision Trees Target variable Label Dependent variable Output space Person ID Age Gender Income Balance Mortgag e payment 123213 32 F 25000 32000 Y 17824 49 M 12000-3000
The Curse of Dimensionality
The Curse of Dimensionality ACAS 2002 p1/66 Curse of Dimensionality The basic idea of the curse of dimensionality is that high dimensional data is difficult to work with for several reasons: Adding more
A Systematic Overview of Data Mining Algorithms. Sargur Srihari University at Buffalo The State University of New York
A Systematic Overview of Data Mining Algorithms Sargur Srihari University at Buffalo The State University of New York 1 Topics Data Mining Algorithm Definition Example of CART Classification Iris, Wine
Classification and Regression Trees
Classification and Regression Trees David S. Rosenberg New York University April 3, 2018 David S. Rosenberg (New York University) DS-GA 1003 / CSCI-GA 2567 April 3, 2018 1 / 51 Contents 1 Trees 2 Regression
Uninformed Search Methods. Informed Search Methods. Midterm Exam 3/13/18. Thursday, March 15, 7:30 9:30 p.m. room 125 Ag Hall
Midterm Exam Thursday, March 15, 7:30 9:30 p.m. room 125 Ag Hall Covers topics through Decision Trees and Random Forests (does not include constraint satisfaction) Closed book 8.5 x 11 sheet with notes
Topics in Machine Learning
Topics in Machine Learning Gilad Lerman School of Mathematics University of Minnesota Text/slides stolen from G. James, D. Witten, T. Hastie, R. Tibshirani and A. Ng Machine Learning - Motivation Arthur
Machine Learning: Algorithms and Applications Mockup Examination
Machine Learning: Algorithms and Applications Mockup Examination 14 May 2012 FIRST NAME STUDENT NUMBER LAST NAME SIGNATURE Instructions for students Write First Name, Last Name, Student Number and Signature
CSE 573: Artificial Intelligence Autumn 2010
CSE 573: Artificial Intelligence Autumn 2010 Lecture 16: Machine Learning Topics 12/7/2010 Luke Zettlemoyer Most slides over the course adapted from Dan Klein. 1 Announcements Syllabus revised Machine
Data Mining. Lecture 03: Nearest Neighbor Learning
Data Mining Lecture 03: Nearest Neighbor Learning Theses slides are based on the slides by Tan, Steinbach and Kumar (textbook authors) Prof. R. Mooney (UT Austin) Prof E. Keogh (UCR), Prof. F. Provost
Task Description: Finding Similar Documents. Document Retrieval. Case Study 2: Document Retrieval
Case Study 2: Document Retrieval Task Description: Finding Similar Documents Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade April 11, 2017 Sham Kakade 2017 1 Document
Ensemble Learning. Another approach is to leverage the algorithms we have via ensemble methods
Ensemble Learning Ensemble Learning So far we have seen learning algorithms that take a training set and output a classifier What if we want more accuracy than current algorithms afford? Develop new learning
SOCIAL MEDIA MINING. Data Mining Essentials
SOCIAL MEDIA MINING Data Mining Essentials Dear instructors/users of these slides: Please feel free to include these slides in your own material, or modify them as you see fit. If you decide to incorporate
Practice EXAM: SPRING 2012 CS 6375 INSTRUCTOR: VIBHAV GOGATE
Practice EXAM: SPRING 0 CS 6375 INSTRUCTOR: VIBHAV GOGATE The exam is closed book. You are allowed four pages of double sided cheat sheets. Answer the questions in the spaces provided on the question sheets.
7. Nearest neighbors. Learning objectives. Foundations of Machine Learning École Centrale Paris Fall 2015
Foundations of Machine Learning École Centrale Paris Fall 2015 7. Nearest neighbors Chloé-Agathe Azencott Centre for Computational Biology, Mines ParisTech chloe agathe.azencott@mines paristech.fr Learning
6.034 Quiz 2, Spring 2005
6.034 Quiz 2, Spring 2005 Open Book, Open Notes Name: Problem 1 (13 pts) 2 (8 pts) 3 (7 pts) 4 (9 pts) 5 (8 pts) 6 (16 pts) 7 (15 pts) 8 (12 pts) 9 (12 pts) Total (100 pts) Score 1 1 Decision Trees (13
Fall 09, Homework 5
5-38 Fall 09, Homework 5 Due: Wednesday, November 8th, beginning of the class You can work in a group of up to two people. This group does not need to be the same group as for the other homeworks. You
Big Data Methods. Chapter 5: Machine learning. Big Data Methods, Chapter 5, Slide 1
Big Data Methods Chapter 5: Machine learning Big Data Methods, Chapter 5, Slide 1 5.1 Introduction to machine learning What is machine learning? Concerned with the study and development of algorithms that
Fall 2018 CSE 482 Big Data Analysis: Exam 1 Total: 36 (+3 bonus points)
Fall 2018 CSE 482 Big Data Analysis: Exam 1 Total: 36 (+3 bonus points) Name: This exam is open book and notes. You can use a calculator but no laptops, cell phones, nor other electronic devices are allowed.
Learning Rules. CS 391L: Machine Learning: Rule Learning. Raymond J. Mooney. Rule Learning Approaches. Decision-Trees to Rules. Sequential Covering
Learning Rules CS 9L: Machine Learning: Rule Learning Raymond J. Mooney University of Texas at Austin If-then rules in logic are a standard representation of knowledge that have proven useful in expert-systems
Machine Learning. Topic 5: Linear Discriminants. Bryan Pardo, EECS 349 Machine Learning, 2013
Machine Learning Topic 5: Linear Discriminants Bryan Pardo, EECS 349 Machine Learning, 2013 Thanks to Mark Cartwright for his extensive contributions to these slides Thanks to Alpaydin, Bishop, and Duda/Hart/Stork
Feature Extractors. CS 188: Artificial Intelligence Fall Some (Vague) Biology. The Binary Perceptron. Binary Decision Rule.
CS 188: Artificial Intelligence Fall 2008 Lecture 24: Perceptrons II 11/24/2008 Dan Klein UC Berkeley Feature Extractors A feature extractor maps inputs to feature vectors Dear Sir. First, I must solicit
ADVANCED CLASSIFICATION TECHNIQUES
Admin ML lab next Monday Project proposals: Sunday at 11:59pm ADVANCED CLASSIFICATION TECHNIQUES David Kauchak CS 159 Fall 2014 Project proposal presentations Machine Learning: A Geometric View 1 Apples
Model Selection Introduction to Machine Learning. Matt Gormley Lecture 4 January 29, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Model Selection Matt Gormley Lecture 4 January 29, 2018 1 Q&A Q: How do we deal
CSE 158. Web Mining and Recommender Systems. Midterm recap
CSE 158 Web Mining and Recommender Systems Midterm recap Midterm on Wednesday! 5:10 pm 6:10 pm Closed book but I ll provide a similar level of basic info as in the last page of previous midterms CSE 158
Text classification II CE-324: Modern Information Retrieval Sharif University of Technology
Text classification II CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2015 Some slides have been adapted from: Profs. Manning, Nayak & Raghavan (CS-276, Stanford)
k-nearest Neighbors + Model Selection
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University k-nearest Neighbors + Model Selection Matt Gormley Lecture 5 Jan. 30, 2019 1 Reminders
Machine Learning - Lecture 2: Nearest-neighbour methods
Machine Learning - Lecture 2: Nearest-neighbour methods Chris Thornton January 8, 22 Brighton pier Switchback Data Data Vertical axis is age; horizontal axis is alchohol consumption per week. Data Vertical
Data Mining Lecture 8: Decision Trees
Data Mining Lecture 8: Decision Trees Jo Houghton ECS Southampton March 8, 2019 1 / 30 Decision Trees - Introduction A decision tree is like a flow chart. E. g. I need to buy a new car Can I afford it?
INTRODUCTION TO MACHINE LEARNING. Measuring model performance or error
INTRODUCTION TO MACHINE LEARNING Measuring model performance or error Is our model any good? Context of task Accuracy Computation time Interpretability 3 types of tasks Classification Regression Clustering
Classifiers and Detection. D.A. Forsyth
Classifiers and Detection D.A. Forsyth Classifiers Take a measurement x, predict a bit (yes/no; 1/-1; 1/0; etc) Detection with a classifier Search all windows at relevant scales Prepare features Classify
K Nearest Neighbor Wrap Up K- Means Clustering. Slides adapted from Prof. Carpuat
K Nearest Neighbor Wrap Up K- Means Clustering Slides adapted from Prof. Carpuat K Nearest Neighbor classification Classification is based on Test instance with Training Data K: number of neighbors that
Midterm Examination CS540-2: Introduction to Artificial Intelligence
Midterm Examination CS540-2: Introduction to Artificial Intelligence March 15, 2018 LAST NAME: FIRST NAME: Problem Score Max Score 1 12 2 13 3 9 4 11 5 8 6 13 7 9 8 16 9 9 Total 100 Question 1. [12] Search
Geometric data structures:
Geometric data structures: Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade Sham Kakade 2017 1 Announcements: HW3 posted Today: Review: LSH for Euclidean distance Other
Supervised Learning. CS 586 Machine Learning. Prepared by Jugal Kalita. With help from Alpaydin s Introduction to Machine Learning, Chapter 2.
Supervised Learning CS 586 Machine Learning Prepared by Jugal Kalita With help from Alpaydin s Introduction to Machine Learning, Chapter 2. Topics What is classification? Hypothesis classes and learning
5 Learning hypothesis classes (16 points)
5 Learning hypothesis classes (16 points) Consider a classification problem with two real valued inputs. For each of the following algorithms, specify all of the separators below that it could have generated
Decision Tree (Continued) and K-Nearest Neighbour. Dr. Xiaowei Huang
Decision Tree (Continued) and K-Nearest Neighbour Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Recap basic knowledge Decision tree learning How to split Identify the best feature to
06: Logistic Regression
06_Logistic_Regression 06: Logistic Regression Previous Next Index Classification Where y is a discrete value Develop the logistic regression algorithm to determine what class a new input should fall into
COMP 465: Data Mining Classification Basics
Supervised vs. Unsupervised Learning COMP 465: Data Mining Classification Basics Slides Adapted From : Jiawei Han, Micheline Kamber & Jian Pei Data Mining: Concepts and Techniques, 3 rd ed. Supervised
Data Mining. 3.5 Lazy Learners (Instance-Based Learners) Fall Instructor: Dr. Masoud Yaghini. Lazy Learners
Data Mining 3.5 (Instance-Based Learners) Fall 2008 Instructor: Dr. Masoud Yaghini Outline Introduction k-nearest-neighbor Classifiers References Introduction Introduction Lazy vs. eager learning Eager
7. Nearest neighbors. Learning objectives. Centre for Computational Biology, Mines ParisTech
Foundations of Machine Learning CentraleSupélec Paris Fall 2016 7. Nearest neighbors Chloé-Agathe Azencot Centre for Computational Biology, Mines ParisTech chloe-agathe.azencott@mines-paristech.fr Learning
Lecture 2 :: Decision Trees Learning
Lecture 2 :: Decision Trees Learning 1 / 62 Designing a learning system What to learn? Learning setting. Learning mechanism. Evaluation. 2 / 62 Prediction task Figure 1: Prediction task :: Supervised learning
Feature Extractors. CS 188: Artificial Intelligence Fall Nearest-Neighbor Classification. The Perceptron Update Rule.
CS 188: Artificial Intelligence Fall 2007 Lecture 26: Kernels 11/29/2007 Dan Klein UC Berkeley Feature Extractors A feature extractor maps inputs to feature vectors Dear Sir. First, I must solicit your
COSC160: Detection and Classification. Jeremy Bolton, PhD Assistant Teaching Professor
COSC160: Detection and Classification Jeremy Bolton, PhD Assistant Teaching Professor Outline I. Problem I. Strategies II. Features for training III. Using spatial information? IV. Reducing dimensionality
CS229 Lecture notes. Raphael John Lamarre Townshend
CS229 Lecture notes Raphael John Lamarre Townshend Decision Trees We now turn our attention to decision trees, a simple yet flexible class of algorithms. We will first consider the non-linear, region-based