INTRODUCTION TO MACHINE LEARNING. Measuring model performance or error
|
|
|
- Hilary Carpenter
- 6 years ago
- Views:
Transcription
1 INTRODUCTION TO MACHINE LEARNING Measuring model performance or error
2 Is our model any good? Context of task Accuracy Computation time Interpretability 3 types of tasks Classification Regression Clustering
3 Classification Accuracy and Error System is right or wrong Accuracy goes up when Error goes down correctly classified instances Accuracy = total amount of classified instances Error = 1 - Accuracy
4 Example Squares with 2 features: small/big and solid/dotted Label: colored/not colored Binary classification problem
5 Example Truth Predicted = 3 = 5 60%
6 Example Truth Predicted 3 = = 5 60%
7 Limits of accuracy Classifying very rare heart disease Classify all as negative (not sick) Predict 99 correct (not sick) and miss 1 Accuracy: 99% Bogus you miss every positive case!
8 Confusion matrix Rows and columns contain all available labels Each cell contains frequency of instances that are classified in a certain way
9 Confusion matrix Binary classifier: positive or negative (1 or 0) Prediction P N Truth p TP FN n FP TN
10 Confusion matrix Binary classifier: positive or negative (1 or 0) Prediction True Positives Prediction: P Truth: P Truth P N p TP FN n FP TN
11 Confusion matrix Binary classifier: positive or negative (1 or 0) Prediction True Negatives Prediction: N Truth: N Truth P N p TP FN n FP TN
12 Confusion matrix Binary classifier: positive or negative (1 or 0) Prediction False Negatives Prediction: N Truth: P Truth P N p TP FN n FP TN
13 Confusion matrix Binary classifier: positive or negative (1 or 0) Prediction False Positives Prediction: P Truth: N Truth P N p TP FN n FP TN
14 Ratios in the confusion matrix Accuracy Precision Recall Prediction P N Truth p TP FN n FP TN
15 Ratios in the confusion matrix Accuracy Precision Recall Prediction Precision TP/(TP+FP) P N Truth p TP FN n FP TN
16 Ratios in the confusion matrix Accuracy Precision Recall Prediction Precision TP/(TP+FP) P N Truth p TP FN n FP TN
17 Ratios in the confusion matrix Accuracy Precision Recall Prediction Recall TP/(TP+FN) P N Truth p TP FN n FP TN
18 Ratios in the confusion matrix Accuracy Precision Recall Prediction Recall TP/(TP+FN) P N Truth p TP FN n FP TN
19 Back to the squares Prediction P N Truth Predicted Truth p 1 1 n 1 2
20 Back to the squares Prediction P N Truth Predicted Truth p 1 1 n 1 2
21 Back to the squares Prediction P N Truth Predicted Truth p 1 1 n 1 2
22 Back to the squares Prediction P N Truth Predicted Truth p 1 1 n 1 2
23 Back to the squares Prediction P N Truth Predicted Truth p 1 1 n 1 2
24 Back to the squares Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/( ) = 60% Precision: TP/(TP+FP) = 1/(1+1) = 50% Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth Predicted Truth p 1 1 n 1 2
25 Back to the squares Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/( ) = 60% Precision: TP/(TP+FP) = 1/(1+1) = 50% Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth Predicted Truth p 1 1 n 1 2
26 Back to the squares Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/( ) = 60% Precision: TP/(TP+FP) = 1/(1+1) = 50% Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth Predicted Truth p 1 1 n 1 2
27 Back to the squares Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/( ) = 60% Precision: TP/(TP+FP) = 1/(1+1) = 50% Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth Predicted Truth p 1 1 n 1 2
28 Rare heart disease Accuracy: 99/(99+1) = 99% Recall: 0/1 = 0% Precision: undefined no positive predictions Prediction P N Truth p 0 1 n 0 99
29 Regression: RMSE Root Mean Squared Error (RMSE) Mean distance between estimates and regression line X1 X2
30 Clustering No label information Need distance metric between points
31 Clustering Performance measure consists of 2 elements Similarity within each cluster Similarity between clusters
32 X1 X2 Within cluster similarity Within sum of squares (WSS) Diameter Minimize
33 X1 X2 Between cluster similarity Between cluster sum of squares (BSS) Intercluster distance Maximize
34 Dunn s index X1 X2 minimal intercluster distance maximal diameter
35 INTRODUCTION TO MACHINE LEARNING Let s practice!
36 INTRODUCTION TO MACHINE LEARNING Training set and test set
37 Machine learning - statistics Predictive power vs. descriptive power Supervised learning: model must predict unseen observations Classical statistics: model must fit data explain or describe data
38 Predictive model Training not on complete dataset training set Test set to evaluate performance of model Sets are disjoint: NO OVERLAP Model tested on unseen observations -> Generalization!
39 Split the dataset N instances (=observations): X K features: F Class labels: y f1 f2 fk y x1 x1,1 x1,2 x1,k y1 x2 x2,1 x2,2 x2,k y2 Training set xr xr,1 xr,2 xr,k yr xr+1 xr+1,1 xr+1,2 xr+1,k yr+1 xr+2 xr+2,1 xr+2,2 xr+2,k yr+2 Test set xn xn,1 xn,2 xn,k yn
40 Split the dataset N instances (=observations): X K features: F Class labels: y f1 f2 fk y x1 x1,1 x1,2 x1,k y1 x2 x2,1 x2,2 x2,k y2 Training set xr xr,1 xr,2 xr,k yr xr+1 xr+1,1 xr+1,2 xr+1,k yr+1 xr+2 xr+2,1 xr+2,2 xr+2,k yr+2 Test set xn xn,1 xn,2 xn,k yn
41 Split the dataset N instances (=observations): X K features: F Class labels: y f1 f2 fk y x1 x1,1 x1,2 x1,k y1 x2 x2,1 x2,2 x2,k y2 Training set xr xr,1 xr,2 xr,k yr xr+1 xr+1,1 xr+1,2 xr+1,k yr+1 xr+2 xr+2,1 xr+2,2 xr+2,k yr+2 Test set xn xn,1 xn,2 xn,k yn
42 Split the dataset N instances (=observations): X K features: F Class labels: y f1 f2 fk y x1 x1,1 x1,2 x1,k y1 x2 x2,1 x2,2 x2,k y2 Training set xr xr,1 xr,2 xr,k yr xr+1 xr+1,1 xr+1,2 xr+1,k yr+1 xr+2 xr+2,1 xr+2,2 xr+2,k yr+2 Test set xn xn,1 xn,2 xn,k yn
43 Split the dataset f1 f2 fk y x1 x1,1 x1,2 x1,k y1 x2 x2,1 x2,2 x2,k y2 Training set xr xr,1 xr,2 xr,k yr xr+1 xr+1,1 xr+1,2 xr+1,k yr+1 xr+2 xr+2,1 xr+2,2 xr+2,k yr+2 Test set xn xn,1 xn,2 xn,k yn
44 Split the dataset f1 f2 fk y x1 x1,1 x1,2 x1,k y1 x2 x2,1 x2,2 x2,k y2 Training set xr xr,1 xr,2 xr,k yr xr+1 xr+1,1 xr+1,2 xr+1,k yr+1 xr+2 xr+2,1 xr+2,2 xr+2,k yr+2 Test set xn xn,1 xn,2 xn,k yn Use to predict y: ŷ
45 Split the dataset f1 f2 fk y x1 x1,1 x1,2 x1,k y1 x2 x2,1 x2,2 x2,k y2 Training set xr xr,1 xr,2 xr,k yr xr+1 xr+1,1 xr+1,2 xr+1,k yr+1 xr+2 xr+2,1 xr+2,2 xr+2,k yr+2 Test set xn xn,1 xn,2 xn,k yn Use to predict y: ŷ real y compare them
46 When to use training/test set? Supervised learning Not for unsupervised (clustering) Data not labeled
47 Predictive power of model Train model Use model Test model Performance measure Predictive power Training set Test set
48 How to split the sets? Which observations go where? Training set larger test set Typically about 3/1 Quite arbitrary Generally: more data = better model Test set not too small
49 Distribution of the sets Classification classes must have similar distributions avoid a class not being available in a set Classification & regression shuffle dataset before splitting
50 Effect of sampling Sampling can affect performance measures Add robustness to these measures: cross-validation Idea: sample multiple times, with different separations
51 Cross-validation E.g.: 4-fold cross-validation Training set Test set Training set Test set Test set Training set Test set Training set
52 Cross-validation E.g.: 4-fold cross-validation Training set Test set Training set Test set Test set Training set Test set Training set
53 Cross-validation E.g.: 4-fold cross-validation Training set Test set Training set Test set Test set Training set Test set Training set
54 Cross-validation E.g.: 4-fold cross-validation Training set Test set Training set Test set Test set Training set Test set Training set aggregate results for robust measure
55 n-fold cross-validation Fold test set over dataset n times Each test set is 1/n size of total dataset
56 INTRODUCTION TO MACHINE LEARNING Let s practice!
57 INTRODUCTION TO MACHINE LEARNING Bias and Variance
58 What you ve learned? Accuracy and other performance measures Training and test set
59 Knitting it all together Effect of splitting dataset (train/test) on accuracy Over- and underfitting
60 Introducing BIAS VARIANCE
61 Bias and Variance Main goal of supervised learning: prediction Prediction error ~ reducible + irreducible error
62 Irreducible - reducible error Irreducible: noise don t minimize Reducible: error due to unfit model minimize Reducible error is split into bias and variance
63 Bias Error due to bias: wrong assumptions Difference predictions and truth using models trained by specific learning algorithm
64 Example Introduction to Machine Learning
65 Example Quadratic data
66 Example Quadratic data Assumption: data is linear use linear regression
67 Example Quadratic data Assumption: data is linear use linear regression Error due to bias is high: more restrictions on model
68 Bias Complexity of model More restrictions lead to high bias
69 Variance Error due to variance: error due to the sampling of the training set Model with high variance fits training set closely
70 Example Quadratic data Few restrictions: fit polynomial perfectly through training set If you change training set, model will change completely high variance : generalizes bad to test set
71 Bias-variance tradeoff BIAS VARIANCE low bias - high variance low variance - high bias
72 Overfitting Accuracy will depend on dataset split (train/test) High variance will heavily depend on split Overfitting = model fits training set a lot better than test set Too specific
73 Underfitting Restricting your model too much High bias Too general
74 Example - spam or not? Truth no A lot of capital letters? no spam capital letters s training set exclamation marks yes A lot of exclamation marks? no no spam yes exception with spam 50 capital letters 30 exclamation marks is no spam
75 Example - spam or not? Overfit no A lot of capital letters? no spam s training set capital letters exclamation marks yes A lot of exclamation marks? no no spam yes no 50 capital letters? spam exception with 50 capital letters 30 exclamation marks is no spam yes 30 exclamation marks? spam yes no spam no too specific!
76 Example - spam or not? s training set capital letters exclamation marks Underfit More than 10 capital letters? yes spam no no spam too general!
77 INTRODUCTION TO MACHINE LEARNING Let s practice!
Evaluating Classifiers
Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts
Evaluating Classifiers
Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts
Weka ( )
Weka ( http://www.cs.waikato.ac.nz/ml/weka/ ) The phases in which classifier s design can be divided are reflected in WEKA s Explorer structure: Data pre-processing (filtering) and representation Supervised
CS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING Clustering Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu November 7, 2017 Learnt Clustering Methods Vector Data Set Data Sequence Data Text
CS4491/CS 7265 BIG DATA ANALYTICS
CS4491/CS 7265 BIG DATA ANALYTICS EVALUATION * Some contents are adapted from Dr. Hung Huang and Dr. Chengkai Li at UT Arlington Dr. Mingon Kang Computer Science, Kennesaw State University Evaluation for
CSE Data Mining Concepts and Techniques STATISTICAL METHODS (REGRESSION) Professor- Anita Wasilewska. Team 13
CSE 634 - Data Mining Concepts and Techniques STATISTICAL METHODS Professor- Anita Wasilewska (REGRESSION) Team 13 Contents Linear Regression Logistic Regression Bias and Variance in Regression Model Fit
Large Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Machine Learning Basics - 1 U Kang Seoul National University U Kang 1 In This Lecture Overview of Machine Learning Capacity, overfitting, and underfitting
Evaluation Measures. Sebastian Pölsterl. April 28, Computer Aided Medical Procedures Technische Universität München
Evaluation Measures Sebastian Pölsterl Computer Aided Medical Procedures Technische Universität München April 28, 2015 Outline 1 Classification 1. Confusion Matrix 2. Receiver operating characteristics
Artificial Intelligence. Programming Styles
Artificial Intelligence Intro to Machine Learning Programming Styles Standard CS: Explicitly program computer to do something Early AI: Derive a problem description (state) and use general algorithms to
Network Traffic Measurements and Analysis
DEIB - Politecnico di Milano Fall, 2017 Sources Hastie, Tibshirani, Friedman: The Elements of Statistical Learning James, Witten, Hastie, Tibshirani: An Introduction to Statistical Learning Andrew Ng:
CS6375: Machine Learning Gautam Kunapuli. Mid-Term Review
Gautam Kunapuli Machine Learning Data is identically and independently distributed Goal is to learn a function that maps to Data is generated using an unknown function Learn a hypothesis that minimizes
Introduction to Automated Text Analysis. bit.ly/poir599
Introduction to Automated Text Analysis Pablo Barberá School of International Relations University of Southern California pablobarbera.com Lecture materials: bit.ly/poir599 Today 1. Solutions for last
Data Mining: Classifier Evaluation. CSCI-B490 Seminar in Computer Science (Data Mining)
Data Mining: Classifier Evaluation CSCI-B490 Seminar in Computer Science (Data Mining) Predictor Evaluation 1. Question: how good is our algorithm? how will we estimate its performance? 2. Question: what
Classification. Instructor: Wei Ding
Classification Part II Instructor: Wei Ding Tan,Steinbach, Kumar Introduction to Data Mining 4/18/004 1 Practical Issues of Classification Underfitting and Overfitting Missing Values Costs of Classification
Distributed Logistic Model Trees, Stratio Intelligence Mateo Álvarez and Antonio
Distributed Logistic Model Trees, Stratio Intelligence Mateo Álvarez and Antonio Soriano @StratioBD Aerospace Engineer, MSc in Propulsion Systems (UPM), Master in Data Science (URJC). Working as data scientist
2. On classification and related tasks
2. On classification and related tasks In this part of the course we take a concise bird s-eye view of different central tasks and concepts involved in machine learning and classification particularly.
Evaluation Metrics. (Classifiers) CS229 Section Anand Avati
Evaluation Metrics (Classifiers) CS Section Anand Avati Topics Why? Binary classifiers Metrics Rank view Thresholding Confusion Matrix Point metrics: Accuracy, Precision, Recall / Sensitivity, Specificity,
Stages of (Batch) Machine Learning
Evalua&on Stages of (Batch) Machine Learning Given: labeled training data X, Y = {hx i,y i i} n i=1 Assumes each x i D(X ) with y i = f target (x i ) Train the model: model ß classifier.train(x, Y ) x
Evaluating Classifiers
Evaluating Classifiers Charles Elkan elkan@cs.ucsd.edu January 18, 2011 In a real-world application of supervised learning, we have a training set of examples with labels, and a test set of examples with
Contents Machine Learning concepts 4 Learning Algorithm 4 Predictive Model (Model) 4 Model, Classification 4 Model, Regression 4 Representation
Contents Machine Learning concepts 4 Learning Algorithm 4 Predictive Model (Model) 4 Model, Classification 4 Model, Regression 4 Representation Learning 4 Supervised Learning 4 Unsupervised Learning 4
Machine Learning and Bioinformatics 機器學習與生物資訊學
Molecular Biomedical Informatics 分子生醫資訊實驗室 機器學習與生物資訊學 Machine Learning & Bioinformatics 1 Evaluation The key to success 2 Three datasets of which the answers must be known 3 Note on parameter tuning It
INTRODUCTION TO DATA MINING. Daniel Rodríguez, University of Alcalá
INTRODUCTION TO DATA MINING Daniel Rodríguez, University of Alcalá Outline Knowledge Discovery in Datasets Model Representation Types of models Supervised Unsupervised Evaluation (Acknowledgement: Jesús
Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates?
Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to
DATA MINING INTRODUCTION TO CLASSIFICATION USING LINEAR CLASSIFIERS
DATA MINING INTRODUCTION TO CLASSIFICATION USING LINEAR CLASSIFIERS 1 Classification: Definition Given a collection of records (training set ) Each record contains a set of attributes and a class attribute
CISC 4631 Data Mining
CISC 4631 Data Mining Lecture 03: Introduction to classification Linear classifier Theses slides are based on the slides by Tan, Steinbach and Kumar (textbook authors) Eamonn Koegh (UC Riverside) 1 Classification:
Classification Part 4
Classification Part 4 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville Model Evaluation Metrics for Performance Evaluation How to evaluate
MIT 801. Machine Learning I. [Presented by Anna Bosman] 16 February 2018
![MIT 801. Machine Learning I. [Presented by Anna Bosman] 16 February 2018](/thumbs/77/76295965.jpg "MIT 801. Machine Learning I. [Presented by Anna Bosman] 16 February 2018")
MIT 801 [Presented by Anna Bosman] 16 February 2018 Machine Learning What is machine learning? Artificial Intelligence? Yes as we know it. What is intelligence? The ability to acquire and apply knowledge
INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering
INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering Erik Velldal University of Oslo Sept. 18, 2012 Topics for today 2 Classification Recap Evaluating classifiers Accuracy, precision,
Lecture 25: Review I
Lecture 25: Review I Reading: Up to chapter 5 in ISLR. STATS 202: Data mining and analysis Jonathan Taylor 1 / 18 Unsupervised learning In unsupervised learning, all the variables are on equal standing,
CS249: ADVANCED DATA MINING
CS249: ADVANCED DATA MINING Classification Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu April 24, 2017 Homework 2 out Announcements Due May 3 rd (11:59pm) Course project proposal
ECLT 5810 Evaluation of Classification Quality
ECLT 5810 Evaluation of Classification Quality Reference: Data Mining Practical Machine Learning Tools and Techniques, by I. Witten, E. Frank, and M. Hall, Morgan Kaufmann Testing and Error Error rate:
Application of Support Vector Machine Algorithm in Spam Filtering
Application of Support Vector Machine Algorithm in E-Mail Spam Filtering Julia Bluszcz, Daria Fitisova, Alexander Hamann, Alexey Trifonov, Advisor: Patrick Jähnichen Abstract The problem of spam classification
CCRMA MIR Workshop 2014 Evaluating Information Retrieval Systems. Leigh M. Smith Humtap Inc.
CCRMA MIR Workshop 2014 Evaluating Information Retrieval Systems Leigh M. Smith Humtap Inc. leigh@humtap.com Basic system overview Segmentation (Frames, Onsets, Beats, Bars, Chord Changes, etc) Feature
Pattern recognition (4)
Pattern recognition (4) 1 Things we have discussed until now Statistical pattern recognition Building simple classifiers Supervised classification Minimum distance classifier Bayesian classifier (1D and
Evaluating Machine-Learning Methods. Goals for the lecture
Evaluating Machine-Learning Methods Mark Craven and David Page Computer Sciences 760 Spring 2018 www.biostat.wisc.edu/~craven/cs760/ Some of the slides in these lectures have been adapted/borrowed from
Business Club. Decision Trees
Business Club Decision Trees Business Club Analytics Team December 2017 Index 1. Motivation- A Case Study 2. The Trees a. What is a decision tree b. Representation 3. Regression v/s Classification 4. Building
CSE 158. Web Mining and Recommender Systems. Midterm recap
CSE 158 Web Mining and Recommender Systems Midterm recap Midterm on Wednesday! 5:10 pm 6:10 pm Closed book but I ll provide a similar level of basic info as in the last page of previous midterms CSE 158
Predictive Analysis: Evaluation and Experimentation. Heejun Kim
Predictive Analysis: Evaluation and Experimentation Heejun Kim June 19, 2018 Evaluation and Experimentation Evaluation Metrics Cross-Validation Significance Tests Evaluation Predictive analysis: training
Model Complexity and Generalization
HT2015: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Generalization Learning Curves Underfit Generalization
Model s Performance Measures
Model s Performance Measures Evaluating the performance of a classifier Section 4.5 of course book. Taking into account misclassification costs Class imbalance problem Section 5.7 of course book. TNM033:
Machine Learning nearest neighbors classification. Luigi Cerulo Department of Science and Technology University of Sannio
Machine Learning nearest neighbors classification Luigi Cerulo Department of Science and Technology University of Sannio Nearest Neighbors Classification The idea is based on the hypothesis that things
Lecture on Modeling Tools for Clustering & Regression
Lecture on Modeling Tools for Clustering & Regression CS 590.21 Analysis and Modeling of Brain Networks Department of Computer Science University of Crete Data Clustering Overview Organizing data into
Classification of Subject Motion for Improved Reconstruction of Dynamic Magnetic Resonance Imaging
1 CS 9 Final Project Classification of Subject Motion for Improved Reconstruction of Dynamic Magnetic Resonance Imaging Feiyu Chen Department of Electrical Engineering ABSTRACT Subject motion is a significant
Supervised vs unsupervised clustering
Classification Supervised vs unsupervised clustering Cluster analysis: Classes are not known a- priori. Classification: Classes are defined a-priori Sometimes called supervised clustering Extract useful
I211: Information infrastructure II
Data Mining: Classifier Evaluation I211: Information infrastructure II 3-nearest neighbor labeled data find class labels for the 4 data points 1 0 0 6 0 0 0 5 17 1.7 1 1 4 1 7.1 1 1 1 0.4 1 2 1 3.0 0 0.1
Evaluation. Evaluate what? For really large amounts of data... A: Use a validation set.
Evaluate what? Evaluation Charles Sutton Data Mining and Exploration Spring 2012 Do you want to evaluate a classifier or a learning algorithm? Do you want to predict accuracy or predict which one is better?
Partitioning Data. IRDS: Evaluation, Debugging, and Diagnostics. Cross-Validation. Cross-Validation for parameter tuning
Partitioning Data IRDS: Evaluation, Debugging, and Diagnostics Charles Sutton University of Edinburgh Training Validation Test Training : Running learning algorithms Validation : Tuning parameters of learning
ML4Bio Lecture #1: Introduc3on. February 24 th, 2016 Quaid Morris
ML4Bio Lecture #1: Introduc3on February 24 th, 216 Quaid Morris Course goals Prac3cal introduc3on to ML Having a basic grounding in the terminology and important concepts in ML; to permit self- study,
CISC 4631 Data Mining
CISC 4631 Data Mining Lecture 05: Overfitting Evaluation: accuracy, precision, recall, ROC Theses slides are based on the slides by Tan, Steinbach and Kumar (textbook authors) Eamonn Koegh (UC Riverside)
Lecture 6 K- Nearest Neighbors(KNN) And Predictive Accuracy
Lecture 6 K- Nearest Neighbors(KNN) And Predictive Accuracy Machine Learning Dr.Ammar Mohammed Nearest Neighbors Set of Stored Cases Atr1... AtrN Class A Store the training samples Use training samples
Data Mining Concepts & Techniques
Data Mining Concepts & Techniques Lecture No. 03 Data Processing, Data Mining Naeem Ahmed Email: naeemmahoto@gmail.com Department of Software Engineering Mehran Univeristy of Engineering and Technology
DATA MINING LECTURE 9. Classification Decision Trees Evaluation
DATA MINING LECTURE 9 Classification Decision Trees Evaluation 10 10 Illustrating Classification Task Tid Attrib1 Attrib2 Attrib3 Class 1 Yes Large 125K No 2 No Medium 100K No 3 No Small 70K No 4 Yes Medium
List of Exercises: Data Mining 1 December 12th, 2015
List of Exercises: Data Mining 1 December 12th, 2015 1. We trained a model on a two-class balanced dataset using five-fold cross validation. One person calculated the performance of the classifier by measuring
Global Probability of Boundary
Global Probability of Boundary Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues Martin, Fowlkes, Malik Using Contours to Detect and Localize Junctions in Natural
Using Machine Learning to Optimize Storage Systems
Using Machine Learning to Optimize Storage Systems Dr. Kiran Gunnam 1 Outline 1. Overview 2. Building Flash Models using Logistic Regression. 3. Storage Object classification 4. Storage Allocation recommendation
Evaluating Machine Learning Methods: Part 1
Evaluating Machine Learning Methods: Part 1 CS 760@UW-Madison Goals for the lecture you should understand the following concepts bias of an estimator learning curves stratified sampling cross validation
Topics in Machine Learning
Topics in Machine Learning Gilad Lerman School of Mathematics University of Minnesota Text/slides stolen from G. James, D. Witten, T. Hastie, R. Tibshirani and A. Ng Machine Learning - Motivation Arthur
Big Data Methods. Chapter 5: Machine learning. Big Data Methods, Chapter 5, Slide 1
Big Data Methods Chapter 5: Machine learning Big Data Methods, Chapter 5, Slide 1 5.1 Introduction to machine learning What is machine learning? Concerned with the study and development of algorithms that
Classification: Feature Vectors
Classification: Feature Vectors Hello, Do you want free printr cartriges? Why pay more when you can get them ABSOLUTELY FREE! Just # free YOUR_NAME MISSPELLED FROM_FRIEND... : : : : 2 0 2 0 PIXEL 7,12
Machine Learning for. Artem Lind & Aleskandr Tkachenko
Machine Learning for Object Recognition Artem Lind & Aleskandr Tkachenko Outline Problem overview Classification demo Examples of learning algorithms Probabilistic modeling Bayes classifier Maximum margin
An Intelligent Clustering Algorithm for High Dimensional and Highly Overlapped Photo-Thermal Infrared Imaging Data
An Intelligent Clustering Algorithm for High Dimensional and Highly Overlapped Photo-Thermal Infrared Imaging Data Nian Zhang and Lara Thompson Department of Electrical and Computer Engineering, University
Advanced Video Content Analysis and Video Compression (5LSH0), Module 8B
Advanced Video Content Analysis and Video Compression (5LSH0), Module 8B 1 Supervised learning Catogarized / labeled data Objects in a picture: chair, desk, person, 2 Classification Fons van der Sommen
K- Nearest Neighbors(KNN) And Predictive Accuracy
Contact: mailto: Ammar@cu.edu.eg Drammarcu@gmail.com K- Nearest Neighbors(KNN) And Predictive Accuracy Dr. Ammar Mohammed Associate Professor of Computer Science ISSR, Cairo University PhD of CS ( Uni.
CS 584 Data Mining. Classification 3
CS 584 Data Mining Classification 3 Today Model evaluation & related concepts Additional classifiers Naïve Bayes classifier Support Vector Machine Ensemble methods 2 Model Evaluation Metrics for Performance
1 Machine Learning System Design
Machine Learning System Design Prioritizing what to work on: Spam classification example Say you want to build a spam classifier Spam messages often have misspelled words We ll have a labeled training
Leveling Up as a Data Scientist. ds/2014/10/level-up-ds.jpg
Model Optimization Leveling Up as a Data Scientist http://shorelinechurch.org/wp-content/uploa ds/2014/10/level-up-ds.jpg Bias and Variance Error = (expected loss of accuracy) 2 + flexibility of model
COMPUTATIONAL INTELLIGENCE SEW (INTRODUCTION TO MACHINE LEARNING) SS18. Lecture 6: k-nn Cross-validation Regularization
COMPUTATIONAL INTELLIGENCE SEW (INTRODUCTION TO MACHINE LEARNING) SS18 Lecture 6: k-nn Cross-validation Regularization LEARNING METHODS Lazy vs eager learning Eager learning generalizes training data before
Classification and K-Nearest Neighbors
Classification and K-Nearest Neighbors Administrivia o Reminder: Homework 1 is due by 5pm Friday on Moodle o Reading Quiz associated with today s lecture. Due before class Wednesday. NOTETAKER 2 Regression
Naïve Bayes Classification. Material borrowed from Jonathan Huang and I. H. Witten s and E. Frank s Data Mining and Jeremy Wyatt and others
Naïve Bayes Classification Material borrowed from Jonathan Huang and I. H. Witten s and E. Frank s Data Mining and Jeremy Wyatt and others Things We d Like to Do Spam Classification Given an email, predict
Machine Learning (CSE 446): Concepts & the i.i.d. Supervised Learning Paradigm
Machine Learning (CSE 446): Concepts & the i.i.d. Supervised Learning Paradigm Sham M Kakade c 2018 University of Washington cse446-staff@cs.washington.edu 1 / 17 Review 1 / 17 Decision Tree: Making a
FMA901F: Machine Learning Lecture 3: Linear Models for Regression. Cristian Sminchisescu
FMA901F: Machine Learning Lecture 3: Linear Models for Regression Cristian Sminchisescu Machine Learning: Frequentist vs. Bayesian In the frequentist setting, we seek a fixed parameter (vector), with value(s)
Machine Learning Techniques for Data Mining
Machine Learning Techniques for Data Mining Eibe Frank University of Waikato New Zealand 10/25/2000 1 PART V Credibility: Evaluating what s been learned 10/25/2000 2 Evaluation: the key to success How
Machine learning in fmri
Machine learning in fmri Validation Alexandre Savio, Maite Termenón, Manuel Graña 1 Computational Intelligence Group, University of the Basque Country December, 2010 1/18 Outline 1 Motivation The validation
CS 229 Midterm Review
CS 229 Midterm Review Course Staff Fall 2018 11/2/2018 Outline Today: SVMs Kernels Tree Ensembles EM Algorithm / Mixture Models [ Focus on building intuition, less so on solving specific problems. Ask
K-fold cross validation in the Tidyverse Stephanie J. Spielman 11/7/2017
K-fold cross validation in the Tidyverse Stephanie J. Spielman 11/7/2017 Requirements This demo requires several packages: tidyverse (dplyr, tidyr, tibble, ggplot2) modelr broom proc Background K-fold
Slides for Data Mining by I. H. Witten and E. Frank
Slides for Data Mining by I. H. Witten and E. Frank 7 Engineering the input and output Attribute selection Scheme-independent, scheme-specific Attribute discretization Unsupervised, supervised, error-
CSE 573: Artificial Intelligence Autumn 2010
CSE 573: Artificial Intelligence Autumn 2010 Lecture 16: Machine Learning Topics 12/7/2010 Luke Zettlemoyer Most slides over the course adapted from Dan Klein. 1 Announcements Syllabus revised Machine
Machine Learning in Python. Rohith Mohan GradQuant Spring 2018
Machine Learning in Python Rohith Mohan GradQuant Spring 2018 What is Machine Learning? https://twitter.com/myusuf3/status/995425049170489344 Traditional Programming Data Computer Program Output Getting
DATA MINING LECTURE 9. Classification Basic Concepts Decision Trees Evaluation
DATA MINING LECTURE 9 Classification Basic Concepts Decision Trees Evaluation What is a hipster? Examples of hipster look A hipster is defined by facial hair Hipster or Hippie? Facial hair alone is not
CS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING 08: Classification Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu October 24, 2017 Learnt Prediction and Classification Methods Vector Data
Feature Extractors. CS 188: Artificial Intelligence Fall Some (Vague) Biology. The Binary Perceptron. Binary Decision Rule.
CS 188: Artificial Intelligence Fall 2008 Lecture 24: Perceptrons II 11/24/2008 Dan Klein UC Berkeley Feature Extractors A feature extractor maps inputs to feature vectors Dear Sir. First, I must solicit
CSC411/2515 Tutorial: K-NN and Decision Tree
CSC411/2515 Tutorial: K-NN and Decision Tree Mengye Ren csc{411,2515}ta@cs.toronto.edu September 25, 2016 Cross-validation K-nearest-neighbours Decision Trees Review: Motivation for Validation Framework:
Knowledge Discovery and Data Mining
Knowledge Discovery and Data Mining Lecture 10 - Classification trees Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey
SOFTWARE DEFECT PREDICTION USING IMPROVED SUPPORT VECTOR MACHINE CLASSIFIER
International Journal of Mechanical Engineering and Technology (IJMET) Volume 7, Issue 5, September October 2016, pp.417 421, Article ID: IJMET_07_05_041 Available online at http://www.iaeme.com/ijmet/issues.asp?jtype=ijmet&vtype=7&itype=5
Tree-based methods for classification and regression
Tree-based methods for classification and regression Ryan Tibshirani Data Mining: 36-462/36-662 April 11 2013 Optional reading: ISL 8.1, ESL 9.2 1 Tree-based methods Tree-based based methods for predicting
Cross- Valida+on & ROC curve. Anna Helena Reali Costa PCS 5024
Cross- Valida+on & ROC curve Anna Helena Reali Costa PCS 5024 Resampling Methods Involve repeatedly drawing samples from a training set and refibng a model on each sample. Used in model assessment (evalua+ng
DATA MINING OVERFITTING AND EVALUATION
DATA MINING OVERFITTING AND EVALUATION 1 Overfitting Will cover mechanisms for preventing overfitting in decision trees But some of the mechanisms and concepts will apply to other algorithms 2 Occam s
Ester Bernadó-Mansilla. Research Group in Intelligent Systems Enginyeria i Arquitectura La Salle Universitat Ramon Llull Barcelona, Spain
Learning Classifier Systems for Class Imbalance Problems Research Group in Intelligent Systems Enginyeria i Arquitectura La Salle Universitat Ramon Llull Barcelona, Spain Aim Enhance the applicability
MIT Samberg Center Cambridge, MA, USA. May 30 th June 2 nd, by C. Rea, R.S. Granetz MIT Plasma Science and Fusion Center, Cambridge, MA, USA
Exploratory Machine Learning studies for disruption prediction on DIII-D by C. Rea, R.S. Granetz MIT Plasma Science and Fusion Center, Cambridge, MA, USA Presented at the 2 nd IAEA Technical Meeting on
Data Mining. Practical Machine Learning Tools and Techniques. Slides for Chapter 5 of Data Mining by I. H. Witten, E. Frank and M. A.
Data Mining Practical Machine Learning Tools and Techniques Slides for Chapter 5 of Data Mining by I. H. Witten, E. Frank and M. A. Hall Credibility: Evaluating what s been learned Issues: training, testing,
Probabilistic Classifiers DWML, /27
Probabilistic Classifiers DWML, 2007 1/27 Probabilistic Classifiers Conditional class probabilities Id. Savings Assets Income Credit risk 1 Medium High 75 Good 2 Low Low 50 Bad 3 High Medium 25 Bad 4 Medium
INF 4300 Classification III Anne Solberg The agenda today:
INF 4300 Classification III Anne Solberg 28.10.15 The agenda today: More on estimating classifier accuracy Curse of dimensionality and simple feature selection knn-classification K-means clustering 28.10.15
DATA MINING LECTURE 11. Classification Basic Concepts Decision Trees Evaluation Nearest-Neighbor Classifier
DATA MINING LECTURE 11 Classification Basic Concepts Decision Trees Evaluation Nearest-Neighbor Classifier What is a hipster? Examples of hipster look A hipster is defined by facial hair Hipster or Hippie?
CSC 411 Lecture 4: Ensembles I
CSC 411 Lecture 4: Ensembles I Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 04-Ensembles I 1 / 22 Overview We ve seen two particular classification algorithms:
Announcements. CS 188: Artificial Intelligence Spring Classification: Feature Vectors. Classification: Weights. Learning: Binary Perceptron
CS 188: Artificial Intelligence Spring 2010 Lecture 24: Perceptrons and More! 4/20/2010 Announcements W7 due Thursday [that s your last written for the semester!] Project 5 out Thursday Contest running
Hyperparameters and Validation Sets. Sargur N. Srihari
Hyperparameters and Validation Sets Sargur N. srihari@cedar.buffalo.edu 1 Topics in Machine Learning Basics 1. Learning Algorithms 2. Capacity, Overfitting and Underfitting 3. Hyperparameters and Validation
Machine Learning in Biology
Università degli studi di Padova Machine Learning in Biology Luca Silvestrin (Dottorando, XXIII ciclo) Supervised learning Contents Class-conditional probability density Linear and quadratic discriminant
10 Classification: Evaluation
CSE4334/5334 Data Mining 10 Classification: Evaluation Chengkai Li Department of Computer Science and Engineering University of Texas at Arlington Fall 2018 (Slides courtesy of Pang-Ning Tan, Michael Steinbach
CPSC 340: Machine Learning and Data Mining. Non-Parametric Models Fall 2016
CPSC 340: Machine Learning and Data Mining Non-Parametric Models Fall 2016 Assignment 0: Admin 1 late day to hand it in tonight, 2 late days for Wednesday. Assignment 1 is out: Due Friday of next week.
Preface to the Second Edition. Preface to the First Edition. 1 Introduction 1
Preface to the Second Edition Preface to the First Edition vii xi 1 Introduction 1 2 Overview of Supervised Learning 9 2.1 Introduction... 9 2.2 Variable Types and Terminology... 9 2.3 Two Simple Approaches