k-nn classification & Statistical Pattern Recognition
|
|
- Bethany Hines
- 6 years ago
- Views:
Transcription
1 k-nn classification & Statistical Pattern Recognition Andreas C. Kapourani (Credit: Hiroshi Shimodaira) February 27 k-nn classification In classification, the data consist of a training set and a test set. The training set is a set of N feature vectors and their class labels; and a learning algorithm is used to train a classifier using the training set. The test set is a set of feature vectors to which the classifier must assign labels. An intuitive way to decide how to classify an unlabelled test item is to look at the training data points nearby, and make the classification according to the classes of those nearby labelled data points. This intuition is formalised in a classification approach called K-nearest neighbour (k-nn) classification. The k-nn approach looks at the K points in the training set that are closest to the test point; the test point is then classified according to the class to which the majority of the K-nearest neighbours belong. The training of the K-NN classifier is simple, we just need to store all the training set! However, testing is much slower, since it involves measuring the distance between each test point and every training point. We can write the k-nn algorithm precisely as follows, where X is the training data set with class labels so that X = {(x, c)}, Z is the test set, there are C possible classes, and r is the distance metric (typically the Euclidean distance): For each test example z Z: Compute the distance r(z, x) between z and each training example (x, c) X Select U k (z) X, the set of the k nearest training examples to z. Assign test point z to class c, where the majority of the K-nearest neighbours belongs. To illustrate how the k-nn algorithm works, we will use the Lemon-Orange dataset described in the lecture slides. Since we do not have class labels for this data set, we will use the output of the K- means algorithm as ground truth labels for the training data; then, based on this training set we will run k-nn algorithm to classify new test points. Exercise Download the lemon-orange.txt file from the course website, and load the data in MAT- LAB. Save the actual data in matrix A. Run MATLAB s K-means algorithm for K = clusters and plot the data together with the cluster means. The result should look like Figure. Note: Before running k-means type rng(2). Check what rng command does.
2 9 8 height width Figure : Lemon-Orange dataset after running k-means algorithm for K =. Based on k-means output, we will assign each data point to that specific class using the following code. % Orange-lemons data training_data = A; % set seed for repeatable results rng(2); % Number of clusters K = ; % Corresponding class labels for each training point C = kmeans(training_data, K); % we use Matlab s built in function % Concatenate labels to the training data (type: help cat) training_data = cat(2, training_data, C); We can create some random test data using the following code: % Random test data test_data = [6. 7.6; ; 8. 9]; % show test data on the same plot with the training data plot(training_data(:,), training_data(:,2), + ); hold on; xlabel(col_headers{}); ylabel(col_headers{2}); % show test data as red circles plot(test_data(:,), test_data(:,2), ro ); axis([ ]); hold off; 2
3 9 8 height width Figure 2: Lemon-Orange training dataset together with some random test data shown in red circles. Now we can implement k-nn algorithm for different K and observe on which class each test point will be assigned. The following code illustrates the k-nn algorithm for the first test point: % distance between first test point and each training observation % NOTE: we need the square_dist function from previous labs r_zx = square_dist(a, test_data(,:)); % Sort the distances in ascending order [r_zx,idx] = sort(r_zx, 2, ascend ); Knn = 3; % K nearest neighbors, e.g. Knn = 3 r_zx = r_zx(:knn); % keep the first Knn distances idx = idx(:knn); % keep the first Knn indexes % majority vote only on those Knn indexes prediction = mode(c(idx)) % class label 2, belongs to the green colour prediction = 2 Exercise For the same test point, use different number of k-nearest neighbours and check if the class label changes. Write a function simpleknn which will implement a simple k-nn algorithm, similar to what was shown in the lab session. Classify the other two test samples using your k-nn algorithm for K-nearest neighbours =,, 6,. 3
4 . Plot decision boundary We can draw a line such that one side of it corresponds to one class and the other side to the other. Such a line is called a decision boundary. In the case where we need to classify more than two classes a more complex decision boundary will be created. For the Lemon-Orange dataset, we will create two class labels using the K-means algorithm. Then, the decision boundaries using -nearest neighbour and -nearest neighbours will be calculated. % Colormap we will use to colour each classes. cmap = [ ,.68689,.66737;.8766,.8272,.9962;.83393,.6277, ;.83293,.83,.7798; ,.69836,.82; ,.8783,.33927; ,.8627, ; ,.8686,.2968 ;.83622,.77723,.68336;.7968,.7968,.7968]; rng(2); % Set seed Knn = ; % K- nearest neighbours K = ; % Number of clusters C = kmeans(a, K); % Class labels for each training point Xplot = linspace(min(a(:,)), max(a(:,)), ) ; Yplot = linspace(min(a(:,2)), max(a(:,2)), ) ; % Obtain the grid vectors for the two dimensions [Xv Yv] = meshgrid(xplot, Yplot); gridx = [Xv(:), Yv(:)]; % Concatenate to get a 2-D point. classes = length(xv(:)); for i = :length(gridx) % Apply k-nn for each test point dists = square_dist(a, gridx(i, :)) ; % Compute distances [d I] = sort(dists, ascend ); classes(i) = mode(c(i(:knn))); end figure; % This function will draw the decision boundaries [CC,h] = contourf(xplot(:), Yplot(:), reshape(classes, length(xplot ), length(yplot))); set(h, LineColor, none ); colormap(cmap); hold on; % Plot the scatter plots grouped by their classes scatters = gscatter(a(:,), A(:,2), C, [,,], o, ); % Fill in the color of each point according to the class labels. for n = :length(scatters) set(scatters(n), MarkerFaceColor, cmap(n,:)); end
5 Running the above code for 2 and clusters with knn = we obtain Figure (a) (b) Figure 3: Decision boundaries for (a) C = 2 and (b) C = using knn =. Exercise Show the decision boundaries using knn =, 2,,, when we have two clusters. 2 Statistical Pattern Recognition In many real life problems, we have to make decisions based on uncertainty, e.g. due to inaccurate or incomplete information about a problem. The mathematics of probability provides the way to deal with uncertainty, and tells us how to update our knowledge and beliefs if new information becomes available. In this lab session we introduce the use of probability and statistics for pattern recognition and learning. Consider a pattern classification problem in which there are K classes. Let C denote the class, taking values,..., K, and let P(C k ) be the prior probability of class k. The observed input data, which is a D-dimensional feature vector, is denoted by X. Once the training set is used to train the classifier, a new, unlabelled data point x is observed. Let P(x C k ) be the likelihood of class k for the data x. To perform classification we could use Bayes theorem to compute the posterior probabilities P(C k x), for every class k =,..., K; we can then classify x by assigning it to the class with the highest posterior probability. That is, we need to compute: P(C k x) = P(x C k)p(c k ) P(x) = P(x C k)p(c k ) k P(x C k )P(C k ) for every class k and then assign x to the class with the highest posterior probability (i.e. find arg maxp(c k x)). This procedure is sometimes called MAP (maximum a posteriori) decision rule. k Thus, for each class k we need to provide an estimate of the likelihood P(x C k ) and the prior probability P(C k ). () (2)
6 2. Fish Example data To illustrate the use of posterior probabilities to perform classification, we will use a dataset which contains measurements of fish lengths. The dataset comprises 2 observations ( male and female fish), each representing the length of the fish. The objective is to classify the fish as male or female based on their length measurement. You can download the Fish dataset from the course website: You will find a file named fish.txt, download it and save it in your current folder. Note that this file is already pre-processed, and each line corresponds to three columns, the first column denotes the fish length x, the second and the third columns denote the number of male n M (x) and female n F (x) observations for that length, respectively. Exercise Read the file and load the data in MATLAB. Store the fish data in a matrix A. 2.2 Compute prior and likelihood Let class C = M represent male, and C = F represent female fish. The prior probability expresses our beliefs about the sex of the fish before any evidence is taken into account. We can assume that male and female fish have different prior probabilities (e.g. P(C M ) =.6, P(C F ) =.) or we can compute an estimate of those from the actual data by finding the proportion of male and female fish out of the total observations: % Total number of male fish, i.e. N_M = sum(a(:,2)); % Total number of female fish, i.e. N_F = sum(a(:,3)); % total number of observations, i.e. 2 N_total = N_M + N_F; % prior probability of male fish prior_m = N_M / N_total prior_m =. % prior probability for female is -P(M), since P(M) + P(F) =. prior_f = - prior_m prior_f =. We can now estimate the likelihoods P(x C M ) and P(x C F ) as the counts in each class for length x divided by the total number of examples in that class: P(x C M ) n M(x) N M (3) P(x C F ) n F(x) N F () Thus we can estimate the likelihoods of the length of each fish given each class using relative frequencies (i.e. using the training set of examples from each class). Note that we obtain estimates of P(x C M ) and P(x C F ), since N M and N F are finite. We can compute the likelihood for each fish length x, simply by computing the relative frequencies as follows: 6
7 % Likelihood vector for each length x for male fish lik_m = A(:,2)/N_M; % Likelihood vector for each length x for female fish lik_f = A(:,3)/N_F; Let s observe the length distribution for each class. We can do this easily by plotting histograms. Figure shows the length distribution for male and female fish. For each class, also the Cumulative Distribution Function (CDF) is shown. The CDF is the probability that a real-valued random variable X will take a value less than or equal to x, that is, CDF(x) = P(X x), where P denotes the probability..2 Lengths of male fish.2 Lengths of female fish Rel. Freq. Rel. Freq..... (a) (b) cdf. cdf (c) (d) Figure : (a) Relative frequency of lengths of male fish. (b) Relative frequency of lengths of female fish. (c) CDF for male fish. (d) CDF for female fish. The code for creating plots (a) and (c) in Figure is the following: % Create a histogram bar and return a vector of handles to this object hh = bar(a(:,), A(:,2)/N_M, ); % Modify the initial plot set(hh, FaceColor, white ); set(hh, EdgeColor, red ); set(hh, linewidth,.); % Define x and y labels ylabel( Rel. Freq. ); xlabel( ); 7
8 % Create title title( Lengths of male fish ); % Define only x-axis limits xlim([ 2]); % Create CDF plot. Check what the cumsum function does in Matlab hh = plot(a(:,), cumsum(a(:,2))/n_m, -r ); % Modify the initial plot set(hh, linewidth,.); % Define x and y labels ylabel( cdf ); xlabel( ); % Define only x-axis limits xlim([ 2]); We can also plot the likelihood P(x C k ) for each class as shown in Figure ; note that the shapes of each class are similar to Figure, since we computed the likelihood from the relative frequencies. We observe that fish with length around 3cm are most likely to be male fish, since the likelihood is P(x = 2 C M ).22, whereas for female fish is only P(x = 2 C F )...2 P(x M) P(x F).2 Likelihood P(x C)... Figure : Likelihood function for male and female fish lengths. Exercises Plot histogram of female fish lengths as shown in Figure (b). Figures (a) and (b), show relative frequencies. Modify your code so it can show the actual frequencies. Show both the male and female histograms in the same bar plot. Plot the likelihood functions for male and female fish lengths as shown in Figure. 8
9 2.3 Compute posterior probabilities Having computed the prior and likelihood, we can now compute the posterior probabilities using Eq.. First we need to compute the evidence P(x), which can be thought as a normalization constant ensuring that we have actual (posterior) probabilities (i.e. P(C k x) and k P(C k x) = ). % Compute evidence vector for each fish length Px = prior_m * lik_m + prior_f * lik_f; % Compute vector of posterior probabilities for male fish lengths post_m = lik_m * prior_m./ Px; % Compute vector of posterior probabilities for female fish lengths post_f = lik_f * prior_f./ Px; We can now plot the posterior probabilities using the following code: % Posterior probabilities for male fish hh = plot(a(:,), post_m, -r ); set(hh, linewidth,.); ylabel( Posterior P(C x) ); xlabel( ); xlim([ 2]); hold on % Posterior probabilities for female fish hh = plot(a(:,), post_f, -b ); set(hh, linewidth,.); legend( P(M x), P(F x), Location, northwest ); hold on % Show decision boundary dec_bound =.7; plot([dec_bound dec_bound], get(gca, ylim ), --k );.9 P(M x) P(F x).8.7 Posterior P(C x) Figure 6: Posterior probabilities for male and female fish lengths. The vertical black line around x = denotes the decision boundary. 9
10 Figure 6 depicts how the posterior probability changes as a function of the fish length for both the male (red) and female (blue) fish. The vertical black line denotes the decision boundary (i.e. where the posterior probabilities of male and female fish are the same). If we used these probabilities to assign a new (unlabelled) fish, we would classify it as female if its length was on the left of the decision boundary, and male otherwise. Assume that we observe a new fish for which we know that has length x = 8. Should we classify it as male or female fish? % Fish of length x = 8, are in the th element of the likelihood vectors % So we compute the test point likelihood directly from that element >> test_lik_m = lik_m(); >> test_lik_f = lik_f(); % Compute posterior probabilities for each class >> test_post_m = test_lik_m * prior_m / Px() test_post_m =.2 >> test_post_f = test_lik_f * prior_f / Px() test_post_f =.87 Hence the fish would be classified as female, which could be observed directly from Figure Bayes decision rule In the previous section (for the sake of illustration) we computed the actual posterior probabilities for each class and then assigned each example to the class with the maximum posterior probability. However, computing posterior probabilities for real life problems is often impractical, mainly due the denominator in the Bayes theorem (i.e. the evidence). Since our goal is to classify a test example to the most probable class, we can compute their ratio: P(C M x) P(C F x) = P(x C M )P(C M ) P(x) P(x C F )P(C F ) P(x) = P(x C M)P(C M ) P(x C F )P(C F ) If the ratio in the above equation is greater than then x is classified as M, if x is less than then x is classified as F. As you can observe, the denominator term P(x) cancels, so there is no need to compute it at all. Let s compute the ratio of the above example for a test fish of length x = 8. We would expect the ratio to be less than, since the fish should be classified as female. % Compute ratio of posterior probabilities for test example x = 8 >> test_ratio = (test_lik_m * prior_m) / (test_lik_f * prior_f) test_ratio =.29 () Exercises Compute the posterior probabilities for each class using the following prior distributions P(C M ) =.9 and P(C F =.). Create the likelihood and the posterior probability plots as shown in the previous sections. What do you observe? Does the likelihood depend on the prior? Classify the test example x = 8 using the updated posterior probabilities. Assuming equal prior probabilities, classify the following test examples: x = 2, 9, 2, 6.
K-means Clustering & k-nn classification
K-means Clustering & k-nn classification Andreas C. Kapourani (Credit: Hiroshi Shimodaira) 03 February 2016 1 Introduction In this lab session we will focus on K-means clustering and k-nearest Neighbour
More informationK-means Clustering & PCA
K-means Clustering & PCA Andreas C. Kapourani (Credit: Hiroshi Shimodaira) 02 February 2018 1 Introduction In this lab session we will focus on K-means clustering and Principal Component Analysis (PCA).
More informationComputer Vision. Exercise Session 10 Image Categorization
Computer Vision Exercise Session 10 Image Categorization Object Categorization Task Description Given a small number of training images of a category, recognize a-priori unknown instances of that category
More informationCSCI567 Machine Learning (Fall 2014)
CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 9, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 9, 2014 1 / 47
More informationClustering and Visualisation of Data
Clustering and Visualisation of Data Hiroshi Shimodaira January-March 28 Cluster analysis aims to partition a data set into meaningful or useful groups, based on distances between data points. In some
More informationNearest Neighbor Classification
Nearest Neighbor Classification Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms January 11, 2017 1 / 48 Outline 1 Administration 2 First learning algorithm: Nearest
More informationDistribution-free Predictive Approaches
Distribution-free Predictive Approaches The methods discussed in the previous sections are essentially model-based. Model-free approaches such as tree-based classification also exist and are popular for
More informationInf2B assignment 2. Natural images classification. Hiroshi Shimodaira and Pol Moreno. Submission due: 4pm, Wednesday 30 March 2016.
Inf2B assignment 2 (Ver. 1.2) Natural images classification Submission due: 4pm, Wednesday 30 March 2016 Hiroshi Shimodaira and Pol Moreno This assignment is out of 100 marks and forms 12.5% of your final
More informationMathematics of Data. INFO-4604, Applied Machine Learning University of Colorado Boulder. September 5, 2017 Prof. Michael Paul
Mathematics of Data INFO-4604, Applied Machine Learning University of Colorado Boulder September 5, 2017 Prof. Michael Paul Goals In the intro lecture, every visualization was in 2D What happens when we
More informationSimilarity and recommender systems
Similarity and recommender systems Andreas C. Kapourani January 8 Introduction In this lab session we will work with some toy data and implement a simple collaborative filtering recommender system (RS),
More informationMachine Learning: Basic Principles
Machine Learning: Basic Principles Teaching demonstration Kalle Palomäki Department of Signal Processing and Acoustics Aalto University Content 1. Goal 2. Machine learning: definition 3. Classification
More informationSOCIAL MEDIA MINING. Data Mining Essentials
SOCIAL MEDIA MINING Data Mining Essentials Dear instructors/users of these slides: Please feel free to include these slides in your own material, or modify them as you see fit. If you decide to incorporate
More informationKTH ROYAL INSTITUTE OF TECHNOLOGY. Lecture 14 Machine Learning. K-means, knn
KTH ROYAL INSTITUTE OF TECHNOLOGY Lecture 14 Machine Learning. K-means, knn Contents K-means clustering K-Nearest Neighbour Power Systems Analysis An automated learning approach Understanding states in
More informationPerformance Evaluation of Various Classification Algorithms
Performance Evaluation of Various Classification Algorithms Shafali Deora Amritsar College of Engineering & Technology, Punjab Technical University -----------------------------------------------------------***----------------------------------------------------------
More informationMachine Learning. Nonparametric methods for Classification. Eric Xing , Fall Lecture 2, September 12, 2016
Machine Learning 10-701, Fall 2016 Nonparametric methods for Classification Eric Xing Lecture 2, September 12, 2016 Reading: 1 Classification Representing data: Hypothesis (classifier) 2 Clustering 3 Supervised
More informationMODULE 7 Nearest Neighbour Classifier and its variants LESSON 11. Nearest Neighbour Classifier. Keywords: K Neighbours, Weighted, Nearest Neighbour
MODULE 7 Nearest Neighbour Classifier and its variants LESSON 11 Nearest Neighbour Classifier Keywords: K Neighbours, Weighted, Nearest Neighbour 1 Nearest neighbour classifiers This is amongst the simplest
More informationIntroduction to Artificial Intelligence
Introduction to Artificial Intelligence COMP307 Machine Learning 2: 3-K Techniques Yi Mei yi.mei@ecs.vuw.ac.nz 1 Outline K-Nearest Neighbour method Classification (Supervised learning) Basic NN (1-NN)
More informationClassification Algorithms in Data Mining
August 9th, 2016 Suhas Mallesh Yash Thakkar Ashok Choudhary CIS660 Data Mining and Big Data Processing -Dr. Sunnie S. Chung Classification Algorithms in Data Mining Deciding on the classification algorithms
More informationKernels and Clustering
Kernels and Clustering Robert Platt Northeastern University All slides in this file are adapted from CS188 UC Berkeley Case-Based Learning Non-Separable Data Case-Based Reasoning Classification from similarity
More informationk-nearest Neighbors + Model Selection
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University k-nearest Neighbors + Model Selection Matt Gormley Lecture 5 Jan. 30, 2019 1 Reminders
More informationFrequency Distributions
Displaying Data Frequency Distributions After collecting data, the first task for a researcher is to organize and summarize the data so that it is possible to get a general overview of the results. Remember,
More informationA Brief Overview of Audio Information Retrieval. Unjung Nam CCRMA Stanford University
A Brief Overview of Audio Information Retrieval Unjung Nam CCRMA Stanford University 1 Outline What is AIR? Motivation Related Field of Research Elements of AIR Experiments and discussion Music Classification
More informationCase-Based Reasoning. CS 188: Artificial Intelligence Fall Nearest-Neighbor Classification. Parametric / Non-parametric.
CS 188: Artificial Intelligence Fall 2008 Lecture 25: Kernels and Clustering 12/2/2008 Dan Klein UC Berkeley Case-Based Reasoning Similarity for classification Case-based reasoning Predict an instance
More informationCS 188: Artificial Intelligence Fall 2008
CS 188: Artificial Intelligence Fall 2008 Lecture 25: Kernels and Clustering 12/2/2008 Dan Klein UC Berkeley 1 1 Case-Based Reasoning Similarity for classification Case-based reasoning Predict an instance
More informationWhat do we mean by recognition?
Announcements Recognition Project 3 due today Project 4 out today (help session + photos end-of-class) The Margaret Thatcher Illusion, by Peter Thompson Readings Szeliski, Chapter 14 1 Recognition What
More informationWe use non-bold capital letters for all random variables in these notes, whether they are scalar-, vector-, matrix-, or whatever-valued.
The Bayes Classifier We have been starting to look at the supervised classification problem: we are given data (x i, y i ) for i = 1,..., n, where x i R d, and y i {1,..., K}. In this section, we suppose
More informationk-nn classification with R QMMA
k-nn classification with R QMMA Emanuele Taufer file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l1-knn-eng.html#(1) 1/16 HW (Height and weight) of adults Statistics
More informationNearest Neighbor Predictors
Nearest Neighbor Predictors September 2, 2018 Perhaps the simplest machine learning prediction method, from a conceptual point of view, and perhaps also the most unusual, is the nearest-neighbor method,
More informationEE795: Computer Vision and Intelligent Systems
EE795: Computer Vision and Intelligent Systems Spring 2013 TTh 17:30-18:45 FDH 204 Lecture 18 130404 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Object Recognition Intro (Chapter 14) Slides from
More informationIntroduction to Pattern Recognition Part II. Selim Aksoy Bilkent University Department of Computer Engineering
Introduction to Pattern Recognition Part II Selim Aksoy Bilkent University Department of Computer Engineering saksoy@cs.bilkent.edu.tr RETINA Pattern Recognition Tutorial, Summer 2005 Overview Statistical
More informationMS1b Statistical Data Mining Part 3: Supervised Learning Nonparametric Methods
MS1b Statistical Data Mining Part 3: Supervised Learning Nonparametric Methods Yee Whye Teh Department of Statistics Oxford http://www.stats.ox.ac.uk/~teh/datamining.html Outline Supervised Learning: Nonparametric
More informationGenerative and discriminative classification techniques
Generative and discriminative classification techniques Machine Learning and Category Representation 2014-2015 Jakob Verbeek, November 28, 2014 Course website: http://lear.inrialpes.fr/~verbeek/mlcr.14.15
More informationUniversity of Ghana Department of Computer Engineering School of Engineering Sciences College of Basic and Applied Sciences
University of Ghana Department of Computer Engineering School of Engineering Sciences College of Basic and Applied Sciences CPEN 405: Artificial Intelligence Lab 7 November 15, 2017 Unsupervised Learning
More informationCS 343: Artificial Intelligence
CS 343: Artificial Intelligence Kernels and Clustering Prof. Scott Niekum The University of Texas at Austin [These slides based on those of Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley.
More informationSection 2-2 Frequency Distributions. Copyright 2010, 2007, 2004 Pearson Education, Inc
Section 2-2 Frequency Distributions Copyright 2010, 2007, 2004 Pearson Education, Inc. 2.1-1 Frequency Distribution Frequency Distribution (or Frequency Table) It shows how a data set is partitioned among
More informationObject recognition (part 1)
Recognition Object recognition (part 1) CSE P 576 Larry Zitnick (larryz@microsoft.com) The Margaret Thatcher Illusion, by Peter Thompson Readings Szeliski Chapter 14 Recognition What do we mean by object
More informationClassification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University
Classification Vladimir Curic Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University Outline An overview on classification Basics of classification How to choose appropriate
More informationNearest neighbors classifiers
Nearest neighbors classifiers James McInerney Adapted from slides by Daniel Hsu Sept 11, 2017 1 / 25 Housekeeping We received 167 HW0 submissions on Gradescope before midnight Sept 10th. From a random
More informationLast week. Multi-Frame Structure from Motion: Multi-View Stereo. Unknown camera viewpoints
Last week Multi-Frame Structure from Motion: Multi-View Stereo Unknown camera viewpoints Last week PCA Today Recognition Today Recognition Recognition problems What is it? Object detection Who is it? Recognizing
More informationNOTE: Your grade will be based on the correctness, efficiency and clarity.
THE HONG KONG UNIVERSITY OF SCIENCE & TECHNOLOGY Department of Computer Science COMP328: Machine Learning Fall 2006 Assignment 2 Due Date and Time: November 14 (Tue) 2006, 1:00pm. NOTE: Your grade will
More information10-701/15-781, Fall 2006, Final
-7/-78, Fall 6, Final Dec, :pm-8:pm There are 9 questions in this exam ( pages including this cover sheet). If you need more room to work out your answer to a question, use the back of the page and clearly
More informationCIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, :59pm, PDF to Canvas [100 points]
CIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, 2015. 11:59pm, PDF to Canvas [100 points] Instructions. Please write up your responses to the following problems clearly and concisely.
More informationVerification: is that a lamp? What do we mean by recognition? Recognition. Recognition
Recognition Recognition The Margaret Thatcher Illusion, by Peter Thompson The Margaret Thatcher Illusion, by Peter Thompson Readings C. Bishop, Neural Networks for Pattern Recognition, Oxford University
More informationCOMPUTATIONAL STATISTICS UNSUPERVISED LEARNING
COMPUTATIONAL STATISTICS UNSUPERVISED LEARNING Luca Bortolussi Department of Mathematics and Geosciences University of Trieste Office 238, third floor, H2bis luca@dmi.units.it Trieste, Winter Semester
More informationMachine Learning and Pervasive Computing
Stephan Sigg Georg-August-University Goettingen, Computer Networks 17.12.2014 Overview and Structure 22.10.2014 Organisation 22.10.3014 Introduction (Def.: Machine learning, Supervised/Unsupervised, Examples)
More information6. Dicretization methods 6.1 The purpose of discretization
6. Dicretization methods 6.1 The purpose of discretization Often data are given in the form of continuous values. If their number is huge, model building for such data can be difficult. Moreover, many
More informationNotes and Announcements
Notes and Announcements Midterm exam: Oct 20, Wednesday, In Class Late Homeworks Turn in hardcopies to Michelle. DO NOT ask Michelle for extensions. Note down the date and time of submission. If submitting
More informationCSC 411: Lecture 05: Nearest Neighbors
CSC 411: Lecture 05: Nearest Neighbors Raquel Urtasun & Rich Zemel University of Toronto Sep 28, 2015 Urtasun & Zemel (UofT) CSC 411: 05-Nearest Neighbors Sep 28, 2015 1 / 13 Today Non-parametric models
More informationCHAPTER 6. The Normal Probability Distribution
The Normal Probability Distribution CHAPTER 6 The normal probability distribution is the most widely used distribution in statistics as many statistical procedures are built around it. The central limit
More informationChapter 4: Non-Parametric Techniques
Chapter 4: Non-Parametric Techniques Introduction Density Estimation Parzen Windows Kn-Nearest Neighbor Density Estimation K-Nearest Neighbor (KNN) Decision Rule Supervised Learning How to fit a density
More informationMachine Learning Classifiers and Boosting
Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline Different types of learning problems Different types of learning algorithms Supervised learning Decision trees Naïve
More informationCS5670: Computer Vision
CS5670: Computer Vision Noah Snavely Lecture 33: Recognition Basics Slides from Andrej Karpathy and Fei-Fei Li http://vision.stanford.edu/teaching/cs231n/ Announcements Quiz moved to Tuesday Project 4
More informationCCNY. BME 2200: BME Biostatistics and Research Methods. Lecture 4: Graphing data with MATLAB
BME 2200: BME Biostatistics and Research Methods Lecture 4: Graphing data with MATLAB Lucas C. Parra Biomedical Engineering Department CCNY parra@ccny.cuny.edu 1 Content, Schedule 1. Scientific literature:
More informationProblem Set 5. MAS 622J/1.126J: Pattern Recognition and Analysis. Due Monday, 8 November Resubmission due, 12 November 2010
Problem Set 5 MAS 622J/1.126J: Pattern Recognition and Analysis Due Monday, 8 November 2010. Resubmission due, 12 November 2010 Note: All instructions to plot data or write a program should be carried
More informationWeka ( )
Weka ( http://www.cs.waikato.ac.nz/ml/weka/ ) The phases in which classifier s design can be divided are reflected in WEKA s Explorer structure: Data pre-processing (filtering) and representation Supervised
More informationQuestion 1: knn classification [100 points]
CS 540: Introduction to Artificial Intelligence Homework Assignment # 8 Assigned: 11/13 Due: 11/20 before class Question 1: knn classification [100 points] For this problem, you will be building a k-nn
More informationImage classification Computer Vision Spring 2018, Lecture 18
Image classification http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 2018, Lecture 18 Course announcements Homework 5 has been posted and is due on April 6 th. - Dropbox link because course
More informationCSE 573: Artificial Intelligence Autumn 2010
CSE 573: Artificial Intelligence Autumn 2010 Lecture 16: Machine Learning Topics 12/7/2010 Luke Zettlemoyer Most slides over the course adapted from Dan Klein. 1 Announcements Syllabus revised Machine
More informationStatistical Methods in AI
Statistical Methods in AI Distance Based and Linear Classifiers Shrenik Lad, 200901097 INTRODUCTION : The aim of the project was to understand different types of classification algorithms by implementing
More informationMachine Learning Lecture 3
Machine Learning Lecture 3 Probability Density Estimation II 19.10.2017 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Exam dates We re in the process
More informationBayes Risk. Classifiers for Recognition Reading: Chapter 22 (skip 22.3) Discriminative vs Generative Models. Loss functions in classifiers
Classifiers for Recognition Reading: Chapter 22 (skip 22.3) Examine each window of an image Classify object class within each window based on a training set images Example: A Classification Problem Categorize
More informationINF 4300 Classification III Anne Solberg The agenda today:
INF 4300 Classification III Anne Solberg 28.10.15 The agenda today: More on estimating classifier accuracy Curse of dimensionality and simple feature selection knn-classification K-means clustering 28.10.15
More informationData Mining Classification: Alternative Techniques. Lecture Notes for Chapter 4. Instance-Based Learning. Introduction to Data Mining, 2 nd Edition
Data Mining Classification: Alternative Techniques Lecture Notes for Chapter 4 Instance-Based Learning Introduction to Data Mining, 2 nd Edition by Tan, Steinbach, Karpatne, Kumar Instance Based Classifiers
More informationWeek 2: Frequency distributions
Types of data Health Sciences M.Sc. Programme Applied Biostatistics Week 2: distributions Data can be summarised to help to reveal information they contain. We do this by calculating numbers from the data
More informationBasic plotting commands Types of plots Customizing plots graphically Specifying color Customizing plots programmatically Exporting figures
Basic plotting commands Types of plots Customizing plots graphically Specifying color Customizing plots programmatically Exporting figures Matlab is flexible enough to let you quickly visualize data, and
More informationAcoustic to Articulatory Mapping using Memory Based Regression and Trajectory Smoothing
Acoustic to Articulatory Mapping using Memory Based Regression and Trajectory Smoothing Samer Al Moubayed Center for Speech Technology, Department of Speech, Music, and Hearing, KTH, Sweden. sameram@kth.se
More informationK-Nearest Neighbors. Jia-Bin Huang. Virginia Tech Spring 2019 ECE-5424G / CS-5824
K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019 Administrative Check out review materials Probability Linear algebra Python and NumPy Start your HW 0 On your Local machine:
More informationRecap: Gaussian (or Normal) Distribution. Recap: Minimizing the Expected Loss. Topics of This Lecture. Recap: Maximum Likelihood Approach
Truth Course Outline Machine Learning Lecture 3 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Probability Density Estimation II 2.04.205 Discriminative Approaches (5 weeks)
More informationClassifiers for Recognition Reading: Chapter 22 (skip 22.3)
Classifiers for Recognition Reading: Chapter 22 (skip 22.3) Examine each window of an image Classify object class within each window based on a training set images Slide credits for this chapter: Frank
More informationMachine Learning. Classification
10-701 Machine Learning Classification Inputs Inputs Inputs Where we are Density Estimator Probability Classifier Predict category Today Regressor Predict real no. Later Classification Assume we want to
More informationCS1114 Section: SIFT April 3, 2013
CS1114 Section: SIFT April 3, 2013 Object recognition has three basic parts: feature extraction, feature matching, and fitting a transformation. In this lab, you ll learn about SIFT feature extraction
More informationHomework Assignment #3
CS 540-2: Introduction to Artificial Intelligence Homework Assignment #3 Assigned: Monday, February 20 Due: Saturday, March 4 Hand-In Instructions This assignment includes written problems and programming
More informationSupervised vs unsupervised clustering
Classification Supervised vs unsupervised clustering Cluster analysis: Classes are not known a- priori. Classification: Classes are defined a-priori Sometimes called supervised clustering Extract useful
More informationPraxis Elementary Education: Mathematics Subtest (5003) Curriculum Crosswalk. Required Course Numbers. Test Content Categories.
Page 1 I. Numbers and Operations (40%) A. Understands the place value system 1. Writes numbers using base-10 numerals, number names, and expanded form 2. Composes and decomposes multi-digit numbers 3.
More informationMachine Learning: Algorithms and Applications Mockup Examination
Machine Learning: Algorithms and Applications Mockup Examination 14 May 2012 FIRST NAME STUDENT NUMBER LAST NAME SIGNATURE Instructions for students Write First Name, Last Name, Student Number and Signature
More informationPattern recognition. Classification/Clustering GW Chapter 12 (some concepts) Textures
Pattern recognition Classification/Clustering GW Chapter 12 (some concepts) Textures Patterns and pattern classes Pattern: arrangement of descriptors Descriptors: features Patten class: family of patterns
More informationPerformance Analysis of Data Mining Classification Techniques
Performance Analysis of Data Mining Classification Techniques Tejas Mehta 1, Dr. Dhaval Kathiriya 2 Ph.D. Student, School of Computer Science, Dr. Babasaheb Ambedkar Open University, Gujarat, India 1 Principal
More information10/5/2017 MIST.6060 Business Intelligence and Data Mining 1. Nearest Neighbors. In a p-dimensional space, the Euclidean distance between two records,
10/5/2017 MIST.6060 Business Intelligence and Data Mining 1 Distance Measures Nearest Neighbors In a p-dimensional space, the Euclidean distance between two records, a = a, a,..., a ) and b = b, b,...,
More informationArtificial Neural Networks Lab 2 Classical Pattern Recognition
Artificial Neural Networks Lab 2 Classical Pattern Recognition Purpose To implement (using MATLAB) some traditional classifiers and try these on both synthetic and real data. The exercise should give some
More informationSpatial Information Based Image Classification Using Support Vector Machine
Spatial Information Based Image Classification Using Support Vector Machine P.Jeevitha, Dr. P. Ganesh Kumar PG Scholar, Dept of IT, Regional Centre of Anna University, Coimbatore, India. Assistant Professor,
More informationUnsupervised learning: Clustering & Dimensionality reduction. Theo Knijnenburg Jorma de Ronde
Unsupervised learning: Clustering & Dimensionality reduction Theo Knijnenburg Jorma de Ronde Source of slides Marcel Reinders TU Delft Lodewyk Wessels NKI Bioalgorithms.info Jeffrey D. Ullman Stanford
More informationEGR 102 Introduction to Engineering Modeling. Lab 05B Plotting
EGR 102 Introduction to Engineering Modeling Lab 05B Plotting 1 Overview Plotting in MATLAB 2D plotting ( ezplot(), fplot(), plot()) Formatting of 2D plots 3D plotting (surf(), mesh(), plot3()) Formatting
More informationNetwork Traffic Measurements and Analysis
DEIB - Politecnico di Milano Fall, 2017 Introduction Often, we have only a set of features x = x 1, x 2,, x n, but no associated response y. Therefore we are not interested in prediction nor classification,
More informationWild Mushrooms Classification Edible or Poisonous
Wild Mushrooms Classification Edible or Poisonous Yulin Shen ECE 539 Project Report Professor: Hu Yu Hen 2013 Fall ( I allow code to be released in the public domain) pg. 1 Contents Introduction. 3 Practicality
More informationIntroduction to Machine Learning. Xiaojin Zhu
Introduction to Machine Learning Xiaojin Zhu jerryzhu@cs.wisc.edu Read Chapter 1 of this book: Xiaojin Zhu and Andrew B. Goldberg. Introduction to Semi- Supervised Learning. http://www.morganclaypool.com/doi/abs/10.2200/s00196ed1v01y200906aim006
More informationCS 584 Data Mining. Classification 1
CS 584 Data Mining Classification 1 Classification: Definition Given a collection of records (training set ) Each record contains a set of attributes, one of the attributes is the class. Find a model for
More informationModel Selection Introduction to Machine Learning. Matt Gormley Lecture 4 January 29, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Model Selection Matt Gormley Lecture 4 January 29, 2018 1 Q&A Q: How do we deal
More informationClassification: Feature Vectors
Classification: Feature Vectors Hello, Do you want free printr cartriges? Why pay more when you can get them ABSOLUTELY FREE! Just # free YOUR_NAME MISSPELLED FROM_FRIEND... : : : : 2 0 2 0 PIXEL 7,12
More informationECE 662:Pattern Recognition and Decision-Making Processes Homework Assignment Two *************
ECE 662:Pattern Recognition and Decision-Making Processes Homework Assignment Two ************* Collaborators: None April 16, 28 1 1 Question 1: Numerical Experiments with the Fisher Linear discriminant
More informationIntroduction to Machine Learning Prof. Mr. Anirban Santara Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Introduction to Machine Learning Prof. Mr. Anirban Santara Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture - 19 Python Exercise on Naive Bayes Hello everyone.
More informationBayes Classifiers and Generative Methods
Bayes Classifiers and Generative Methods CSE 4309 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 The Stages of Supervised Learning To
More informationPattern recognition systems Lab 8 Bayesian Classifier: Simple Digit Recognition Application
Pattern recognition systems Lab 8 Bayesian Classifier: Simple Digit Recognition Application 1. Objectives In this lab session we will study the naïve Bayes algorithm and we will apply it to a simple recognition
More informationSupervised Learning: K-Nearest Neighbors and Decision Trees
Supervised Learning: K-Nearest Neighbors and Decision Trees Piyush Rai CS5350/6350: Machine Learning August 25, 2011 (CS5350/6350) K-NN and DT August 25, 2011 1 / 20 Supervised Learning Given training
More informationSupervised Learning Classification Algorithms Comparison
Supervised Learning Classification Algorithms Comparison Aditya Singh Rathore B.Tech, J.K. Lakshmipat University -------------------------------------------------------------***---------------------------------------------------------
More informationSegmentation Computer Vision Spring 2018, Lecture 27
Segmentation http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 218, Lecture 27 Course announcements Homework 7 is due on Sunday 6 th. - Any questions about homework 7? - How many of you have
More informationIntroduction to Machine Learning Prof. Anirban Santara Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Introduction to Machine Learning Prof. Anirban Santara Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture 14 Python Exercise on knn and PCA Hello everyone,
More informationMachine Learning A W 1sst KU. b) [1 P] Give an example for a probability distributions P (A, B, C) that disproves
Machine Learning A 708.064 11W 1sst KU Exercises Problems marked with * are optional. 1 Conditional Independence I [2 P] a) [1 P] Give an example for a probability distribution P (A, B, C) that disproves
More informationComputational Statistics The basics of maximum likelihood estimation, Bayesian estimation, object recognitions
Computational Statistics The basics of maximum likelihood estimation, Bayesian estimation, object recognitions Thomas Giraud Simon Chabot October 12, 2013 Contents 1 Discriminant analysis 3 1.1 Main idea................................
More informationK Nearest Neighbor Wrap Up K- Means Clustering. Slides adapted from Prof. Carpuat
K Nearest Neighbor Wrap Up K- Means Clustering Slides adapted from Prof. Carpuat K Nearest Neighbor classification Classification is based on Test instance with Training Data K: number of neighbors that
More informationDATA MINING LECTURE 10B. Classification k-nearest neighbor classifier Naïve Bayes Logistic Regression Support Vector Machines
DATA MINING LECTURE 10B Classification k-nearest neighbor classifier Naïve Bayes Logistic Regression Support Vector Machines NEAREST NEIGHBOR CLASSIFICATION 10 10 Illustrating Classification Task Tid Attrib1
More information