Nearest neighbors classifiers

Transcription

1 Nearest neighbors classifiers James McInerney Adapted from slides by Daniel Hsu Sept 11, / 25

2 Housekeeping We received 167 HW0 submissions on Gradescope before midnight Sept 10th. From a random sample, most look well done and are using the question assignment mechanism on Gradescope correctly. Example assignment submission. I received a few late submission s. This one time only, I will turn the Gradescope submission page on again after class until 8pm EST. We will go over the trickiest parts of the homework at the beginning of the talk this Wednesday. Therefore I have pushed the first office hours to Wednesday. Here are the dates of the two exams for this course: Exam 1: Wednesday October 18th, 2017 Exam 2: Monday December 11th, / 25

3 Example: OCR for digits 1. Classify images of handwritten digits by the actual digits they represent. 2. Classification problem: Y = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} (a discrete set). 3 / 25

4 Nearest neighbor (NN) classifier Given: labeled examples D := {(x i, y i)} n i=1 Predictor: ˆfD : X Y On input x, 1. Find the point x i among {x i} n i=1 that is closest to x (the nearest neighbor). 2. Return y i. 4 / 25

5 How to measure distance? A default choice for distance between points in R d is the Euclidean distance (also called l 2 distance): u v 2 := d (u i v i) 2 i=1 (where u = (u 1, u 2,..., u d ) and v = (v 1, v 2,..., v d )). Grayscale pixel images. Treat as vectors (of 784 real-valued features) that live in R / 25

6 Example: OCR for digits with NN classifier Classify images of handwritten digits by the digits they depict. 6 / 25

7 Example: OCR for digits with NN classifier Classify images of handwritten digits by the digits they depict. X = R 784, Y = {0, 1,..., 9}. 6 / 25

8 Example: OCR for digits with NN classifier Classify images of handwritten digits by the digits they depict. X = R 784, Y = {0, 1,..., 9}. Given: labeled examples D := {(x i, y i)} n i=1 X Y. 6 / 25

9 Example: OCR for digits with NN classifier Classify images of handwritten digits by the digits they depict. X = R 784, Y = {0, 1,..., 9}. Given: labeled examples D := {(x i, y i)} n i=1 X Y. Construct NN classifier ˆf D using D. 6 / 25

10 Example: OCR for digits with NN classifier Classify images of handwritten digits by the digits they depict. X = R 784, Y = {0, 1,..., 9}. Given: labeled examples D := {(x i, y i)} n i=1 X Y. Construct NN classifier ˆf D using D. Question: Is this classifier any good? 6 / 25

11 Error rate Error rate of classifier f on a set of labeled examples D: err D(f) := # of (x, y) D such that f(x) y D (i.e., the fraction of D on which f disagrees with paired label). 7 / 25

12 Error rate Error rate of classifier f on a set of labeled examples D: err D(f) := # of (x, y) D such that f(x) y D (i.e., the fraction of D on which f disagrees with paired label). Sometimes, we ll write this as err(f, D). 7 / 25

13 Error rate Error rate of classifier f on a set of labeled examples D: err D(f) := # of (x, y) D such that f(x) y D (i.e., the fraction of D on which f disagrees with paired label). Sometimes, we ll write this as err(f, D). Question: What is err D( ˆf D)? 7 / 25

14 A better way to evaluate the classifier Split the labeled examples {(x i, y i)} n i=1 into two sets (randomly). Training data S. Test data T. 8 / 25

15 A better way to evaluate the classifier Split the labeled examples {(x i, y i)} n i=1 into two sets (randomly). Training data S. Test data T. Only use training data S to construct NN classifier ˆf S. 8 / 25

16 A better way to evaluate the classifier Split the labeled examples {(x i, y i)} n i=1 into two sets (randomly). Training data S. Test data T. Only use training data S to construct NN classifier ˆf S. Training error rate of ˆfS: err S( ˆf S) = 0%. 8 / 25

17 A better way to evaluate the classifier Split the labeled examples {(x i, y i)} n i=1 into two sets (randomly). Training data S. Test data T. Only use training data S to construct NN classifier ˆf S. Training error rate of ˆfS: err S( ˆf S) = 0%. Use test data T to evaluate accuracy of ˆf S. 8 / 25

18 A better way to evaluate the classifier Split the labeled examples {(x i, y i)} n i=1 into two sets (randomly). Training data S. Test data T. Only use training data S to construct NN classifier ˆf S. Training error rate of ˆfS: err S( ˆf S) = 0%. Use test data T to evaluate accuracy of ˆf S. Test error rate of ˆfS: err T ( ˆf S) = 3.09%. 8 / 25

19 A better way to evaluate the classifier Split the labeled examples {(x i, y i)} n i=1 into two sets (randomly). Training data S. Test data T. Only use training data S to construct NN classifier ˆf S. Training error rate of ˆfS: err S( ˆf S) = 0%. Use test data T to evaluate accuracy of ˆf S. Test error rate of ˆfS: err T ( ˆf S) = 3.09%. Is this good? 8 / 25

20 Diagnostics Some mistakes made by the NN classifier (test point in T, nearest neighbor in S): First mistake (correct label is 2 ) could ve been avoided by looking at the three nearest neighbors (whose labels are 8, 2, and 2 ). test point three nearest neighbors 9 / 25

21 k-nearest neighbors classifier Given: labeled examples D := {(x i, y i)} n i=1 Predictor: ˆfD,k : X Y: On input x, 1. Find the k points x i1, x i2,..., x ik among {x i} n i=1 closest to x (the k nearest neighbors). 2. Return the plurality of y i1, y i2,..., y ik. (Break ties in both steps arbitrarily.) 10 / 25

22 Effect of k Smaller k: smaller training error rate. Larger k: higher training error rate, but predictions are more stable due to voting. OCR digits classification k Test error rate / 25

23 Choosing k The hold-out set approach 1. Pick a subset V S (hold-out set, a.k.a. validation set). 2. For each k {1, 3, 5,... }: Construct k-nn classifier ˆf S\V,k using S \ V. Compute error rate of ˆf S\V,k on V ( hold-out error rate ). 3. Pick the k that gives the smallest hold-out error rate. 12 / 25

24 Other distance functions Lp norm dist(u, v) = x p 1 + x p x p d 1 p 13 / 25

25 Other distance functions Lp norm dist(u, v) = x p 1 + x p x p d 1 p OCR digits classification Distance l 2 l 3 Test error rate 3.09% 2.83% 13 / 25

26 Other distance functions Lp norm Manhattan distance dist(u, v) = x p 1 + x p x p d 1 p OCR digits classification Distance l 2 l 3 Test error rate 3.09% 2.83% dist(u, v) = distance on grid between u and v on strict horizontal/vertical path 13 / 25

27 Other distance functions Lp norm Manhattan distance dist(u, v) = x p 1 + x p x p d 1 p OCR digits classification Distance l 2 l 3 Test error rate 3.09% 2.83% dist(u, v) = distance on grid between u and v on strict horizontal/vertical path String edit distance dist(u, v) = # insertions/deletions/mutations needed to change u to v 13 / 25

28 Bad features Caution: nearest neighbor classifier can be broken by bad/noisy features! Feature 2 y = 0 y = 1 Feature 1 Feature 1 y = 0 y = 1 14 / 25

29 Questions of interest 1. How good is the classifier learned using NN on your problem? 2. Is NN a good learning method in general? 15 / 25

30 Statistical learning theory Basic assumption (main idea): labeled examples {(x i, y i} n i=1 come from same source as future examples. 16 / 25

31 Statistical learning theory Basic assumption (main idea): labeled examples {(x i, y i} n i=1 come from same source as future examples. new (unlabeled) example from P past labeled examples from P learning algorithm learned predictor predicted label More formally: {(x i, y i)} n i=1 is an i.i.d. sample from a probability distribution P over X Y. 16 / 25

32 Prediction error rate Define the (true) error rate of a classifier f : X Y w.r.t. P to be err P (f) := P (f(x) Y ) where (X, Y ) is a pair of random variables with joint distribution P (i.e., (X, Y ) P ). 17 / 25

33 Prediction error rate Define the (true) error rate of a classifier f : X Y w.r.t. P to be err P (f) := P (f(x) Y ) where (X, Y ) is a pair of random variables with joint distribution P (i.e., (X, Y ) P ). Let ˆf S be classifier trained using labeled examples S. 17 / 25

34 Prediction error rate Define the (true) error rate of a classifier f : X Y w.r.t. P to be err P (f) := P (f(x) Y ) where (X, Y ) is a pair of random variables with joint distribution P (i.e., (X, Y ) P ). Let ˆf S be classifier trained using labeled examples S. True error rate of ˆf S is err P ( ˆf S) := P ( ˆf S(X) Y ). 17 / 25

35 Prediction error rate Define the (true) error rate of a classifier f : X Y w.r.t. P to be err P (f) := P (f(x) Y ) where (X, Y ) is a pair of random variables with joint distribution P (i.e., (X, Y ) P ). Let ˆf S be classifier trained using labeled examples S. True error rate of ˆf S is err P ( ˆf S) := P ( ˆf S(X) Y ). We cannot compute this without knowing P. 17 / 25

36 Estimating the true error rate Suppose {(x i, y i) n i=1 (assumed to be an i.i.d. sample from P ) is randomly split into S and T, and ˆf S is based only on S. 18 / 25

37 Estimating the true error rate Suppose {(x i, y i) n i=1 (assumed to be an i.i.d. sample from P ) is randomly split into S and T, and ˆf S is based only on S. ˆfS and T are independent, and the test error rate of ˆf S is an unbiased estimate of the true error rate of ˆf S. 18 / 25

38 Estimating the true error rate Suppose {(x i, y i) n i=1 (assumed to be an i.i.d. sample from P ) is randomly split into S and T, and ˆf S is based only on S. ˆfS and T are independent, and the test error rate of ˆf S is an unbiased estimate of the true error rate of ˆf S. If T = m, then the test error rate err T ( ˆf S) of ˆf S (conditional on S) is a binomial random variable (scaled by 1/m): m err T ( ˆf S) S Bin(m, err P ( ˆf S)). 18 / 25

39 Estimating the true error rate Suppose {(x i, y i) n i=1 (assumed to be an i.i.d. sample from P ) is randomly split into S and T, and ˆf S is based only on S. ˆfS and T are independent, and the test error rate of ˆf S is an unbiased estimate of the true error rate of ˆf S. If T = m, then the test error rate err T ( ˆf S) of ˆf S (conditional on S) is a binomial random variable (scaled by 1/m): m err T ( ˆf S) S Bin(m, err P ( ˆf S)). The expected value of err T ( ˆf S) is err P ( ˆf S). (This means that err T ( ˆf S) is an unbiased estimator of err P ( ˆf S).) 18 / 25

40 Limits of prediction Binary classification: Y = {0, 1}. 19 / 25

41 Limits of prediction Binary classification: Y = {0, 1}. Probability distribution P over X {0, 1}; let (X, Y ) P. 19 / 25

42 Limits of prediction Binary classification: Y = {0, 1}. Probability distribution P over X {0, 1}; let (X, Y ) P. Think of P as being comprised of two parts. 1. Marginal distribution µ of X (a distribution over X ). 2. Conditional distribution of Y given X = x, for each x X : η(x) := P (Y = 1 X = x). 19 / 25

43 Limits of prediction Binary classification: Y = {0, 1}. Probability distribution P over X {0, 1}; let (X, Y ) P. Think of P as being comprised of two parts. 1. Marginal distribution µ of X (a distribution over X ). 2. Conditional distribution of Y given X = x, for each x X : η(x) := P (Y = 1 X = x). If η(x) is 0 or 1 for all x X where µ(x) > 0, then optimal error rate is zero (i.e., min f err P (f) = 0). 19 / 25

44 Limits of prediction Binary classification: Y = {0, 1}. Probability distribution P over X {0, 1}; let (X, Y ) P. Think of P as being comprised of two parts. 1. Marginal distribution µ of X (a distribution over X ). 2. Conditional distribution of Y given X = x, for each x X : η(x) := P (Y = 1 X = x). If η(x) is 0 or 1 for all x X where µ(x) > 0, then optimal error rate is zero (i.e., min f err P (f) = 0). Otherwise it is non-zero. 19 / 25

45 Bayes optimality What is the classifier with smallest true error rate? { f 0 if η(x) 1/2; (x) := 1 if η(x) > 1/2. (Do you see why?) 20 / 25

46 Bayes optimality What is the classifier with smallest true error rate? { f 0 if η(x) 1/2; (x) := 1 if η(x) > 1/2. (Do you see why?) f is called the Bayes (optimal) classifier, and err P (f ) = min err P (f) = E [min { η(x), 1 η(x) }] f which is called the Bayes (optimal) error rate. 20 / 25

47 Bayes optimality What is the classifier with smallest true error rate? { f 0 if η(x) 1/2; (x) := 1 if η(x) > 1/2. (Do you see why?) f is called the Bayes (optimal) classifier, and err P (f ) = min err P (f) = E [min { η(x), 1 η(x) }] f which is called the Bayes (optimal) error rate. Question: How far from optimal is the classifier produced by the NN learning method? 20 / 25

48 Consistency of k-nn We say that a learning algorithm A is consistent if [ lim E err P ( ˆf ] n n) = err(f ), where ˆf n is the classifier learned using A on an i.i.d. sample of size n. Theorem (e.g., Cover and Hart 1967) Assume η is continuous. Then: 1-NN is consistent if min f err P (f) = 0. k-nn is consistent, provided that k := k n is chosen as an increasing but sublinear function of n: k n lim kn =, lim n n n = / 25

49 Key takeaways 1. k-nn learning procedure; role of k, distance functions, features. 2. Training and test error rates. 3. Framework of statistical learning theory; estimating the true error rate; Bayes optimality; high-level idea of consistency. 22 / 25

50 Recap: definitions denote our classifier as f 23 / 25

51 Recap: definitions denote our classifier as f what does f(x) mean? 23 / 25

52 Recap: definitions denote our classifier as f what does f(x) mean? denote the dataset as D 23 / 25

53 Recap: definitions denote our classifier as f what does f(x) mean? denote the dataset as D what is err D(f)? 23 / 25

54 Recap: definitions denote our classifier as f what does f(x) mean? denote the dataset as D what is err D(f)? # of (x, y) D such that f(x) y D 23 / 25

55 Recap: definitions denote our classifier as f what does f(x) mean? denote the dataset as D what is err D(f)? what does P (X, Y ) mean? # of (x, y) D such that f(x) y D 23 / 25

56 Recap: definitions denote our classifier as f what does f(x) mean? denote the dataset as D what is err D(f)? what does P (X, Y ) mean? what is err P (f)? # of (x, y) D such that f(x) y D 23 / 25

57 Recap: definitions denote our classifier as f what does f(x) mean? denote the dataset as D what is err D(f)? what does P (X, Y ) mean? # of (x, y) D such that f(x) y D what is err P (f)? E P [I[f(X) Y ]] 23 / 25

58 Recap: definitions denote our classifier as f what does f(x) mean? denote the dataset as D what is err D(f)? what does P (X, Y ) mean? # of (x, y) D such that f(x) y D what is err P (f)? E P [I[f(X) Y ]] = P (f(x) Y ) 23 / 25

59 Recap: unbiased estimator what is an estimator? 24 / 25

60 Recap: unbiased estimator what is an estimator? using definitions from previous slide, what is err D(f) an estimator of? 24 / 25

61 Recap: unbiased estimator what is an estimator? using definitions from previous slide, what is err D(f) an estimator of? err P (f) 24 / 25

62 Recap: unbiased estimator what is an estimator? using definitions from previous slide, what is err D(f) an estimator of? err P (f) what is an unbiased estimator? 24 / 25

63 Recap: unbiased estimator what is an estimator? using definitions from previous slide, what is err D(f) an estimator of? err P (f) what is an unbiased estimator? E[err D(f)] = err P (f) 24 / 25

64 Recap: unbiased estimator what is an estimator? using definitions from previous slide, what is err D(f) an estimator of? err P (f) what is an unbiased estimator? E[err D(f)] = err P (f) what on earth does that mean? 24 / 25

65 Recap: unbiased estimator what is an estimator? using definitions from previous slide, what is err D(f) an estimator of? err P (f) what is an unbiased estimator? E[err D(f)] = err P (f) what on earth does that mean? if we repeatedly draw datasets of size n, D n P (X, Y ) and calculate err D(f), the average will converge to err P (f) 24 / 25

66 Recap: unbiased estimator what is an estimator? using definitions from previous slide, what is err D(f) an estimator of? err P (f) what is an unbiased estimator? E[err D(f)] = err P (f) what on earth does that mean? if we repeatedly draw datasets of size n, D n P (X, Y ) and calculate err D(f), the average will converge to err P (f) why should we care? 24 / 25

67 Recap: consistent algorithm let ˆf n mean a classifier trained using n examples 25 / 25

68 Recap: consistent algorithm let ˆf n mean a classifier trained using n examples [ what is the property lim n E err P ( ˆf ] n) = err(f )? 25 / 25

69 Recap: consistent algorithm let ˆf n mean a classifier trained using n examples [ what is the property lim n E err P ( ˆf ] n) = err(f )? consistency 25 / 25

70 Recap: consistent algorithm let ˆf n mean a classifier trained using n examples [ what is the property lim n E err P ( ˆf ] n) = err(f )? consistency what is the ideal classifier f also known as? 25 / 25

71 Recap: consistent algorithm let ˆf n mean a classifier trained using n examples [ what is the property lim n E err P ( ˆf ] n) = err(f )? consistency what is the ideal classifier f also known as? the Bayes optimal classifier 25 / 25

72 Recap: consistent algorithm let ˆf n mean a classifier trained using n examples [ what is the property lim n E err P ( ˆf ] n) = err(f )? consistency what is the ideal classifier f also known as? the Bayes optimal classifier how can we define f? 25 / 25

73 Recap: consistent algorithm let ˆf n mean a classifier trained using n examples [ what is the property lim n E err P ( ˆf ] n) = err(f )? consistency what is the ideal classifier f also known as? the Bayes optimal classifier how can we define f? err P (f ) = min f err P (f) = E [min { η(x), 1 η(x) }] 25 / 25