Nearest Neighbor Learning

Transcription

1 Nearest Neighbor Learning Cassify based on oca simiarity Ranges from simpe nearest neighbor to case-based and anaogica reasoning Use oca information near the current query instance to decide the cassification of that instance As such can represent quite compex decision surfaces in a simpe manner Loca mode vs a mode such as an MLP which uses a goba decision surface CS Instance Based Learning 1

2 k-nearest Neighbor Approach Simpy store a (or some representative subset) of the exampes in the training set When desiring to generaize on a new instance, measure the distance from the new instance to a the stored instances and the nearest ones vote to decide the cass of the new instance No need to pre-process a specific hypothesis (Lazy vs. Eager earning) Fast earning Can be sow during execution and require significant storage Some modes index the data or reduce the instances stored to enhance efficiency

3 k-nearest Neighbor (cont.) Naturay supports rea vaued attributes Typicay use Eucidean distance dist(x,y) = (x i y i ) 2 m i=1 Nomina/unknown attributes can just be a 1/0 distance (more on other distance metrics ater) The output cass for the query instance is set to the most common cass of its k nearest neighbors ^ f (xq ) = argmax v V where d(x,y) = 1 if x = y, ese 0 k greater than 1 is more noise resistant, but a arge k woud ead to ess accuracy as ess reevant neighbors have more infuence (common vaues: k=3, k=5) Often choose k by Cross Vaidation (trying different vaues for a task) k i=1 δ(v, f (x i )) 3

4 Decision Surface Linear decision boundary between 2 cosest points of different casses for 1-nn CS Instance Based Learning 4

5 Decision Surface Combining a the appropriate intersections gives a Voronoi diagram Eucidean distance each point a unique cass Same points - Manhattan distance CS Instance Based Learning 5

6 k-nearest Neighbor (cont.) Usuay do distance weighted voting where the strength of a neighbor's infuence is proportiona to its distance ^ f (xq ) = argmax w i δ(v, f (x i )) w i = v V k i=1 Inverse of distance squared is a common weight Gaussian is another common distance weight In this case k vaue more robust, coud et k be even and/or be arger (even a points if desired), because the more distant points have negigibe infuence anyway f (x) = w 0 + h i=1 w i K(d(x i, x)) 1 dist(x q,x i ) 2 CS Instance Based Learning 6

7 k-nearest Neighbor Homework Assume the foowing training set. Assume a new point (.5,.2) For nearest neighbor distance use Manhattan distance What woud the output be for 3-nn with no distance weighting? Show work and vote. What woud the output be for 3-nn with distance weighting? Show work and vote. x y Target.3.8 A B.9 0 B 1 1 A CS Homework 7

8 Regression with k-nn Can do regression by etting the output be mean of the k nearest neighbors CS Instance Based Learning 8

9 Weighted Regression with k-nn Can do weighted regression by etting the output be the weighted mean of the k nearest neighbors For distance weighted regression ^ f (xq ) = k i=1 w i f (x i ) k i=1 w i w i = 1 dist(x q,x i ) 2 Where f(x) is the output vaue for instance x CS Instance Based Learning 9

10 Regression Exampe ^ f (xq ) = k i=1 w i f (x i ) k i=1 w i w i = 1 dist(x q,x i ) 2 What is the vaue of the new instance? Assume dist(x q, n 8 ) = 2, dist(x q, n 5 ) = 3, dist(x q, n 3 ) = 4 f(x q ) = (8/ / /4 2 )/(1/ / /4 2 ) = 2.74/.42 = 6.5 The denominator renormaizes the vaue CS Instance Based Learning 10

11 Attribute Weighting One of the main weaknesses of nearest neighbor is irreevant features, since they can dominate the distance Exampe: assume 2 reevant and 10 irreevant features Can create agorithms which weight the attributes (Note that backprop and ID3 etc. do higher order weighting of features) Coud do attribute weighting - No onger azy evauation since you need to come up with a portion of your hypothesis (attribute weights) before generaizing Sti an open area of research Higher order weighting 1 st order heps, but not enough Even if a features are reevant features, a distances become simiar as number of features increases, since not a features are reevant at the same time, and the currenty irreevant ones can dominate distance Probem with a pure distance based techniques, need higher-order weighting to ignore currenty irreevant features What is the best method, etc.? important research area Dimensionaity reduction can be usefu (feature pre-processing, PCA, NLDR, etc.) CS Instance Based Learning 11

12 Reduction Techniques Create a subset or other representative set of prototype nodes Faster execution, and coud even improve accuracy if noisy instances removed Approaches Leave-one-out reduction - Drop instance if it woud sti be cassified correcty Growth agorithm - Ony add instance if it is not aready cassified correcty - both order dependent, simiar resuts More goba optimizing approaches Just keep centra points ower accuracy (mosty inear Voronoi decision surface), best space savings Just keep border points, best accuracy (pre-process noisy instances Drop5) Drop 5 (Wison & Martinez) maintains amost fu accuracy with approximatey 15% of the origina instances Wison, D. R. and Martinez, T. R., Reduction Techniques for Exempar-Based Learning Agorithms, Machine Learning Journa, vo. 38, no. 3, pp , CS Instance Based Learning 12

13 CS Instance Based Learning 13

14 k-nearest Neighbor Notes Note that fu "Leave one out" CV is easy with k-nn Very powerfu yet simpe scheme which does we on many tasks Overfitting handed by just using arger k Strugges with irreevant inputs Needs better incorporation of feature weighting schemes Issues with distance with very high dimensionaity tasks So many features wash out effects of the specificay important ones (akin to the irreevant feature probem) May need distance metrics other than Eucidean distances Aso can be beneficia to reduce tota # of instances Efficiency Sometimes accuracy CS Instance Based Learning 14

15 Instance Based Learning Assignment See Learning Suite Regression part Normaize output vaues? No need Wi change MSE C++ defaut is RMSE? Don't normaize outputs so that we can have consistent MSEs CS Instance Based Learning 15

16 Distance Metrics Wison, D. R. and Martinez, T. R., Improved Heterogeneous Distance Functions, Journa of Artificia Inteigence Research, vo. 6, no. 1, pp. 1-34, Normaization of features - critica Don't know vaues in nove or data set instances Can do some type of imputation and then norma distance Or have a distance (between 0-1) for don't know vaues Origina main question: How best to hande nomina features CS Instance Based Learning 16

17 CS Instance Based Learning 17

18 Vaue Difference Metric Assume a 2 output cass (A,B) exampe Attribute 1 = Shape (Round, Square, Triange, etc.) 10 tota round instances 6 cass A and 4 cass B 5 tota square instances 3 cass A and 2 cass B Since both attribute vaues suggest the same probabiities for the output cass, the distance between Round and Square woud be 0 If triange and round suggested very different outputs, triange and round woud have a arge distance Distance of two attribute vaues is a measure of how simiar they are in inferring the output cass CS Instance Based Learning 18

19 CS Instance Based Learning 19

20 CS Instance Based Learning 20

21 CS Instance Based Learning 21

22 CS Instance Based Learning 22

23 CS Instance Based Learning 23

24 IVDM Distance Metrics make a difference IVDM aso heps dea with the many/irreevant feature probem of k-nn, because features ony add significanty to the overa distance if that distance eads to different outputs Two features which tend to ead to the same output probabiities (exacty what irreevant features shoud do) wi have 0 or itte distance, whie their Eucidean distance coud have been significanty arger Need to take it further to find distance approaches taking into account higher order combinations between features in the distance metric CS Instance Based Learning 24

25 Radia Basis Function Networks f (x) = w 0 + h i=1 w i K(d(µ i, x)) Where K is RBF (kerne function) and d is distance 25

26 Radia Basis Function (RBF) Networks One inear output node per cass with weights and bias Each hidden (prototype) node computes the distance from itsef to the input instance (Gaussian is common) not ike an MLP hidden node An arbitrary number of prototype nodes form a hidden ayer in the Radia Basis Function network prototype nodes typicay non-adaptive The prototype ayer expands the input space into a new prototype space. Transates the data set into a new set with more features Output ayer weights are earned with a simpe inear mode (e.g. deta rue) Thus, output nodes earn 1 st order prototype weightings for each cass A B y 3 4 x y CS Instance Based Learning 26 x

27 Radia Basis Function Networks Neura Network variation of nearest neighbor agorithm Output ayer execution and weight earning Highest node/cass net vaue wins (or can output confidences) Each node coects weighted votes from prototypes unearned weighting from distance (prototype activation ike k-nn), but unike k-nn, vote vaue is earned Weight earning - Deta rue variations or direct matrix weight cacuation inear or non-inear node activation function Coud use an MLP at the top ayer if desired Key Issue: How many prototype nodes shoud there be and where shoud they be paced (means) Prototype node sphere of infuence Kerne basis function (deviation) ike choosing k for k-nn Too sma ess generaization, shoud have some overap Too arge - saturation, ose oca effects, onger training CS Instance Based Learning 27

28 Node Pacement Random Coverage - Prototypes potentiay paced in areas where instances don't occur, Curse of dimensionaity One prototype node for each instance of the training set Random subset of training set instances Custering - Unsupervised or supervised - k-means stye vs. constructive Genetic Agorithms Node adjustment Adaptive prototypes (Competitive Learning stye) Dynamic addition and deetion of prototype nodes CS Instance Based Learning 28

29 RBF Homework Assume you have an RBF with Two inputs Three output casses A, B, and C (inear units) Three prototype nodes at (0,0), (.5,1) and (1,.5) The radia basis function of the prototype nodes is max(0, 1 Manhattan distance between the node and the instance in question) Assume no bias and initia weights of.6 into output node A, -.4 into output node B, and 0 into output node C Assume top ayer training is the deta rue with LR =.1 Assume we input the singe instance.6.8 Which cass woud be the winner? What woud the weights be updated to if it were a training instance of.6.8 with target cass B? (thus B has target 1 and A has target 0) CS Homework 29

30 RBF vs. BP Line vs. Sphere - mix-and-match approaches Mutipe spheres sti create Voronoi decision surfaces Potentia Faster Training - nearest neighbor ocaization - Yet more data and hidden nodes typicay needed Loca vs Distributed, ess extrapoation (aa BP), have reject capabiity (avoid fase positives) RBF wi have probems with irreevant features just ike nearest neighbor (or any distance based approach which treats a inputs equay) Coud be improved by adding earning into the prototype ayer to earn attribute weighting CS Instance Based Learning 30

31 Distributed vs Loca MLP vs K-NN (RBF) exponentia vs inear representation potentia but how useabe is it? overfit, exponentia training data? Which is best is an open question. Beow are decision surfaces for MLP with 3 hidden nodes, and K-NN with 3 nodes CS Instance Based Learning 31