Instance Based Learning

Transcription

1 Instance Based Learning Vibhav Ggate The University f Texas at Dallas Readings: Mitchell, Chapter 8 surces: curse slides are based n material frm a variety f surces, including Tm Dietterich, Carls Guestrin, Ray Mney, Andrew Mre, Andrew Ng, Padhraic Smyth and thers.

2 Instance Based Learning k-nearest Neighbr Lcally weighted Linear regressin

3 Sme Vcabulary Parametric vs. Nn-parametric: parametric: A particular functinal frm is assumed, e.g., multivariate nrmal, naïve Bayes. Advantage f simplicity easy t estimate and interpret may have high bias because the real data may nt bey the assumed functinal frm. nn-parametric: distributin r density estimate is data-driven and relatively few assumptins are made a priri abut the functinal frm. Other terms: Instance-based, Memry-based, Lazy, Case-based, kernel methds

4 K-Nearest Neighbr Algrithm

5 K-Nearest Neighbr: Example T classify a new input vectr x, examine the k-clsest training data pints t x and assign the bject t the mst frequently ccurring class k=1 x k=5 cmmn values fr k: 3, 5

6 Decisin Bundaries The nearest neighbr algrithm des nt explicitly cmpute decisin bundaries. Hwever, the decisin bundaries frm a subset f the Vrni diagram fr the training data. 1-NN Decisin Surface Each line segment is equidistant between tw pints f ppsite classes. The mre examples that are stred, the mre cmplex the decisin bundaries can becme.

7 Distance-Weighted k-nn

8 Issues What Distance measure t use? Hw t speed up Classificatin? K-NN is a memry-based technique. Must make a pass thrugh the data fr each classificatin. This can be prhibitive fr large data sets. Disadvantages Curse f Dimensinality In high-dimensinal spaces, prblem that the nearest neighbr may nt be very clse at all! Irrelevant Attributes Easily fled by irrelevant attributes.

9 Distance Metrics Euclidean Distance

10 Euclidean Distance: Prblems with Scaling If we scale each attribute arbitrarily, nearest pints may becme farthest pints and vice versa. Example: multiply sme c-rdinate f each pint by an arbitrary cnstant. Scale x crdinate f each pint by 1/3 Black nearest t Blue Red nearest t Blue

11 Euclidean Distance: Practical Cnsideratins

12 Generalizatin f Euclidean Distance

13 The Curse f Dimensinality Nearest neighbr breaks dwn in high-dimensinal spaces because the neighbrhd becmes very large. Suppse we have 5000 pints unifrmly distributed in the unit hypercube and we want t apply the 5-nearest neighbr algrithm. Suppse ur query pint is at the rigin. 1D On a ne dimensinal line, we must g a distance f 5/5000 = n average t capture the 5 nearest neighbrs 2D In tw dimensins, we must g sqrt(0.001) t get a square that cntains f the vlume D In d dimensins, we must g (0.001) 1/d

14 Curse f Dimensinality cnt. With 5000 pints in 10 dimensins, we must g distance alng each attribute in rder t find the 5 nearest neighbrs!

15 K-NN and irrelevant features + + +?

16 K-NN and irrelevant features +?

17 Efficient Indexing: Kd-trees A kd-tree is similar t a decisin tree, except that we split using the median value alng the dimensin having the highest variance, and pints are stred at the leaves. (See Wikipedia article)!

18 Edited Nearest Neighbr String all f the training examples can require a huge amunt f memry. Select a subset f pints that still give gd classificatins. Incremental deletin. Lp thrugh the training data and test each pint t see if it can be crrectly classified given the ther pints. If s, delete it frm the data set. Incremental grwth. Start with an empty data set. Add each pint t the data set nly if it is nt crrectly classified by the pints already stred.

19 KNN Advantages Easy t prgram N ptimizatin r training required Classificatin accuracy can be very gd; can utperfrm mre cmplex mdels

20 Nearest Neighbr Summary Advantages variable-sized hypthesis space Learning is extremely efficient hwever grwing a gd kd-tree can be expensive Very flexible decisin bundaries Disadvantages distance functin must be carefully chsen Irrelevant r crrelated features must be eliminated Typically cannt handle mre than 30 features Cmputatinal csts: Memry and classificatin-time cmputatin

21 Lcally Weighted Linear Regressin: LWLR Idea: k-nn frms lcal apprximatin fr each query pint x Why nt frm an explicit apprximatin f fr regin surrunding x Fit linear functin t k nearest neighbrs Fit quadratic,... Thus prducing ``piecewise apprximatin'' t f Minimize errr ver k nearest neighbrs f x Minimize errr entire set f examples, weighting by distances Cmbine tw abve

22 LWLR: Cntinued

23 LWR Example Lcally-weighted regressin (f2) f1 (simple regressin) Lcally-weighted regressin (f4) Lcally-weighted regressin (f3) Training data Predicted value using simple regressin Predicted value using lcally weighted (piece-wise) regressin Yike Gu, Advanced Knwledge Management, 2000

24 Lazy and Eager Learning Lazy: wait fr query befre generalizing k-nearest Neighbr Eager: generalize befre seeing query ID3, Backprpagatin, etc. Des it matter? Eager learner must create glbal apprximatin Lazy learner can create many lcal apprximatins If they use same H, lazy can represent mre cmplex functins

25 What yu need t knw Instance-based learning nn-parametric trade decreased learning time fr increased classificatin time Issues apprpriate distance metrics curse f dimensinality efficient indexing