12/2/2009. Announcements. Parametric / Non-parametric. Case-Based Reasoning. Nearest-Neighbor on Images. Nearest-Neighbor Classification

Transcription

1 Introducton to Artfcal Intellgence V Fall 2009 Lecture 24: Nearest-Neghbors & Support Vector Machnes Rob Fergus Dept of Computer Scence, Courant Insttute, NYU Sldes from Danel Yeung, John DeNero and Dan Klen Assgnment 3 graded Assgnment 4 partally graded Assgnment 5 s up. Announcements Fnal exam detals are up on webpage I wll be away next week Prof. Geger wll cover Topcs are: (Mon) Clusterng, (Wed) Computer Vson Need to fll n feedback sheets today (at end) Case-Based Reasonng Smlarty for classfcaton Case-based reasonng Predct an nstance s label usng smlar nstances Nearest-neghbor classfcaton 1-NN: copy the label of the most smlar data pont K-NN: let the k nearest neghbors vote (have to devse a weghtng scheme) Key ssue: how to defne smlarty Trade-off: Small k gves relevant neghbors Large k gves smoother functons Sound famlar? Parametrc / Non-parametrc Parametrc models: Fxed set of parameters More data means better settngs Non-parametrc models: Complexty of the classfer ncreases wth data Better n the lmt, often worse n the non-lmt (K)NN s non-parametrc Truth 2 Examples 10 Examples 100 Examples Examples Nearest-Neghbor Classfcaton Nearest neghbor for dgts: Take new mage Compare to all tranng mages Assgn based on closest example Encodng: mage s vector of ntenstes: What s the smlarty functon? Dot product of two mages vectors? Usually normalze vectors so x = 1 mn = 0 (when?), max = 1 (when?) Nearest-Neghbor on Images 80 mllon mage dataset Sngle descrptor for whole mage Compute Eucldean dstance Need lots of data 1

2 Basc Smlarty Invarant Metrcs Many smlartes based on feature dot products: If features are just the pxels: Note: not all smlartes are of ths form Better dstances use knowledge about vson Invarant metrcs: Smlartes are nvarant under certan transformatons Rotaton, scalng, translaton, stroke-thckness E.g: 16 x 16 = 256 pxels; a pont n 256-dm space Small smlarty n R 256 (why?) How to ncorporate nvarance nto smlartes? Ths and next few sldes adapted from Xao Hu, UIUC Rotaton Invarant Metrcs Tangent Famles Each example s now a curve n R 256 Rotaton nvarant smlarty: s =max s( r( ), r( )) E.g. hghest smlarty between mages rotaton lnes Problems wth s : Hard to compute Allows large transformatons (6 9) Tangent dstance: 1st order approxmaton at orgnal ponts. Easy to compute Models small rotatons Template Deformaton Example of Metrcs for Images Deformable templates: An deal verson of each category Best-ft to mage usng mn varance Cost for hgh dstorton of template Cost for mage ponts beng far from dstorted template Used n many commercal dgt recognzers Examples from [Haste 94] Eucldean dstance Flps, Scalngs, Rotatons, Shear Warp + subwndow shfts 2

3 Metrcs for Images Overvew of Nearest-Neghbor Very smple method Retan all tranng data Can be slow n testng Fndng NN n hgh dmensons s slow Metrcs are very mportant Good baselne Support Vector Machne Bascally a 2-class classfer developed by Vapnk and Chervonenks (1992) Whch lne s optmal? Support Vector Machne Tranng vectors : x, =1.n Consder a smple case wth two classes : Defne a vector y y = 1 f x n class 1 = -1 f x n class 2 Ah hyperplane whch hseparates all lldata ρ Separatng plane Margn Class 1 Class 2 Support Vector (Class 1) Support Vector (Class 2) Lnear Separable SVM Label the tranng data Suppose we have some hyper-planes p whch separates the + from - examples (a separatng hyperplane) Lnear Separable SVM Defne two support hyperplane as H1: w T x = b +δ and H2: w T x = b δ To solve over-parameterzed problem, set δ=1 Defne the dstance between OSH and two support hyperplanes as x whch le on the hyperplane, satsfy w s normal to hyperplane, b / w s the perpendcular dstance from hyperplane to orgn Margn = dstance between H1 and H2 = 2/ w 3

4 The Prmal problem of SVM Goal: Fnd a separatng hyperplane wth largest margn. A SVM s to fnd w and b that satsfy (1) mnmze w /2 = w T w/2 Cost (2) y (x w+b)-1 0 Constrant Use Lagrangan formulaton, mn w.r.t w, α: Only a few α are non-zero: support vectors Langrange Mulpler Method a method to fnd the extremum of a multvarate functon f(x 1,x 2, x n ) subject to the constrant g(x 1,x 2, x n ) = 0 For an extremum of f to exst on g, the gradent of f must lne up wth the gradent of g. for all k = 1,...,n, where the constant λs called the Lagrange multpler The Lagrangan transformaton of the problem s Formng the SVM Dual To have, we need to fnd the gradent of L wth respect to w and b. (1) (2) Substtute them nto Lagrangan form, we have a dual problem SVM Dual Dual has two key propertes: Unque global optmum (can be found by Quadratc Prog.) Data only enters n terms of dot products of pars of ponts Inner product form => Can be generalze to nonlnear case by applyng kernel Depends only on alpha s SVM Classfcaton Form of lnear SVM classfer: h( x) = sgn( α y ( x x )) Form of non-lnear SVM classfer: Non-Lnear Separators General dea: the orgnal feature space can always be mapped to some hgher-dmensonal feature space where the tranng set s separable: Φ: x φ(x) h( x) = sgn( α y K( x x )) K(x j,x ) s Kernel Functon 4

5 Φ s a mappng functon. Kernels Snce the tranng algorthm only depend on data through dot products. We can use a kernel functon K such that Dfferent Types of Kernels Kernels must satsfy certan condtons to be vald kernels Lnear kernel: Quadratc kernel: Kernels mplctly map orgnal vectors to hgher dmensonal spaces, take the dot product there, and hand the result back RBF: nfnte dmensonal representaton Analogous to defnng metrcs n Nearest-Neghbor Lke a soft verson of Nearest-Neghbors Non-separable SVM Real world applcaton usually have no OSH. We need to add an error term ζ. Examples of SVMs wth dfferent Kernels Lnear Polynomal => To gve penalty to error term, defne New Lagrangan form s Examples of SVMs wth dfferent Kernels Radal Bass Functons SVM Overvew Tranng relatvely easy Convergence guarentees Scales well to hgh-dmensonal data Choce of Kernel s crtcal Can end up usng many support vectors, whch makes testng slow Lots of software onlne for t Probably default opton for classfer 5