4/14/011 Outlne Dscrmnatve classfers for mage recognton Wednesday, Aprl 13 Krsten Grauman UT-Austn Last tme: wndow-based generc obect detecton basc ppelne face detecton wth boostng as case study Today: dscrmnatve classfers for mage recognton nearest neghbors (+ scene match app) support vector machnes (+ gender, person app) Nearest Neghbor classfcaton Assgn label of nearest tranng data pont to each test data pont K-Nearest Neghbors classfcaton For a new pont, fnd the k closest ponts from tranng data Labels of the k ponts vote to classfy Black = negatve Red = postve from Duda et al. Novel test example Closest to a postve example from the tranng set, so classfy t as postve. Black = negatve Red = postve k= 5 If query lands here, the 5 NN consst of 3 negatves and postves, so we classfy t as negatve. Vorono parttonng of feature space for -category D data Source: D. Lowe Where n the World? A nearest neghbor recognton example [Hays and Efros. mgps: Estmatng Geographc Informaton from a Sngle Image. CVPR 008.] Sldes: James Hays CS 376 Lecture 1
4/14/011 Where n the World? Where n the World? Sldes: James Hays Sldes: James Hays 6+ mllon geotagged photos by 109,788 photographers 6+ mllon geotagged photos by 109,788 photographers Annotated by Flckr users Sldes: James Hays Annotated by Flckr users Sldes: James Hays Whch scene propertes are relevant? Spatal Envelope Theory of Scene Representaton Olva & Torralba (001) A scene s a sngle surface that can be represented by global (statstcal) descrptors Slde Credt: Aude Olva CS 376 Lecture
4/14/011 Global texture: capturng the Gst of the scene Capture global mage propertes whle keepng some spatal nformaton Whch scene propertes are relevant? Gst scene descrptor Color Hstograms L*A*B* 4x14x14 hstograms Texton Hstograms 51 entry, flter bank based Lne Features Hstograms of straght lne stats Gst descrptor Olva & Torralba IJCV 001, Torralba et al. CVPR 003 Scene Matches [Hays and Efros. mgps: Estmatng Geographc Informaton from a Sngle Image. CVPR 008.] Sldes: James Hays Sldes: James Hays Scene Matches [Hays and Efros. mgps: Estmatng Geographc Informaton from a Sngle Image. CVPR 008.] Sldes: James Hays [Hays and Efros. mgps: Estmatng Geographc Informaton from a Sngle Image. CVPR 008.] Sldes: James Hays CS 376 Lecture 3
4/14/011 Scene Matches [Hays and Efros. mgps: Estmatng Geographc Informaton from a Sngle Image. CVPR 008.] Sldes: James Hays [Hays and Efros. mgps: Estmatng Geographc Informaton from a Sngle Image. CVPR 008.] Sldes: James Hays The Importance of Data Feature Performance [Hays and Efros. mgps: Estmatng Geographc Informaton from a Sngle Image. CVPR 008.] Sldes: James Hays Sldes: James Hays Nearest neghbors: pros and cons Pros: Smple to mplement Flexble to feature / dstance choces Naturally handles mult-class cases Can do well n practce wth enough representatve data Cons: Large search problem to fnd nearest neghbors Storage of data Must know we have a meanngful dstance functon Outlne Dscrmnatve classfers Boostng (last tme) Nearest neghbors Support vector machnes CS 376 Lecture 4
4/14/011 Lnear classfers Lnes n R a Let w c x x y ax cy b 0 Lnes n R a x Let w x c y w ax cy b 0 x 0, y 0 D Lnes n R a x Let w x c y w ax cy b 0 w x b 0 w x b 0 x 0, y 0 D Lnes n R a x Let w x c y w ax cy b 0 x 0, y 0 D Lnes n R a x Let w x c y w ax cy b 0 w x b 0 w x b 0 D ax0 cy0 b w x b dstance from a c w pont to lne D ax0 cy0 b w x b dstance from a c w pont to lne CS 376 Lecture 5
4/14/011 Lnear classfers Fnd lnear functon to separate postve and negatve examples x postve : x negatve : x w b 0 x w b 0 Support Vector Machnes (SVMs) Dscrmnatve classfer based on optmal separatng lne (for d case) Whch lne s best? Maxmze the margn between the postve and negatve tranng examples Support vector machnes Want lne that maxmzes the margn. Support vector machnes Want lne that maxmzes the margn. x postve ( y 1) : x negatve( y 1) : x w b 1 x w b 1 x postve ( y 1) : x negatve( y 1) : x w b 1 x w b 1 For support, vectors, x w b 1 For support, vectors, x w b 1 Support vectors Margn Support vectors Margn M Dstance between pont x w b and lne: w For support vectors: Τ w x b 1 1 1 M w w w w w C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1998 Support vector machnes Want lne that maxmzes the margn. x postve ( y 1) : x negatve( y 1) : x w b 1 x w b 1 For support, vectors, x w b 1 Dstance between pont x w b and lne: w Therefore, the margn s / w Support vectors Margn M Fndng the maxmum margn lne 1. Maxmze margn / w. Correctly classfy all tranng data ponts: x postve ( y 1) : x negatve( y 1) : x w b 1 x w b 1 Quadratc optmzaton problem: 1 Mnmze w T w Subect to y (w x +b) 1 CS 376 Lecture 6
4/14/011 Fndng the maxmum margn lne Soluton: learned weght w y x Support vector Fndng the maxmum margn lne Soluton: w yx b = y w x (for any support vector) w x b y x x b Classfcaton functon: f ( x) sgn ( w x b) sgn x x b If f(x) < 0, classfy as negatve, f f(x) > 0, classfy as postve C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1 Questons What f the features are not d? What f the data s not lnearly separable? What f we have more than ust two categores? Questons What f the features are not d? Generalzes to d dmensons replace lne wth hyperplane What f the data s not lnearly separable? What f we have more than ust two categores? Person detecton wth HoG s & lnear SVM s Person detecton wth HoG s & lnear SVM s Map each grd cell n the nput wndow to a hstogram countng the gradents per orentaton. Tran a lnear SVM usng tranng set of pedestran vs. non-pedestran wndows. Dalal & Trggs, CVPR 005 Code avalable: http://pascal.nralpes.fr/soft/olt/ Hstograms of Orented Gradents for Human Detecton, Navneet Dalal, Bll Trggs, Internatonal Conference on Computer Vson & Pattern Recognton - June 005 http://lear.nralpes.fr/pubs/005/dt05/ CS 376 Lecture 7
4/14/011 Questons What f the features are not d? What f the data s not lnearly separable? What f we have more than ust two categores? Non lnear SVMs Datasets that are lnearly separable wth some nose work out great: 0 x But what are we gong to do f the dataset s ust too hard? 0 x How about mappng data to a hgher-dmensonal space: x 0 x Non lnear SVMs: feature spaces General dea: the orgnal nput space can be mapped to some hgher-dmensonal feature space where the tranng set s separable: Φ: x φ(x) The Kernel Trck The lnear classfer reles on dot product between vectors K(x,x )=x T x If every data pont s mapped nto hgh-dmensonal space va some transformaton Φ: x φ(x), the dot product becomes: K(x,x )= φ(x ) T φ(x ) A kernel functon s smlarty functon that corresponds to an nner product n some expanded feature space. Slde from Andrew Moore s tutoral: http://www.autonlab.org/tutorals/svm.html Slde from Andrew Moore s tutoral: http://www.autonlab.org/tutorals/svm.html Example -dmensonal vectors x=[x 1 x ]; let K(x,x )=(1 + x T x ) Need to show that K(x,x )= φ(x ) T φ(x ): K(x,x )=(1 + x T x ), = 1+ x 1 x 1 + x 1 x 1 x x + x x + x 1 x 1 + x x = [1 x 1 x 1 x x x 1 x ] T [1 x 1 x 1 x x x 1 x ] = φ(x ) T φ(x ), where φ(x) = [1 x 1 x 1 x x x 1 x ] from Andrew Moore s tutoral: http://www.autonlab.org/tutorals/svm.html Nonlnear SVMs The kernel trck: nstead of explctly computng the lftng transformaton φ(x), defne a kernel functon K such that K(x,x ) = φ(x ) φ(x ) Ths gves a nonlnear decson boundary n the orgnal feature space: yk ( x, x ) b CS 376 Lecture 8
4/14/011 Examples of kernel functons Lnear: Gaussan RBF: Hstogram ntersecton: K( x, x ) x T x x K( x,x ) exp( K ( x, x ) mn( x ( k), x ( k)) k x ) SVMs for recognton 1. Defne your representaton for each example.. Select a kernel functon. 3. Compute parwse kernel values between labeled examples 4. Use ths kernel matrx to solve for SVM support vectors & weghts. 5. To classfy a new example: compute kernel values between new nput and support vectors, apply weghts, check sgn of output. Example: learnng gender wth SVMs Face algnment processng Moghaddam and Yang, Learnng Gender wth Support Faces, TPAMI 00. Processed faces Moghaddam and Yang, Face & Gesture 000. Moghaddam and Yang, Learnng Gender wth Support Faces, TPAMI 00. Learnng gender wth SVMs Support Faces Tranng examples: 1044 males 713 females Experment wth varous kernels, select Gaussan RBF x x K( x, x ) exp( ) Moghaddam and Yang, Learnng Gender wth Support Faces, TPAMI 00. CS 376 Lecture 9
4/14/011 Gender percepton experment: How well can humans do? Subects: 30 people ( male, 8 female) Ages md-0 s to md-40 s Test data: 54 face mages (6 males, 4 females) Low res and hgh res versons Task: Classfy as male or female, forced choce No tme lmt Moghaddam and Yang, Learnng Gender wth Support Faces, TPAMI 00. Moghaddam and Yang, Face & Gesture 000. Gender percepton experment: How well can humans do? Human vs. Machne SVMs performed better than any sngle human test subect, at ether resoluton Error Error Moghaddam and Yang, Face & Gesture 000. Hardest examples for humans Questons What f the features are not d? What f the data s not lnearly separable? What f we have more than ust two categores? Moghaddam and Yang, Face & Gesture 000. CS 376 Lecture 10
4/14/011 Mult-class SVMs Acheve mult-class classfer by combnng a number of bnary classfers One vs. all Tranng: learn an SVM for each class vs. the rest Testng: apply each SVM to test example and assgn to t the class of the SVM that returns the hghest decson value One vs. one Tranng: learn an SVM for each par of classes Testng: each learned SVM votes for a class to assgn to the test example SVMs: Pros and cons Pros Many publcly avalable SVM packages: http://www.kernel-machnes.org/software http://www.cse.ntu.edu.tw/~cln/lbsvm/ Kernel-based framework s very powerful, flexble Often a sparse set of support vectors compact at test tme Work very well n practce, even wth very small tranng sample szes Cons No drect mult-class SVM, must combne two-class SVMs Can be trcky to select best kernel functon for a problem Computaton, memory Durng tranng tme, must compute matrx of kernel values for every par of examples Learnng can take a very long tme for large-scale problems Adapted from Lana Lazebnk Comng up Part-based models Vdeo processng: moton, trackng, actvty CS 376 Lecture 11