Announcements. Recognizing object categories. Today 2/10/2016. Recognition via feature matching+spatial verification. Kristen Grauman UT-Austin

Transcription

1 Announcements Recognzng object categores Krsten Grauman UT-Austn Remnder: Assgnment 1 due Feb 19 on Canvas Remnder: Optonal CNN/Caffe tutoral on Monday Feb 15, 5-7 pm Presentatons: Choose paper, coordnate Experment and paper can overlap Be very mndful of tme lmt Last tme: Recognzng nstances Last tme: Recognzng nstances 1. Bascs n feature extracton: flterng 2. Invarant local features 3. Recognzng object nstances Recognton va feature matchng+spatal verfcaton Pros: Ef f ectve when we are able to f nd relable f eatures wthn clutter Great results f or matchng specf c nstances Cons: Scalng wth number of models Spatal v erf caton as post-processng not seamless, expensv e f or large-scale problems Not suted f or category recognton. Today Intro to categorzaton problem Object categorzaton as dscrmnatve classfcaton Boostng + fast face detecton example Nearest neghbors + scene recognton example Support vector machnes + pedestran detecton example Pyramd match kernels, spatal pyramd match Convolutonal neural networks + ImageNet example Some new representatons along the way Rectangular flters GIST HOG Krsten Grauman 1

2 What does recognton nvolve? Detecton: are there people? Fe-Fe L Actvty: What are they dong? Object categorzaton mountan tree banner buldng street lamp people vendor Instance recognton Potala Palace Scene and context categorzaton outdoor cty A partcular sgn 2

3 Vsual Perceptual Object and Recognton Sensory Augmented Tutoral Computng Vsual Perceptual Object and Recognton Sensory Augmented Tutoral Computng Vsual Perceptual Object and Recognton Sensory Augmented Tutoral Computng Vsual Perceptual Object and Recognton Sensory Augmented Tutoral Computng 2/10/2016 Attrbute recognton Object Categorzaton Task Descrpton Gven a small number of tranng mages of a category, recognze a-pror unknown nstances of that category and assgn the correct category label. Whch categores are feasble vsually? made of fabrc gray crow ded Fdo German shepherd dog anmal lvng beng flat K. Grauman, B. Lebe Vsual Object Categores Basc Lev el Categores n human categorzaton [Rosch 76, Lakoff 87] The hghest level at whch category members have smlar perceved shape The hghest level at whch a sngle mental mage reflects the entre category The level at whch human subjects are usually fastest at dentfyng category members The frst level named and understood by chldren The hghest level at whch a person uses smlar motor actons for nteracton wth category members K. Grauman, B. Lebe Vsual Object Categores Basc-lev el categores n humans seem to be defned pr edomnantly vsually. There s evdence that humans (usually) star t wth basc-level categorzaton before dong dentfcaton. Basc-level categorzaton s easer and faster for humans than object dentfcaton! How does ths transfer to automatc classfcaton algorthms? Basc level Indvdual level K. Grauman, B. Lebe Abstract levels dog German shepherd Fdo anmal quadruped cat Doberman cow Other Types of Categores Functonal Categores e.g. chars = somethng you can st on Challenges: robustness Illumnaton Object pose Clutter Occlusons Intra-class appearance Vewpont K. Grauman, B. Lebe 3

4 Challenges: context and human experence Challenges: complexty Mllons of pxels n an mage 30,000 human recognzable object categores 30+ degrees of freedom n the pose of artculated objects (humans) Bllons of mages onlne 144K hours of new vdeo on YouTube daly About half of the cerebral cortex n prmates s devoted to processng vsual nformaton [Felleman and van Essen 1991] Context cues Functon Dy namcs Vdeo credt: J. Davs Challenges: learnng wth Less mnmal supervson More Evoluton of methods Hand-crafted models 3D geometry Hypothesze and algn Hand-crafted features Learned models Data-drven End-to-end learnng of features and models*,** Generc category recognton: basc framework Buld/tran object model Wndow -based object detecton: recap Tranng: 1. Obtan tranng data 2. Defne features 3. Defne classfer (Choose a representaton) Learn or f t parameters of model / classf er Generate canddates n new mage Score the canddates Gven new mage: 1. Slde wndow 2. Score by classfer Feature extracton Tranng examples Car /non-car Classfer Krsten Grauman 4

5 Issues What classfer? Factors n choosng: Generatv e or dscrmnatv e model? Data resources how much tranng data? How s the labeled data prepared? Tranng tme allowance Test tme requrements real-tme? Ft wth the representaton Dscrmnatve classfer constructon Nearest neghbor 10 6 examples Shakhnarovch, Vola, Darrell 2003 Berg, Berg, Malk Support Vector Machnes Guyon, Vapnk Hesele, Serre, Poggo, 2001, Boostng Neural networks Vola, Jones 2001, Torralba et al. 2004, Opelt et al. 2006, LeCun, Bottou, Bengo, Haffner 1998 Rowley, Baluja, Kanade 1998 Condtonal Random Felds McCallum, Fretag, Perera 2000; Kumar, Hebert 2003 Krsten Grauman Krsten Grauman Slde adapted from Antono Torralba Issues What categores are amenable? Wndow-based models: Three landmark case studes Smlar to specfc object matchng, we expect spatal lay out to be f arly rgdly preserv ed. Unlke specfc object matchng, by tranng classf ers we attempt to capture ntra-class v araton or determne requred dscrmnatv e f eatures. Boostng + f ace detecton NN + scene Gst classf caton SVM + person detecton Vola & Jones e.g., Hays & Efros e.g., Dalal & Trggs Krsten Grauman Vola-Jones face detector Boostng ntuton Man dea: Represent local texture wth ef f cently computable rectangular f eatures wthn wndow of nterest Select dscrmnatv e f eatures to be weak classf ers Weak Classfer 1 Use boosted combnaton of them as f nal classf er Form a cascade of such classf ers, rejectng clear negatv es quckly Krsten Grauman Slde credt: Paul Vola 5

6 Boostng llustraton Boostng llustraton Weghts Increased Weak Classfer 2 Boostng: tranng Intally, weght each tranng example equally In each boostng round: Fnd the weak learner that acheves the lowest weghted tranng error Rase weghts of tranng examples msclassfed by current weak learner Compute f nal classf er as lnear combnaton of all weak learners (weght of each learner s drectly proportonal to ts accuracy ) Exact f ormulas f or re-weghtng and combnng weak learners depend on the partcular boostng scheme (e.g., AdaBoost) Slde credt: Lana Lazebnk Boostng: pros and cons Advantages of boostng Integrates classfcaton wth feature selecton Complexty of tranng s lnear n the number of tranng examples Flexblty n the choce of weak learners, boostng scheme Testng s fast Easy to mplement Dsadvantages Needs many tranng examples Often found not to work as well as an alternatve dscrmnatve classfer, support vector machne (SVM) especally for many-class problems Slde credt: Lana Lazebnk Vola-Jones detector: features Ef f cently computable wth ntegral mage: any sum can be computed n constant tme. Rectangular flters Feature output s df f erence between adjacent regons Value at (x,y) s sum of pxels above and to the left of (x,y) Integral mage Computng sum wthn a rectangle Let A,B,C,D be the values of the ntegral mage at the corners of a rectangle Then the sum of orgnal mage values w thn the rectangle can be computed as: sum = A B C + D Only 3 addtons are requred for any sze of rectangle! D C B A Krsten Grauman Lana Lazebnk 6

7 Vsual Perceptual Object and Recognton Sensory Augmented Tutoral Computng 2/10/2016 Vola-Jones detector: features Vola-Jones detector: features Ef f cently computable wth ntegral mage: any sum can be computed n constant tme Av od scalng mages scale f eatures drectly f or same cost Krsten Grauman Rectangular flters Feature output s df f erence between adjacent regons Value at (x,y) s sum of pxels above and to the left of (x,y) Integral mage Krsten Grauman Whch subset of these features should we use to determne f a wndow has a face? Consderng all possble f lter parameters: poston, scale, and ty pe: 180,000+ possble f eatures assocated wth each 24 x 24 wndow Use AdaBoost both to select the nformatve features and to form the classfer Vola-Jones detector: AdaBoost Want to select the sngle rectangle feature and threshold that best separates postve (faces) and negatve (nonfaces) tranng examples, n terms of weghted error. Resultng weak classfer: Vola-Jones Face Detector: Results Frst two features selected Outputs of a possble rectangle feature on faces and non-faces. For next round, reweght the examples accordng to errors, choose another flter/threshold combo. Krsten Grauman Cascadng classfers for detecton Vola-Jones detector: summary Tran cascade of classfers wth AdaBoost Faces New mage Form a cascade wth low f alse negatv e rates early on Apply less accurate but f aster classf ers frst to mmedately dscard wndows that clearly appear to be negatv e Krsten Grauman Non-faces Selected features, thresholds, and weghts Tran wth 5K postves, 350M negatves Real-tme detector usng 38 layer cascade 6061 features n all layers [Implementaton avalable n OpenCV: Krsten Grauman 7

8 Vola-Jones detector: summary Wndow-based models: Three landmark case studes A semnal approach to real-tme object detecton Tranng s slow, but detecton s v ery f ast Key deas Integral mages f or f ast f eature ev aluaton Boostng f or f eature selecton Attentonal cascade of classf ers f or fast rejecton of nonf ace wndows Boostng + f ace detecton NN + scene Gst classf caton SVM + person detecton Vola & Jones e.g., Hays & Efros e.g., Dalal & Trggs P. Vola and M. Jones. Rapd object detecton usng a boosted cascade of smple features. CVPR P. Vola and M. Jones. Robust real-tme face detecton. IJCV 57(2), Nearest Neghbor classfcaton Assgn label of nearest tranng data pont to each test data pont K-Nearest Neghbors classfcaton For a new pont, f nd the k closest ponts f rom tranng data Labels of the k ponts v ote to classf y Black = negatve Red = postve from Duda et al. Nov el test example Closest to a postve example f rom the tranng set, so classfy t as postve. Black = negatve Red = postve k = 5 If query lands here, the 5 NN consst of 3 negatves and 2 postves, so we classfy t as negatve. Vorono parttonng of feature space for 2-category 2D data Source: D. Lowe 80M Tny Images [Torralba et al. 2008] Another nearest neghbor recognton example 8

9 Where n the World? 6+ mllon geotagged photos by 109,788 photographers [Hays and Efros. m2gps: Estmatng Geographc Informaton from a Sngle Image. CVPR 2008.] Annotated by Flckr users Spatal Envelope Theory of Scene Representaton Olva & Torralba (2001) Global texture: capturng the Gst of the scene Capture global mage propertes whle keepng some spatal nf ormaton A scene s a sngle surface that can be represented by global (statstcal) descrptors Slde Credt: Aude Olva Olva & Torralba IJCV 2001, Torralba et al. CVPR 2003 Gst descrptor Whch scene propertes are relevant? Scene Matches Gst scene descrptor Color Hstograms - L*A*B* 4x14x14 hstograms Texton Hstograms 512 entry, flter bank based Lne Features Hstograms of straght lne stats [Hays and Efros. m2gps: Estmatng Geographc Informaton from a Sngle Image. CVPR 2008.] 9

10 Scene Matches [Hays and Efros. m2gps: Estmatng Geographc Informaton from a Sngle Image. CVPR 2008.] The Importance of Data [Hays and Efros. m2gps: Estmatng Geographc Informaton from a Sngle Image. CVPR 2008.] [Hays and Efros. m2gps: Estmatng Geographc Informaton from a Sngle Image. CVPR 2008.] Nearest neghbors: pros and cons Pros: Smple to mplement Flexble to f eature / dstance choces Naturally handles mult-class cases Can do well n practce wth enough representatv e data Cons: Large search problem to f nd nearest neghbors Storage of data Must know we hav e a meanngf ul dstance f uncton Wndow-based models: Three landmark case studes Boostng + f ace detecton Vola & Jones NN + scene Gst classf caton e.g., Hays & Efros SVM + person detecton e.g., Dalal & Trggs Krsten Grauman 10

11 Support Vector Machnes (SVMs) Support vector machnes Want lne that maxmzes the margn. Dscrmnatve classfer based on optmal separatng lne (for 2d case) Maxmze the margn betw een the postve and negatve tranng examples Support vectors x postve ( y 1): x negatve ( y 1): For support vectors: Τ w x b 1 M Margn M w w x w b 1 x w b 1 For support, vectors, x w b 1 Dstance between pont x w b and lne: w w w w Fndng the maxmum margn lne 1. Maxmze margn 2/ w 2. Correctly classfy all tranng data ponts: x postve ( y 1): x negatve ( y 1): x w b 1 x w b 1 Quadratc optmzaton problem: 1 Mnmze w T w 2 Subject to y (w x +b) 1 Fndng the maxmum margn lne Soluton: w y x b = y w x (for any support vector) w x b y x x b Classfcaton functon: f ( x) sgn ( w x b) sgn y x x b C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1 Person detecton wth HoG s & lnear SVM s HoG descrptor Map each grd cell n the nput w ndow to a hstogram countng the gradents per orentaton. Tran a lnear SVM usng tranng set of pedestran vs. non-pedestran w ndows. Dalal & Trggs, CVPR 2005 Code avalable: Dalal & Trggs, CVPR 2005 Code avalable: 11

12 Person detecton wth HoGs & lnear SVMs Non-lnear SVMs Datasets that are lnearly separable wth some nose work out great: 0 x But what are we gong to do f the dataset s just too hard? 0 x How about mappng data to a hgher-dmensonal space: x 2 Hstograms of Orented Gradents for Human Detecton, Navneet Dalal, Bll Trggs, Internatonal Conference on Computer Vson & Pattern Recognton - June x Nonlnear SVMs The kernel trck: nstead of explctly computng the lftng transformaton φ(x), defne a kernel functon K such that K(x,x j j) = φ(x ) φ(x j ) Ths gves a nonlnear decson boundary n the orgnal feature space: yk ( x, x ) b Example 2-dmensonal vectors x=[x 1 x 2 ]; let K(x,x j )=(1 + x T x j ) 2 Need to show that K(x,x j )= φ(x ) T φ(x j ): K(x,x j )=(1 + x T x j ) 2, 2 2 = 1+ x 12 x j1 + 2 x 1 x j1 x 2 x j2 + x 22 x j2 + 2x 1 x j1 + 2x 2 x j2 = [1 x x 1 x 2 x 2 2 2x 1 2x 2 ] T [1 x 2 j1 2 x j1 x j2 x 2 j2 2x j1 2x j2 ] = φ(x ) T φ(x j ), where φ(x) = [1 x x 1 x 2 x 2 2 2x 1 2x 2 ] Examples of kernel functons Lnear: Gaussan RBF: Hstogram ntersecton: K( x, x ) x j T x x j K( x,x j ) exp( 2 2 K ( x, x j ) mn( x ( k), x j ( k)) k x j 2 ) SVMs for recognton 1. Def ne y our representaton f or each example. 2. Select a kernel f uncton. 3. Compute parwse kernel v alues between labeled examples 4. Use ths kernel matrx to solv e f or SVM support v ectors & weghts. 5. To classf y a new example: compute kernel v alues between new nput and support v ectors, apply weghts, check sgn of output. Krsten Grauman 12

13 What about a matchng kernel? Partally matchng sets of features Optmal match: O(m 3 ) Greedy match: O(m 2 log m) Pyramd match: O(m) (m=num pts) We ntroduce an approxmate matchng kernel that makes t practcal to compare large sets of f eatures based on ther partal correspondences. Krsten Grauman Local feature correspondence useful smlarty measure for generc object categores Krsten Grauman [Prevous work: Indyk & Thaper, Bartal, Charkar, Agarwal & Varadarajan, ] Pyramd match: man dea Pyramd match: man dea Feature space parttons serv e to match the local descrptors wthn successv ely wder regons. descrptor space Krsten Grauman Krsten Grauman Hstogram ntersecton counts number of possble matches at a gv en parttonng. Pyramd match kernel Pyramd match kernel Optmal match: O(m 3 ) Pyramd match: O(mL) measures dffculty of a match at level number of newly matched pars at level For smlarty, weghts nv ersely proportonal to bn sze (or may be learned) Normalze these kernel v alues to av od f av orng large sets optmal partal matchng [Grauman & Darrell, ICCV 2005] Krsten Grauman 13

14 Unordered sets of local features: No spatal layout preserved! Spatal pyramd match Make a pyramd of bag-of-words hstograms. Provdes some loose (global) spatal layout nformaton Too much? Too lttle? [Lazebnk, Schmd & Ponce, CVPR 2006] Spatal pyramd match Make a pyramd of bag-of-words hstograms. Provdes some loose (global) spatal layout nformaton Spatal pyramd match Can capture scene categores well---texture-lke patterns but wth some v arablty n the postons of all the local peces. Sum ov er PMKs computed n mage coordnate space, one per word. [Lazebnk, Schmd & Ponce, CVPR 2006] Spatal pyramd match Can capture scene categores well---texture-lke patterns but wth some v arablty n the postons of all the local peces. Senstv e to global shf ts of the v ew Mult-class SVMs Achev e mult-class classf er by combnng a number of bnary classf ers One vs. all Tranng: learn an SVM f or each class v s. the rest Testng: apply each SVM to test example and assgn to t the class of the SVM that returns the hghest decson v alue Confuson table One vs. one Tranng: learn an SVM f or each par of classes Testng: each learned SVM v otes f or a class to assgn to the test example Krsten Grauman 14

15 SVMs: Pros and cons Pros Kernel-based framework s very powerful, flexble Often a sparse set of support vectors compact at test tme Work very well n practce, even wth very small tranng sample szes Basc recognton models so far Cons No drect mult-class SVM, must combne two-class SVMs Can be trcky to select best kernel functon for a problem Computaton, memory Durng tranng tme, must compute matrx of kernel values for every par of examples Learnng can take a very long tme for large-scale problems Adapted from Lana Lazebnk Instances: recognton by algnment Categores: Holstc appearance models (and sldng wndow detecton) Krsten Grauman Summary so far Basc ppelne for wndow-based detecton Model/representaton/classfer choce Sldng wndow and classfer scorng Dscrmnatve classfers for wndow-based representatons Boostng Vola-Jones face detector example Nearest neghbors Scene recognton example 80M Tny Images studes Support v ector machnes HOG person detecton example Pyramd match kernel Hand-crafted models 3D geometry Hypothesze and algn Evoluton of methods Hand-crafted features Learned models Data-drven End-to-end learnng of features and models*,** Next Convolutonal neural networks Guest lecture by Dnesh Jayaraman 15