N pels Scrpt Computer Vson Lecture 4 Recognton wt Local Features We ve created a scrpt for te part of te lecture on object recognton & categorzaton K. Grauman, Vsual Object Recognton Morgan & Clapool publsers, 2 6.2.24 Bastan Lebe RWH acen ttp://www.vson.rwt-aacen.de/ Capter 3: Local Feature Etracton (Last 2 lectures) Capter 4: Matcng (uesda s topc) Capter 5: Geometrc Verfcaton (oda s topc) lebe@vson.rwt-aacen.de valable on te L2P nnouncements(2) Course Outlne Lecture evaluaton Please fll out te forms... Image Processng Bascs Segmentaton & Groupng Object Recognton Object Categorzaton I Sldng Wndow based Object Detecton Local Features & Matcng Local Features Detecton and Descrpton Recognton wt Local Features Indeng & Vsual Vocabulares 3 Object Categorzaton II 3D Reconstructon Moton and rackng 4 Recap: Local Feature Matcng Outlne Recap: utomatc Scale Selecton. Fnd a set of dstnctve keponts Functon responses for ncreasng scale (scale sgnature) 2. Defne a regon around eac kepont N pels f e.g. color Smlart measure d( f, f ) B f B e.g. color 3. Etract and normalze te regon content 4. Compute a local descrptor from te normalzed regon 5. Matc local descrptors 5 f ( I (, )) f ( I (, )) m Slde credt: Krstan Mkolajczk m 6
Resample Blur Subtract Recap: Laplacan-of-Gaussan (LoG) Recap: LoG Detector Responses Interest ponts: Local mama n scale space of Laplacan-of- Gaussan 5 4 L ( ) L ( ) 3 2 Lst of (,, σ) Slde adapted from Krstan Mkolajczk Slde credt: Svetlana Lazebnk 8 Recap: Ke pont localzaton wt DoG Effcent mplementaton ppromate LoG wt a dfference of Gaussans (DoG) Recap: Harrs-Laplace [Mkolajczk ]. Intalzaton: Multscale Harrs corner detecton 2. Scale selecton based on Laplacan (same procedure wt Hessan Hessan-Laplace) pproac: DoG Detector Detect mama of dfferenceof-gaussan n scale space Reject ponts wt low contrast (tresold) Elmnate edge responses Harrs ponts Canddate keponts: lst of (,,σ) Image source: Davd Lowe Slde adapted from Krstan Mkolajczk Harrs-Laplace ponts opcs of s Lecture Local Descrptors Local Descrptors SIF pplcatons Recognton wt Local Features Matcng local features Fndng consstent confguratons lgnment: lnear transformatons ffne estmaton Homograp estmaton We know ow to detect ponts Net queston: How to descrbe tem for matcng?? Dealng wt Outlers RNSC Generalzed Houg ransform Pont descrptor sould be:. Invarant 2. Dstnctve 3 2
Local Descrptors Smplest descrptor: lst of ntenstes wtn a patc. Wat s ts gong to be nvarant to? Feature Descrptors Dsadvantage of patces as descrptors: Small sfts can affect matcng score a lot Soluton: stograms 4 Slde credt: Svetlana Lazebnk 2 p 5 Feature Descrptors: SIF Scale Invarant Feature ransform Descrptor computaton: Dvde patc nto 44 sub-patces: 6 cells Compute stogram of gradent orentatons (8 reference angles) for all pels nsde eac sub-patc Resultng descrptor: 448 = 28 dmensons Overvew: SIF Etraordnarl robust matcng tecnque Can andle canges n vewpont up to ~6 deg. out-of-plane rotaton Can andle sgnfcant canges n llumnaton Sometmes even da vs. ngt (below) Fast and effcent can run n real tme Lots of code avalable ttp://people.csal.mt.edu/albert/ladpack/wk/nde.pp/known_mplementatons_of_sif Davd G. Lowe. "Dstnctve mage features from scale-nvarant keponts. IJCV 6 (2), pp. 9-, 24. Slde credt: Svetlana Lazebnk 6 Slde credt: Steve Setz 7 Workng wt SIF Descrptors Local Descrptors: SURF One mage elds: n 2D ponts gvng postons of te patces [n 2 matr] n scale parameters specfng te sze of eac patc [n vector] n orentaton parameters specfng te angle of te patc [n vector] n 28-dmensonal descrptors: eac one s a stogram of te gradent orentatons wtn a patc [n 28 matr] Fast appromaton of SIF dea Effcent computaton b 2D bo flters & ntegral mages 6 tmes faster tan SIF Equvalent qualt for object dentfcaton ttp://www.vson.ee.etz.c/~surf GPU mplementaton avalable Feature etracton @ Hz (detector + descrptor, 64 48 mg) ttp://omes.esat.kuleuven.be/~ncornel/gpusurf/ Slde credt: Steve Setz 8 9 [Ba, ECCV 6], [Cornels, CVGPU 8] 3
You Can r It t Home For most local feature detectors, eecutables are avalable onlne: ttp://robots.o.ac.uk/~vgg/researc/affne ttp://www.cs.ubc.ca/~lowe/keponts/ ttp://www.vson.ee.etz.c/~surf ttp://omes.esat.kuleuven.be/~ncornel/gpusurf/ 2 ttp://www.robots.o.ac.uk/~vgg/researc/affne/detectors.tml#bnares opcs of s Lecture pplcatons of Local Invarant Features Local Descrptors SIF pplcatons Recognton wt Local Features Matcng local features Fndng consstent confguratons lgnment: lnear transformatons ffne estmaton Homograp estmaton Dealng wt Outlers RNSC Generalzed Houg ransform Wde baselne stereo Moton trackng Panoramas Moble robot navgaton 3D reconstructon Recognton Specfc objects etures Categores 22 23 Wde-Baselne Stereo utomatc Mosacng 24 Image from. utelaars ECCV 26 tutoral 25 [Brown & Lowe, ICCV 3] 4
Panorama Sttcng Recognton of Specfc Objects, Scenes Scmd and Mor 997 Svc and Zsserman, 23 Pone verson ttp://www.cs.ubc.ca/~mbrown/autosttc/autosttc.tml avalable 26 [Brown, Szelsk, and Wnder, 25] Rotganger et al. 23 Lowe 22 27 Recognton of Categores Value of Local Features Constellaton model Bags of words dvantages Crtcal to fnd dstnctve and repeatable local regons for multvew matcng. Complet reducton va selecton of dstnctve ponts. Descrbe mages, objects, parts wtout requrng segmentaton; robustness to clutter & occluson. Robustness: smlar descrptors n spte of moderate vew canges, nose, blur, etc. Weber et al. (2) Fergus et al. (23) Csurka et al. (24) Dorko & Scmd (25) Svc et al. (25) Lazebnk et al. (26), How can we use local features for suc applcatons? Net: matcng and recognton Slde credt: Svetlana Lazebnk 28 Slde adapted from Krsten Grauman 29 opcs of s Lecture Recognton wt Local Features Local Descrptors SIF pplcatons Recognton wt Local Features Matcng local features Fndng consstent confguratons lgnment: lnear transformatons ffne estmaton Homograp estmaton Image content s transformed nto local features tat are nvarant to translaton, rotaton, and scale Goal: Verf f te belong to a consstent confguraton Dealng wt Outlers RNSC Generalzed Houg ransform Local Features, e.g. SIF 3 Slde credt: Davd Lowe 32 5
Concepts: Warpng vs. lgnment Parametrc (Global) Warpng Warpng: Gven a source mage and a transformaton, wat does te transformed output look lke? lgnment: Gven two mages wt correspondng features, wat s te transformaton between tem? 33 p = (,) p = (, ) ransformaton s a coordnate-cangng macne: p = (p) Wat does t mean tat s global? It s te same for an pont p It can be descrbed b just a few numbers (parameters) Let s represent as a matr: p = Mp, Slde credt: leej Efros ' M ' 34 Wat Can be Represented b a 22 Matr? Wat Can be Represented b a 22 Matr? 2D Scalng? ' s * ' s * ' s ' s 2D Mrror about as? ' ' ' ' 2D Rotaton around (,)? ' cos * sn * ' sn * cos * ' cos sn ' sn cos 2D Mrror over (,)? ' ' ' ' 2D Searng? ' s * ' s * ' s ' s 2D ranslaton? ' t ' t NO! Slde credt: leej Efros 35 Slde credt: leej Efros 36 2D Lnear ransforms Onl lnear 2D transformatons can be represented wt a 22 matr. Lnear transformatons are combnatons of Scale, Rotaton, Sear, and Mrror ' a b ' c d Homogeneous Coordnates Q: How can we represent translaton as a 33 matr usng omogeneous coordnates? ' t ' t : Usng te rgtmost column: t ranslaton t Slde credt: leej Efros 37 Slde credt: leej Efros 38 6
Basc 2D ransformatons 2D ffne ransformatons Basc 2D transformatons as 33 matrces ' t ' t ranslaton ' s ' s Scalng ' a b c ' d e f w w ffne transformatons are combnatons of Lnear transformatons, and ranslatons ' cos sn ' sn cos Rotaton ' s ' s Searng Parallel lnes reman parallel Slde credt: leej Efros 39 Slde credt: leej Efros 4 Projectve ransformatons lgnment Problem ' a ' d w' g b e c f w We ave prevousl consdered ow to ft a model to mage evdence E.g., a lne to edge ponts Projectve transformatons: ffne transformatons, and Projectve warps In algnment, we wll ft te parameters of some transformaton accordng to a set of matcng feature pars ( correspondences ). Parallel lnes do not necessarl reman parallel ' Slde credt: leej Efros 4 42 Let s Start wt ffne ransformatons Fttng an ffne ransformaton Smple fttng procedure (lnear least squares) ppromates vewpont canges for rougl planar objects and rougl ortograpc cameras Can be used to ntalze fttng for more comple models ffne model appromates perspectve projecton of planar objects Slde credt: Svetlana Lazebnk 43 44 Image source: Davd Lowe 7
Fttng an ffne ransformaton ssumng we know te correspondences, ow do we get te transformaton? m m3 (, ) m2 t m 4 t2 B, ) ( 45 Recall: Least Squares Estmaton ' ' ' Set of data ponts: ( X, X),( X 2, X 2),( X3, X3) Goal: a lnear functon to predct X s from Xs: ' Xa b X We want to fnd a and b. ' How man ( X, X ) pars do we need? ' ' Xa b X X a X ' ' B X 2a b X 2 X 2 b X 2 Wat f te data s nos? ' X X Overconstraned Soluton: ' problem X 2 a X 2 mn k Bk 2 = + B ' X 3 b X 3 Least-squares Matlab:......... mnmzaton = nb Slde credt: leej Efros 46 Fttng an ffne ransformaton ssumng we know te correspondences, ow do we get te transformaton? m m3 (, ) m2 t m 4 t2 B (, ) m m2 m3 m 4 t t2 47 Fttng an ffne ransformaton m m2 m3 m 4 t t2 How man matces (correspondence pars) do we need to solve for te transformaton parameters? Once we ave solved for te parameters, ow do we compute te coordnates of te correspondng pont for, new )? ( new 48 Homograp Homograp projectve transform s a mappng between an two perspectve projectons wt te same center of projecton. I.e. two planes n 3D along te same sgt ra Propertes Rectangle sould map to arbtrar quadrlateral PP2 Parallel lnes aren t but must preserve stragt lnes s s called a omograp w' * w' * w * p Slde adapted from leej Efros * * * H * * * p PP 49 projectve transform s a mappng between an two perspectve projectons wt te same center of projecton. I.e. two planes n 3D along te same sgt ra Propertes Rectangle sould map to arbtrar quadrlateral Parallel lnes aren t but must preserve stragt lnes s s called a omograp w' 2 3 w' 2 22 23 w 3 32 p H p Slde adapted from leej Efros Set scale factor to 8 parameters left. 8
Fttng a Homograp Estmatng te transformaton Fttng a Homograp Estmatng te transformaton B B B 2 3 Homogenous coordnates ' ' 2 3 2 22 32 3 23 Image coordnates ' ' Matr notaton ' H '' ' B 2 3 Homogenous coordnates ' ' 2 3 2 22 32 3 23 Image coordnates ' ' Matr notaton ' H '' ' Slde credt: Krstan Mkolajczk 5 Slde credt: Krstan Mkolajczk 52 Fttng a Homograp Estmatng te transformaton Fttng a Homograp Estmatng te transformaton B B B 2 3 Homogenous coordnates ' ' 3 23 2B 3 3 32B Slde credt: Krstan Mkolajczk 2 3 2 22 32 Image coordnates ' ' Matr notaton ' H '' ' 53 B 2 3 Homogenous coordnates Image coordnates ' ' 2B 3 3 32B Slde credt: Krstan Mkolajczk 2 3 2 22 32 3 ' 23 ' 2 22B 23 3 B 32 B Matr notaton ' H '' ' 54 Fttng a Homograp Fttng a Homograp Estmatng te transformaton Estmatng te transformaton B B B 2 3 Homogenous coordnates Image coordnates 2B 3 2 22B 23 3 32B 3 32B Slde credt: Krstan Mkolajczk B B B B 3 32 2 3 55 B 2 3 Homogenous coordnates Image coordnates 2B 3 2 22B 23 3 32B 3 32B 3 32 B 2 B 3 2B 3 3 32B 2 22B 23 3 32B Slde credt: Krstan Mkolajczk 56 9
utomatc rectfcaton Fttng a Homograp Estmatng te transformaton 2B 3 3 32B 2 22B 23 3 32B B 2 3... B.. B. B.. B.............. 2 3 2. 22.. 23... 3 32 57 Slde credt: Krstan Mkolajczk. B B. B B Fttng a Homograp Estmatng te transformaton Soluton: Null-space vector of B 2 3 SVD Slde credt: Krstan Mkolajczk d v v9 UDV? U d v v 99 9 99 B 58 Fttng a Homograp Image Warpng wt Homograpes Estmatng te transformaton Soluton: Null-space vector of Corresponds to smallest sngular vector B 2 3 SVD Slde credt: Krstan Mkolajczk d v v9 UDV U d v v 99 9 99 v9,, v v 99 99 B Mnmzes least square error 59 p Slde credt: Steve Setz Image plane n front Black area were no pel maps to p mage plane below 6 Uses: nalzng Patterns and Sapes nalzng Patterns and Sapes Wat s te sape of te b/w floor pattern? From Martn Kemp e Scence of rt (manual reconstructon) Slde credt: ntono Crmns e floor (enlarged) 6 Slde credt: ntono Crmns 62
opcs of s Lecture Problem: Outlers Recognton wt Local Features Matcng local features Fndng consstent confguratons lgnment: lnear transformatons ffne estmaton Homograp estmaton Outlers can urt te qualt of our parameter estmates, e.g., n erroneous par of matcng ponts from two mages feature pont tat s nose or doesn t belong to te transformaton we are fttng. Dealng wt Outlers RNSC Generalzed Houg ransform Indeng wt Local Features Inverted fle nde Vsual Words Vsual Vocabular constructon tf-df wegtng 63 64 Eample: Least-Squares Lne Fttng Outlers ffect Least-Squares Ft ssumng all te ponts tat belong to a partcular lne are known 65 Source: Forst & Ponce 66 Source: Forst & Ponce Outlers ffect Least-Squares Ft Strateg : RNSC [Fscler8] RNdom Smple Consensus pproac: we want to avod te mpact of outlers, so let s look for nlers, and use onl tose. Intuton: f an outler s cosen to compute te current ft, ten te resultng lne won t ave muc support from rest of te ponts. 67 Source: Forst & Ponce 68
RNSC RNSC Lne Fttng Eample RNSC loop: ask: Estmate te best lne. Randoml select a seed group of ponts on wc to base transformaton estmate (e.g., a group of matces) How man ponts do we need to estmate te lne? 2. Compute transformaton from seed group 3. Fnd nlers to ts transformaton 4. If te number of nlers s suffcentl large, recompute least-squares estmate of transformaton on all of te nlers Keep te transformaton wt te largest number of nlers 69 Slde credt: Jnang Ca 7 RNSC Lne Fttng Eample ask: Estmate te best lne RNSC Lne Fttng Eample ask: Estmate te best lne Sample two ponts Ft a lne to tem Slde credt: Jnang Ca 7 Slde credt: Jnang Ca 72 RNSC Lne Fttng Eample ask: Estmate te best lne RNSC Lne Fttng Eample ask: Estmate te best lne 7 nler ponts otal number of ponts wtn a tresold of lne. otal number of ponts wtn a tresold of lne. Slde credt: Jnang Ca 73 Slde credt: Jnang Ca 74 2
RNSC Lne Fttng Eample ask: Estmate te best lne RNSC Lne Fttng Eample ask: Estmate te best lne nler ponts Repeat, untl we get a good result. Repeat, untl we get a good result. Slde credt: Jnang Ca 75 Slde credt: Jnang Ca 76 RNSC: How man samples? RNSC: Computed k (p=.99) How man samples are needed? Suppose w s fracton of nlers (ponts from lne). n ponts needed to defne potess (2 for lnes) k samples cosen. Prob. tat a sngle sample of n ponts s correct: Prob. tat all k samples fal s: n w ( w ) Coose k g enoug to keep ts below desred falure rate. n k Sample sze n Proporton of outlers 5% % 2% 25% 3% 4% 5% 2 2 3 5 6 7 7 3 3 4 7 9 9 35 4 3 5 9 3 7 34 72 5 4 6 2 7 26 57 46 6 4 7 6 24 37 97 293 7 4 8 2 33 54 63 588 8 5 9 26 44 78 272 77 Slde credt: Davd Lowe 77 Slde credt: Davd Lowe 78 fter RNSC RNSC dvdes data nto nlers and outlers and elds estmate computed from mnmal set of nlers. Improve ts ntal estmate wt estmaton over all nlers (e.g. wt standard least-squares mnmzaton). But ts ma cange nlers, so alternate fttng wt reclassfcaton as nler/outler. Eample: Fndng Feature Matces Fnd best stereo matc wtn a square searc wndow (ere 3 pels 2 ) Global transformaton model: eppolar geometr Images from Hartle & Zsserman Slde credt: Davd Lowe 79 Slde credt: Davd Lowe 8 3
Eample: Fndng Feature Matces Fnd best stereo matc wtn a square searc wndow (ere 3 pels 2 ) Global transformaton model: eppolar geometr before RNSC after RNSC Problem wt RNSC In man practcal stuatons, te percentage of outlers (ncorrect putatve matces) s often ver g (9% or above). lternatve strateg: Generalzed Houg ransform Images from Hartle & Zsserman Slde credt: Davd Lowe 8 Slde credt: Svetlana Lazebnk 82 Strateg 2: Generalzed Houg ransform Strateg 2: Generalzed Houg ransform Suppose our features are scale- and rotaton-nvarant en a sngle feature matc provdes an algnment potess (translaton, scale, orentaton). Suppose our features are scale- and rotaton-nvarant en a sngle feature matc provdes an algnment potess (translaton, scale, orentaton). Of course, a potess from a sngle matc s unrelable. Soluton: let eac matc vote for ts potess n a Houg space wt ver coarse bns. model model Slde credt: Svetlana Lazebnk 83 Slde credt: Svetlana Lazebnk 84 Pose Clusterng and Verfcaton wt SIF Indeng Local Features o detect nstances of objects from a model base:. Inde descrptors Dstnctve features narrow down possble matces New mage 85 Image source: Davd Lowe Model base Image source: Davd Lowe 4
Pose Clusterng and Verfcaton wt SIF Object Recognton Results o detect nstances of objects from a model base:. Inde descrptors Dstnctve features narrow down possble matces 2. Generalzed Houg transform to vote for poses Keponts ave record of parameters relatve to model coordnate sstem 3. ffne ft to ceck for agreement between model and mage features Ft and verf usng features from Houg bns wt 3+ votes 87 Image source: Davd Lowe Background subtract for model boundares Objects recognzed Recognton n spte of occluson 88 Image source: Davd Lowe Locaton Recognton ranng Recall: Dffcultes of Votng Nose/clutter can lead to as man votes as true target. Bn sze for te accumulator arra must be cosen carefull. (Recall Houg ransform) In practce, good dea to make broad bns and spread votes to nearb bns, snce verfcaton stage can prune bad vote peaks. [Lowe, IJCV 4] 89 Slde credt: Davd Lowe 9 Summar References and Furter Readng Recognton b algnment: lookng for object and pose tat fts well wt mage Use good correspondences to desgnate poteses. Invarant local features offer more relable matces. Fnd consstent nler confguratons n clutter Generalzed Houg ransform RNSC detaled descrpton of local feature etracton and recognton can be found n Capters 3-5 of Grauman & Lebe (avalable on te L2P). K. Grauman, Vsual Object Recognton Morgan & Clapool publsers, 2 lgnment approac to recognton can be effectve f we fnd relable features wtn clutter. pplcaton: large-scale mage retreval pplcaton: recognton of specfc (mostl planar) objects Move posters Books CD covers 92 R. Hartle,. Zsserman Multple Vew Geometr n Computer Vson 2nd Ed., Cambrdge Unv. Press, 24 More detals on RNSC can also be found n Capter 4.7 of Hartle & Zsserman. 5