Computer Vision Lecture 12

Transcription

1 N pels Course Outlne Computer Vson Lecture 2 Recognton wt Local Features 5226 Bastan Lebe RWH acen ttp://wwwvsonrwt-aacende/ lebe@vsonrwt-aacende Image Processng Bascs Segmentaton & Groupng Object Recognton Object Categorzaton I Sldng Wndow based Object Detecton Local Features & Matcng Local Features Detecton and Descrpton Recognton wt Local Features Indeng & Vsual Vocabulares Object Categorzaton II 3D Reconstructon Moton and rackng 3 Recap: Local Feature Matcng Outlne Recap: utomatc Scale Selecton Fnd a set of dstnctve keponts Functon responses for ncreasng scale (scale sgnature) 2 Defne a regon around eac kepont N pels f eg color Smlart measure d( f, f ) B f B eg color 3 Etract and normalze te regon content 4 Compute a local descrptor from te normalzed regon 5 Matc local descrptors 4 f ( I (, )) f ( I (, )) m Slde credt: Krstan Mkolajczk m 5 Recap: Laplacan-of-Gaussan (LoG) Interest ponts: Local mama n scale space of Laplacan-of- Gaussan L ( ) L ( ) Recap: Harrs-Laplace [Mkolajczk ] Intalzaton: Multscale Harrs corner detecton 2 Scale selecton based on Laplacan (same procedure wt Hessan Hessan-Laplace) Harrs ponts 2 Lst of (,, σ) Slde adapted from Krstan Mkolajczk Slde adapted from Krstan Mkolajczk Harrs-Laplace ponts 7

2 Recap: SIF Feature Descrptor opcs of s Lecture Scale Invarant Feature ransform Descrptor computaton: Dvde patc nto 44 sub-patces: 6 cells Compute stogram of gradent orentatons (8 reference angles) for all pels nsde eac sub-patc Resultng descrptor: 448 = 28 dmensons Recognton wt Local Features Matcng local features Fndng consstent confguratons lgnment: lnear transformatons ffne estmaton Homograp estmaton Dealng wt Outlers RNSC Generalzed Houg ransform Davd G Lowe "Dstnctve mage features from scale-nvarant keponts IJCV 6 (2), pp 9-, 24 Slde credt: Svetlana Lazebnk 8 Indeng wt Local Features Inverted fle nde Vsual Words Vsual Vocabular constructon tf-df wegtng Recognton wt Local Features Image content s transformed nto local features tat are nvarant to translaton, rotaton, and scale Goal: Verf f te belong to a consstent confguraton Concepts: Warpng vs lgnment Warpng: Gven a source mage and a transformaton, wat does te transformed output look lke? Slde credt: Davd Lowe Local Features, eg SIF 2 lgnment: Gven two mages wt correspondng features, wat s te transformaton between tem? 3 Parametrc (Global) Warpng Wat Can be Represented b a 22 Matr? p = (,) p = (, ) 2D Scalng? s * s * s s ransformaton s a coordnate-cangng macne: p = (p) Wat does t mean tat s global? It s te same for an pont p It can be descrbed b just a few numbers (parameters) Let s represent as a matr: p = Mp, Slde credt: leej Efros M 4 2D Rotaton around (,)? cos * sn * cos sn sn * cos * sn cos 2D Searng? s * s * Slde credt: leej Efros s s 5 2

3 Wat Can be Represented b a 22 Matr? 2D Mrror about as? 2D Mrror over (,)? 2D ranslaton? t t NO! 2D Lnear ransforms Onl lnear 2D transformatons can be represented wt a 22 matr Lnear transformatons are combnatons of Scale, Rotaton, Sear, and Mrror a b c d Slde credt: leej Efros 6 Slde credt: leej Efros 7 Homogeneous Coordnates Basc 2D ransformatons Q: How can we represent translaton as a 33 matr usng omogeneous coordnates? t t Basc 2D transformatons as 33 matrces t s t s ranslaton Scalng : Usng te rgtmost column: t ranslaton t cos sn sn cos Rotaton s s Searng Slde credt: leej Efros 8 Slde credt: leej Efros 9 2D ffne ransformatons Projectve ransformatons a d w b e c f w a d w g b e c f w ffne transformatons are combnatons of Lnear transformatons, and ranslatons Projectve transformatons: ffne transformatons, and Projectve warps Parallel lnes reman parallel Parallel lnes do not necessarl reman parallel Slde credt: leej Efros 2 Slde credt: leej Efros 2 3

4 lgnment Problem We ave prevousl consdered ow to ft a model to mage evdence Eg, a lne to edge ponts In algnment, we wll ft te parameters of some transformaton accordng to a set of matcng feature pars ( correspondences ) Let s Start wt ffne ransformatons Smple fttng procedure (lnear least squares) ppromates vewpont canges for rougl planar objects and rougl ortograpc cameras Can be used to ntalze fttng for more comple models 22 Slde credt: Svetlana Lazebnk 23 Fttng an ffne ransformaton Fttng an ffne ransformaton ssumng we know te correspondences, ow do we get te transformaton? (, ) B, ) ( ffne model appromates perspectve projecton of planar objects m m3 m2 t m 4 t2 24 Image source: Davd Lowe 25 Recall: Least Squares Estmaton Set of data ponts: ( X, X),( X 2, X 2),( X3, X3) Goal: a lnear functon to predct X s from Xs: Xa b X We want to fnd a and b How man ( X, X ) pars do we need? Xa b X X a X B X 2a b X 2 X 2 b X 2 Wat f te data s nos? X X Overconstraned Soluton: problem X 2 a X 2 mn k Bk 2 = + B X 3 b X 3 Least-squares Matlab: mnmzaton = nb Slde credt: leej Efros 26 Fttng an ffne ransformaton ssumng we know te correspondences, ow do we get te transformaton? m m3 (, ) m2 t m 4 t2 B (, ) m m2 m3 m 4 t t2 27 4

5 Fttng an ffne ransformaton m m2 m3 m 4 t t2 How man matces (correspondence pars) do we need to solve for te transformaton parameters? Once we ave solved for te parameters, ow do we compute te coordnates of te correspondng pont for, new )? ( new 28 Homograp projectve transform s a mappng between an two perspectve projectons wt te same center of projecton Ie two planes n 3D along te same sgt ra Propertes Rectangle sould map to arbtrar quadrlateral PP2 Parallel lnes aren t but must preserve stragt lnes s s called a omograp w * w * w * p Slde adapted from leej Efros * * * H * * * p PP 29 Homograp Fttng a Homograp projectve transform s a mappng between an two perspectve projectons wt te same center of projecton Ie two planes n 3D along te same sgt ra Propertes Rectangle sould map to arbtrar quadrlateral Parallel lnes aren t but must preserve stragt lnes s s called a omograp w 2 3 w w 3 p H p Slde adapted from leej Efros Set scale factor to 8 parameters left Estmatng te transformaton B 2 3 B Homogenous coordnates Image coordnates Slde credt: Krstan Mkolajczk Matr notaton H 3 Fttng a Homograp Estmatng te transformaton Fttng a Homograp Estmatng te transformaton B B B 2 3 Homogenous coordnates 3 23 Slde credt: Krstan Mkolajczk Image coordnates Matr notaton H B 2 3 Homogenous coordnates B 3 3 B Slde credt: Krstan Mkolajczk Image coordnates Matr notaton H 33 5

6 Fttng a Homograp Estmatng te transformaton Fttng a Homograp Estmatng te transformaton B B B 2 3 Homogenous coordnates Image coordnates 2B 3 3 B Slde credt: Krstan Mkolajczk B 23 3 B B Matr notaton H 34 B 2 3 2B 3 Slde credt: Krstan Mkolajczk 3 B B 2 22B 23 B B B B B B 35 Fttng a Homograp Estmatng te transformaton Fttng a Homograp Estmatng te transformaton 2B 3 3 B 2 22B 23 3 B B B 2 3 B Homogenous coordnates Image coordnates 2B B 23 3 B 3 B 3 B 2 B 3 2B 3 3 B 2 22B 23 3 B Slde credt: Krstan Mkolajczk 36 B 2 3 Slde credt: Krstan Mkolajczk B B B B B B B Fttng a Homograp Fttng a Homograp Estmatng te transformaton Soluton: Null-space vector of B 2 3 SVD Slde credt: Krstan Mkolajczk d v v9 UDV? U d v v B 38 Estmatng te transformaton Soluton: Null-space vector of Corresponds to smallest sngular vector B 2 3 SVD Slde credt: Krstan Mkolajczk d v v9 UDV U d v v v9,, v v B Mnmzes least square error 39 6

7 utomatc rectfcaton Image Warpng wt Homograpes Uses: nalzng Patterns and Sapes Wat s te sape of te b/w floor pattern? p p Slde credt: Steve Setz Image plane n front Black area were no pel maps to mage plane below 4 Slde credt: ntono Crmns e floor (enlarged) 4 nalzng Patterns and Sapes opcs of s Lecture Recognton wt Local Features Matcng local features Fndng consstent confguratons lgnment: lnear transformatons ffne estmaton Homograp estmaton Dealng wt Outlers RNSC Generalzed Houg ransform From Martn Kemp e Scence of rt (manual reconstructon) Indeng wt Local Features Inverted fle nde Vsual Words Vsual Vocabular constructon tf-df wegtng Slde credt: ntono Crmns Problem: Outlers Eample: Least-Squares Lne Fttng Outlers can urt te qualt of our parameter estmates, eg, n erroneous par of matcng ponts from two mages feature pont tat s nose or doesn t belong to te transformaton we are fttng ssumng all te ponts tat belong to a partcular lne are known Source: Forst & Ponce 7

8 Outlers ffect Least-Squares Ft Outlers ffect Least-Squares Ft 46 Source: Forst & Ponce 47 Source: Forst & Ponce Strateg : RNSC [Fscler8] RNSC RNdom Smple Consensus RNSC loop: pproac: we want to avod te mpact of outlers, so let s look for nlers, and use onl tose Randoml select a seed group of ponts on wc to base transformaton estmate (eg, a group of matces) Intuton: f an outler s cosen to compute te current ft, ten te resultng lne won t ave muc support from rest of te ponts 2 Compute transformaton from seed group 3 Fnd nlers to ts transformaton 4 If te number of nlers s suffcentl large, recompute least-squares estmate of transformaton on all of te nlers Keep te transformaton wt te largest number of nlers RNSC Lne Fttng Eample ask: Estmate te best lne How man ponts do we need to estmate te lne? RNSC Lne Fttng Eample ask: Estmate te best lne Sample two ponts Slde credt: Jnang Ca 5 Slde credt: Jnang Ca 5 8

9 RNSC Lne Fttng Eample ask: Estmate te best lne RNSC Lne Fttng Eample ask: Estmate te best lne Ft a lne to tem otal number of ponts wtn a tresold of lne Slde credt: Jnang Ca 52 Slde credt: Jnang Ca 53 RNSC Lne Fttng Eample ask: Estmate te best lne RNSC Lne Fttng Eample ask: Estmate te best lne 7 nler ponts otal number of ponts wtn a tresold of lne Repeat, untl we get a good result Slde credt: Jnang Ca 54 Slde credt: Jnang Ca 55 RNSC Lne Fttng Eample RNSC: How man samples? ask: Estmate te best lne How man samples are needed? Suppose w s fracton of nlers (ponts from lne) n ponts needed to defne potess (2 for lnes) nler ponts k samples cosen Prob tat a sngle sample of n ponts s correct: n w Prob tat all k samples fal s: ( w ) n k Repeat, untl we get a good result Coose k g enoug to keep ts below desred falure rate Slde credt: Jnang Ca 56 Slde credt: Davd Lowe 57 9

10 RNSC: Computed k (p=99) fter RNSC Sample sze n Proporton of outlers 5% % 2% 25% 3% 4% 5% RNSC dvdes data nto nlers and outlers and elds estmate computed from mnmal set of nlers Improve ts ntal estmate wt estmaton over all nlers (eg wt standard least-squares mnmzaton) But ts ma cange nlers, so alternate fttng wt reclassfcaton as nler/outler Slde credt: Davd Lowe 58 Slde credt: Davd Lowe 59 Eample: Fndng Feature Matces Eample: Fndng Feature Matces Fnd best stereo matc wtn a square searc wndow (ere 3 pels 2 ) Global transformaton model: eppolar geometr Fnd best stereo matc wtn a square searc wndow (ere 3 pels 2 ) Global transformaton model: eppolar geometr before RNSC after RNSC Images from Hartle & Zsserman Images from Hartle & Zsserman Slde credt: Davd Lowe 6 Slde credt: Davd Lowe 6 Problem wt RNSC Strateg 2: Generalzed Houg ransform In man practcal stuatons, te percentage of outlers (ncorrect putatve matces) s often ver g (9% or above) lternatve strateg: Generalzed Houg ransform Suppose our features are scale- and rotaton-nvarant en a sngle feature matc provdes an algnment potess (translaton, scale, orentaton) model Slde credt: Svetlana Lazebnk 62 Slde credt: Svetlana Lazebnk 63

11 Strateg 2: Generalzed Houg ransform Pose Clusterng and Verfcaton wt SIF Suppose our features are scale- and rotaton-nvarant en a sngle feature matc provdes an algnment potess (translaton, scale, orentaton) Of course, a potess from a sngle matc s unrelable Soluton: let eac matc vote for ts potess n a Houg space wt ver coarse bns o detect nstances of objects from a model base: Inde descrptors Dstnctve features narrow down possble matces model Slde credt: Svetlana Lazebnk Image source: Davd Lowe Indeng Local Features Pose Clusterng and Verfcaton wt SIF o detect nstances of objects from a model base: Model base New mage Image source: Davd Lowe Inde descrptors Dstnctve features narrow down possble matces 2 Generalzed Houg transform to vote for poses Keponts ave record of parameters relatve to model coordnate sstem 3 ffne ft to ceck for agreement between model and mage features Ft and verf usng features from Houg bns wt 3+ votes 67 Image source: Davd Lowe Object Recognton Results Locaton Recognton ranng Background subtract for model boundares Objects recognzed Recognton n spte of occluson 68 Image source: Davd Lowe [Lowe, IJCV 4] 69 Slde credt: Davd Lowe

12 Recall: Dffcultes of Votng Summar Nose/clutter can lead to as man votes as true target Bn sze for te accumulator arra must be cosen carefull (Recall Houg ransform) Recognton b algnment: lookng for object and pose tat fts well wt mage Use good correspondences to desgnate poteses Invarant local features offer more relable matces Fnd consstent nler confguratons n clutter In practce, good dea to make broad bns and spread votes to nearb bns, snce verfcaton stage can prune bad vote peaks Generalzed Houg ransform RNSC lgnment approac to recognton can be effectve f we fnd relable features wtn clutter pplcaton: large-scale mage retreval pplcaton: recognton of specfc (mostl planar) objects 7 Move posters Books CD covers 72 References and Furter Readng detaled descrpton of local feature etracton and recognton can be found n Capters 3-5 of Grauman & Lebe (avalable on te L2P) K Grauman, Vsual Object Recognton Morgan & Clapool publsers, 2 R Hartle, Zsserman Multple Vew Geometr n Computer Vson 2nd Ed, Cambrdge Unv Press, 24 More detals on RNSC can also be found n Capter 47 of Hartle & Zsserman 2