Fttng and Algnment Computer Vson Ja-Bn Huang, Vrgna Tech Many sldes from S. Lazebnk and D. Hoem
Admnstratve Stuffs HW 1 Competton: Edge Detecton Submsson lnk HW 2 wll be posted tonght Due Oct 09 (Mon) 23:59 PM
Anonymous feedback HW s too hard I wll adjust accordngly. Please come to offce hours and see TA/me after classes Gong over starter code used n HWs n class. Great dea! Wll do ths startng HW2. Not a lot of correlaton between class content and HW. Hybrd mage: spatal and frequency vews of flters Image pyramd: Gaussan/Laplacan pyramd Edge detecton: Gradent flter
Where are we? Interest ponts Fnd dstnct and repeatable ponts n mages Harrs-> corners, DoG -> blobs SIFT -> feature descrptor Feature trackng and optcal flow Fnd moton of a keypont/pxel over tme Lucas-Kanade: brghtness consstency, small moton, spatal coherence Handle large moton: teratve update + pyramd search Fttng and algnment (ths class) fnd the transformaton parameters that best algn matched ponts Object nstance recognton (next class) Keypont-based object nstance recognton and search
Revew: Harrs Corner Detector Second moment matrx 2 I x ( D ) I xi y ( D) ( I, D) g( ) 2 1. Image I I xi y ( D ) I y ( D ) dervatves (optonally, blur frst) I x I y det M trace M 1 2 1 2 2. Square of dervatves 3. Gaussan flter g( I ) I x 2 I y 2 I x I y g(i x2 ) g(i y2 ) g(i x I y ) 4. Cornerness functon both egenvalues are strong 2 har det[ (, )] [trace( (, )) ] g I D 2 2 2 2 2 ( I x ) g( I y ) [ g( I xi y )] [ g( I x ) g( I y I D )] 2 5. 5 Non-maxma suppresson har
Revew: Fnd local maxma n postonscale space of Dfference-of-Gaussan 5 4 L xx ( ) L ( ) yy 3 2 Lst of (x, y, s) K. Grauman, B. Lebe
Revew: SIFT Descrptor [Lowe, ICCV 1999] K. Grauman, B. Lebe Hstogram of orented gradents Captures mportant texture nformaton Robust to small translatons / affne deformatons
Revew: Lucas-Kanade Tracker Brghtness consstency I(x,y,t) I(x,y,t+1) ), ( ),, ( 1, t v y u x I t y x I t y x I v I u I t y x I t v y u x I ),, ( 1),, ( Small moton 0 t y x I v I u I Spatal coherence
Dealng wth larger movements: Iteratve refnement Orgnal (x,y) poston 1. Intalze (x,y ) = (x,y) 2. Compute (u,v) by I t = I(x, y, t+1) - I(x, y, t) 2 nd moment matrx for feature patch n frst mage dsplacement 3. Shft wndow by (u, v): x =x +u; y =y +v; 4. Recalculate I t 5. Repeat steps 2-4 untl small change Use nterpolaton for subpxel values
Dealng wth larger movements: coarse-to-fne regstraton run teratve L-K upsample run teratve L-K... mage J1 mage I2 Gaussan pyramd of mage 1 (t) Gaussan pyramd of mage 2 (t+1)
Fttng [ˈfdNG]: fnd the parameters of a model that best ft the data Algnment [əˈlīnmənt]: fnd the parameters of the transformaton that best algn matched ponts
Fttng and algnment Choose a parametrc model to represent a set of features smple model: lnes smple model: crcles complcated model: face shape complcated model: car Source: K. Grauman
Fttng and Algnment -Desgn challenges Desgn a sutable goodness of ft measure Smlarty should reflect applcaton goals Encode robustness to outlers and nose Desgn an optmzaton method Avod local optma Fnd best parameters quckly
Fttng and Algnment: Methods Global optmzaton / Search for parameters Least squares ft Robust least squares Iteratve closest pont (ICP) Hypothesze and test Generalzed Hough transform RANSAC
Smple example: Fttng a lne
Least squares lne fttng Data: (x 1, y 1 ),, (x n, y n ) Lne equaton: y = m x + b Fnd (m, b) to mnmze 0 2 2 y A Ap A T T dp de ) ( ) ( ) 2( 1 1 1 2 2 1 1 1 2 Ap Ap y Ap y y y Ap T T T n n n y y b m x x y b m x E n b mx y E 1 2 ) ( (x, y ) y=mx+b y A A A p y A Ap A T T T T 1 Matlab: p = A \ y; Modfed from S. Lazebnk
Problem wth vertcal least squares Not rotaton-nvarant Fals completely for vertcal lnes Slde from S. Lazebnk
Total least squares If (a 2 +b 2 =1) then Dstance between pont (x, y ) and lne ax+by+c=0 s ax + by + c E n 1 Slde modfed from S. Lazebnk 2 ( ax(x by, y d ) ax+by+c=0 Unt normal: N=(a, b) proof: http://mathworld.wolfram.com/pont- LneDstance2-Dmensonal.html
Total least squares If (a 2 +b 2 =1) then Dstance between pont (x, y ) and lne ax+by+c=0 s ax + by + c E n 1 Slde modfed from S. Lazebnk 2 ( ax(x by, y d ) ax+by+c=0 Unt normal: N=(a, b) Fnd (a, b, c) to mnmze the sum of squared perpendcular dstances E n 1 ( ax b y c) 2
Total least squares Fnd (a, b, c) to mnmze the sum of squared perpendcular dstances E n 1 ( ax b y c) 2 E n 1 Slde modfed from S. Lazebnk 2 ( ax(x by, y d ) ax+by+c=0 Unt normal: N=(a, b) E n a 2( ) 0 1 ax b y c c c n 2 x1 x y1 y n a E 2 a x p x b y y ( ( ) ( )) 1 b xn x yn y T T T T T p A Ap mnmze p A Ap s.t. p p 1 mnmze T p p n b n x y ax b y 1 1 n Soluton s egenvector correspondng to smallest egenvalue of A T A T A T Ap See detals on Ralegh Quotent: http://en.wkpeda.org/wk/raylegh_quotent
Recap: Two Common Optmzaton Problems Problem statement mnmze least squares soluton to Ax b 2 Soluton Ax b T 1 T x A A A b x A \ b (matlab) Problem statement T T mnmze x A Ax s.t. T T x A Ax mnmze T x x x T x 1 non - trval lsq soluton to Ax 0 Soluton [ v, ] 1 2.. n : eg( A x T A) v 1
Least squares (global) optmzaton Good Clearly specfed objectve Optmzaton s easy Bad May not be what you want to optmze Senstve to outlers Bad matches, extra ponts Doesn t allow you to get multple good fts Detectng multple objects, lnes, etc.
Robust least squares (to deal wth outlers) General approach: mnmze u (x, θ) resdual of th pont w.r.t. model parameters θ ρ robust functon wth scale parameter σ u x, ; u 2 n 1 ( y mx b) 2 The robust functon ρ Favors a confguraton wth small resduals Constant penalty for large resduals Slde from S. Savarese
Robust Estmator 1.5 medan error 1. Intalze: e.g., choose θ by least squares ft and 2. Choose params to mnmze: E.g., numercal optmzaton 2 error(, data error(, data ) 2 ) 2 3. Compute new 1.5 medan error 4. Repeat (2) and (3) untl convergence
Other ways to search for parameters (for when no closed form soluton exsts) Lne search 1. For each parameter, step through values and choose value that gves best ft 2. Repeat (1) untl no parameter changes Grd search 1. Propose several sets of parameters, evenly sampled n the jont set 2. Choose best (or top few) and sample jont parameters around the current best; repeat Gradent descent 1. Provde ntal poston (e.g., random) 2. Locally search for better parameters by followng gradent
Hypothesze and test 1. Propose parameters Try all possble Each pont votes for all consstent parameters Repeatedly sample enough ponts to solve for parameters 2. Score the gven parameters Number of consstent ponts, possbly weghted by dstance 3. Choose from among the set of parameters Global or local maxmum of scores 4. Possbly refne parameters usng nlers
Hough Transform: Outlne 1. Create a grd of parameter values 2. Each pont votes for a set of parameters, ncrementng those values n grd 3. Fnd maxmum or local maxma n grd
Hough transform P.V.C. Hough, Machne Analyss of Bubble Chamber Pctures, Proc. Int. Conf. Hgh Energy Accelerators and Instrumentaton, 1959 Gven a set of ponts, fnd the curve or lne that explans the data ponts best y m x y = m x + b Hough space b Slde from S. Savarese
Hough transform y m x b y m 3 5 3 3 2 2 3 7 11 10 4 3 2 3 1 4 5 2 2 1 0 1 3 3 Slde from S. Savarese x b
Hough transform P.V.C. Hough, Machne Analyss of Bubble Chamber Pctures, Proc. Int. Conf. Hgh Energy Accelerators and Instrumentaton, 1959 Issue : parameter space [m,b] s unbounded Use a polar representaton for the parameter space y x Hough space x cos ysn Slde from S. Savarese
Hough transform - experments features votes Slde from S. Savarese
Hough transform - experments Nosy data features votes Need to adjust grd sze or smooth Slde from S. Savarese
Hough transform - experments features Issue: spurous peaks due to unform nose votes Slde from S. Savarese
1. Image Canny
2. Canny Hough votes
3. Hough votes Edges Fnd peaks and post-process
Hough transform example http://ostatc.com/fles/mages/ss_hough.jpg
Hough transform for crcles Crcle: center (a,b) and radus r For a fxed radus r 2 2 2 ( x a) ( y b) r b Equaton of crcle? Equaton of set of crcles that all pass through a pont? Image space Hough space a Adapted by Dev Parkh from: Krsten Grauman 38
Hough transform for crcles Crcle: center (a,b) and radus r For a fxed radus r 2 2 2 ( x a) ( y b) r Intersecton: most votes for center occur here. Image space Hough space 39 Krsten Grauman
Hough transform for crcles Crcle: center (a,b) and radus r For an unknown radus r 2 2 2 ( x a) ( y b) r r? Image space a Hough space b 40 Krsten Grauman
Hough transform for crcles Crcle: center (a,b) and radus r For an unknown radus r 2 2 2 ( x a) ( y b) r r b Image space a Hough space 41 Krsten Grauman
Hough transform for crcles Crcle: center (a,b) and radus r 2 2 2 ( x a) ( y b) r For an unknown radus r, known gradent drecton x θ Image space Hough space 42 Krsten Grauman
Example: detectng crcles wth Hough Orgnal Edges Votes: Penny Note: a dfferent Hough transform (wth separate accumulators) was used for each crcle radus (quarters vs. penny). Slde credt: Krsten Grauman 43
Example: detectng crcles wth Hough Combned Orgnal detectons Edges Votes: Quarter Slde credt: Krsten Grauman 44 Con fndng sample mages from: Vvek Kwatra
Generalzed Hough for object detecton Instead of ndexng dsplacements by gradent orentaton, ndex by matched local patterns. tranng mage vsual codeword wth dsplacement vectors B. Lebe, A. Leonards, and B. Schele, Combned Object Categorzaton and Segmentaton wth an Implct Shape Model, ECCV Workshop on Statstcal Learnng n Computer Vson 2004 45 Source: L. Lazebnk
Generalzed Hough for object detecton Instead of ndexng dsplacements by gradent orentaton, ndex by vsual codeword test mage B. Lebe, A. Leonards, and B. Schele, Combned Object Categorzaton and Segmentaton wth an Implct Shape Model, ECCV Workshop on Statstcal Learnng n Computer Vson 2004 46 Source: L. Lazebnk
Hough transform conclusons Good Robust to outlers: each pont votes separately Farly effcent (much faster than tryng all sets of parameters) Provdes multple good fts Bad Some senstvty to nose Bn sze trades off between nose tolerance, precson, and speed/memory Can be hard to fnd sweet spot Not sutable for more than a few parameters grd sze grows exponentally Common applcatons Lne fttng (also crcles, ellpses, etc.) Object nstance recognton (parameters are poston/scale/orentaton) Object category recognton (parameters are poston/scale)
RANSAC (RANdom SAmple Consensus) : Fschler & Bolles n 81. Algorthm: 1. Sample (randomly) the number of ponts requred to ft the model 2. Solve for model parameters usng samples 3. Score by the fracton of nlers wthn a preset threshold of the model Repeat 1-3 untl the best model s found wth hgh confdence
RANSAC Lne fttng example Algorthm: 1. Sample (randomly) the number of ponts requred to ft the model (#=2) 2. Solve for model parameters usng samples 3. Score by the fracton of nlers wthn a preset threshold of the model Repeat 1-3 untl the best model s found wth hgh confdence Illustraton by Savarese
RANSAC Lne fttng example Algorthm: 1. Sample (randomly) the number of ponts requred to ft the model (#=2) 2. Solve for model parameters usng samples 3. Score by the fracton of nlers wthn a preset threshold of the model Repeat 1-3 untl the best model s found wth hgh confdence
RANSAC Lne fttng example N I 6 Algorthm: 1. Sample (randomly) the number of ponts requred to ft the model (#=2) 2. Solve for model parameters usng samples 3. Score by the fracton of nlers wthn a preset threshold of the model Repeat 1-3 untl the best model s found wth hgh confdence
RANSAC Algorthm: 1. Sample (randomly) the number of ponts requred to ft the model (#=2) 2. Solve for model parameters usng samples 3. Score by the fracton of nlers wthn a preset threshold of the model Repeat 1-3 untl the best model s found wth hgh confdence N I 14
How to choose parameters? Number of samples N Choose N so that, wth probablty p, at least one random sample s free from outlers (e.g. p=0.99) (outler rato: e ) Number of sampled ponts s Mnmum number needed to ft the model Dstance threshold Choose so that a good pont wth nose s lkely (e.g., prob=0.95) wthn threshold Zero-mean Gaussan nose wth std. dev. σ: t 2 =3.84σ 2 N log 1 p/ log1 1 e s proporton of outlers e s 5% 10% 20% 25% 30% 40% 50% 2 2 3 5 6 7 11 17 3 3 4 7 9 11 19 35 4 3 5 9 13 17 34 72 5 4 6 12 17 26 57 146 6 4 7 16 24 37 97 293 7 4 8 20 33 54 163 588 8 5 9 26 44 78 272 1177 modfed from M. Pollefeys
RANSAC conclusons Good Robust to outlers Applcable for larger number of objectve functon parameters than Hough transform Optmzaton parameters are easer to choose than Hough transform Bad Computatonal tme grows quckly wth fracton of outlers and number of parameters Not as good for gettng multple fts (though one soluton s to remove nlers after each ft and repeat) Common applcatons Computng a homography (e.g., mage sttchng) Estmatng fundamental matrx (relatng two vews)
RANSAC Song
What f you want to algn but have no pror matched pars? Hough transform and RANSAC not applcable Important applcatons Medcal magng: match bran scans or contours Robotcs: match pont clouds
Iteratve Closest Ponts (ICP) Algorthm Goal: estmate transform between two dense sets of ponts 1. Intalze transformaton (e.g., compute dfference n means and scale) 2. Assgn each pont n {Set 1} to ts nearest neghbor n {Set 2} 3. Estmate transformaton parameters e.g., least squares or robust least squares 4. Transform the ponts n {Set 1} usng estmated parameters 5. Repeat steps 2-4 untl change s very small
Example: solvng for translaton A 1 A 2 A 3 B 1 B 2 B 3 Gven matched ponts n {A} and {B}, estmate the translaton of the object x y B B x y A A t t x y
Example: solvng for translaton A 1 A 2 A B 1 3 (t x, t y ) B 2 B 3 Least squares soluton 1. Wrte down objectve functon 2. Derved soluton a) Compute dervatve b) Compute soluton 3. Computatonal soluton a) Wrte n form Ax=b b) Solve usng pseudo-nverse or egenvalue decomposton x y 1 0 1 0 B B x y 0 1 t t 0 1 x y A A x y x y B 1 B 1 B n B n t t x y x y A 1 A 1 A n A n x y
Example: solvng for translaton A 1 A 5 B 4 A 2 A B 1 3 (t x, t y ) A 4 B 2 B 3 B 5 Problem: outlers RANSAC soluton 1. Sample a set of matchng ponts (1 par) 2. Solve for transformaton parameters 3. Score parameters wth number of nlers 4. Repeat steps 1-3 N tmes x y B B x y A A t t x y
Example: solvng for translaton B 4 A 1 B 5 B 6 A 2 A B 1 3 (t x, t y ) A 4 B 2 B 3 A 5 A 6 Problem: outlers, multple objects, and/or many-to-one matches Hough transform soluton 1. Intalze a grd of parameter values 2. Each matched par casts a vote for consstent values 3. Fnd the parameters wth the most votes 4. Solve usng least squares wth nlers x y B B x y A A t t x y
Example: solvng for translaton (t x, t y ) Problem: no ntal guesses for correspondence ICP soluton 1. Fnd nearest neghbors for each pont 2. Compute transform usng matches 3. Move ponts usng transform 4. Repeat steps 1-3 untl convergence x y B B x y A A t t x y
HW 2 Feature tracker Keypont detecton Compute second moment matrx Harrs corner crteron Threshold Non-maxmum suppresson Trackng Kanade-Lucas-Tomas trackng Show keypont trajectores Show ponts have moved out of frames
HW 2 Shape Algnment Global transformaton (smlarty, affne, perspectve) Iteratve closest pont algorthm
HW 2 Local Feature Matchng Implement dstance rato test feature matchng algorthm
Thngs to remember Least Squares Ft closed form soluton robust to nose not robust to outlers Robust Least Squares mproves robustness to nose requres teratve optmzaton Hough transform robust to nose and outlers can ft multple models only works for a few parameters (1-4 typcally) RANSAC robust to nose and outlers works wth a moderate number of parameters (e.g, 1-8) Iteratve Closest Pont (ICP) For local algnment only: does not requre ntal correspondences
Next week Object nstance recognton