Algnment and Object Instance Recognton Computer Vson Ja-Bn Huang, Vrgna Tech Man sldes from S. Lazebnk and D. Hoem
Admnstratve Stuffs HW 2 due 11:59 PM Oct 9
Anonmous feedback Lectures Mcrophone on our shrt. A lttle quck to follow Make the topcs easer to understand Thanks for the feedback. I wll adjust the contents. Homeworks Mnor mstakes n HW1 Too man assgnments. You are expectng too much. Wll provde explct starter code structure and hnts. Other aspects Real-world demo frst, then explan the foundaton Wll do.
Toda s class Revew fttng Algnment Object nstance recognton Gong over HW 2
Prevous class Global optmzaton / Search for parameters Least squares ft Robust least squares Iteratve closest pont (ICP) Hpothesze and test Generalzed Hough transform RANSAC
Least squares lne fttng Data: (x 1, 1 ),, (x n, n ) Lne equaton: = m x + b Fnd (m, b) to mnmze 0 2 2 A Ap A T T dp de ) ( ) ( ) 2( 1 1 1 2 2 1 1 1 2 Ap Ap Ap Ap T T T n n n b m x x b m x E n b mx E 1 2 ) ( (x, ) =mx+b A A A p A Ap A T T T T 1 Matlab: p = A \ ; Modfed from S. Lazebnk
Least squares lne fttng functon [m, b] = lsqft(x, ) % = mx + b % fnd lne that best predcts gven x % mnmze sum_ (m*x_ + b - _).^2 A = [x(:) ones(numel(x), 1)]; b = (:); p = A\b; m = p(1); b = p(2); A p
Total least squares Fnd (a, b, c) to mnmze the sum of squared perpendcular dstances E n 1 ( ax b c) 2 E n 1 Slde modfed from S. Lazebnk 2 ( ax(x b, d ) ax+b+c=0 Unt normal: N=(a, b) E n a 2( ) 0 1 ax b c c c n 2 x1 x 1 n a E 2 a x p x b ( ( ) ( )) 1 b xn x n T T T T T p A Ap mnmze p A Ap s.t. p p 1 mnmze T p p n b n x ax b 1 1 n Soluton s egenvector correspondng to smallest egenvalue of A T A T A T Ap See detals on Ralegh Quotent: http://en.wkpeda.org/wk/ralegh_quotent
Total least squares functon [m, b, err] = total_lsqft(x, ) % ax + b + c = 0 % dstance to lne for (a^2+b^2=1): dst_sq = (ax + b + c).^2 A = [x(:)-mean(x) (:)-mean()]; [v, d] = eg(a'*a); p = v(:, 1); % egenvector corr. to smaller egenvalue % get a, b, c parameters a = p(1); b = p(2); c = -(a*mean(x)+b*mean()); err = (a*x+b*+c).^2; % convert to slope-ntercept (m, b) m = -a/b; b = -c/b; % note: ths b s for slope-ntercept now A p
Robust Estmator 1. Intalze: e.g., choose θ b least squares ft and 1.5 medan 2. Choose params to mnmze: E.g., numercal optmzaton error 2 error(, data error(, data ) 2 ) 2 3. Compute new 1.5 medan error 4. Repeat (2) and (3) untl convergence
functon [m, b] = robust_lsqft(x, ) % teratve robust ft = mx + b % fnd lne that best predcts gven x % mnmze sum_ (m*x_ + b - _).^2 [m, b] = lsqft(x, ); p = [m ; b]; err = sqrt((-p(1)*x-p(2)).^2); sgma = medan(err)*1.5; for k = 1:7 p = fmnunc(@(p)geterr(p,x,,sgma), p); err = sqrt((-p(1)*x-p(2)).^2); sgma = medan(err)*1.5; end m = p(1); b = p(2);
Hough transform P.V.C. Hough, Machne Analss of Bubble Chamber Pctures, Proc. Int. Conf. Hgh Energ Accelerators and Instrumentaton, 1959 Use a polar representaton for the parameter space x cos sn x Hough space Slde from S. Savarese
functon [m, b] = houghft(x, ) % = mx + b % x*cos(theta) + *sn(theta) = r % fnd lne that best predcts gven x % mnmze sum_ (m*x_ + b - _).^2 thetas = (-p+p/50):(p/100):p; costhetas = cos(thetas); snthetas = sn(thetas); mnr = 0; stepr = 0.005; maxr = 1; % count hough votes counts = zeros(numel(thetas),(maxr-mnr)/stepr+1); for k = 1:numel(x) end r = x(k)*costhetas + (k)*snthetas; % onl count parameters wthn the range of r nrange = fnd(r >= mnr & r <= maxr); rnum = round((r(nrange)-mnr)/stepr)+1; nd = sub2nd(sze(counts), nrange, rnum); counts(nd) = counts(nd) + 1; % smooth the bn counts counts = mflter(counts, fspecal('gaussan', 5, 0.75)); % get best theta, rho and show counts [maxval, maxnd] = max(counts(:)); [thetand, rnd] = nd2sub(sze(counts), maxnd); theta = thetas(thetand); r = mnr + stepr*(rnd-1); % convert to slope-ntercept b = r/sn(theta); m = -cos(theta)/sn(theta);
RANSAC Algorthm: 1. Sample (randoml) the number of ponts requred to ft the model (#=2) 2. Solve for model parameters usng samples 3. Score b the fracton of nlers wthn a preset threshold of the model Repeat 1-3 untl the best model s found wth hgh confdence N I 14
functon [m, b] = ransacft(x, ) % = mx + b N = 200; thresh = 0.03; bestcount = 0; for k = 1:N rp = randperm(numel(x)); tx = x(rp(1:2)); t = (rp(1:2)); m = (t(2)-t(1))./ (tx(2)-tx(1)); b = t(2)-m*tx(2); nn = sum(abs(-m*x-b)<thresh); f nn > bestcount bestcount = nn; nlers = (abs( - m*x - b) < thresh); end end % total least square fttng on nlers [m, b] = total_lsqft(x(nlers), (nlers));
Lne fttng demo demo_lneft(npts, outlers, nose, method) npts: number of ponts outlers: number of outlers nose: nose level Method lsq: least squares tlsq: total least squares rlsq: robust least squares hough: hough transform ransac: RANSAC
Whch algorthm should I use? If we know whch ponts belong to the lne, how do we fnd the optmal lne parameters? Least squares What f there are outlers? Robust fttng, RANSAC What f there are man lnes? Votng methods: RANSAC, Hough transform Slde credt: S. Lazebnk
Algnment as fttng Prevous lectures: fttng a model to features n one mage M x Fnd model M that mnmzes resdual( x, M ) Algnment: fttng a model to a transformaton between pars of features (matches) n two mages x T x ' Fnd transformaton T that mnmzes resdual( T ( x ), x )
What f ou want to algn but have no pror matched pars? Hough transform and RANSAC not applcable Important applcatons Medcal magng: match bran scans or contours Robotcs: match pont clouds
Iteratve Closest Ponts (ICP) Algorthm Goal: estmate transform between two dense sets of ponts 1. Intalze transformaton (e.g., compute dfference n means and scale) 2. Assgn each pont n {Set 1} to ts nearest neghbor n {Set 2} 3. Estmate transformaton parameters e.g., least squares or robust least squares 4. Transform the ponts n {Set 1} usng estmated parameters 5. Repeat steps 2-4 untl change s ver small
Example: solvng for translaton A 1 A 2 A 3 B 1 B 2 B 3 Gven matched ponts n {A} and {B}, estmate the translaton of the object x B B x A A t t x
Example: solvng for translaton A 1 A 2 A B 1 3 (t x, t ) B 2 B 3 Least squares soluton 1. Wrte down objectve functon 2. Derved soluton a) Compute dervatve b) Compute soluton 3. Computatonal soluton a) Wrte n form Ax=b b) Solve usng pseudo-nverse or egenvalue decomposton x 1 0 1 0 B B x 0 1 t t 0 1 x A A x x B 1 B 1 B n B n t t x x A 1 A 1 A n A n x
Example: solvng for translaton A 1 A 5 B 4 A 2 A B 1 3 (t x, t ) A 4 B 2 B 3 B 5 Problem: outlers RANSAC soluton 1. Sample a set of matchng ponts (1 par) 2. Solve for transformaton parameters 3. Score parameters wth number of nlers 4. Repeat steps 1-3 N tmes x B B x A A t t x
Example: solvng for translaton B 4 A 1 B 5 B 6 A 2 A B 1 3 (t x, t ) A 4 B 2 B 3 A 5 A 6 Problem: outlers, multple objects, and/or man-to-one matches Hough transform soluton 1. Intalze a grd of parameter values 2. Each matched par casts a vote for consstent values 3. Fnd the parameters wth the most votes 4. Solve usng least squares wth nlers x B B x A A t t x
Example: solvng for translaton (t x, t ) Problem: no ntal guesses for correspondence ICP soluton 1. Fnd nearest neghbors for each pont 2. Compute transform usng matches 3. Move ponts usng transform 4. Repeat steps 1-3 untl convergence x B B x A A t t x
Algorthm Summar Least Squares Ft closed form soluton robust to nose not robust to outlers Robust Least Squares mproves robustness to nose requres teratve optmzaton Hough transform robust to nose and outlers can ft multple models onl works for a few parameters (1-4 tpcall) RANSAC robust to nose and outlers works wth a moderate number of parameters (e.g, 1-8) Iteratve Closest Pont (ICP) For local algnment onl: does not requre ntal correspondences
Algnment Algnment: fnd parameters of model that maps one set of ponts to another Tpcall want to solve for a global transformaton that accounts for most true correspondences Dffcultes Nose (tpcall 1-3 pxels) Outlers (often 30-50%) Man-to-one matches or multple objects
Parametrc (global) warpng T p = (x,) p = (x, ) Transformaton T s a coordnate-changng machne: p = T(p) What does t mean that T s global? Is the same for an pont p can be descrbed b just a few numbers (parameters) For lnear transformatons, we can represent T as a matrx p = Tp x' x T '
Common transformatons orgnal Transformed translaton rotaton aspect affne perspectve Slde credt (next few sldes): A. Efros and/or S. Setz
Scalng Scalng a coordnate means multplng each of ts components b a scalar Unform scalng means ths scalar s the same for all components: 2
Scalng Non-unform scalng: dfferent scalars per component: X 2, Y 0.5
Scalng Scalng operaton: Or, n matrx form: x' ' ax b x' ' a 0 0 b x scalng matrx S
2-D Rotaton (x, ) (x, ) x = x cos() - sn() = x sn() + cos()
2-D Rotaton f (x, ) (x, ) Polar coordnates x = r cos (f) = r sn (f) x = r cos (f + ) = r sn (f + ) Trg Identt x = r cos(f) cos() r sn(f) sn() = r sn(f) cos() + r cos(f) sn() Substtute x = x cos() - sn() = x sn() + cos()
2-D Rotaton Ths s eas to capture n matrx form: x' ' sn x cos sn cos R Even though sn() and cos() are nonlnear functons of, x s a lnear combnaton of x and s a lnear combnaton of x and What s the nverse transformaton? Rotaton b For rotaton matrces 1 T R R
Basc 2D transformatons x' ' s x 0 0 s x x' ' 1 x 1 x Scale Shear x' ' cos sn x sn cos Rotate x 1 0 0 1 Translate t t x x 1 x a d b e Affne x c f 1 Affne s an combnaton of translaton, scale, rotaton, shear
Affne Transformatons Affne transformatons are combnatons of Lnear transformatons, and Translatons Propertes of affne transformatons: Lnes map to lnes Parallel lnes reman parallel Ratos are preserved Closed under composton 1 x f e d c b a x 1 1 0 0 1 ' ' x f e d c b a x or
Projectve Transformatons Projectve transformatons are combos of Affne transformatons, and Projectve warps x' ' w' a d g b e h c x f w Propertes of projectve transformatons: Lnes map to lnes Parallel lnes do not necessarl reman parallel Ratos are not preserved Closed under composton Models change of bass Projectve matrx s defned up to a scale (8 DOF)
Projectve Transformatons (homograph) The transformaton between two vews of a planar surface The transformaton between mages from two cameras that share the same center
Applcaton: Panorama sttchng Source: Hartle & Zsserman
Applcaton: document scannng
2D mage transformatons (reference table)
Object Instance Recognton 1. Match keponts to object model 2. Solve for affne transformaton parameters 3. Score b nlers and choose solutons wth score above threshold A 1 A 2 A 3 Matched keponts Affne Parameters # Inlers Ths Class Choose hpothess wth max score above threshold
N pxels Overvew of Kepont Matchng 1. Fnd a set of dstnctve keponts A 1 A 2 A 3 2. Defne a regon around each kepont N pxels f A e.g. color d( f A, f B ) T f B e.g. color K. Grauman, B. Lebe 3. Extract and normalze the regon content 4. Compute a local descrptor from the normalzed regon 5. Match local descrptors
Fndng the objects (overvew) Input Image Stored Image 1. Match nterest ponts from nput mage to database mage 2. Matched ponts vote for rough poston/orentaton/scale of object 3. Fnd poston/orentaton/scales that have at least three votes 4. Compute affne regstraton and matches usng teratve least squares wth outler check 5. Report object f there are at least T matched ponts
Matchng Keponts Want to match keponts between: 1. Quer mage 2. Stored mage contanng the object Gven descrptor x 0, fnd two nearest neghbors x 1, x 2 wth dstances d 1, d 2 x 1 matches x 0 f d 1 /d 2 < 0.8 Ths gets rd of 90% false matches, 5% of true matches n Lowe s stud
Affne Object Model Accounts for 3D rotaton of a surface under orthographc projecton
Fttng an affne transformaton Assume we know the correspondences, how do we get the transformaton? ( x, ) x, ) ( x m m 1 3 m m 2 4 x t t 1 2 x Mx t Want to fnd M, t to mnmze n 1 x Mx t 2
Fttng an affne transformaton ), ( x ), ( x 2 1 4 3 2 1 t t x m m m m x x t t m m m m x x 2 1 4 3 2 1 1 0 0 0 0 1 0 0 Assume we know the correspondences, how do we get the transformaton?
Fttng an affne transformaton Lnear sstem wth sx unknowns Each match gves us two lnearl ndependent equatons: need at least three to solve for the transformaton parameters x t t m m m m x x 2 1 4 3 2 1 1 0 0 0 0 1 0 0
Fndng the objects (n detal) 1. Match nterest ponts from nput mage to database mage 2. Get locaton/scale/orentaton usng Hough votng In tranng, each pont has known poston/scale/orentaton wrt whole object Matched ponts vote for the poston, scale, and orentaton of the entre object Bns for x,, scale, orentaton Wde bns (0.25 object length n poston, 2x scale, 30 degrees orentaton) Vote for two closest bn centers n each drecton (16 votes total) 3. Geometrc verfcaton For each bn wth at least 3 keponts Iterate between least squares ft and checkng for nlers and outlers 4. Report object f > T nlers (T s tpcall 3, can be computed to match some probablstc threshold)
Examples of recognzed objects
Vew nterpolaton Tranng Gven mages of dfferent vewponts Cluster smlar vewponts usng feature matches Lnk features n adjacent vews Recognton Feature matches ma be spread over several tranng vewponts Use the known lnks to transfer votes to other vewponts [Lowe01] Slde credt: Davd Lowe
Applcatons Son Abo (Evoluton Robotcs) SIFT usage Recognze dockng staton Communcate wth vsual cards Other uses Place recognton Loop closure n SLAM K. Grauman, B. Lebe 54 Slde credt: Davd Lowe
Locaton Recognton Tranng [Lowe04] Slde credt: Davd Lowe
Another applcaton: categor recognton Goal: dentf what tpe of object s n the mage Approach: algn to known objects and choose categor wth best match? Shape matchng and object recognton usng low dstorton correspondence, Berg et al., CVPR 2005: http://www.cnbc.cmu.edu/cns/papers/berg-cvpr05.pdf
Examples of Matches
Examples of Matches
Other deas worth beng aware of Thn-plate splnes: combnes global affne warp wth smooth local deformaton Robust non-rgd pont matchng: A new pont matchng algorthm for non-rgd regstraton, CVIU 2003 (ncludes code, demo, paper)
HW 2: Feature Trackng functon featuretrackng % Man functon for feature trackng folder = '.\mages'; m = readimages(folder, 0:50); tau = 0.06; [pt_x, pt_] = getkeponts(m{1}, tau); % Threshold for harrs corner detecton % Prob 1.1: kepont detecton ws = 7; [track_x, track_] =... trackponts(pt_x, pt_, m, ws); % Trackng ws x ws patches % Prob 1.2 Kepont trackng % Vsualzng the feature tracks on the frst and the last frame fgure(2), magesc(m{1}), hold off, axs mage, colormap gra hold on, plot(track_x', track_', 'r');
HW 2 Feature/Kepont detecton Compute second moment matrx Harrs corner crteron Threshold Non-maxmum suppresson ) ( ) ( ) ( ) ( ) ( ), ( 2 2 D D x D x D x I D I I I I I I I g 2 2 2 2 2 2 )] ( ) ( [ )] ( [ ) ( ) ( x x x I g I g I I g I g I g ] )), ( [trace( )], ( det[ 2 D I D I har 0.04
functon [kexs, keys] = getkeponts(m, tau) % m: nput mage; tau: threshold % kexs, keys: detected keponts, wth dmenson [N] x [1] % 0. Smooth mage (optonal) % 1. Compute mage gradents. Hnt: can use gradent % 2. Compute Ix*Ix, I*I, Ix*I % 3. Smooth Ix*Ix, I*I, Ix*I (locall weghted average) % 4. Compute Harrs corner score. Normalze the max to 1 % 5. Non-maxmum suppresson. Hnt: use ordflt2 % 6. Fnd postons whose values larger than tau. Hnt: use end fnd
functon [track_x, track_] = trackponts(pt_x, pt_, m, ws) % Trackng ntal ponts (pt_x, pt_) across the mage sequence % track_x: [Number of keponts] x [nm] % track_: [Number of keponts] x [nm] % Intalzaton N = numel(pt_x); % Number of keponts nm = numel(m); % Number of mages track_x = zeros(n, nm); track_ = zeros(n, nm); track_x(:, 1) = pt_x(:); track_(:, 1) = pt_(:); for t = 1:nm-1 % Trackng ponts from t to t+1 [track_x(:, t+1), track_(:, t+1)] =... getnextponts(track_x(:, t), track_(:, t), m{t}, m{t+1}, ws); end
Iteratve L-K algorthm Orgnal (x,) poston 1. Intalze (x, ) = (x,) 2. Compute (u,v) b I t = I(x,, t+1) - I(x,, t) 2 nd moment matrx for feature patch n frst mage dsplacement 3. Shft wndow b (u, v): x =x +u; = +v; 4. Recalculate I t 5. Repeat steps 2-4 untl small change Use nterpolaton for subpxel values
functon [x2, 2] = getnextponts(x,, m1, m2, ws) % Iteratve Lucas-Kanade feature trackng % (x, ) : ntalzed kepont poston n m1; (x2, 2): tracked kepont postons n m2 % ws: patch wndow sze % 1. Compute gradents from Im1 (get Ix and I) % 2. Grab patches of Ix, I, and m1. Hnt 1: use [X, Y] = meshgrd(-hw:hw,-hw:hw); to get patch ndex, where hw = floor(ws/2); Hnt 2: use nterp2 to sample non-nteger postons. for ter = 1:numIter % 5 teratons should be suffcent % Check f tracked patch are outsde the mage. Onl track vald patches. % For each kepont % - grab patch1 (centered at x1, 1), grab patch2 (centered at x2,2) % - compute It = patch2 patch1 % - grab Ix, I (centered at x1, 1) % - Set up matrx A and vector b % - Solve lnear sstem d = A\b. % - x2(p)=x2(p)+d(1); 2(p)=2(p)+d(1); -> Update the ncrement end
HW 2 Shape Algnment Global transformaton (smlart, affne, perspectve) Iteratve closest pont algorthm
functon Tfm = algn_shape(m1, m2) % m1: nput edge mage 1 % m2: nput edge mage 2 % Output: transformaton Tfm [3] x [3] % 1. Fnd edge ponts n m1 and m2. Hnt: use fnd % 2. Compute ntal transformaton (e.g., compute translaton and scalng b center of mass, varance wthn each mage) for = 1: 50 % 3. Get nearest neghbors: for each pont p fnd correspondng match() = argmn dst(p, qj) j % 4. Compute transformaton T based on matches % 5. Warp ponts p accordng to T end
HW 2 Local Feature Matchng Implement dstance rato test feature matchng algorthm
functon featurematchng m1 = mread('stop1.jpg'); m2 = mread('stop2.jpg'); % Load pre-computed SIFT features load('sift_features.mat'); % Descrptor1, Descrptor2: SIFT features % Frame1, Frame2: poston, scale, rotaton % For ever descrptor n m1, fnd the 1 st nearest neghbor and the 2 nd nearest neghbor n m2. % Compute dstance rato score. % Threshold and get the matches: a 2 x N arra of ndces that ndcates whch keponts from mage1 match whch ponts n mage 2 fgure(1), hold off, clf plotmatches(m2double(m1),m2double(m2),frame1,frame2,mat ches); % Dspla the matched keponts
Thngs to remember Algnment Hough transform RANSAC ICP Object nstance recognton Fnd keponts, compute descrptors Match descrptors Vote for / ft affne parameters Return object f # nlers > T
What have we learned? Interest ponts Fnd dstnct and repeatable ponts n mages Harrs-> corners, DoG -> blobs SIFT -> feature descrptor Feature trackng and optcal flow Fnd moton of a kepont/pxel over tme Lucas-Kanade: brghtness consstenc, small moton, spatal coherence Handle large moton: teratve update + pramd search Fttng and algnment fnd the transformaton parameters that best algn matched ponts Object nstance recognton Kepont-based object nstance recognton and search
Next week Perspectve and 3D Geometr