In ths lecture, we re gong to talk about a number of problems related to fttng and matchng. We wll formulate these problems formally and our dscusson wll nvolve Least Squares methods, RANSAC and Hough votng. We wll conclude the lecture wth a few remarks on how fttng can be often used to solve a matchng problem. Lecture 9 Fttng and Matchng Problem formulaton Least square methods RANSAC Hough transforms Mult- model fttng Fttng helps matchng! Readng: [HZ] Chapter: 4 Estmaton D projectve transformaton Chapter: 11 Computaton of the fundamental matrx F [FP] Chapter:10 Groupng and model fttng Some sldes of ths lectures are courtesy of profs. S. Lazebnk & K. Grauman Lecture 8 - Slvo Savarese 4-Feb-15 Fttng Goals: Choose a parametrc model to ft a certan quantty from data Estmate model parameters Fttng s about tryng to ft observed data nto a parametrc model that we are assumng holds. The process of fttng such model to the data nvolves estmatng the parameters that descrbe the model such that the fttng error s mnmzed. A classc example s lne fttng: gven a set of ponts n D, the goal s to fnd the parameters that descrbe the lne so as to best ft the set of D ponts. Smlar problems can be defned for other geometrcal quanttes such as curves, homographc transformatons, fundamental matrces or even object shapes. - Lnes - Curves - Homographc transformaton - Fundamental matrx - Shape model Example: fttng lnes (for computng vanshng ponts) The next couple of sldes show some examples of models we could be tryng to ft our data to. In ths example, the goal s ft a lne gven a number of pont observatons (red dots). Fttng lnes n an mage can be useful, for nstance, to estmate the scene vanshng ponts.
Example: Estmatng an homographc transformaton Here s another example, where we have ponts (red dots) that le on the same planar regon (the façade of the buldng) and that are observed from two dfferent vews. Because they le on the same planar regons, we expect them to be related by a homographc transformaton. So the goal here would be to ft (and estmate) such a homographc transformaton, gven the set of correspondng ponts. H Example: Estmatng F In ths example, the goal s ft a fundamental matrx transformaton F that relates (at least) 7 pont correspondences. Example: fttng a D shape template One more example s fttng a shape template (the letter A on the top-rght) wth an observaton (mage) of the letter A. A
Example: fttng a 3D object model A fnal example s fttng a 3D shape template of a car wth an observaton (mage) of the car. Notce that n ths case the fttng problem can also be nterpreted as a recognton or matchng problem. In fact, fttng, matchng and recognton are nterconnected problems. Fttng, matchng and recognton are nterconnected problems Fttng Fttng becomes problematc n presence of: - nosy data - outlers Crtcal ssues: - nosy data - outlers - mssng data - mssng data We dscuss ths n detals n the next few sldes.
Crtcal ssues: nosy data Crtcal ssues: nosy data (ntra-class varablty) Data may not always ft the model exactly because of measurement or detecton nose. Also, the model may not ft exactly the data because of ntra-class varablty the shape of the template may be dfferent from the observaton. Ths s stll true even f there s no measurement nose. A Crtcal ssues: outlers H Another class of problems stem from the presence of outlers. An outler s a data pont that s not explaned by the model but t stll appears among the data pont we wsh to ft. In ths example, the outler s the par of correspondng ponts shown n blue. Ths par of correspondng ponts s clearly ncorrect but nevertheless t may be produced by mstake by the algorthm that generates pont correspondences. The fttng procedure should somehow be able to account for t.
Crtcal ssues: mssng data (occlusons) A thrd class of problems s due to occlusons. Ths takes place when some of the ponts we wsh to ft are not vsble. Occlusons become a crtcal ssue n shape fttng (or matchng) problems whereby the observaton of the object we want to match aganst s not fully vsble. Fttng Goal: Choose a parametrc model to ft a certan quantty from data Technques: Least square methods RANSAC Hough transform There are several technques that we can use to ft a parametrc model to the data: Least square methods RANSAC Hough transform EM (Expectaton Maxmzaton) In ths lecture we wll explan the detals of the frst 3 methods; we don t cover the 4 th one n ths course. Let s start wth the least square methods. EM (Expectaton Maxmzaton) [not covered] Data: (x 1, y 1 ),, (x n, y n ) Least squares methods - fttng a lne - y=mx+b We llustrate the least square technque usng the smple example of lne fttng. In ths problem, our goal s to ft a parametrc model that descrbes the equaton of the lne to a seres of ponts labeled as (x 1,y 1 ), (x,y ),, (x n,y n ). If we model the lne as y = mx + b, our goal s to estmate the parameters m, b so as to mnmze the value of E gven n (Eq. ). E s the sum of resduals or how far off our estmate s for each pont. We can nterpret ths as choosng a lne to mnmze the dstance between the lne and the observed data ponts. Lne equaton: y m x b = 0 [Eq. 1] (x, y ) Fnd (m, b) to mnmze E = n = 1 ( y mx b) [Eq. ]
E = n = 1 Least squares methods - fttng a lne - ( y mx b) [Eq. ] ' y1 $ ' x1 1$ n - ' m$ * ' m$ E = y [ ] 1 x 1 % " % " + ( = % b " = %! " %!! "% b ", & #) % y " % n x n 1" & # & # & # = Y XB [Eq. 3] We can formulate ths easly as a least squares problem. Eq. descrbes the objectve functon we are tryng to mnmze. We can re-wrte ths equaton n the matrx form expressed n Eq. 3, where B = [m b] T and X and Y collect the coordnates of the data ponts as ndcated n the equaton. By expandng the square of the norm of Y X B, we obtan the expresson n Eq. 4. Our goal s to fnd the parameters m and b that mnmzes E. These can be found by takng the dervatve of E wth respect to B and equal t to zero (Eq. 5). Solvng ths equaton allows to fnd B as expressed n Eq. 6. Ths soluton can be nterpreted as the optmal (n the least squares sense) parameters m, b that mnmze the resduals, or, equvalently, choosng a lne that comes closest to contanng all of the ponts. Eq. 7 s typcally called the normal equaton. T T T T = (Y XB) (Y XB) = Y Y (XB) Y + (XB) (XB) [Eq. 4] Fnd B=[m, b] T that mnmzes E de T T = X Y + X XB = 0 [Eq. 5] db T T X XB = X Y [Eq. 7] Normal equaton B = T 1 T ( X X) X Y [Eq. 6] Least squares methods - fttng a lne - There s a major lmtaton to ths smple method t fals completely for vertcal lnes. In ths case m would be nfnty (or some very large number) whch may ntroduce numercal error n our solutons. E = B n = 1 (y mx b) T 1 T ( X X) X Y & m# = B = $! % b " [Eq. 6] y=mx+b (x, y ) Lmtatons Fals completely for vertcal lnes Dstance between pont (x n, y n ) and lne ax+by=d Least squares methods - fttng a lne - ax+by=d To remedy ths ssue, we can use a very smlar set up wth a small tweak. Instead of the slope-ntercept form of a lne y = mx + b, we can parameterze our lne equaton usng ax + by = d. Ths new parameterzaton leads us to a new expresson for E whch s expressed n Eq. 8. Ths s very smlar to Eq. 1 earler, but parameterzed to avod havng nfnte slope. It can be shown that mnmzng E wth respect to a, b and d s equvalent to solvng the homogenous system expressed n Eq. 9, where h =[a b d] T and the matrx A collects all the coordnates of the data ponts. We leave the detals of ths dervaton as an exercse. We can solve Eq. 9 usng SVD. Fnd (a, b, d) to mnmze the sum of squared perpendcular dstances E = n = 1 ( ax + b y [Eq. 8] d) (x, y ) A h = 0 [Eq. 9] data model parameters
A h = 0 Mnmze A h subject to h = 1 T A = UDV Least squares methods - fttng a lne - A s rank defcent As we have seen n prevous lectures, the soluton h of the system n Eq. 9 can be computed as the last column of V, where U*D*V = svd(a). A few more detals kndly offered by Phl: When we take the SVD of a matrx, we are separatng the matrx nto three peces, U (drectons of output), D (a dagonal matrx wth scalng amounts for dfferent drectons), and V (drectons of nput). In ths case, nput to the matrx means beng multpled by a vector on the rght. We can nterpret the SVD as when we have an nput to A that s n the drecton of the nth column of V, t wll get scaled by the nth value of D, and be output as the nth column of U. Thus, when the matrx of scalng factors D s sorted (as s custom/ what matlab does), the last column of V dctates the nput drecton of A that wll have the least output. Therefore, that s the drecton of nput to A that wll get the output closest to zero and s the vector that wll mnmze Ah. h = last column of V y Least squares methods - fttng an homography - H y Another popular problem s the one of fttng the homographc transformaton H to pars of correspondng ponts (see fgure). It can be shown that fndng the H that best fts all the data s equvalent to solvng the homogenous system n Eq. 10, where A collects all the data ponts (.e., coordnates of correspondng ponts n the two mages) and h collects the 9 unknown parameters that descrbe the homographc transformaton H (.e. the coeffcent of H). Remember that H has 8 degrees of freedom n that t s known up to scale. As we dd before, h can be found n the least square sense, through SVD. x x A h = 0 [Eq. 10] data model parameters Least squares: Robustness to nose Least squares methods can work well when the data are nosy. The fgure shows the result of a smulaton where we used least square to ft all the red ponts wth a lne.
Least squares: Robustness to nose However, least square s n general not very robust to the presence of outlers as the result shown here demonstrates. The outler marked n blue sgnfcantly alters the qualty of the fttng process. outler! Crtcal ssues: outlers Another example s shown here for the case of fttng homographc transformatons. H CONCLUSION: Least square s not robust w.r.t. outlers [Eq. 1] Least squares: Robust estmators Instead of mnmzng E = E = n = 1 ( u ) ρ ;σ ( ax + b y d) We mnmze [Eq. 11] u = error (resdual) of th pont w.r.t. model parameters h = (a,b,d) ρ = robust functon of u wth scale parameter σ (ρ=rho) ρ [Eq. 8] u = ax + b y d Robust functon ρ: When u s large, ρ saturates to 1 When u s small, ρ s a functon of u To make our estmator less senstve to the presence of outlers, we can use a slghtly modfed cost functon whch s called robust estmator. Instead of mnmzng the sum of the squares of the resduals, we look to mnmze Eq. 11, where the u s are the orgnal resduals that we have been usng and ρ(u ; σ) s a robust functon of the resduals and of the scale parameter σ (Eq. 1). In the lower part of the slde, as an example, we see a plot of ρ(u ; σ) as functon of u and σ. Ths functon s more robust n that t penalzes sets of large resduals and prefer very small resduals. Indeed: When u s large, ρ saturates to 1 When u s small, ρ s a functon of u u In concluson: Favors a confguraton wth small resduals Penalzes large resduals
Least squares: Robust estmators Instead of mnmzng E = E = n = 1 ( u ) ρ ;σ ( ax + b y d) We mnmze [Eq. 11] u = error (resdual) of th pont w.r.t. model parameters h = (a,b,d) [Eq. 8] u = ax + b y d The scale parameter σ regulates how much weght we gve to potental outlers. Indeed a small value of sgma hghly penalzes large resduals (whch means we wll end up gvng less weght to the outlers). A large value of sgma mldly penalzes large resduals and act smlarly to Eq. 8 (.e., lke n a standard least square soluton, outlers wll affect our result more). The plot n the fgure shows dfferent profles of ρ(u ; σ) for dfferent values of σ (e.g., σ=0.1, σ=1 and σ=10). ρ = robust functon of u wth scale parameter σ ρ [Eq. 1] Small sgma à hghly penalze large resduals Large sgma à mldly penalze large resdual (lke LSQR) u Least squares: Robust estmators A downsde to usng the robust estmator s that now we have one more parameter to choose what value of sgma to use. Here, we have an example of a good choce of sgma for the earler data. Good scale parameter σ The effect of the outler s elmnated Least squares: Robust estmators Here s an example of a bad choce of sgma. σ s too small and we end up favorng a soluton for the lne that only fts ponts locally (.e., most of the ponts are consdered as outlers). Ths n general produces very unstable results. Bad scale parameter σ (too small!) Fts only locally
Least squares: Robust estmators Bad scale parameter σ (too large!) Same as standard LSQ Here s another example of a bad choce of sgma. σ s too large and we end up wth a smlar result to the orgnal least squares functon. In concluson, the robust estmator s useful when we have pror knowledge about what σ should be or some way t to fnd out. Generally, n practce, people use least squares as a good startng pont and use an teratve approach to fnd the optmal robust fttng method (whch s a non lnear problem). CONCLUSION: Robust estmator useful f pror nfo about the dstrbuton of ponts s known Robust fttng s a nonlnear optmzaton problem (teratve soluton) Least squares soluton provdes good ntal condton Fttng Next, we wll talk about RANSAC whch stands for Random Sample Consensus. RANSAC was ntroduced by Fschler & Bolles n 1981. Goal: Choose a parametrc model to ft a certan quantty from data Technques: Least square methods RANSAC Hough transform Basc phlosophy (votng scheme) Data elements are used to vote for one (or multple) models Robust to outlers and mssng data The basc dea s that we are gong to set up a votng procedure where each data ponts counts for a vote for one or more models. Ideally, ponts that are nosy wll not vote for the same model and ponts that are relevant wll vote for consstent models. Ths would make our system robust to outlers and mssng data (two of the challenges we mentoned earler). Ths basc dea s shared by RANSAC and Hough Votng (as we shall see next). RANSAC s desgned to be robust to outlers and mssng data and follows two major assumptons: 1: Nosy data ponts wll not vote consstently for any sngle model ( few outlers) : There are enough data ponts to agree on a good model ( few mssng data) Assumpton 1: Nosy data ponts wll not vote consstently for any sngle model ( few outlers) Assumpton : There are enough data ponts to agree on a good model ( few mssng data)
RANSAC (RANdom SAmple Consensus) : Fschler & Bolles n 81. We start we a very hgh level overvew of RANSAC. Suppose we have the standard lne fttng problem n presence of outlers. We can formulate ths problem as follows: we want to fnd the best partton of ponts n the nler set P and outler set O such that we keep as many nler ponts as necessary, but that they also result n small resduals (that s, a resdual that s smaller than δ for every nler pont). In Eq.1, r(p,h) s the resdual assocated to the nler pont P and the model parameter h. δ { P O} π : I, such that: mn O r(p, h) < δ, P P r(p, h) = resdual [Eq. 1] π Model parameters RANSAC In practce, RANSAC can be formulated as follows: As before, we have a set of D ponts that we want to ft a lne to. There are three major steps to RANSAC. Sample set = set of ponts n D Algorthm: 1. Select random sample of mnmum requred sze to ft model. Compute a putatve model from sample set 3. Compute the set of nlers to ths model from whole data set Repeat 1-3 untl model wth the most nlers over all samples s found RANSAC Frst, we select a random sample of the mnmum requred sze to ft the model. In ths case, a lne s determned by at least two ponts. Thus, to ft a model of a lne, we select two ponts at random whch consttutes the random sample. In ths example, the random sample s made up by the two green ponts. Sample set = set of ponts n D Algorthm: 1. Select random sample of mnmum requred sze to ft model [?]. Compute a putatve model from sample set 3. Compute the set of nlers to ths model from whole data set Repeat 1-3 untl model wth the most nlers over all samples s found
RANSAC Next, we calculate a model from ths sample set. Usng the two green ponts that we have already randomly sampled, we calculate the lne that these two represent. Sample set = set of ponts n D Algorthm: 1. Select random sample of mnmum requred sze to ft model [?]. Compute a putatve model from sample set 3. Compute the set of nlers to ths model from whole data set Repeat 1-3 untl model wth the most nlers over all samples s found RANSAC From here, we see how much of our entre set of D ponts agrees wth ths model up to a tolerance δ. In ths case, we see that all of the ponts that are now blue agree wth the model hypotheszed by the two green ponts. Thus, we have an nler set P of 6 ( green + 4 blue) and an outler set (denoted by O n the slde) of 14. δ Sample set = set of ponts n D Algorthm: 1. Select random sample of mnmum requred sze to ft model [?]. Compute a putatve model from sample set 3. Compute the set of nlers to ths model from whole data set Repeat 1-3 untl model wth the most nlers over all samples s found O =? =14 P =? = 6 RANSAC We repeat these steps untl the number of ponts n the nler set s mnmzed or we meet some other crtera (to be dscussed n the followng sldes). In ths example, we ve chosen the two ponts n green. The model that these two ponts mply would nclude all of the blue ponts. Ths gves us 14 nlers and 6 outlers, whch looks to be a reasonable model. Indeed, ths s the model that maxmzes the nler set. Algorthm: δ O = 6 P =14 1. Select random sample of mnmum requred sze to ft model [?]. Compute a putatve model from sample set 3. Compute the set of nlers to ths model from whole data set Repeat 1-3 untl model wth the most nlers over all samples s found
How many samples? Number N of samples requred to ensure, wth a probablty p, that at least one random sample produces an nler set that s free from real outlers. Usually, p=0.99 It s often computatonally unnecessary (and nfeasble) to explore the entre sample space. A typcal assumpton s that only N samples are drawn, where N s the number of samples that s requred to ensure, wth a probablty p, that at least one random sample produces an nler set that s free from real outlers. Usually, p=0.99. Example Here we see an example of a random sample that produces an nler set that contans real outlers. real outler rato s 6/0 = 30% δ Here a random sample s gven by two green ponts Estmated nler set s gven by the green+blue ponts How many real outlers we have here? Example Here we see an example of a random sample that produces an nler set that does not contan any real outler (t s free from real outlers). real outler rato s 6/0 = 30% δ Random sample s gven by two green ponts Estmated nler set s gven by the green+blue ponts How many real outlers we have here? 0
How many samples? Number N of samples requred to ensure, wth a probablty p, that at least one random sample produces an nler set that s free from real outlers for a gven s and e. E.g., p=0.99 N = log 1 Estmatng H by RANSAC ( ) ( p) / log 1 ( 1 e) s proporton of outlers e s 5% 10% 0% 5% 30% 40% 50% 3 5 6 7 11 17 3 3 4 7 9 11 19 35 4 3 5 9 13 17 34 7 5 4 6 1 17 6 57 146 6 4 7 16 4 37 97 93 7 4 8 0 33 54 163 588 8 5 9 6 44 78 7 1177 e = outler rato s = mnmum number needed to ft the model Note: ths table assumes neglgble measurement nose [Eq. 13] N can be estmated usng equaton 13, where e denotes the outler rato (6/0 = 30% n the prevous example), s s the mnmum number of ponts needed to ft the model ( n the prevous example of a lne), and p s the probablty that at least one random sample produces an nler set that s free from real outlers for a gven s and e. The chart shows numbers of samples needed N for a 99% chance that at least one random sample produces an nler set that s free from real outlers. To read ths chart, we have s ponts needed to ft the model n the frst column and the proporton of outlers e n the top row. The crcled example shows that for a model that requres ponts and data that has 30% outlers, we need 7 samples (of two ponts each) to have a 99% chance that at least one random sample produces an nler set that s free from real outlers. A justfcaton of Eq 13 s reported below (kndly provded by Phllp Lee): We can say that p = (1-e)^s. Ths s the chance that f we pcked one set of samples (s ponts), all of them wll be nlers (w/ replacement). Then, we know that 1-p = 1-(1-e)^s. Ths s the chance that f we pcked one set of samples (s ponts), at least one of them wll be an outler. Then, (1-p) = (1-(1-e)^s)^n. Ths s the chance that f we pcked n sets of samples, at least one of them wll contan at least one outler. Wth some algebra, we see that: log(1-p) = log(1-(1-e)^s)^n) Log(1-p) = n*log(1-(1-e)^s) N = log(1-p)/log(1-(1-e)^s) Here are some other examples of when RANSAC can be useful. In ths example the goal s to ft the homographc transformaton H gven a set of pars of correspondng ponts (correspondences) n presence of outlers. In ths example, the blue dashed lnes ndcate correct correspondences whereas the red ones are outlers. The steps of the algorthm are the same as before. The only dfference s the sze of the random sample whch s 4 n ths case. Why? H has 8 degrees of freedom, so we need 4 correspondences (because each pont has two coordnates) to compute a mnmum model. Thus, n ths case s=4. H 8 DOF Need 4 correspondences Outler match Sample set = set of matches between mages Algorthm: 1. Select a random sample of mnmum requred sze [?]. Compute a putatve model from these 3. Compute the set of nlers to ths model from whole sample space Repeat 1-3 untl model wth the most nlers over all samples s found Estmatng F by RANSAC Outler matches In ths example the goal s to ft the fundamental matrx F gven a set of pars of correspondng ponts (correspondences) n presence of outlers. The outlers n ths case are related to the dark red and cyan ponts, whereas all the other ponts are n correct correspondence. In ths case, the random sample s 8. Why? F has 7 degrees of freedom, so we need 7 correspondences (n fact, 8 f we use the 8 ponts algorthm) to compute a mnmum model. Thus, n ths case s=7(8). F 7 DOF Need 7 (8) correspondences Sample set = set of matches between mages Algorthm: 1. Select a random sample of mnmum requred sze [?]. Compute a putatve model from these 3. Compute the set of nlers to ths model from whole sample space Repeat 1-3 untl model wth the most nlers over all samples s found
RANSAC - conclusons Good: In concluson, RANSAC s an easly mplementable method to estmate a model that often works well n practce. However, there are many parameters to tune; t may take a long tme to get to the accuracy that you need, and requres the nler to outler rato to be reasonable. Emprcal studes show that RANSAC doesn t work well f the nler/outler raton s greater than 50%. Smple and easly mplementable Successful n dfferent contexts Bad: Many parameters to tune Trade-off accuracy-vs-tme Cannot be used f rato nlers/outlers s too small Fttng The Hough transform s the last method that we ll dscuss today. Goal: Choose a parametrc model to ft a certan quantty from data Technques: Least square methods RANSAC Hough transform Hough transform P.V.C. Hough, Machne Analyss of Bubble Chamber Pctures, Proc. Int. Conf. Hgh Energy Accelerators and Instrumentaton, 1959 Gven a set of ponts, fnd the curve or lne that explans the data ponts best Hough votng s a votng procedure as well, where each data ponts counts for a vote for one or more models that we want to ft to our data. Let s consder the example of lne fttng agan. Our goal s to estmate the parameter m and n of the lne (n dashed red color) that fts our data ponts (n red). Hough votng uses the concept of parameter (or dual) space. If the model we want to ft (.e. the lne) s represented parametrcally by, for nstance, m and n, then we can establsh a relatonshp between the orgnal space where the data ponts le, and the dual parameter space whch s defned by the parameters (.e., m and n) that descrbe the model we want to ft the data to (each axs corresponds to the a model parameter). y y = m x + n x
Hough transform P.V.C. Hough, Machne Analyss of Bubble Chamber Pctures, Proc. Int. Conf. Hgh Energy Accelerators and Instrumentaton, 1959 Gven a set of ponts, fnd the curve or lne that explans the data ponts best y (x, y ) (x 1, y 1 ) y = m x + n x m m n y 1 = m x 1 + n y = m x + n n Hough space In ths slde, on the left we have the orgnal mage x,y (Cartesan) coordnates and on the rght, we have the parameter (Hough) space. We want to show that the parameters that descrbe the lne y = m x + n (.e. m and n ) can be computed by ntersectng lnes n the Hough space. Let s pck up a pont (x 1, y 1 ) that belongs to the lne y = m x + n n the orgnal mage. Ths pont corresponds to the lne y 1 = m x 1 + n wth parameters x 1 and y 1 n the Hough space. Let s pck up a second pont (x, y ) that belongs to y = m x + n n the orgnal mage. Ths corresponds to the lne y = m x + n wth parameters x and y n the Hough space. Thus, t s easy to realze that the pont of ntersecton of y 1 = m x 1 + n and y = m x + n n the Hough space returns the coordnates m and n of the lne y = m x + n n the orgnal space. Ths t s true for any arbtrary pont we pck up from the orgnal space as long as t belongs to the lne y = m x + n. Thus, ths mechansm can be used for fttng lnes to ponts: gven a set of data ponts we want to ft a lne to, we assocate for each of these ponts a lne n the Hough space. Then, we ntersect these lnes n the Hough space. The pont of ntersectons returns the parameter of the lne fttng the ponts n the orgnal space. y = m x + n Hough transform As we already dscussed for the least square case, the parameter space [m,n] s unbounded; that s, m and n can range from mnus nfnty to nfnty (thnk of a vertcal lne). P.V.C. Hough, Machne Analyss of Bubble Chamber Pctures, Proc. Int. Conf. Hgh Energy Accelerators and Instrumentaton, 1959 Any Issue? The parameter space [m,n] s unbounded Hough transform P.V.C. Hough, Machne Analyss of Bubble Chamber Pctures, Proc. Int. Conf. Hgh Energy Accelerators and Instrumentaton, 1959 Any Issue? The parameter space [m,n] s unbounded Use a polar representaton for the parameter space Thus, t s common to consder the polar parameterzaton of the lne as shown n Eq. 13, nstead of y = mx + n. Now all possble lnes n Cartesan space have a snusodal profle n hough space. The Hough votng procedure s stll the same n that the parameters rho and theta of the lne (fttng the data ponts n the orgnal space) are estmated as the pont of ntersecton of all the snusodal profles n the Hough space. y ρ ρ θ Orgnal space x Hough space θ x cosθ + ysnθ = ρ [Eq. 13]
Hough transform - experments ρ Here s an example of the Hough transform n acton. Orgnal space Hough space θ Hough transform - experments Nosy data ρ When the data ponts are nosy, the correspondng snusodal profles don t ntersect at the exactly same pont. So how do we compute the parameters of the lne n ths case? Let s call a pont of ntersecton of two snusods a vote. The dea s to dvde the Hough space nto grd and count how many votes we have n each cell of the grd. The cell that contans the largest number of votes (the yellow cell n the fgure) returns the parameters of the lne we want to ft. For nstance, such parameters can be estmated as the coordnates of the center of the cell that contans the largest number of vote. The name Hough votng comes from ths concept of countng votes n the Hough space. Orgnal space How to compute the ntersecton pont? IDEA: ntroduce a grd a count ntersecton ponts n each cell Issue: Grd sze needs to be adjusted Hough space θ Whle the dea of usng a grd helps dealng wth the case of nosy data, the grd sze s an addtonal parameter that we need to tune when runnng Hough votng. Small grd szes makes t harder to fnd the cell that contans the largest number of votes. Large grd szes decreases the accuracy n estmatng the parameters of the model (all of the values of θ and ρ wthn the most voted cell are good canddates for descrbng the lne). Hough transform - experments ρ An nterestng case s the one where we have a more or less unform dstrbuton of ponts n the mage (unform nose). In presence of unform nose, we don t fnd a clear consensus of an approprate model n the Hough space. However, random aggregatons of ponts along a lne n the orgnal space may produce consstent votng n certan cells whch result n lowerng the overall sgnal to nose rato. Orgnal space Hough space θ Issue: spurous peaks due to unform nose
Hough transform - conclusons Good: All ponts are processed ndependently, so can cope wth occluson/outlers Some robustness to nose: nose ponts unlkely to contrbute consstently to any sngle cell Of course, the Hough transform has pros and cons. An mportant pros s that t can successfully handle some degree of outlers and nose: An outler just produces an addtonal snusod that s very unlkely to matter as we count the number of votes n each cell. Ths s true as long as the outlers don t get aggregated n a way that they consstently vote for another model. The cons nclude the problem of havng unform nose to produce spurous peaks that may decrease the sgnal to nose rato. Also, t s not easy to fnd the rght sze of the grd unless we have some pror knowledge about how data are dstrbuted. Bad: Spurous peaks due to unform nose Trade-off nose-grd sze (hard to fnd sweet pont) Applcatons lane detecton A typcal applcaton of Hough votng (for lnes) s to detect lanes n mages for autonomous navgaton applcatons. Courtesy of Mnchae Lee Applcatons computng vanshng ponts Another typcal applcaton of Hough votng (for lnes) s to estmate vanshng lnes (and thus vanshng ponts).
Generalzed Hough transform [more on forthcomng lectures] D. Ballard, Generalzng the Hough Transform to Detect Arbtrary Shapes, Pattern Recognton 13(), 1981 Parameterze a shape by measurng the locaton of ts parts and shape centrod Gven a set of measurements, cast a vote n the Hough (parameter) space We can generalze the Hough votng scheme to fnd shapes that are much more complcated than a straght lne. The general concept s to parameterze a shape n term of the locaton of ts parts wth respect to the shape centrod. Gven a set of measurements, we can cast votes n the Hough (parameter) space; the cell wth the largest number of votes returns the probablty that the shape has been found. Ths dea has been used n desgnng methods for object detecton (see for nstance the work by Lebe et al., 004). We wll dscuss ths n detals n one of the upcomng lectures. Used n object recognton! (the mplct shape model) B. Lebe, A. Leonards, and B. Schele, Combned Object Categorzaton and Segmentaton wth an Implct Shape Model, ECCV Workshop on Statstcal Learnng n Computer Vson 004 Lecture 9 Fttng and Matchng We now dscuss the case of fttng data when these data can be explaned by more than one model. Problem formulaton Least square methods RANSAC Hough transforms Mult- model fttng Fttng helps matchng! Slvo Savarese Lecture 8-4-Feb-15 Fttng multple models Here s an example: gven ths dstrbuton of D ponts, we want to: - Dscover how many lnes explan the ponts say. - Estmate the parameters of these lnes that best fts the ponts. All of ths n presence of nose and outlers. There are many approaches to ths problem. Incremental fttng (sweepng through possble numbers of lnes), Expectaton maxmzaton (not covered n ths class, but usng the ft of a model to estmate how many lnes should be ft, then usng the estmate to ft the model n an teratve fashon), and the Hough transform (to vote for how many lnes are needed whle estmatng ther parameters). Incremental fttng E.M. (probablstc fttng) Hough transform
Incremental lne fttng Scan data pont sequentally (usng localty constrants) The dea of ncremental lne fttng s farly straghtforward. We ft a lne to N pont and calculate our resduals. Then we add a new pont and re-ft the lne and resdual. If the resdual ever goes over a threshold, we add another lne to our model. We contnue untl all (or enough) ponts are accounted for. Perform followng loop: 1. Select N pont and ft lne to N ponts. Compute resdual R N 3. Add a new pont, re-ft lne and re-compute R N+1 4. Contnue whle lne fttng resdual s small enough, When resdual exceeds a threshold, start fttng new model (lne) Courtesy of unknown Hough transform Hough votng provdes an appealng alternatve to ncremental fttng. The problem s essentally solved usng the same procedure as dscussed before. If data ponts are explaned by multple lnes (two n ths case), we wll observe more than one cell that contans a number of votes that s sgnfcantly hgher than those due to nose. The fgure shows an example where Hough votng s appled to the pont dstrbuton on the left hand sde. In the rght fgure, the n-axs and m-axs descrbe the Hough votng space, and the vertcal axs (counter) reports the number of votes that appears n each cell. We can clearly see two peaks (hence two lnes) emergng from all the other peaks that are due to nose. The parameters assocated to these peaks characterze the two lnes that ft the ponts n the orgnal space. Same cons and pros as before Lecture 9 Fttng and Matchng We wll wrap up ths lecture wth how fttng (estmatng models) nteract wth matchng (fndng correspondences between mages). Problem formulaton Least square methods RANSAC Hough transforms Mult- model fttng Fttng helps matchng! Slvo Savarese Lecture 8-4-Feb-15
Fttng helps matchng! Image 1 Image A common task n vson s to match feature ponts between mages. Each mage n the fgure shows a number of detected ponts obtaned, for nstance, usng a feature detector; these are are depcted as whte ponts. Feature ponts can be matched across mages usng a smlarty crteron such as the cross correlaton (see lecture 6) that s computed over a small wndow around the feature of nterest. wndow wndow Features are matched (for nstance, based on correlaton) Fttng helps matchng! Image 1 Image Unfortunately, for a number of reasons that we already extensvely dscussed durng lecture 6, ths matchng process s faulty and can produce a large number of msmatched (outlers) whch are ndcated n red n the fgure. A technque such as RANSAC can help. Suppose that these mages are related by an homographc transformaton H. Ths s actually true n ths case, snce the scene we are observng s very far from the camera. Then, we can run RANSAC to estmate the H that relates correspondng ponts. Not only does RANSAC produce a robust estmaton of H by rejectng the outlers, but also returns the lst of outlers (.e., the msmatches). Ths example shows how matchng and fttng are very much nterrelated problems. Matches based on appearance only Green: good matches Red: bad matches Idea: Fttng an homography H (by RANSAC) mappng features from mages 1 to Bad matches wll be labeled as outlers (hence rejected)! Fttng helps matchng! Usng our homography, we can be use to blend the two mages on top of each other after.
Recognsng Panoramas M. Brown and D. G. Lowe. Recognsng Panoramas. In Proceedngs of the 9th Internatonal Conference on Computer Vson -- ICCV003 Ths approach can be used to sttch mages so as to form panoramc mages and t s currently commonly used n many commercal applcatons for panorama processng. Next lecture: Feature detectors and descrptors Back up sldes for Least squares methods. Please see Lnear algebra
Least squares methods - fttng a lne - We saw earler that the Least Squares soluton to Ax = b s x = (A^T A)^-1 A^T b. Ths means that gven an A and b, ths s the value of x that comes closest to Ax=b wth respect to the cost functon Ax-b. X may not be an exact soluton when A s tall and full rank (meanng we have more equatons than unknowns). Ax = b More equatons than unknowns Look for soluton whch mnmzes Ax-b = (Ax-b) T (Ax-b) T Solve ( Ax b) ( Ax b) = 0 x LS soluton x = ( A A) T 1 A T b Least squares methods - fttng a lne - Another way to look at the least squares soluton s as a sngular valude decomposton (SVD) problem. The pseudo-nverse A-dagger, where x = (A-dagger) y can also be derved from SVD. Solvng x = ( A A) t 1 t A b A + t 1 = (A A) A = U V t A t = pseudo-nverse of A = SVD decomposton of A A A = V 1 1 + = V + U U + 1 wth equal to for all nonzero sngular values and zero otherwse Least squares methods - fttng an homography - To ft a homography, wth 4 ponts, we fnd the least squares soluton to Ah=0, where A and h are defned above. The soluton to ths wll be the last rght sngular vector of A. From n>=4 correspondng ponts: A h = 0 & h $ $ h $! $ % h 1,1 1, 3,3 #!! = 0!! "
Hough transform - experments Here s an example of very nosy data. It does not seem lke a lne should ft most of the ponts n Cartesan space. Accordngly, n Hough space, there does not seem to be a clear consensus of an approprate model. However there does seem to be a few ponts that seem to be potental lnes. These may correspond to lnes that would ft some of the data ponts, but not all of them. features votes Issue: spurous peaks due to unform nose