Multiple View Geometry Frank Dellaert
Outline Intro Camera Review Stereo triangulation Geometry of 2 views Essential Matrix Fundamental Matrix Estimating E/F from point-matches
Why Consider Multiple Views? P X P' x x' Answer: To extract 3D structure via triangulation.
Stereo Rig Top View Matches on Scanlines Convenient when searching for correspondences.
Feature Matching!
Real World Challenges Bad News: Good correspondences are hard to find Good news: Geometry constrains possible correspondences. 4 DOF between x and x'; only 3 DOF in X. Constraint is manifest in the Fundamental matrix F can be calculated either from camera matrices or a set of good correspondences.
Geometry of 2 views? What if we do not know R,t? Caveat: My exposition follows book conventions but more intuitive (IMHO) Different from Hartley & Zisserman! F&P use [R T -R T t] camera matrices H&Z uses [R t]
Epipolar Geometry Where can p appear? P p t C C M =[R T -R T t] M=[I 0]? p
Image of Camera Center epipole M =[R T -R T t] M=[I 0]
Example:Cameras Point at Each Other Top View Epipolar Lines
Epipoles Camera Center C in first view: [ ] t 1 e = I 0 " $ # Origin C in second view: % ' = t & # e'= [ R T "R T t] 0 & % ( = "R T t $ 1'
Image of Camera Ray? epipole M =[R T -R T t] M=[I 0]
Point at infinity Given p, what is corresponding point at infinity [x 0]? Answer for any camera M =[A a]: # p'= [ A a] " x & % ( = Ax ) x = A *1 p' $ 0' A -1 = Infinite homography In our case M =[R T -R T t]: x = Rp'
Sidebar: Infinite Homographies Homography between image plane plane at infinity Navigation by the stars: Image of stars = function of rotation R only! Traveling on a sphere rotates viewer
Essential Matrix
Epipolar Line Calculation 1) Point 1 = epipole e=t 2) Point 2 = point at infinity [ ] # Rp' p " = I 0 3) Epipolar line = join of points 1 and 2 $ & % 0 l = t " Rp' ' ) = Rp' (
P Epipolar Lines P p C e e=t C M =[R T -R T t] M=[I 0] p l = t! p =Rp Rp'
Epipolar lines e=t l = t! Rp' p =Rp
Epipolar Plane P p l' l p C e e=t C M =[R T -R T t] M=[I 0]
Essential Matrix mapping from p to l l = t " Rp'= [ t] R # p'= E # p' " E = 3*3 matrix Because p is on l, we have p T Ep'= 0
E s Degrees of Freedom R,t = 6 DOF However, scale ambiguity! = 5 DOF
Fundamental Matrix FUNDAMENTAL
P Uncalibrated Case P p p =A -1 p p [ ]! e A 1 p' C e =a e=-a -1 a C M =K [R [A a] -1 K [R T -R -R T t] t] M=K[I 0] M=[I 0] l = "
Uncalibrated Case, Forsyth & Ponce Version Fundamental Matrix (Faugeras and Luong, 1992)
Fundamental Matrix mapping from p to l l = e " A #1 p'= [ e] A #1 $ p'= F $ p' " F = 3*3 matrix Because p is on l, we have p T Fp'= 0
Properties of the Fundamental Matrix Fp is the epipolar line associated with p. F T p is the epipolar line associated with p. F T e=0 and Fe =0. F is singular.
The Eight-Point Algorithm (Longuet-Higgins, 1981) Minimize: under the constraint 2 F =1.
Non-Linear Least-Squares Approach (Luong et al., 1993) Minimize with respect to the coefficients of F, using an appropriate rank-2 parameterization.
The Normalized Eight-Point Algorithm (Hartley, 1995) Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels: q i = T p i q i = T p i. Use the eight-point algorithm to compute F from the points q i and q i. Enforce the rank-2 constraint. Output T -1 F T.
RANSAC Frank Dellaert Slides all my own this time
Motivation Estimating motion models Typically: points in two images Candidates: Translation Homography Fundamental matrix
Mosaicking: Homography www.cs.cmu.edu/~dellaert/mosaicking
Fundamental Matrix
Omnidirectional example Images by Branislav Micusik, Tomas Pajdla, cmp.felk.cvut.cz/ demos/fishepip/
Simpler Example Fitting a straight line
Discard Outliers No point with d>t RANSAC: RANdom SAmple Consensus Fischler & Bolles 1981 Copes with a large proportion of outliers
Main Idea Select 2 points at random Fit a line Support = number of inliers Line with most inliers wins
Why will this work?
Best Line has most support More support -> better fit
In General Fit a more general model Sample = minimal subset Translation:? Homography? Fundamental Matrix?
RANSAC Objective: Robust fit of a model to data S Algorithm Randomly select s points Instantiate a model Get consensus set Si If Si >T, terminate and return model Repeat for N trials, return model with max Si
Distance Threshold Requires noise distribution Gaussian noise with σ Chi-squared distribution with DOF m 95% cumulative: Line, F: m=1, t=3.84 σ 2 Translation, homography: m=2, t=5.99\ σ 2 I.e. -> 95% prob that d<t is inlier
How many samples? We want: at least one sample with all inliers Can t guarantee: probability p E.g. p =0.99
Calculate N If w = proportion of inliers = 1-etha P(sample with all inliers)=w s P(sample with an outlier)=1-w s P(N samples an outlier)=(1-w s )^N We want P(N samples an outlier)<1-p (1-w s )^N<1-p N>log(1-p)/log(1-w s )
Example P=0.99 s=2, etha=5% => N=2 s=2, etha=50% => N=17 s=4, etha=5% => N=3 s=4, etha=50% => N=72 s=8, etha=5% => N=5 s=8, etha=50% => N=1177
Remarks N = f(etha), not the number of points N increases steeply with s
Threshold T Remember: terminate if Si >T Rule of thumb: T #inliers So, T=(1-etha)n
Adaptive N When etha is unknown? Start with etha=50%, N=inf Repeat: Sample s, fit model -> update etha as outliers /n -> set N=f(etha,s,p) Terminate when N samples seen