Automatic Estimation of Epipolar Geometry

Transcription

1 Robust line estimation Automatic Estimation of Epipolar Geometry Fit a line to 2D data containing outliers c b a d There are two problems: (i) a line fit to the data ;, (ii) a classification of the data into inliers (valid points) outliers Problem Statement RANdom SAmple onsensus (RANSA) Given Image pair [Fischler Bolles, 1981] Objective Robust fit of a model to a data set which contains outliers Algorithm Find The fundamental matrix ompute image points ompute correspondences ompute epipolar geometry correspondences (i) Romly select a sample of from this subset data points from instantiate the model (ii) Determine the set of data points which are within a distance threshold of the model The set is the consensus set of the sample defines the inliers of (iii) If the size of (the number of inliers) is greater than some threshold re-estimate the model using all the points in terminate (iv) If the size of is less than (v) After trials the largest consensus set re-estimated using all the points in the subset, select a new subset repeat the above is selected, the model is,

2 Robust ML estimation orrelation matching A B D A B D An improved fit by A better minimal set Robust MLE: instead of Match each corner to most similar looking corner in the other image Many wrong matches (10-50%), but enough to compute the fundamental matrix Feature extraction: orner detection orrespondences consistent with epipolar geometry Interest points [Harris] 100s of points per image Use RANSA robust estimation algorithm Obtain correspondences Guided matching by epipolar line Typically: final number of matches is about , with distance error of 02 pixels

3 Automatic Estimation of correspondences Adaptive RANSA Algorithm based on RANSA [Torr] (i) Interest points: ompute interest points in each image (ii) Putative correspondences: use cross-correlation proximity (iii) RANSA robust estimation: Repeat (a) Select rom sample of 7 correspondences (b) ompute (c) Measure support (number of inliers) hoose the with the largest number of inliers (iv) MLE: re-estimate from inlier correspondences (v) Guided matching: generate additional matches, sample count While sample count Repeat hoose a sample count the number of inliers Set Set from with Increment the sample count by one Terminate eg for a sample size of 4 Number of 1 - Adaptive inliers 6 2% % % % % % 43 How many samples? For probability of no outliers: eg for, number of samples, size of sample set, proportion of outliers Sample size Proportion of outliers 5% 10% 20% 25% 30% 40% 50% This page left empty

4 Two View Reconstruction Ambiguity Part 2 : Three-view Multiple-view Geometry omputing a Metric Reconstruction Given: image point correspondences compute a reconstruction:, with Ambiguity is an equivalent Projective Reconstruction Reconstruction from two views Metric Reconstruction Given only image points their correspondence, what can be determined? projective metric orrect: angles, length ratios

5 Algebraic Representation of Metric Reconstruction Direct Metric Reconstruction ompute Use 5 or more 3D points with known Euclidean coordinates to determine Projective Reconstruction Metric Reconstruction How? Remaining ambiguity is rotation (3), translation (3) scale (1) Only 8 parameters required to rectify entire sequence ( alibration points: position of 5 scene points Scene geometry: eg parallel lines/planes, orthogonal lines/planes, length ratios Auto-calibration: eg camera aspect ratio constant for sequence ) Projective Reconstruction Stratified Reconstruction Given a projective reconstruction, compute a metric reconstruction via an intermediate affine reconstruction (i) affine reconstruction: Determine the vector An affine reconstruction is obtained as which defines with (ii) Metric reconstruction: is obtained as with

6 Stratified Reconstruction Stratified reconstruction Start with a projective reconstruction Find transformation to upgrade to affine reconstruction Equivalent to finding the plane at infinity Find transformation to upgrade to metric (Euclidean) reconstruction Equivalent to finding the absolute conic Equivalent to camera calibration If camera calibration is known then metric reconstruction is possible Metric reconstruction implies knowledge of angles camera calibration (i) Apply the transformations one after the other : Projective transformation reduce to affine ambiguity Affine transformation reduce to metric ambiguity Metric ambiguity of scene remains Anatomy of a 3D projective transformation Reduction to affine ma- General 3D projective transformation represented by a trix metric affine projective Affine reduction using scene constraints - parallel lines

7 Reduction to affine Metric Reconstruction Other scene constraints are possible : Ratios of distances of points on line (eg equally spaced points) Ratios of distances on parallel lines Points lie in front of the viewing camera onstrains the position of the plane at infinity Linear-programming problem can be used to set bounds on the plane at infinity Gives so-called quasi-affine reconstruction Reference : Hartley-Azores Reduction to affine Metric Reconstruction ommon calibration of cameras With 3 or more views, one can find (in principle) the position of the plane at infinity Iteration over the entries of projective transform : Not always reliable Generally reduction to affine is difficult Assume plane at infinity is known Wish to make the step to metric reconstruction Apply a transformation of the form Linear solution exists in many cases

8 The Absolute onic Example of calibration Absolute conic is an imaginary conic lying on the plane at infinity Defined by ontains only imaginary points X Y Z Determines the Euclidean geometry of the space Represented by matrix diag Image of the absolute conic (IA) under camera given by Basic fact : T is Images taken with a non-translating camera: is unchanged under camera motion Using the infinite homography Mosaiced image showing projective transformations (i) When a camera moves, the image of a plane undergoes a projective transformation (ii) If we have affine reconstruction, we can compute the transformation of the plane at infinity between two images (iii) Absolute conic lies on the plane at infinity, but is unchanged by this image transformation : (iv) Transformation rule for dual conic (v) Linear equations on the entries of (vi) Given three images, solve for the entries of (vii) ompute by holeski factorization of

9 omputation of hanging internal camera parameters alibration matrix of camera is found as follows : ompute the homographies (2D projective transformations) between images Form equations The previous calibration procedure (affine-to-metric) may be generalized to case of changing internal parameters See paper tomorrow given by Agapito Solve for the entries of holeski factorization of gives Affine to metric upgrade Principal is the same for non-stationary cameras once principal plane is known is the infinite homography (ie via the plane at infinity) between images May be computed directly from affinely-correct camera matrices Given camera matrices This page left empty Infinite homography is given by Algorithm proceeds as for fixed cameras

10 Nice Video Nice alibration Results Pan angle Video Here pan (deg) PAN linear non linear frame Nice alibration Results Focal Length Nice alibration Results Tilt angle Focal length (pixels) FOAL LENGTH linear non linear Frame tilt (deg) TILT linear non linear frame

11 Nice alibration Results Focal Length Keble alibration Results Focal Length v_o (pixels) PRINIPAL POINT linear non linear Focal length (pixels) FOAL LENGTH linear non linear u_o (pixels) Frame Keble ollege Video Keble alibration Results Pan angle Video Here pan (deg) PAN linear non linear frame

12 Keble alibration Results Tilt angle 10 8 TILT linear non linear tilt (deg) This page left empty frame Keble alibration Results Focal Length PRINIPAL POINT v_o (pixels) linear non linear This page left empty u_o (pixels)

13 Geometry of three views Point-line-line incidence L The Trifocal Tensor x l / l / / / / / orrespondence The Trifocal Tensor Geometry of three views (i) Defined for three views (ii) Plays a similar rôle to Fundmental matrix for two views (iii) Unlike fundamental matrix, trifocal tensor also relates lines in three views (iv) Mixed combinations of lines points are also related Let General line Four plane are be two lines that meet in back-projects to a plane,, The four planes meet in a point L l / / x / / l / /

14 The trifocal relationship Tensor Notation Four planes meet in a point means determinant is zero Line coordinates Line is represented by a vector In new coordinate system, line has coordinate vector, This is a linear relationship in the line coordinates Also (less obviously) linear in the entries of the point This is the trifocal tensor relationship Line coordinates transform according to Preserves incidence relationship Point lies on line if : Terminology : transforms covariantly Use lower indices for covariant quantities Tensor Notation Summation notation Point coordinates onsider basis set Point is represented by a vector New basis : With respect to new basis represented by where Repeated index in upper lower positions implies summation Example Incidence relation is written Transformation of covariant contravariant indices ontravariant transformation If basis is transformed according to transform according to Terminology : transforms contravariantly Use upper indices for contravariant quantities, then point coordinates ovariant transformation

15 More transformation examples Basic Trifocal constraint amera mapping has one covariant one contravariant index : Transformation rule is Basic relation is a point-line-line incidence L Trifocal tensor Transformation rule : has one covariant two contravariant indices x l / l / / / / / Point lines in image 1 in images 2 3 The tensor Basic Trifocal constraint Tensor : L Defined for unless, are distinct if is an even permutation of if is an odd permutation of x l / / l / / / / Related to the cross-product : Let The four lines be two lines that meet in point,, The four planes meet in a point back-project to planes in space

16 / Line Transfer Basic relation is Interpretation : meets intersection of back-projected Back projected ray from planes from to a line passing Line in space projects to lines through L l / / / / x l / Line transfer Denote lies on the projection of the intersection when See that of the planes represents the transferred line corresponding to Thus We write ross-product of the two sides are equal : by by replacing Derived from basic relation Derivation of the basic three-view relationship -th row is where back-projects to plane Line Four planes are coincident if means determinant where 4-way wedge Thus gives Multiply by constant : is the point Intersection (cross-product) of Definition of the trifocal tensor Basic relationship is Define Point-line-line relation is ), contravariant in the other two is covariant in one index (

17 Point transfer via a plane ontraction of trifocal tensor with a line 2 1 image 2 / l x x // 3 Write Then represents the homography from image 1 to image 3 via the plane of the line image 1 X π / image 3 2 image 2 Line Ray from back-projects to a plane meets in a point This point projects to point For fixed, mapping in the third image is a homography 1 image 1 / l x x // X image 3 π / 3 Point-transfer the trifocal tensor Three-point correspondence If is any line through, then trifocal condition holds Given a triple correspondence must represent the point hoose any lines Trifocal condition holds passing through Alternatively (cross-product of the two sides) 2 image 2 Derived from basic relation by replacing by 1 x / l x/ // l x // 3 X image 1 L image 3 π /

18 Geometry of the three-point correspondence Summary of incidence relations 2 (i) Point in first view, lines in second third image 2 1 x / l x/ X // l x // 3 (ii) Point in first view, point in second line in third image 1 L image 3 π / (iii) Point in first view, line in second point in third 4 choices of lines May also be written as 4 equations (iv) Point in three views Gives 9 equations, only 4 linearly independent Summary of transfer formulas Degeneracies of line transfer (i) Point transfer from first to third view via a plane in the second L π π / (ii) Point transfer from first to second view via a plane in the third l l / (iii) Line transfer from third to first view via a plane in the second; or, from second to first view via a plane in the third 1 e e / epipolar 2 plane Degeneracy of line transfer for corresponding epipolar lines When the line lies in an epipolar plane, its position can not be inferred from two views Hence it can not be transferred to a third view

19 Degeneracy of point transfer Finding epipolar lines 1 x e 12 B 12 e21 l / image 1 image 3 X π / 2 image 2 Transferring points from images 1 2 to image 3 : Only points that can not be transferred are those points on the baseline between centres of cameras 1 2 For transfer with fundamental matrix, points in the trifocal plane can not be transferred B 23 x // 3 To find the epipolar line corresponding to a point Transfer to third image via plane back-projected from Epipolar line satisfies For all Epipolar line corresponding to for each such : found by solving Result : Epipole is the common perpendicular to the null-space of all ontraction on a point In write Represents a mapping from line to the point : As varies Epipolar line in third image traces out the projection of the ray through Epipole is the intersection of these lines for varying This page left empty x // x / l / / /

20 Where does this formula come from? Formula for trifocal tensor Extraction of camera matrices from trifocal tensor Notation : means omit row Example, when Formula for Trifocal tensor Extraction of the camera matrices Trifocal tensor is independent of projective transformation May assume that first camera is Other cameras are Formula Note : represent the epipoles : entre of the first camera is Epipole is image of camera centre Basic formula Entries of But if epipoles Strategy : are quadratic in the entries of camera matrices Estimate the epipoles are known, entries are linear Solve linearly for the remaining entries of 27 equations in 18 unknowns Exact formulae are possible, but not required for practical computation

21 This page left empty This page left empty Matrix formulas involving trifocal tensor Given the trifocal tensor written in matrix notation as (i) Retrieve the epipoles Let be the left right null vectors respectively of ie, Then the epipoles are obtained as the null-vectors to the following matrices, (ii) Retrieve the fundamental matrices ) (with (iii) Retrieve the camera matrices Normalize the epipoles to unit norm Then This page left empty

22 Summary of relations Other point or line correspondences also yield constraints on omputation of the trifocal tensor orrespondence Relation number of equations three points 4 two points, one line 2 one point, two lines three lines Trilinear Relations 1 2 Linear equations for the trifocal tensor Solving the equations Given a 3-point correspondence The trifocal tensor relationship is Relationship is linear in the entries of each correspondence gives 9 equations, 4 linearly independent has 27 entries defined up to scale 7 point correspondences give 28 equations Linear or least-squares solution for the entries of Given 26 equations we can solve for the 27 entries of Need 7 point correspondences or 13 line correspondences or some mixture Total set of equations has the form With 26 equations find an exact solution With more equations, least-squares solution

23 Solving the equations What are the constraints Solution : Take the SVD : corresponding to smallest sin- Solution is the last column of gular value) Minimizes subject to Normalization of data is essential Some of the constraints are easy to find (i) Each must have rank 2 (ii) Their null spaces must lie in a plane (iii) This gives 4 constraints in all (iv) 4 other constraints are not so easily formulated onstraints onstraints through parametrization has 27 entries, defined only up to scale Geometry only has 18 degrees of freedom 3 camera matrices account for dof Invariant under 3D projective transformation (15 dof) Total of 18 dof must satisfy several constraints to be a geometrically valid trifocal tensor To get good results, one must take account of these constraints (cf Fundamental matrix case) Define Only valid Recall formula for in terms of a set of parameters s may be generated from parameters : Only valid trifocal tensors are generated by this formula Parameters are the entries Over-parametrized : 24 parameters in all

24 Algebraic Estimation of Minimization knowing the epipoles Similar to the algebraic method of estimation of Minimize the algebraic error (i) (ii) (iii) is the vector of entries of is of the form subject to Difficulty is this constraint is a quadratic constraint in terms of the parameters Minimization problem becomes Minimize Minimize subject to subject to Exactly the same problem as with the fundamental matrix Linear solution using the SVD Reference : Hartley Royal Society paper Minimization knowing the epipoles Algebraic estimation of amera matrices, omplete algebraic estimation algorithm is As with fundamental matrix, image is linear in terms of the other parame- If are known, then ters We may write are the epipoles of the first (i) Find a solution for method (ii) Estimated using the normalized linear (7-point) will not satisfy constraints (iii) ompute the two epipoles (a) Find the left (respectively right) null spaces of each (b) Epipole is the common perpendicular to the null spaces (iv) Reestimate epipoles by algebraic method assuming values for the is the matrix of 18 remaining entries of camera matrices is the 27-vector of entries of is a matrix

25 Iterative Algebraic Method Find the trifocal tensor that minimizes subject to oncept : error Vary epipoles defines a minimi- Remark : Each choice of epipoles mum error vector as above to minimize the algebraic Use Levenberg-Marquardt method to minimize this error Simple minimization problem 6 inputs the entries of the two epipoles 27 outputs the algebraic error vector Each step requires estimation of using algebraic method This page left empty This page left empty This page left empty

26 Reconstruction from three views Automatic Estimation of a Projective Reconstruction for a Sequence Given: image point correspondences compute a projective reconstruction: with, What is new? X verify correspondences 1 x 3 x 2 x 3 O O 1 3-view tensor: the trifocal tensor ompute from 6 image point correspondences Automatic algorithm similar to [Torr & Zisserman] 2 O Outline Automatic Estimation of the trifocal tensor correspondences (i) Pairwise matches: ompute point matches between view pairs using robust estimation (ii) Putative correspondences: over three views from two view matches (iii) RANSA robust estimation: Repeat (a) Select rom sample of 6 correspondences first frame of video (i) Projective reconstruction: 2-views, 3-views, N-views (ii) Obtaining correspondences over N-views (b) ompute (1 or 3 solutions) (c) Measure support (number of inliers) hoose the with the largest number of inliers (iv) MLE: re-estimate from inlier correspondences (v) Guided matching: generate additional matches

27 Projective Reconstruction for a Sequence Interest points computed for each frame (i) ompute all 2-view reconstructions for consecutive frames (ii) ompute all 3-view reconstructions for consecutive frames (iii) Extend to sequence by hierarchical merging: (iv) Bundle-adjustment: minimize reprojection error About 500 points per frame first frame of video (v) Automatic algorithm [Fitzgibbon & Zisserman ] ameras Scene Geometry for an Image Sequence Point tracking: orrelation matching Given video first frame of video Point correspondence (tracking) Projective Reconstruction 10-50% wrong matches first frame of video

28 Point tracking: Epipolar-geometry guided matching Reconstruction from Point Tracks ompute 3D points cameras from point tracks first frame of video ompute so that matches consistent with epipolar geometry Many fewer false matches, but still a loose constraint a frame of the video Hierarchical merging of sub-sequences Bundle adjustment Point tracking: Trifocal tensor guided matching Application I: Graphical Models ompute VRML piecewise planar model first frame of video ompute trifocal tensor so that matches consistent with 3-views Tighter constraint, so even fewer false matches Three views is the last significant improvement

29 Example II: Extended Sequence Application II: Augmented Reality 140 frames of a 340 frame sequence Using computed cameras scene geometry, insert virtual objects into the sequence a frame of the video a frame of the video 330 frames Metric Reconstruction 3D Insertion 140 frames of a 340 frame sequence a frame of the video a frame of the video

30 Further Reading Some of these papers are available from Beardsley, P, Torr, P H S, Zisserman, A 3D model acquisition from extended image sequences In B Buxton ipolla R, editors, Proc 4th European onference on omputer Vision, LNS 1065, ambridge, pages Springer Verlag, 1996 Deriche R, Zhang Z, Luong Q T, Faugeras O, Robust recovery of the epipolar geometry for an uncalibrated stereo rig In J O Eckl, editor, Proc 3rd European onference on omputer Vision, LNS 800/801, Stockholm, pages Springer-Verlag, 1994 Faugeras, O What can be seen in three dimensions with an uncalibrated stereo rig?, Proc EV, LNS 588 Springer-Verlag, 1992 vgg Faugeras, O, Three-Dimensional omputer Vision, MIT Press, 1993 Faugeras, O Stratification of three-dimensional vision: projective, affine, metric representation, J Opt Soc Am, A12: , 1995 Fischler, M A Bolles, R, Rom sample consensus: A paradigm for model fitting with applications to image analysis automated cartography, Semple, J Kneebone, G Algebraic Projective Geometry Oxford University Press, 1979 Shashua, A Trilinearity in visual recognition by alignment volume 1, pages , May 1994 In Proc EV, Spetsakis, M E Aloimonos, J, Structure from motion using line correspondences IJV, 4(3): , 1990 Torr P H S, Outlier Detection Motion Segmentation PhD thesis, University of Oxford, Engineering Dept, 1995 Torr, P H S Zisserman, A, Robust parameterization computation of the trifocal tensor Image Vision omputing, 15: , 1997 ommam, 24, 6, , 1981 Fitzgibbon, A Zisserman, A, Automatic camera recovery for closed or open image sequences In Proc EV, pages Springer-Verlag, June 1998 Hartley, R, Gupta, R, hang, T Stereo from uncalibrated cameras, Proc VPR, 1992 Hartley, R I, In defence of the 8-point algorithm In IV5, 1995 Hartley, R I, A linear method for reconstruction from lines points Proc IV, pages , 1995 Hartley, R I Zisserman, A, Multiple View Geometry in omputer Vision, ambridge University Press, to appear Kanatani, K, Geometric omputation for Machine Vision Oxford University Press, Oxford, 1992 Longuet-Higgins H, A computer algorithm for reconstructing a scene from two projections Nature, vol293: , 1981 Luong, Q T Vieville, T, anonical representations for the geometries of multiple projective views VIU, 64(2): , September 1996 Mundy, J Zisserman, A, Geometric invariance in computer vision, MIT Press, 1992 Introduction hapter 23 (projective geometry) In Acknowledgements The images in this document were created by Paul Beardsley, Antonio riminisi, Andrew Fitzgibbon, David Liebowitz, Luc Robert, Phil Torr