Part I: Single and Two View Geometry Internal camera parameters

Transcription

1 !! 43 1!???? Imaging eometry Multiple View eometry Perspective projection Richard Hartley Andrew isserman O p y VPR June 1999 where image plane This can be written as a linear mapping between homogeneous coordinates (the equation is only up to a scale factor) where a projection matri represents a map from 3D to D Image oordinate System Part I Single Two View eometry Internal camera parameters The main points covered in this part are A perspective (central) projection camera is represented by a matri The most general perspective transformation transformation between two planes (a world plane the image plane, or two image planes induced by a world plane) is a plane projective transformation This can be computed from the correspondence of four (or more) points The epipolar geometry between two views is represented by the fundamental matri This can be computed from the correspondence of seven (or more) points " cam cam where the units of are 0+,,- + (*) '& < =9> ; A9B " D E9F! y y 0 p cam cam < =9> ycam cam cam cam < =9> where " "

2 4!! -, *+ " is a matri amera alibration Matri upper triangular matri, called the camera calibration There are four A9B (i) The scaling in the image " D E9F! directions, (ii) The principal point, which is the point where the optic ais intersects the image plane The aspect ratio is "! " oncatenating the three matrices, < = > which defines the an image A B Note, the camera centre is at D E F projection matri from Euclidean 3-space to < = = =9> In the following it is often only the rather than its decomposition form of that is important, World oordinate System A Projective amera Eternal camera parameters cam cam cam < = = =9> < = = =9> cam cam O cam Euclidean transformation between world camera coordinates cam R, t O The camera model for perspective projection is a linear map between homogeneous point coordinates &!" Image Point ' () 01 & Scene Point is a is a rotation matri translation vector The camera centre is the null-vector of eg if then the centre is 4 has 11 degrees of freedom (essential parameters) has rank 3

3 amera alibration (Resectioning) (DLT) Problem Statement is a scene point, where iven correspondences its image ompute such that The algorithm for camera calibration has two parts from a set of point correspondences (i) ompute the matri decomposition via the into (ii) Decompose (DLT) Algorithm step 1 ompute the matri Each correspondence generates two equations Multiplying out gives equations linear in the matri elements of These equations can be rearranged as a 1-vector with What does calibration give? provides the transformation between an image point a ray in Euclidean 3-space is known the camera is termed calibrated Once A calibrated camera is a direction sensor, able to measure the direction of rays like a D protractor < =9> cam 6 798! cam E9F! " A9B < =9> ; cam Angle between rays d 1 1 θ - d - where

4 ! 0 Algorithm step 1 continued Algorithm step Decompose into Solving for (i) oncatenate the equations from ( ) correspondences to generate simultaneous equations, which can be written, where is a matri (ii) In general this will not have an eact solution, but a (linear) solution which minimises, subject to is obtained from the eigenvector with least eigenvalue of Or equivalently from the vector corresponding to the smallest singular value of the SVD of (iii) This linear solution is then used as the starting point for a non-linear minimisation of the difference between the measured projected point ( 0 The first submatri,, of triangular rotation matri matri decomposition This de- (i) Factor into termines (ii) Then using the is the product ( ) of an upper Note, this produces a matri with an etra skew parameter A B " D E F! the angle between the image aes Eample - alibration Object Determine accurate corner positions by (i) Etract link edges using anny edge operator (ii) Fit lines to edges using orthogonal regression (iii) Intersect lines to obtain corners to sub-piel accuracy The final error between measured projected points is typically less than 00 piels

5 Weak Perspective Plane projective transformations Track back, whilst zooming to keep image size fied π π perspective weak perspective The imaging rays become parallel, the result A B A generalization is the affine camera D E F hoose the world coordinate system such that the plane of the points has zero coordinate Then the matri reduces to < = A B D E F < =9= = A B D E F which is a matri representing a general plane to plane projective transformation < = > The Affine amera Projective transformations continued The matri has rank A9B D E9F Projection under an affine camera is a linear mapping on non-homogeneous coordinates composed with a translation < =9> The point is the image of the world origin The centre of the affine camera is at infinity An affine camera has 8 degrees of freedom It models weak-perspective para-perspective or < A9B, where is a D E9F < =9> O π non-singular homogeneous matri This is the most general transformation between the world image plane under imaging by a perspective camera It is often only the form of the matri that is important in establishing properties of this transformation A projective transformation is also called a homography a collineation has 8 degrees of freedom Π

6 Four points define a projective transformation The one of Rays iven ompute point correspondences such that Each point correspondence gives two constraints multiplying out generates two linear equations for the elements of An image is the intersection of a plane with the cone of rays between points in 3-space the optical centre Any two such images (with the same camera centre) are related by a planar projective transformation If (no three points collinear), then is determined uniquely The converse of this is that it is possible to transform any four points in general position to any other four points in general position by a projectivity eg rotation about the camera centre Eample 1 Removing Perspective Distortion Eample Synthetic Rotations iven the coordinates of four points on the scene plane Find a projective rectification of the plane This rectification does not require knowledge of any of the camera s parameters or the pose of the plane It is not always necessary to know coordinates for four points The synthetic images are produced by projectively warping the original image so that four corners of an imaged rectangle map to the corners of a rectangle Both warpings correspond to a synthetic rotation of the camera about the (fied) camera centre

7 orrespondence eometry Two View eometry iven the image of a point in one view, what can we say about its position in another? ameras such that?? Baseline between the cameras is non-zero l iven an image point in the first view, where is the corresponding point in the second view? e e epipolar line for What is the relative position of the cameras? What is the 3D geometry of the scene? A point in one image generates a line in the other image This line is known as an epipolar line, the geometry which gives rise to it is known as epipolar geometry Images of Planes Epipolar eometry Projective transformation between images induced by a plane Epipolar Plane Π π π O Left epipolar line e e O Right epipolar line can be computed from the correspondence of four points on the plane The epipolar line is the image of the ray through The epipole is the point of intersection of the line joining the camera centres the baseline with the image plane The epipole is also the image in one camera of the centre of the other camera All epipolar lines intersect in the epipole

8 Epipolar pencil Homogeneous Notation Interlude A line is represented by the homogeneous 3-vector < =9> e e baseline As the position of the 3D point varies, the epipolar planes rotate about the baseline This family of planes is known as an epipolar pencil All epipolar lines intersect at the epipole for the line Only the ratio of the homogeneous line coordinates is significant point on line or two points define a line or p q l l two lines define a point m Epipolar geometry eample Matri notation for vector product e at e at The vector product can be represented as a matri multiplication infinity infinity 4 where D E F A B 4 " Epipolar geometry depends only on the relative pose (position orientation) internal parameters of the two cameras, ie the position of the camera centres image planes It does not depend on structure (3D points eternal to the camera) 4 is a is the null-vector of skew-symmetric matri of rank 4, since 4

9 Algebraic representation - the Fundamental Matri Properties of is a rank homogeneous matri It has 7 degrees of freedom ounting ompute from 7 image point correspondences is a rank homogeneous matri with 7 degrees of freedom Point correspondence If then are corresponding image points, Epipolar lines is the epipolar line corresponding to Epipoles is the epipolar line corresponding to omputation from camera matrices, where is the pseudo-inverse of is the centre of the first camera Note, 4 anonical cameras,, where 4 4 4, 4, Fundamental matri - sketch derivation Plane induced homographies given π iven the fundamental matri induced by a world plane is between two views, the homography O Hπ e e l O 4 where is the inhomogeneous 3-vector which parametrizes the 3- parameter family of planes Step 1 Point transfer via a plane Step onstruct the epipolar line 4 eg compute plane from 3 point correspondences iven a homography induced by a particular world plane, then a homography induced by any plane may be computed as 4 4 This shows that is a rank matri

10 Projective ambiguity of reconstruction Projective Reconstruction from views Solution is not unique without camera calibration Solution is unique up to a projective mapping Then verify Same problem holds however many views we have Statement of the problem Projective Distortion demo iven Projective distortion demo orresponding points in two images Find ameras 3D points such that

11 Basic Theorem Details of Projective Reconstruction - omputation of iven sufficiently many points to compute unique fundamental matri 8 points in general position 7 points not on a ruled quadric with camera centres Then 3D points may be constructed from two views Up to a 3D projective transformation Ecept for points on the line between the camera centres Methods of computation of left until later Several methods are available (i) Normalized 8-point algorithm (ii) Algebraic minimization (iii) Minimization of epipolar distance (iv) Minimization of symmetric epipolar distance (v) Maimum Likelihood (old-stard) method (vi) Others, Steps of projective reconstruction Factorization of the fundamental matri Reconstruction takes place in the following steps ompute the fundamental matri from point correspondences SVD method Factor the fundamental matri as (i) Define The two camera matrices are A9B D E9F 4 4 (ii) ompute the SVD ompute the points by triangulation where diag (iii) Factorization is Simultaneously corrects to a singular matri

12 4 4 Factorization of the fundamental matri Triangulation Direct formula Let be the epipole Triangulation Knowing Knowing ompute such that Solve Specific formula 4 4 This solution is identical to the SVD solution 4 d d O e e O Non-uniqueness of factorization Triangulation in presence of noise Factorization of the fundamental matri is not unique eneral formula for varying In the presence of noise, back-projected lines do not intersect 4 Difference factorizations give configurations varying by a projective transformation 4-parameter family of solutions with 4 O O Rays do not intersect in space l = F l = F e image 1 image Measured points do not lie on corresponding epipolar lines e

13 Which 3D point to select? Linear triangulation methods Mid-point of common perpendicular to the rays? Not a good choice in projective environment oncepts of mid-point perpendicular are meaningless under projective distortion Weighted point on common perpendicular, weighted by distance from camera centres? Distance is also undefined concept Some algebraic distance? Write down projection equations solve? Linear least squares solution Minimizes nothing meaningful Direct analogue of the linear method of camera resectioning iven equations are the rows of Write as linear equations in Solve A A9A B D E E9E F eneralizes to point match in several images Minimizes no meaningful quantity not optimal Problem statement Minimizing geometric error Assume camera matrices are given without error, up to projective distortion Hence is known A pair of matched points in an image are given Possible errors in the position of matched points Find 3D point that minimizes suitable error metric Method must be invariant under 3D projective transformation Point images in space maps to projected points Measured points are in the two Find that minimizes difference between projected measured points

14 eometric error Minimization method d d O e e O Our strategy for minimizing cost function is as follows (i) Parametrize the pencil of epipolar lines in the first image by a parameter Epipolar line is (ii) Using the fundamental matri, compute the corresponding epipolar line in the second image (iii) Epress the distance function as a function of (iv) Find the value of that minimizes this function eplicitly ost function Different formulation of the problem Minimization method Minimization problem may be formulated differently Minimize range over all choices of corresponding epipolar lines is the closest point on the line Same for to Find the minimum of a function of a single variable, Problem in elementary calculus Derivative of cost reduces to a -th degree polynomial in Find roots of derivative eplicitly compute cost function Provides global minimum cost (guaranteed best solution) Details See Hartley-Sturm Triangulation l = F d d l = F θ(t) e θ (t) image 1 image e

15 Multiple local minima ost function may have local minima Shows that gradient-descent minimization may fail 1 This page left empty Left Eample of a cost function with three minima Right ost function for a perfect point match with two minima Uncertainty of reconstruction This page left empty Uncertainty of reconstruction The shape of the uncertainty region depends on the angle between the rays

16 Single point equation - Fundamental matri ives an equation < = = = = = = =9= = = = = =9>??????? ? 7 798? where????????? holds the entries of the Fundamental matri Total set of equations < =9= =9= = = = = = = =9= = >??????? ? 7? 8 E9F A9B omputation of the Fundamental Matri Basic equations iven a correspondence The basic incidence relation is May be written?????????

17 + + +! + Solving the Equations omputing from 7 points Solution is determined up to scale only Need 8 equations 8 points 8 points unique solution points least-squares solution has 9 entries but is defined only up to scale Singularity condition has 3 rows has only 7 degrees of freedom It is possible to solve for gives a further constraint is a cubic constraint from just 7 point correspondences Least-squares solution (i) Form equations (ii) Take SVD (iii) Solution is last column of (iv) Minimizes subject to (corresp smallest singular value) The singularity constraint 7-point algorithm Fundamental matri has rank omputation of from 7 point correspondences (i) Form the set of equations (ii) System has a -dimensional solution set (iii) eneral solution (use SVD) has form Left Uncorrected epipolar lines are not coincident Right Epipolar lines from corrected (iv) In matri terms (v) ondition! gives cubic equation in (vi) Either one or three real solutions for ratio

18 orrecting using the Singular Value Decomposition The normalized 8-point algorithm If is computed linearly from 8 or more correspondences, singularity condition does not hold SVD Method (i) SVD (ii) (iii) (iv) Set (v) Resulting are orthogonal, diag is singular diag (vi) Minimizes the Frobenius norm of (vii) is the closest singular matri to Raw 8-point algorithm performs badly in presence of noise Normalization of data 8-point algorithm is sensitive to origin of coordinates scale Data must be translated scaled to canonical coordinate frame Normalizing transformation is applied to both images Translate so centroid is at origin Scale so that RMS distance of points from origin is Average point is omplete 8-point algorithm Normalized 8-point algorithm (i) Normalization Transform the image coordinates 8 point algorithm has two steps (i) Linear solution Solve to find (ii) onstraint enforcement Replace Warning This algorithm is unstable should never be used with unnormalized data (see net slide) by (ii) Solution ompute from the matches (iii) Singularity constraint Find closest singular (iv) Denormalization to

19 Statue images omparison of Normalized Unnormalized Algorithms Lifia House images renoble Museum

20 Oford Basement Testing methodology (i) Point matches found in image pairs outliers discarded (ii) Fundamental matri was found from varying number ( points was tested against other matched points not used to com- (iii) pute it ) of (iv) Distance of a point from predicted epipolar line was the metric (v) 100 trials for each value of (vi) Average error is plotted against alibration object omparison of normalized unnormalized 8-point algorithms Average Error House Average Error Statue N N Distance of points from epipolar lines Top Unnormalized 8-point algorithm Bottom Normalized 8-point algorithm

21 omparison of normalized unnormalized 8-point algorithms Illustration of Effect of Normalization Average Error Museum Average Error N N Distance of points from epipolar lines Top Unnormalized 8-point algorithm Bottom Normalized 8-point algorithm alibration Similar problem omputation of a D projective transformation given point matches in two images (i) Homography is computed from noisy point matches (ii) Homography is applied to a further (6th) point (iii) Noise level approimately equal to with of lines in the crosses (net page) (iv) Repeated 100 times (v) Spread of the transformed 6th point is shown in relation to the data points (vi) 9 ellipses are plotted omparison of normalized unnormalized 8-point algorithms Normalization D homography computation 3 3 Average Error orridor Unnormalized data N Distance of points from epipolar lines Top Unnormalized 8-point algorithm Bottom Normalized 8-point algorithm Normalized data

22 Normalization D homography computation Unnormalized data Algebraic Minimization Algorithm Normalized data ondition number The algebraic minimization algorithm Bad condition number the reason for poor performance ondition number ondition number , ratio of singular values of House N no normalization with normalization Enforcing the singularity constraint SVD method minimizes simple rapid Not optimal Treats all entries of equally However, some entries of data are more tightly constrained by the Bad conditioning acts as a noise amplifier Normalization improves the condition number by a factor of Reference Hartley In defence of the 8-point algorithm

23 + + + Minimize The Algebraic Method subject to is a cubic constraint AND Requires an iterative solution However, simple iterative method works Write Minimize Solution assuming known epipole - continued subject to This is a linear least-squares estimation problem Non-iterative algorithm involving SVD is possible Reference Hartley, Minimizing Algebraic Error, Royal Society Proceedings, 1998 Solution assuming known epipole Iterative Algebraic Estimation We may write 4 of this form is singular Assume Write??????? ? 7 798? < = = = = = = = =9= =9= = =9> is known, find A A A A A A A A9A A9A A A9B as, where is epipole D E E E E E E E E9E E9E E E9F < = = = = = = = =9= =9= = =9> Find the fundamental matri subject to oncept Vary epipole Remark Each choice of epipole vector as above that minimizes the algebraic error to minimize the algebraic error defines a minimimum error Use Levenberg-Marquardt method to minimize this error Simple minimization problem 3 inputs the coordinates of the epipole 9 outputs the algebraic error vector Each step requires estimation of using SVD method Tricks can be used to avoid SVD (see Hartley-Royal-Society)

24 The old Stard (ML) Method Minimization of eometric Error Assumes a aussian distributed noise Measured correspondences Estimated correspondences subject to eactly for some Simultaneous estimation of d d l l = F = F Minimization of eometric Error The old Stard (ML) Method Algebraic error vector has no clear geometric meaning Should be minimizing geometric quantities Errors derive from incorrect measurements of match points Minimizing the old-stard error function Initial 3D reconstruction 4 d l = F l = F d ompute Iterate over 4 T T to minimize cost function We should be measuring distances from epipolar lines Total of parameters 1 parameters for the camera matri 3 parameters for each point Once 4 is found, compute 4

25 +??? Sparse Levenberg-Marquardt Parametrization of rank- matrices Reference Hartley- Azores Both epipoles as parameters The resulting form of is (i) oordinates of do not affect or (for (ii) Sparse LM takes advantage of sparseness (iii) Linear time in (number of points) (iv) Reference Hartley- Azores A9B D E9F Parametrization of rank- matrices Epipolar distance Estimation of Parametrize may be done by parameter minimization such that Various parametrizations have been used Overparametrization d l = F l = F d Write 4 3 parameters for Epipolar parametrization A B D E F Point correspondence Point epipolar line Epipolar distance is distance of point Write Distance is to epipolar line To achieve a minimum set of parameters, set one of the elements, for instance to 1

26 Epipolar distance - continued Luong hang s other error function Epipolar distance may be written as Total cost function Represents a first-order approimation to geometric error Total cost sum over all Minimize this cost function over parametrization of Symmetric epipolar distance Epipolar distance function is not symmetric Prefer sum of distances in both images Symmetric cost function is Sum over all points This page left empty ost Problem Points near the epipole have a disproportionate influence Small deviation in point makes big difference to epipolar line

27 Eperimental procedure More Algorithm omparison (i) Find matched points in image pair (ii) Select matched points at rom (iii) ompute the fundamental matri (iv) ompute epipolar distance for all other points (v) Repeat 100 times for each collect statistics The error is defined as ie Average symmetric epipolar distance Eperimental Evaluation of the Algorithms Results Three of the algorithms compared (i) The normalized 8-point algorithm (ii) Minimization of algebraic error whilst imposing the singularity constraint (iii) The old Stard geometric algorithm Error 4 3 houses Error statue Number of points Number of Points Normalized 8-point, geometric algebraic error

28 4 Results - continued 0 Error 1 1 orridor Error calibration ovariance computation Number of Points Number of Points Normalized 8-point, geometric algebraic error Results - continued ovariance of To compute the covariance matri of the entries of Error 3 1 museum (i) Define (ii) ompute the derivative matrices (iii) ompute in steps vector of meaurments in both images Number of Points Normalized 8-point, geometric algebraic error (pseudo-inverse)

29 ovariance of Special cases of -computation To compute the covariance of (i) iven (ii) ompute (iii) 4, then 4 Using special motions can simplify the computation of the fundamental matri ovariance of epipolar line corresponding to (i) Epipolar line is (ii) iven (iii) compute The envelope of epipolar lines Pure translation May compute envelope of epipolar lines is a hyperbola that contains the epipolar line with a given probability chosen such that, represents the cumulative with probability distribution, the lines lie within this region an assume 4 4 is skew-symmetric has dof 4 Being skew-symmetric, automatically has rank e epipolar line demonstration here For a pure translation the epipole can be estimated from the image motion of two points image

30 ameras with the same principal plane Points on a ruled quadric Principal plane of the camera is the third row of ameras have the same third row Affine cameras - last row is Simple correspondences eist for any has the following A9B D E9F (i) If all the points the two camera centres lie on a ruled quadric, then there are three possible fundamental matrices (ii) points lie in a plane The correspondences lead to a 3-parameter family of possible fundamental matrices (note, one of the parameters accounts for scaling the matri so there is only a two-parameter family of homogeneous matrices) (iii) Two cameras at the same point The fundamental matri does not eist There is no such thing as an epipolar plane, epipolar lines are not defined orrespondences give at least a -parameter family of Degeneracies orrespondences are degenerate if they satisfy more than one This page left empty