Unit 3 Multiple View Geometry

Transcription

1 Unit 3 Multiple View Geometry Relations between images of a scene Recovering the cameras Recovering the scene structure

2 3D structure from images Recover 3D structure from 2D images need at least two images from different viewpoints e.g., we have 2 eyes

3 3D structure from images Recover 3D structure from 2D images need at least two images from different viewpoints e.g., we have 2 eyes

4 3D structure from images Recover 3D structure from 2D images need at least two images from different viewpoints e.g., we have 2 eyes

5 3D structure from images Shape from Motion / Shape from Multiple Views / Photogrammetry Given: corresponding image points Unknown: camera positions, orientations, intrinsics, scene.. Multiple cameras Moving scene Moving camera

6 Two-view relations Epipolar geometry Place 2 cameras in space, so that the projection centers are different Each camera sees the other cameras proj.center => the epipole

7 Two-view relations Epipolar geometry Project an arbitrary scene point into both images The point and both projection centers define a plane => epipolar plane

8 Two-view relations Epipolar geometry Images of the epipolar plane / the projection rays => epipolar lines All epipolar lines pass through the epipole

9 Two-view relations Epipolar constraint: If an image point x in the first image and an image point x' in the second image originate from the same 3D scene point, then the corresponding rays must be coplanar For any two camera positions and orientations For any two sets of internal camera parameters For any viewed scene Equivalent: point x must lie on the epipolar line l Invariant to projective transformation

10 Algebraic Formulation Epipolar constraint is given by the fundamental matrix F F is a homogeneous (3x3) matrix with rank 2 Epipolar constraint is bilinear in x,x' 9 entries, scale is arbitrary, 1 rank constraint => 7 unknowns The epipolar geometry can be recovered only from image correspondences!

11 Algebraic Formulation Properties of matrix F (3x3) matrix rank(f)=2 Epipolar lines: l'=fx, l=ftx' Epipoles: Fe=0, FTe'=0 Constraint bilinear in x,x' Relation to cameras F=[e']xP'P+... P+=PT(PPT) 1 e'=p'c... PC=0

12 Estimating epipolar geometry 8-point algorithm Each correspondence gives one equation x'tfx=0 Normalize image points: y=t x, y'=t' x'... centroid=(0,0), rms-dist.=1 Equation system for elements of F: A f=0 Solve using SVD: f' corresponding to smallest singular value of A Enforce rank constraint: [U,diag(r,s,t),VT]=svd(F'), F''=U diag(r,s,0) VT Denormalize: F=T'T F'' T

13 Estimating epipolar geometry For canonical coordinates (known cameras) F reduces to the essential matrix E Replace u=k 1x, u'=k' 1x' then E=K'TFK and u'teu=0 The essential matrix encodes only translation and rotation There is no projective ambiguity, only a 4-fold ambiguity 8-point algorithm: E''=U diag( (r+s)/2, (r+s)/2, 0 ) VT

14 Reconstruction Extracting camera matrices From F Set first camera to canonical camera P=[I 0] The second camera is given by P'=[ [e']xf e'] with e'tf=0 From E Set coordinate origin to first camera center P=[I 0] Compute the SVD and set scale: [U,diag(1,1,0),VT]=svd(E) No projective ambiguity. Four possible choices for P' Only one where the a reconstructed point X is in front of both cameras

15 Important observation Given two views of a static scene, in which we can find a small number of corresponding points. If they were taken with any pinhole camera, we can reconstruct the scene up to a projective ambiguity, i.e., we can determine line intersections, coplanarity. If the camera has been internally calibrated, we can reconstruct the scene up to scale, i.e., we can determine angles and relative distances => 3D modeling If the distance between the camera centers is known, this determines the global scale, i.e., we can determine absolute distances => 3D navigation

16 Camera calibration Principle: from known correspondences between 2D image points and 3D world points, compute the camera matrix (also known as resection) normalize both point sets: x = Ty, X=UY Projection equations y~qy, so x x QX=0 Reordering for unknowns: Solve with SVD (improve with non-linear error minimization) Denormalize P=T 1QU (Factor into intrinsic and extrinsic calibration) P=[KR Krt]

17 Stereo rig calibration Principle: from known correspondences between 2D image points and 3D world points, compute both camera matrices Do the same as before, but for two cameras mounted on a stable rig (e.g., on a robot) Both camera matrices are known => can directly triangulate world points..

18 Triangulation Triangulation of scene points Linear method, geometrically not optimal Geometrically optimal method is non-linar (e.g., Hartley and Zisserman)

19 Self-calibration Upgrading projective model without known calibration Assumptions for self-calibration Scene constraints Parallel lines, vanishing points, angles, lengths, 3D points Camera constraints Extrinsic parameters for 5 views Intrinsic constraints: skew, aspect ratio, principal point, focal length

20 Self-calibration Example: 3 orthogonal vanishing points Estimate projective transformation, which brings all 3 vanishing points to infinity aligns them with the coordinate axes Example: Only focal length of cameras unknown (xp=yp=a=s=0) Leads to a linear set of constraints

21 Reconstruction The whole story [Pollefeys 2000]

22 More two-view geometry Homography x2=hx1 Projective mapping between two planes: H is a homogeneous (3x3) matrix, need 4 points to estimate it 2 cases: planar scene, or pure rotation around viewpoint Application: rectification of planes => can perform measurements

23 More two-view geometry Homography Images of camera rotating around center - panorama stitching need 4 correspondences to solve for H=KRK' 1

24 Cameras at infinity Projection center at (or near) infinity, rays are parallel Simplifications of general perspective camera Affine cameras Weak perspective orthographic

25 Cameras at infinity Weak perspective Principal point not defined Parallel projection, independent of Z Orthographic No skew, no aspect ratio

26 Multiple View Geometry For 3 views F generalizes to trifocal tensor T123=[T1,T2,T3] More constraints if views are added: 3 images of a line reproject to a line in 3D space Can be used for feature transfer: given coordinates of a point or line in 2 views, the coordinates in the third view can be found

27 Summary Two-view geometry Epipolar geometry Fundamental matrix / essential matrix Homographies Structure&Motion Recovery Extracting cameras Triangulation Self-calibration Affine cameras 3 views: trifocal tensor

28 Thank you for your attention!