Today. Stereo (two view) reconstruction. Multiview geometry. Today. Multiview geometry. Computational Photography

Transcription

1 Computational Photography Matthias Zwicker University of Bern Fall 2009 Today From 2D to 3D using multiple views Introduction Geometry of two views Stereo matching Other applications Multiview geometry Recover 3D structure from multiple images Exploit observation of individual scene points from multiple viewpoints Point correspondences: location a in image b shows same 3D point as location c in image d Problem is easy if relative position of cameras, corresponding pixels, and mapping of pixels to rays is known Triangulation Stereo (two view) reconstruction image plane focal point (center of projection) scene point corresponding pixels image plane focal point (center of projection) Assume relative camera pose, corresponding pixels, mapping of pixels to rays is known Find 3D point by triangulation Intersection point of rays Multiview geometry Want to derive relative camera pose, mapping of pixels to rays from correspondence information only Mathematical tool that allows to relate corresponding points in multiple images Describes how 3D points are mapped to 2D images Today From 2D to 3D Introduction Geometry of two views Stereo matching Other applications 1

2 The projective plane Why homogeneous coordinates? Represent points at infinity, homographies, perspective projection, multi-view relationships Geometric intuition: projective plane Projective Lines Point is a ray in projective space image plane in vector notation : 0 x z a b c y l T p f (fx,fy,f) (x,y,1) (0,0,0) image plane (projective) focal point Cartesian plane Each point (x,y) on plane represented by a ray s(x,y,1) Cartesian coordinates (x,y,z)(x/z, y/z) Lineis plane of rays All rays (x,y,z) satisfying ax + by + cz = 0 Lines also represented as homogeneous 3-vectors l Point and line duality Line l is homogeneous 3-vector (a ray) l is to every point (ray) p on it: l T p=0 l p 1 p 2 l 1 Line l spanned by points p 1 and p 2 l is to p 1 and p 2 l = p 1 p 2 l is the plane normal Intersection of two lines l 1 and l 2 p is to l 1 and l 2 p = l 1 l 2 Points and lines are dual every property of points also applies to lines (e.g., cross-ratio) l 2 p Homographies of points & lines Synonyms Homographies Planar perspective transformations Texture-mapping transformations (graphics) Collineations (straight lines are preserved) Computed by 3x3 matrix multiplication Transform a point: p = Hp Transform a line: l T p=0 l T p =0 0 =l T p = l T H -1 Hp = l T H -1 p l T = l T H -1 Lines are transformed by (H -1 ) T 3D projective geometry Natural generalization of 2D homogeneous coordinates to 3D Projective 3D points: X = (X,Y,Z,W) Duality A plane also represented ese by a 4-vector Points and planes are dual in 3D: T P=0 Projective transformations Represented by 4x4 matrices T Transformation of points P = TP Transformation of planes = (T -1 ) T Cross-ratio of planes 3D projective geometry However Can t use cross-products in 4D Need new tools Grassman-Cayley Algebra Generalization of cross product Allows interactions between points, lines, and planes via meet and join operators Won t get into this stuff today 2

3 Applications of projective geometry Homogeneous coordinates in computer graphics Metrology Single view Multi-view Camera calibration Stereo correspondence View interpolation, transformation, panorama stitching Invariants Object recognition Pose estimation Others... Today From 2D to 3D Introduction Geometry of two views Stereo matching Other applications Camera model Extrinsic parameters Describe camera pose relative to world reference frame (rotation, translation) Camera model Intrinsic parameters Describe relation of image locations to rays (focal length, image center, pixel aspect ratio) World coordinate Camera coordinate 1D image Image center Camera coordinate World coordinate Focal distance Camera coordinate World coordinate Camera model Intrinsic parameters Describe relation of image locations to rays (focal length, image center, pixel aspect ratio) 1D image World coordinate Focal distance Image center Camera coordinate Same extrinisic parameters, different intrinsic parameters 3D to 2D: perspective projection Matrix projection: 3D to 2D homogeneous coordinates Note sx * * s * * X * * Y Z * * 1 p sy * * * * ΠP Camera matrix Camera matrix invariant under uniform scale 11 degrees of freedom instead of 12 3

4 3D to 2D: perspective projection Decompose camera matrix into T R A Π sx tx 1 0 sy t y f 0 0 Intrinsic parameters, A T World, reference coordinate R 0 0 R 0 0 3x x x1 I T 3x3 3x1 1 01x3 1 Extrinsic parameters Mapping of rays to pixels Orientation and location of camera in world Projection of point P: p ΠP AR(PT) A Camera matrix Extrinsic parameters or extrinsic calibration Rotation and translation relative to reference coordinate (pose) Intrinsic parameters or intrinsic calibration Focal length Image center Aspect ratio Terminology Uncalibrated case Recover 3D structure only from point correspondences in multiple images Neither intrinsic nor extrinsic camera parameters known a priori Requires auto-calibration Calibrated case Intrinsic camera parameters are known Obtained in separate camera calibration step Camera calibration Problem statement: recover camera parameters given one or several images Often, camera calibration refers to determining intrinsic parameters Calibrated camera means intrinsic i i parameters are known Finding extrinsic parameters also known as pose estimation Camera calibration Simplest approach: determine camera matrix from known 3D calibration object Camera matrix calibration Direct linear calibration Find unknown entries m ij of camera matrix from known image points (u i, v i ) and 3D locations (X i, Y i, Z i ) One correspondence pair Fix uniform scale Calibration object 4

5 Direct linear calibration Direct linear calibration Assemble of linear equations from n correspondences Find least squares solution Direct linear calibration Advantages Very simple to formulate and solve Disadvantages Doesn t tell you the camera parameters Doesn t model lens distortion Doesn t minimize the right error function Need 3D calibration object Nonlinear methods are preferred Error function measures distance between projected 3D points and image positions Nonlinear function of camera parameters Use nonlinear optimization techniques Multi-plane calibration Use several images of planar target at unknown orientations Recover intrinsic & extrinsic parameters separately Also deals with lens distortion Code available, very popular Matlab In OpenCV Input images Un-distorted Today From 2D to 3D Introduction Geometry of two views Stereo matching Other applications Stereo (two view) reconstruction focal point scene point image plane How to relate point positions in two views? Triangulation Need point correspondences Need relative camera pose Would like to infer pose from correspondences! 5

6 Multi-view projective geometry provides powerful tools Constraints between two or more images Equations to transfer points from one image to another Given image point correspondences only, can find camera parameters and 3D position of points Two views: epipolar geometry What does one view tell us about another? Point positions in 2 nd view must lie along a known line epipolar line epipolar plane epipoles epipolar line Epipolar Constraint Extremely useful for stereo matching (see later) Reduces correspondence problem to 1D search along conjugate epipolar lines Demo Transfer from epipolar lines What does one view tell us about another? Point positions in 2 nd view must lie along a known line input image output image input image Two views determine point position in a third image Doesn t work if point is in the trifocal plane spanned by all three camera centers Bad case: three cameras are colinear Epipolar algebra How do we compute epipolar lines? Can trace out lines, reproject. But that is overkill Z Y X p T R Because p, p are same 3D point p = Rp + T Note that p is to Tp So 0 = p T Tp = p T T(Rp + T) = p T T(Rp) p Y X Z Simplifying p T T(Rp) = 0 We can write a cross-product ab as a matrix equation a b = A b Therefore: where x 0 z y z 0 z y x y x 0 Where E = T R is the 33 3x3 essential matrix Holds whenever p and p correspond to same scene point Properties of E 0 p' T Ep Ep is the epipolar line of p; p T E is the epipolar line of p p T E p = 0 for every pair of corresponding points 0 = Ee = e T E where e and e are the epipoles E has rank < 3, has 5 independent parameters E tells us everything about the epipolar geometry Essential Matrix 0 = p T E p First derived by Longuet-Higgins, Nature 1981 Can be computed from 8 correspondences (p, p) Each defines a constraint 0 = p T E p Only applies in the calibrated case Intrinsic camera parameters must be known a priori Focal length, aspect ratio, image center Can compute camera R and T matrices from E Can derive relative camera pose only based on correspondences E has only 5 free parameters Three rotation angles Two translation directions 6

7 Fundamental Matrix 0 = p T F p Generalization of the essential matrix F = (A -1 ) T E A -1, where A 3x3 and A 3x3 contain the intrinsic parameters Essential matrix is special case where A is known a priori Gives epipoles, epipolar lines like Essential Matrix F (like E) is defined only up to a scale factor and has rank 2 7 free parameters Can t uniquely solve for camera pose and intrinsic parameters (R, T, A and A ) from F Can be computed using linear methods R. Hartley, In Defence of the 8-point Algorithm, ICCV 95 Or nonlinear methods Xu & Zhang, Epipolar Geometry in Stereo, Motion and Object Recognition, 1996 The Trifocal Tensor What if you have three views? Can compute 3 pairwise fundamental matrices However there are more constraints Should be possible to resolve the trifocal problem Answer: the trifocal tensor Introduced d by Shashua, h Hartley in 1994/1995 3x3x3 matrix T (27 parameters) Gives all constraints between 3 views Can use to generate new views without trifocal probs. [Shai & Avidan] Linearly computable from point correspondences How about four,five, N views? There is a quadrifocal tensor [Faugeras & Morrain, Triggs, 1995] But: all the constraints are expressed in the trifocal tensors, obtained by considering every subset of 3 cameras Today From 2D to 3D Introduction Geometry of two views Stereo matching Other applications Stereo matching Assumption: calibrated camera pair Extrinsic & intrinsic E.g., extrinsic using Essential Matrix Goal: find dense depth map For each point in first image, find corresponding point in second image Then triangulate to find 3D position Epipolar geometry Matching points lie in conjugate epipolar lines 1D search! Epipolar line Epipolar plane Stereo matching Finding matching points Assume brightness constancy: brightness of point on 3D object does not change from one view to the other True for diffuse surfaces, but not specular ones (mirrors, shiny metals, etc.) Simplest criterion: matching points have same color Tough problem to solve robustly Numerous approaches A good survey and evaluation: Stereo image rectification Before stereo matching, reproject image planes onto a common plane parallel to line between optical centers Two homographies (3x3 transform), one for each input image reprojection Epipolar lines become horizontal Motion of corresponding points (disparity) is horizontal Homographies (reprojection) to common plane Horizontal disparity in rectified images 7

8 Stereo image rectification Note that common plane is not unique! Given Fundamental Matrix, find homographies (common plane) such that image distortion is minimized C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE CVPR, Homographies (reprojection) to common plane Depth from disparity After rectification, triangulation is trivial Disparity inversely related to depth X z x x f f C baseline C Horizontal disparity in rectified images Focal length f Stereo matching Window size Input image pair Disparity map Disparity map Window for matching For each epipolar line For each pixel in the left image Compare with every pixel on same epipolar line in right image Pick pixel with minimum match cost (e.g., squared pixel diff.) Improvement: match windows Match cost: e.g., sum of squared pixel diff. over window Effect of window size Smaller window + Larger window + W = 3 W = 20 Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): , July 1998 Stereo results Data from University of Tsukuba Similar results on other images without ground truth Results with window search Scene Ground truth Window-based matching (best window size) Ground truth 8

9 Summary Steps 1. Calibrate cameras 2. Rectify images 3. Compute disparity 4. Estimate depth What will cause errors? Camera calibration errors Poor image resolution Occlusions Violations of brightness constancy (specular reflections) Large motions Low-contrast image regions Stereo as energy minimization What defines a good stereo correspondence? 1. Match quality: want each pixel to find good match in other image 2. Smoothness: if two pixels are adjacent, should (usually) move about same amount Stereo as energy minimization 1. Match quality: want each pixel to find a good match in the other image Stereo as energy minimization 2. Smoothness: if two pixels are adjacent, they should (usually) move about the same amount We want to minimize This is a special type of energy function known as an MRF (Markov Random Field) Effective and fast algorithms have been recently developed Graph cuts, belief propagation, For more details (and code) Great tutorials available online (including video of talks) State of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September Ground truth For the latest and greatest: Video view interpolation Real-time stereo Nomad robot searches for meteorites in Antartica Used for robot navigation (and other tasks) Several software-based real-time stereo techniques have been developed (most based on simple discrete search) 9

10 Commercial products Factory calibrated Active stereo with structured light Stereo matching does not work well if surfaces are uniform, have little texture Active stereo with structured light Project structured light patterns onto object Simplifies correspondence problem camera 1 Li Zhang s one-shot stereo camera 1 Active stereo with structured light Observed projected light pattern allows identification of corresponding ray of light source Many variations for types of patterns etc. projector projector camera 2 Laser scanning Laser scanned models Object Laser sheet Direction of travel CCD image plane Cylindrical lens Laser CCD Digital Michelangelo Project Optical triangulation Project a single stripe of laser light Scan it across the surface of the object Very precise version of structured light scanning The Digital Michelangelo Project, Levoy et al. 10

11 Laser scanned models Laser scanned models The Digital Michelangelo Project, Levoy et al. The Digital Michelangelo Project, Levoy et al. Laser scanned models Laser scanned models The Digital Michelangelo Project, Levoy et al. The Digital Michelangelo Project, Levoy et al. Spacetime stereo Use 3D space-time window for matching Today From 2D to 3D Introduction Geometry of two views Stereo matching Other applications Zhang et al., Spacetime faces 11

12 Other applications Structure from motion Structure : 3D geometry Motion : camera motion relative to scene, observation of scene from multiple viewpoints Calibrated case Intrinsic calibration of cameras obtained a priori Uncalibrated case Recover intrinsic calibration at the same time as geometry Examples On-line model acquisition from images Examples Using large photo collections From online image databases Building rome in a day Microsoft Photosynth Industrial applications Camera tracking, reconstructing geometry,etc. Entertainement, surveillance, UAVs, etc. More information Multiple View Geometry in Computer Vision, Hartley & Zisserman 12