Lecture 6 Stereo Systems Multi-view geometry Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 6-5-Feb-4
Lecture 6 Stereo Systems Multi-view geometry Stereo systems Rectification Correspondence problem Multi-view geometry The SFM problem Affine SFM Reading: [AZ] Chapter: 4 Estimation 2D perspective transformations Chapter: 9 Epipolar Geometry and the Fundamental Matrix T Chapter: Computation of the Fundamental Matrix F [FP] Chapters: The geometry of multiple views Silvio Savarese Lecture 5-5-Feb-4
Epipolar geometry X For details see CS3A Lecture 9 x x 2 e e 2 Epipolar Plane Epipoles e, e 2 Baseline Epipolar Lines O O 2 = intersections of baseline with image planes = projections of the other camera center
Epipolar Constraint P p p O O p T E p E = Essential Matrix (Longuet-Higgins, 98) E T R
Epipolar Constraint P p p O O p T F p F K T T R K F = Fundamental Matrix (Faugeras and Luong, 992)
Example: Parallel image planes X e 2 x x 2 e 2 y z K K x O O 2 Epipolar lines are horizontal Epipoles go to infinity y-coordinates are equal x x y x x2 2 y
Rectification: making two images parallel For details on how to rectify two views see CS3A Lecture 9 H Courtesy figure S. Lazebnik
Why are parallel images useful? Makes the correspondence problem easier Makes triangulation easy Enables schemes for image interpolation
Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 996, 2-3
Morphing without using geometry
Rectification P I I s î y x I î S y x C y x î C s C
From its reflection!
Why are parallel images useful? Makes the correspondence problem easier Makes triangulation easy Enables schemes for image interpolation
Point triangulation P v v e 2 p p e 2 y u u K z O x O K
Computing depth P z u u f f O Baseline B O u u' B z f = disparity Note: Disparity is inversely proportional to depth
Disparity maps u u' B z f u u Stereo pair Disparity map / depth map http://vision.middlebury.edu/stereo/ Disparity map with occlusions
Why are parallel images useful? Makes the correspondence problem easier Makes triangulation easy Enables schemes for image interpolation
Correspondence problem P p p O O 2 Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]
Correspondence problem P p p O O 2 Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]
Correspondence problem A Cooperative Model (Marr and Poggio, 976) Correlation Methods (97--) Multi-Scale Edge Matching (Marr, Poggio and Grimson, 979-8) [FP] Chapters:
Correlation Methods (97--) p p 2 95 5 2
Correlation Methods (97--) p Pick up a window around p(u,v)
Correlation Methods (97--) Pick up a window around p(u,v) Build vector w Slide the window along v line in image 2 and compute w Keep sliding until w w is maximized.
Correlation Methods (97--) Normalized Correlation; minimize: (w w)(w (w w)(w w ) w )
Correlation methods Left Right scanline Credit slide S. Lazebnik Norm. corr
Correlation methods Window size = 3 Window size = 2 Smaller window - More detail - More noise Larger window - Smoother disparity maps - Less prone to noise Credit slide S. Lazebnik
Fore shortening effect Issues O O Occlusions O O
It is desirable to have small B/z ratio! Issues Small error in measurements implies large error in estimating depth O O
Issues Homogeneous regions Hard to match pixels in these regions
Repetitive patterns Issues
Correspondence problem is difficult! - Occlusions - Fore shortening - Baseline trade-off - Homogeneous regions - Repetitive patterns Apply non-local constraints to help enforce the correspondences
Results with window search Data Ground truth Window-based matching Credit slide S. Lazebnik
Lecture 6 Stereo Systems Multi-view geometry Stereo systems Rectification Correspondence problem Multi-view geometry The SFM problem Affine SFM Silvio Savarese Lecture 5-5-Feb-4
Structure from motion problem Courtesy of Oxford Visual Geometry Group
Structure from motion problem X j x j M m M x mj x 2j M 2 Given m images of n fixed 3D points x ij = M i X j, i =,, m, j =,, n
Structure from motion problem X j x j M m M x mj x 2j M 2 From the mxn correspondences x ij, estimate: m projection matrices M i n 3D points X j motion structure
Affine structure from motion (simpler problem) Image World Image From the mxn correspondences x ij, estimate: m projection matrices M i (affine cameras) n 3D points X j
Finite cameras p q r O Q R x K R M T X P
Question: R T??
Finite cameras p q r R Q P O X T x K R T R K M 3 3x Canonical perspective projection matrix Affine homography (in 3D) Affine Homography (in 2D) y x s K o y o x
p q r R Q O P Affine cameras y α x s α o y o x K T R K M Canonical affine projection matrix (points at infinity are mapped as points at infinity) R Q P What s the main difference???
Transformation in 2D Affinities: y x H y x t A y' x' a
Affine cameras X T x K R y x K T R K M b A 4affine] [4 3affine] 3 [ 2 23 22 2 3 2 b a a a b a a a M 2 23 22 2 3 2 X b AX x Euc M b b Z Y X a a a a a a y x [Homogeneous] [non-homogeneous image coordinates] b A M M Euc ; P M Euc
Affine cameras p P p M = camera matrix To recap: from now on we define M as the camera matrix for the affine case p u v AP b M P ; M A b
The Affine Structure-from-Motion Problem Given m images of n fixed points P j (=X i ) we can write N of cameras N of points Problem: estimate the m 24 matrices M i and the n positions P j from the mn correspondences p ij. How many equations and how many unknown? 2m n equations in 8m+3n unknowns Two approaches: - Algebraic approach (affine epipolar geometry; estimate F; cameras; points) - Factorization method
A factorization method Tomasi & Kanade algorithm C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2):37-54, November 992. Centering the data Factorization
Next lecture Multiple view geometry