Lecture 6 Stereo Systems Multi-view geometry

Transcription

1 Lecture 6 Stereo Systems Multi-view geometry Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 6-5-Feb-4

2 Lecture 6 Stereo Systems Multi-view geometry Stereo systems Rectification Correspondence problem Multi-view geometry The SFM problem Affine SFM Reading: [AZ] Chapter: 4 Estimation 2D perspective transformations Chapter: 9 Epipolar Geometry and the Fundamental Matrix T Chapter: Computation of the Fundamental Matrix F [FP] Chapters: The geometry of multiple views Silvio Savarese Lecture 5-5-Feb-4

3 Epipolar geometry X For details see CS3A Lecture 9 x x 2 e e 2 Epipolar Plane Epipoles e, e 2 Baseline Epipolar Lines O O 2 = intersections of baseline with image planes = projections of the other camera center

4 Epipolar Constraint P p p O O p T E p E = Essential Matrix (Longuet-Higgins, 98) E T R

5 Epipolar Constraint P p p O O p T F p F K T T R K F = Fundamental Matrix (Faugeras and Luong, 992)

6 Example: Parallel image planes X e 2 x x 2 e 2 y z K K x O O 2 Epipolar lines are horizontal Epipoles go to infinity y-coordinates are equal x x y x x2 2 y

7 Rectification: making two images parallel For details on how to rectify two views see CS3A Lecture 9 H Courtesy figure S. Lazebnik

8 Why are parallel images useful? Makes the correspondence problem easier Makes triangulation easy Enables schemes for image interpolation

9 Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 996, 2-3

10 Morphing without using geometry

11 Rectification P I I s î y x I î S y x C y x î C s C

12

13

14

15

16 From its reflection!

17 Why are parallel images useful? Makes the correspondence problem easier Makes triangulation easy Enables schemes for image interpolation

18 Point triangulation P v v e 2 p p e 2 y u u K z O x O K

19 Computing depth P z u u f f O Baseline B O u u' B z f = disparity Note: Disparity is inversely proportional to depth

20 Disparity maps u u' B z f u u Stereo pair Disparity map / depth map Disparity map with occlusions

21 Why are parallel images useful? Makes the correspondence problem easier Makes triangulation easy Enables schemes for image interpolation

22 Correspondence problem P p p O O 2 Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]

23 Correspondence problem P p p O O 2 Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]

24 Correspondence problem A Cooperative Model (Marr and Poggio, 976) Correlation Methods (97--) Multi-Scale Edge Matching (Marr, Poggio and Grimson, 979-8) [FP] Chapters:

25 Correlation Methods (97--) p p

26 Correlation Methods (97--) p Pick up a window around p(u,v)

27 Correlation Methods (97--) Pick up a window around p(u,v) Build vector w Slide the window along v line in image 2 and compute w Keep sliding until w w is maximized.

28 Correlation Methods (97--) Normalized Correlation; minimize: (w w)(w (w w)(w w ) w )

29 Correlation methods Left Right scanline Credit slide S. Lazebnik Norm. corr

30 Correlation methods Window size = 3 Window size = 2 Smaller window - More detail - More noise Larger window - Smoother disparity maps - Less prone to noise Credit slide S. Lazebnik

31 Fore shortening effect Issues O O Occlusions O O

32 It is desirable to have small B/z ratio! Issues Small error in measurements implies large error in estimating depth O O

33 Issues Homogeneous regions Hard to match pixels in these regions

34 Repetitive patterns Issues

35 Correspondence problem is difficult! - Occlusions - Fore shortening - Baseline trade-off - Homogeneous regions - Repetitive patterns Apply non-local constraints to help enforce the correspondences

36 Results with window search Data Ground truth Window-based matching Credit slide S. Lazebnik

37 Lecture 6 Stereo Systems Multi-view geometry Stereo systems Rectification Correspondence problem Multi-view geometry The SFM problem Affine SFM Silvio Savarese Lecture 5-5-Feb-4

38 Structure from motion problem Courtesy of Oxford Visual Geometry Group

39 Structure from motion problem X j x j M m M x mj x 2j M 2 Given m images of n fixed 3D points x ij = M i X j, i =,, m, j =,, n

40 Structure from motion problem X j x j M m M x mj x 2j M 2 From the mxn correspondences x ij, estimate: m projection matrices M i n 3D points X j motion structure

41 Affine structure from motion (simpler problem) Image World Image From the mxn correspondences x ij, estimate: m projection matrices M i (affine cameras) n 3D points X j

42 Finite cameras p q r O Q R x K R M T X P

43 Question: R T??

44 Finite cameras p q r R Q P O X T x K R T R K M 3 3x Canonical perspective projection matrix Affine homography (in 3D) Affine Homography (in 2D) y x s K o y o x

45 p q r R Q O P Affine cameras y α x s α o y o x K T R K M Canonical affine projection matrix (points at infinity are mapped as points at infinity) R Q P What s the main difference???

46 Transformation in 2D Affinities: y x H y x t A y' x' a

47 Affine cameras X T x K R y x K T R K M b A 4affine] [4 3affine] 3 [ b a a a b a a a M X b AX x Euc M b b Z Y X a a a a a a y x [Homogeneous] [non-homogeneous image coordinates] b A M M Euc ; P M Euc

48 Affine cameras p P p M = camera matrix To recap: from now on we define M as the camera matrix for the affine case p u v AP b M P ; M A b

49 The Affine Structure-from-Motion Problem Given m images of n fixed points P j (=X i ) we can write N of cameras N of points Problem: estimate the m 24 matrices M i and the n positions P j from the mn correspondences p ij. How many equations and how many unknown? 2m n equations in 8m+3n unknowns Two approaches: - Algebraic approach (affine epipolar geometry; estimate F; cameras; points) - Factorization method

50 A factorization method Tomasi & Kanade algorithm C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2):37-54, November 992. Centering the data Factorization

51 Next lecture Multiple view geometry