EECS 442 Computer vision. Stereo systems. Stereo vision Rectification Correspondence problem Active stereo vision systems

Transcription

1 EECS 442 Computer vision Stereo systems Stereo vision Rectification Correspondence problem Active stereo vision systems Reading: [HZ] Chapter: 11 [FP] Chapter: 11

2 Stereo vision P p p O 1 O 2 Goal: estimate the position of P given the observation of P from two view points Assumptions: known camera parameters and position (K, R, T)

3 Stereo vision P p p O 1 O 2 Subgoals: - Solve the correspondence problem - Use corresponding observations to triangulate

4 Correspondence problem P p p O 1 O 2 Given a point in 3d, discover corresponding observations in left and right images

5 Triangulation P p p O 1 O 2 Intersecting the two lines of sight gives rise to P

6 Parallel image planes P p p O O When views are parallel these two steps becomes much easier!

7 Epipolar geometry P p p e 1 e 2 Epipolar Plane Epipoles e 1, e 2 Baseline Epipolar Lines O 1 O 2 = intersections of baseline with image planes = projections of the other camera center

8 Parallel image planes P v v e 2 p p e 2 y u u K O z x O K Parallel epipolar lines Epipoles at infinity v = v Rectification: making two images parallel

9 Parallel image planes P v v e 2 p p e 2 y u u K O z x O K K 1 =K 2 = known x parallel to O 1 O 2 E [ t ] R

10 b a b a ] [ z y x x y x z y z b b b a a a a a a Cross product as matrix multiplication

11 Parallel image planes P v v e 2 p p e 2 y u u K O z x O K K 1 =K 2 = known x parallel to O 1 O 2 E [ t ] R 0 0 T 0 T 0 v = v?

12 Parallel image planes v T Tv Tv T v u v u T T v u O O P e 2 p p e 2 K K x y z u v u v 0 E p p T

13 Making image planes parallel P p p H O O GOAL: Estimate the perspective transformation H that makes the images parallel

14 Projective transformation H x i xi x H x i i Now we don t have the destination image

15 Making image planes parallel P p p H O O GOAL: Estimate the perspective transformation H that makes images parallel Impose v =v This leaves degrees of freedom for determining H If not appropriate H is chosen, severe projective distortions on image take place We impose a number of restriction while computing H

16 Making image planes parallel P K I 0 P P P K R T P p p e e O R,T O 0. Compute epipoles e K R T T [e 1 e 2 1] T e K T

17 Making image planes parallel P p p e e O O 1. Map e to the x-axis at location [1,0,1] T (normalization) e [e 1 T 1 e2 1] 0 1 T H R T 1 H H

18 Making image planes parallel P p p e O O 2. Send epipole to infinity: H e T T Minimizes the distortion in a neighborhood (approximates id. mapping)

19 Making image planes parallel P H = H 2 H 1 e p p e O O 3. Define: H = H 2 H 1 4. Align epipolar lines

20 Projective transformation of a line (in 2D) H A v t b l H T l

21 Making image planes parallel P H = H 2 H 1 e p l l p e O O 3. Define: H = H 2 H 1 HT l H T l 4. Align epipolar lines [HZ] Chapters: 11 (sec ) These are called matched pair of transformation

22 Making image planes parallel H Courtesy figure S. Lazebnik

23 Why rectification is useful? Makes the correspondence problem easier Makes triangulation easy

24 Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30

25 Morphing without using geometry

26 Rectification

27

28 Stereo vision P p p O 1 O 2 Subgoals: - Solve the correspondence problem - Use corresponding observations to triangulate

29 Computing depth P v v e 2 p p e 2 y u u K z O x O K

30 Computing depth P z x x f f O Baseline B O x x Bf z = disparity Note: Disparity is inversely proportional to depth

31 Disparity maps x x Bf z x x Stereo pair Disparity map / depth map Disparity map with occlusions

32 Stereo vision P p p O 1 O 2 Subgoals: - Solve the correspondence problem - Use corresponding observations to triangulate

33 Correspondence problem P p p O 1 O 2 Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]

34 Correspondence problem A Cooperative Model (Marr and Poggio, 1976) Correlation Methods (1970--) Multi-Scale Edge Matching (Marr, Poggio and Grimson, ) [FP] Chapters: 11

35 Correlation Methods (1970--) p p

36 Correlation Methods (1970--) p Pick up a window around p(u,v)

37 Correlation Methods (1970--) Pick up a window around p(u,v) Build vector W Slide the window along v line in image 2 and compute w Keep sliding until w w is maximized.

38 Correlation Methods (1970--) Normalized Correlation; minimize: (w w)(w (w w)(w w ) w )

39 Correlation methods Left Right scanline Credit slide S. Lazebnik Norm. corr

40 Correlation methods Window size = 3 Window size = 20 Smaller window - More detail - More noise Larger window - Smoother disparity maps - Less prone to noise Credit slide S. Lazebnik

41 Fore shortening effect Issues O O Occlusions O O

42 It is desirable to have small B/z ratio! Issues Small error in measurements implies large error in estimating depth O O

43 Issues Homogeneous regions Hard to match pixels in these regions

44 Repetitive patterns Issues

45 Correspondence problem is difficult! - Occlusions - Fore shortening - Baseline trade-off - Homogeneous regions - Repetitive patterns Apply non-local constraints to help enforce the correspondences

46 Results with window search Data Ground truth Window-based matching Credit slide S. Lazebnik

47 Improving correspondence: Non-local constraints Uniqueness For any point in one image, there should be at most one matching point in the other image

48 Improving correspondence: Non-local constraints Uniqueness For any point in one image, there should be at most one matching point in the other image Ordering Corresponding points should be in the same order in both views Not always in presence of occlusions! courtesy slide to J. Ponce

49 Dynamic Programming (Baker and Binford, 1981) [Uses ordering constraint] Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals. Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. courtesy slide to J. Ponce

50 Dynamic Programming (Baker and Binford, 1981) courtesy slide to J. Ponce

51 Dynamic Programming (Ohta and Kanade, 1985) Reprinted from Stereo by Intra- and Intet-Scanline Search, by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2): (1985) IEEE.

52 Improving correspondence: Non-local constraints Uniqueness For any point in one image, there should be at most one matching point in the other image Ordering Corresponding points should be in the same order in both views Smoothness Disparity is typically a smooth function of x (expect in occluding boundaries)

53 Smoothness

54 Stereo matching as energy minimization Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01 I 1 I 2 D W 1 (i) W 2 (i+d(i)) D(i ) E Edata ( I1, I2, D) Esmooth ( D) E W ( i) W ( i D( i 2 data 1 2 )) i Esmooth D( i) D( j) neighborsi, j Energy functions of this form can be minimized using graph cuts Credit slide S. Lazebnik

55 Stereo matching as energy minimization Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01 Ground truth Window-based matching Graph cuts

56 Two-frame stereo correspondence algorithms Click here

57 Stereo SDK stereo vision software development kit. A. Criminisi, A. Blake and D. Robertson

58 Application foreground/background separation V. Kolmogorov, A. Criminisi, A. Blake, G. Cross and C. Rother. Bi-layer segmentation of binocular stereo video CVPR

59 Application 3D Urban Scene Modeling 3D Urban Scene Modeling Integrating Recognition and Reconstruction, N. Cornelis, B. Leibe, K. Cornelis, L. Van Gool, IJCV 08.

60 Stereo systems Stereo vision Rectification Correspondence problem Active stereo vision systems

61 Active stereo (point) P p projector O 2 Replace one of the two cameras by a projector - Single camera - Projector geometry calibrated - What s the advantage of having the projector? Correspondence problem solved!

62 Active stereo (stripe) projector O -Projector and camera are parallel - Correspondence problem solved!

63 Laser scanning Digital Michelangelo Project Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning Source: S. Seitz

64 Laser scanning The Digital Michelangelo Project, Levoy et al. Source: S. Seitz

65 Active stereo (shadows) J. Bouguet & P. Perona, 99 Light source O - 1 camera,1 light source - very cheap setup - calibrated the light source

66 Active stereo (shadows)

67 Active stereo (color-coded stripes) L. Zhang, B. Curless, and S. M. Seitz 2002 S. Rusinkiewicz & Levoy 2002 projector - Dense reconstruction - Correspondence problem again - Get around it by using color codes O

68 L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002

69 L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002 Rapid shape acquisition: Projector + stereo cameras

70 Next lecture Affine Structure from Motion

71

72 Human Stereopsis Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., Credit slide J. Ponce

73 Human Stereopsis: Reconstruction d=0 Disparity: d = r-l = D-F; In 3D, the horopter. d<0 Credit slide J. Ponce

74 Human Stereopsis: Reconstruction What if F is not known? Helmoltz (1909): There is evidence showing the vergence angles cannot be measured precisely. Humans get fooled by bas-relief sculptures. Credit slide J. Ponce

75 Human Stereopsis: Binocular Fusion Credit slide J. Ponce

76 Human Stereopsis: Binocular Fusion How are the correspondences established? Julesz (1971): Is the mechanism for binocular fusion a monocular process or a binocular one?? There is anecdotal evidence for the latter (camouflage). Random dot stereograms provide an objective answer Credit slide J. Ponce

77 Issues O O