Lecture 9 & 10: Stereo Vision - PDF Free Download

Lecture 9 & 10: Stereo Vision Professor Fei- Fei Li Stanford Vision Lab 1

What we will learn today? IntroducEon to stereo vision Epipolar geometry: a gentle intro Parallel images Image receficaeon Solving the correspondence problem AcEve stereo vision system Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10 2

Recovering 3D from Images How can we automaecally compute 3D geometry from images? What cues in the image provide 3D informaeon? Real 3D world 2D image? Point of observation 4

Visual Cues for 3D Shading Merle Norman Cosme2cs, Los Angeles Slide credit: J. Hayes 5

Visual Cues for 3D Shading Texture The Visual Cliff, by William Vandivert, 1960 Slide credit: J. Hayes 6

Visual Cues for 3D Shading Texture Focus From The Art of Photography, Canon Slide credit: J. Hayes 7

Visual Cues for 3D Shading Texture Focus MoEon Slide credit: J. Hayes 8

Visual Cues for 3D Shading Texture Focus MoEon Others: Highlights Shadows Silhoue`es Inter- refleceons Symmetry Light PolarizaEon... Shape From X X = shading, texture, focus, moeon,... We ll focus on the moeon cue Slide credit: J. Hayes 9

Stereo ReconstrucEon The Stereo Problem Shape from two (or more) images Biological moevaeon known camera viewpoints Slide credit: J. Hayes 10

Why do we have two eyes? Cyclope vs. Odysseus Slide credit: J. Hayes 11

1. Two is be`er than one Slide credit: J. Hayes 12

2. Depth from Convergence Human performance: up to 6-8 feet Slide credit: J. Hayes 13

3. Depth from binocular disparity P: converging point C: object nearer projects to the outside of the P, disparity = + Sign and magnitude of disparity F: object farther projects to the inside of the P, disparity = - Slide credit: J. Hayes 14

Epipolar geometry P p 1 p 2 e 1 e 2 Epipolar Plane Epipoles e 1, e 2 Baseline Epipolar Lines O 1 O 2 = interseceons of baseline with image planes = projeceons of the other camera center = vanishing points of camera moeon direceon 16

Example: Converging image planes 17

Epipolar Constraint - Two views of the same object - Suppose I know the camera posieons and camera matrices - Given a point on lel image, how can I find the corresponding point on right image? 18

Epipolar Constraint PotenEal matches for p have to lie on the corresponding epipolar line l. PotenEal matches for p have to lie on the corresponding epipolar line l. 19

Epipolar Constraint P p M P = u v 1 p p p M ʹ P = uʹ vʹ 1 O R, T O M = K [ 0] 1 I M '= K2 [ R T ] 20

Epipolar Constraint P p p O R, T O M = K [ 0] 1 I K 1 and K 2 are known (calibrated cameras) M'= K2 [ R T] M = [ I 0] M ' = [ R T] 21

Epipolar Constraint P p p O R, T O T ( R p ʹ ) Perpendicular to epipolar plane [ T (R p )] 0 p T ʹ = 22

23 Cross product as matrix muleplicaeon b a b a ] [ 0 0 0 = = z y x x y x z y z b b b a a a a a a skew symmetric matrix

Epipolar Constraint P p p O R, T O [ T (R p )] 0 p T ʹ = [ T ] R p 0 p T ʹ = (Longuet- Higgins, 1981) E = esseneal matrix 24

Epipolar Constraint P p 1 l 1 l 2 e 1 e 2 p 2 O 1 O 2 E p 2 is the epipolar line associated with p 2 (l 1 = E p 2 ) E T p 1 is the epipolar line associated with p 1 (l 2 = E T p 1 ) E is singular (rank two) E e 2 = 0 and E T e 1 = 0 E is 3x3 matrix; 5 DOF 25

Simplest Case: Parallel images Image planes of cameras are parallel to each other and to the baseline Camera centers are at same height Focal lengths are the same Slide credit: J. Hayes 27

Simplest Case: Parallel images Image planes of cameras are parallel to each other and to the baseline Camera centers are at same height Focal lengths are the same Then, epipolar lines fall along the horizontal scan lines of the images Slide credit: J. Hayes 28

Essential matrix for parallel images p T E! p = 0, E = [t ]R = = 0 0 0 0 0 0 0 ] [ T T R t E = 0 0 0 ] [ x y x z y z a a a a a a a R = I t = (T, 0, 0) Epipolar constraint: t p p Reminder: skew symmetric matrix Slide credit: J. Hayes 29

( ) ( ) v T Tv Tv T v u v u T T v u ʹ = = ʹ = ʹ ʹ 0 0 1 0 1 0 0 0 0 0 0 0 1 The y-coordinates of corresponding points are the same! t p p Essential matrix for parallel images p T E! p = 0, E = [t ]R = = 0 0 0 0 0 0 0 ] [ T T R t E R = I t = (T, 0, 0) Epipolar constraint: Slide credit: J. Hayes 30

Triangulation -- depth from disparity P p p z f f O Baseline O B disparity = u u " = B f z Disparity is inversely proportional to depth! Slide credit: J. Hayes 31

Stereo image rectification Slide credit: J. Hayes 32

Stereo image rectification Algorithm: Re-project image planes onto a common plane parallel to the line between optical centers Pixel motion is horizontal after this transformation Two transformation matrices, one for each input image reprojection Ø C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999. Slide credit: J. Hayes 33

Rectification example 34

Applica2on: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30 35

Applica2on: view morphing 36

Applica2on: view morphing 37

Applica2on: view morphing 38

Removing perspective distortion (rectification) H p 39

Reminder: transformaeons in 2D Special case from lecture 2 (planar rotaeon & translaeon)! # # # " u' v' 1 $ &! & = # R t & " 0 1 %! $ # &# %# " u v 1 $! u & # & = H e # v & # % " 1 $ & & & % - 3 DOF - Preserve distance (areas) - Regulate motion of rigid object

Reminder: transformaeons in 2D Generic case (rotaeon in 3D, scale & translaeon)! # # # " u' v' 1 $! & # & = # & # % "# a 1 a 2 a 3 a 4 a 5 a 6 a 7 a 8 a 9 $! &# &# &# %& " u v 1 $ & & = H & %! # # # " u v 1 $ & & & % - 8 DOF - Preserve colinearity

Computing H H - 8 DOF - how many points do I need to esemate H? At least 4 points! (8 equations) - There are several algorithms 44

DLT algorithm (Direct Linear Transformation) H p i! p i p! i = H p i 45

DLT algorithm (direct Linear Transformation) Unknown [9x1] p! H p = 0 i i A h = 0 i! # H = # # "# h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 $ & & & %&! # # h = # # "# h 1 h 2 h 9 $ & & & & %& Function of measurements [2x9] 9x1 2 independent equaeons 46

DLT algorithm (direct Linear Transformation) A 2x9 h 9x1 H A h = i 0 x i xʹ i A h = 1 0 A h = 2 A h = N 0 0 A h N 9 9 1 2 = 0 Over determined Homogenous system 47

DLT algorithm (direct Linear Transformation) A h N 9 9 1 How to solve? 2 = Singular Value Decomposition (SVD)! 0 48

DLT algorithm (direct Linear Transformation) A h N 9 9 1 How to solve? 2 = Singular Value Decomposition (SVD)! 0 U Σ V T 2 n 9 9 9 9 9 Last column of V gives h! H! Why? See pag 593 of AZ 49

DLT algorithm (direct Linear Transformation) How to solve h 0? A = 2 N 9 9 1 50

Clarification about SVM P = U D V T m n m n n n n n Has n orthogonal columns Orthogonal matrix This is one of the possible SVD decompositions This is typically used for efficiency The classic SVD is actually: P = U D V T m n m m m n n n orthogonal Orthogonal 51

Stereo matching: solving the correspondence problem Multiple matching hypotheses satisfy the epipolar constraint, but which one is correct? 53

Basic stereo matching algorithm For each pixel in the first image Find corresponding epipolar line in the right image Examine all pixels on the epipolar line and pick the best match Triangulate the matches to get depth information Simplest case: epipolar lines are scanlines When does this happen? Slide credit: J. Hayes 54

Basic stereo matching algorithm If necessary, rectify the two stereo images to transform epipolar lines into scanlines For each pixel x in the first image Find corresponding epipolar scanline in the right image Examine all pixels on the scanline and pick the best match x Compute disparity x-x and set depth(x) = 1/(x-x ) 55

Correspondence problem Let s make some assumptions to simplify the matching problem The baseline is relatively small (compared to the depth of scene points) Then most scene points are visible in both views Also, matching regions are similar in appearance Slide credit: J. Hayes 56

Correspondence search with similarity constraint Left Right scanline Matching cost disparity Slide a window along the right scanline and compare contents of that window with the reference window in the left image Matching cost: SSD or normalized correlation Slide credit: J. Hayes 57

Correspondence search with similarity constraint Left Right scanline SSD 58

Correspondence search with similarity constraint Left Right scanline Norm. corr 59

Effect of window size Smaller window + More detail More noise W = 3 W = 20 Larger window + Smoother disparity maps Less detail 60

The similarity constraint Corresponding regions in two images should be similar in appearance and non-corresponding regions should be different When will the similarity constraint fail? Slide credit: J. Hayes 61

Limitations of similarity constraint Textureless surfaces Occlusions, repetition Specular surfaces Slide credit: J. Hayes 62

Results with window search Left image Right image Window-based matching Ground truth 63

Better methods exist... (CS231a) Graph cuts Ground truth Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001 64

The role of the baseline Small Baseline Large Baseline Small baseline: large depth error Large baseline: difficult search problem Slide credit: S. Seitz 65

Problem for wide baselines: Foreshortening Matching with fixed-size windows will fail! Possible solution: adaptively vary window size Another solution: model-based stereo (CS231a) Slide credit: J. Hayes 66

Ac2ve stereo (point) P projector p Replace one of the two cameras by a projector - Single camera - Projector geometry calibrated - What s the advantage of having the projector? O 2 Correspondence problem solved! 68

Ac2ve stereo (stripe) projector - Projector and camera are parallel - Correspondence problem solved! O 69

Ac2ve stereo (shadows) J. Bouguet & P. Perona, 99 S. Savarese, J. Bouguet & Perona, 00 Light source O - 1 camera,1 light source - very cheap setup - calibrated light source 70

Ac2ve stereo (shadows) J. Bouguet & P. Perona, 99 S. Savarese, J. Bouguet & Perona, 00 71

Ac2ve stereo (color- coded stripes) L. Zhang, B. Curless, and S. M. Seitz 2002 S. Rusinkiewicz & Levoy 2002 projector - Dense reconstruceon - Correspondence problem again - Get around it by using color codes O 72

Ac2ve stereo (color- coded stripes) Rapid shape acquisieon: Projector + stereo cameras L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape AcquisiEon Using Color Structured Light and MulE- pass Dynamic Programming. 3DPVT 2002 73

Ac2ve stereo (stripe) Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/ OpEcal triangulaeon Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning 74

Laser scanned models The Digital Michelangelo Project, Levoy et al. 75 Slide credit: S. Seitz

Laser scanned models The Digital Michelangelo Project, Levoy et al. 76 Slide credit: S. Seitz

Laser scanned models The Digital Michelangelo Project, Levoy et al. 77 Slide credit: S. Seitz

Laser scanned models The Digital Michelangelo Project, Levoy et al. 78 Slide credit: S. Seitz

Laser scanned models 1.0 mm resolueon (56 million triangles) The Digital Michelangelo Project, Levoy et al. 79 Slide credit: S. Seitz

What we have learned today? IntroducEon to stereo vision Epipolar geometry: a gentle intro Parallel images Image receficaeon Solving the correspondence problem AcEve stereo vision system Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10 80