Last time: Disparity. Lecture 11: Stereo II. Last time: Triangulation. Last time: Multi-view geometry. Last time: Epipolar geometry

Transcription

1 Last time: Disarity Lecture 11: Stereo II Thursday, Oct 4 CS 378/395T Prof. Kristen Grauman Disarity: difference in retinal osition of same item Case of stereo rig for arallel image lanes and calibrated cameras: deth (Z) is inversely related to disarity (xr-xl). Last time: Multi-view geometry Last time: Triangulation P Scene oint in 3d Left image Right image O O Estimate scene oint based on camera relationshis and corresondence. Last time: Eiolar geometry Key idea: geometry imoses constraints on which oints may corresond. Last time: Eiolar geometry If a oint feature x is observed in one image, its location x in the other image must lie on the eiolar line. Eiolar Plane Eioles Baseline Eiolar Lines Figure from Gee & Ciolla 1999 Adated from M. Pollefeys, UNC

2 Last time: Eiolar constraint Today How do we comute those eiolar lines? How do we relate corresonding oints algebraically? Essential matrix What other constraints can we use besides geometry? Potential matches for have to lie on the corresonding eiolar line l. Still assuming calibrated cameras for now. Potential matches for have to lie on the corresonding eiolar line l. Slide credit: M. Pollefeys, UNC Calibrated cameras If fully calibrated, we know how to rotate and translate camera reference frame 1 to get to camera reference frame 2. how to ma ixel coordinates to image lane coordinates Stereo geometry, with calibrated cameras Vector in second coord. sys. has coordinates R in the first one. Camera 2 frame Camera 1 frame Camera-centered coordinate systems are related by known rotation R and translation T. Recall: Cross roduct From geometry to algebra Vector cross roduct takes two vectors and returns a third vector that s erendicular to both inuts. Also colanar, so dot roduct with normal is 0 Normal to this lane Colanar vectors

3 From geometry to algebra Matrix form of cross roduct Vector in second coord. sys. has coordinates R in the first one. Can be exressed as a matrix multilication. [ T ( R )] = 0 From geometry to algebra Essential matrix and eiolar lines Let E [ T ( R )] = 0 [ Tx] R = 0 = [ Tx] R Τ E = 0 Τ E = 0 Eiolar constraint: if we observe oint in one image, then its osition in second image must satisfy this equation. E Τ is the coordinate vector reresenting the eiolar line for oint E is the essential matrix, which relates corresonding image oints [Longuet-Higgins 1981] E is the coordinate vector reresenting the eiolar line for oint Essential matrix roerties Relates image of corresonding oints in both cameras, given rotation and translation Assuming intrinsic arameters are known Essential matrix examle: arallel cameras R = I T = [ T,0,0] E = [T x]r = Τ T 0 T 0

4 Essential matrix examle: arallel cameras Τ E = 0 R = I T = [ T,0,0] E = [T x]r = Image of any oint must lie on same horizontal line in each image lane! T T T -Ty Τ T 0 T 0 Stereo reconstruction for fully calibrated cameras Image air Detect some features Comute E from R and T Match features using the eiolar and other constraints (coming u) Triangulate for 3d structure Disarity, deth mas Stereo image rectification image I(x,y) Disarity ma D(x,y) image I (x,y ) Motivation: make the lines to be searched corresond to scanlines in images (x,y )=(x+d(x,y),y) reroject image lanes onto a common lane arallel to the line between otical centers ixel motion is horizontal after this transformation two homograhies (3x3 transforms), one for each inut image rerojection Adated from Li Zhang Corresondence roblem Multile match hyotheses satisfy eiolar constraint, but which is correct? Corresondence roblem To find matches in the image air, we will assume Most scene oints visible from both views Image regions for the matches are similar in aearance Ok when distance of fixation oint >> baseline (But, we can t guarantee) Figure from Gee & Ciolla 1999

5 Additional corresondence constraints Similarity Uniqueness Ordering Figural continuity Disarity gradient Dense corresondence search For each eiolar line For each ixel / window in the left image comare with every ixel / window on same eiolar line in right image ick osition with minimum match cost (e.g., SSD, correlation) Adated from Li Zhang Examle: window search Examle: window search Data from University of Tsukuba Effect of window size Sarse corresondence search Figures from Li Zhang W = 3 W = 20 Want window large enough to have sufficient intensity variation, yet small enough to contain only ixels with about the same disarity. Restrict search to sarse set of detected features Rather than ixel values (or lists of ixel values) use feature descritor and an associated feature distance Still narrow search further by eiolar geometry What would make good features?

6 Dense vs. sarse Sarse Efficiency Can have more reliable feature matches, less sensitive to illumination than raw ixels But, have to know enough to ick good features; sarse info Dense Simle rocess More deth estimates, can be useful for surface reconstruction But, breaks down in textureless regions anyway, raw ixel distances can be brittle, not good with very different viewoints Difficulties in similarity constraint Untextured surfaces???? Occlusions Uniqueness For oaque objects, u to one match in right image for every oint in left image Ordering Points on same surface (oaque object) will be in same order in both views Figure from Gee & Ciolla 1999 Figure from Gee & Ciolla 1999 Figural continuity When interest oints lie on image contours Disarity gradient Assume iecewise continuous surface, so want disarity estimates to be locally smooth Figure from Gee & Ciolla 1999 Figure from Gee & Ciolla 1999

7 Additional corresondence constraints Similarity Uniqueness Ordering Figural continuity Disarity gradient Stereo reconstruction for fully calibrated cameras Image air Detect some features Comute E from R and T Match features using the eiolar and other constraints Triangulate for 3d structure Sources of error in corresondences Low-contrast / textureless image regions Occlusions Camera calibration errors Poor image resolution Violations of brightness constancy (secular reflections) Large motions Model-based body tracking, stereo inut Fitting! David Demirdjian, MIT Vision Interface Grou Model-based body tracking, stereo inut Deth for segmentation Edges in disarity in conjunction with image edges enhances contours found David Demirdjian, MIT Vision Interface Grou Danijela Markovic and Margrit Gelautz, Interactive Media Systems Grou, Vienna University of Technology

8 Deth for segmentation Uncalibrated case What if we don t know the extrinsic camera arameters? What if we don t even know the intrinsic arameters? We can still reconstruct 3d structure, u to certain ambiguities, if we can find corresondences between oints Danijela Markovic and Margrit Gelautz, Interactive Media Systems Grou, Vienna University of Technology Coming u Exam Tuesday Oct 9 (next class) Thursday (Oct 11): Finish u multi-view geometry and stereo Following week (Oct 16 and 18): Guest lectures Dana Ballard Michael Ryoo & Shalini Guta