Computer Vision I. Announcement

Transcription

1 Announcement Stereo I HW3: Coming soon Stereo! CSE5A Lecture 3 Binocular Stereopsis: Mars Gien two images of a scene where relatie locations of cameras are known, estimate depth of all common scene points. Epipolar Geometry Two images of Mars (Viking Lander Baseline Epipoles Epipolar Plane Epipolar Lines Family of epipolar Planes Skew Symmetric Matrix Cross Product The cross product a x b of two ectors a and b can be expressed a matrix ector product [a x ]b where[a x ] is the skew symmetric matrix: 0 a 3 a [ a ] = a 3 0 a a a 0 ( O O A matrix S is skew symmetric iff S = -S T Family of planes π and lines l and l Intersection in e and e

2 Epipolar Constraint: Calibrated Case The ectors Op, OO, and O p are coplanar Essential Matrix (Longuet-Higgins, 98 The Eight-Point Algorithm (Longuet-Higgins, 98 Much more on multi-iew in CSE5B!! E E E 3 u [,] [ uu u u u u ] E E E 3 = 0 E 3 E 3 E 33 Set E 33 to Use 8 points (u i, i, i=..8 u u u u u u E u u u u u u E u 3 u 3 u 3 3 u 3 3 u u 3 3 E 3 u 4 u 4 u 4 4 u 4 4 u u 4 4 E = u 5 u 5 u 5 5 u 5 5 u u 5 5 E u 6 u 6 u 6 6 u 6 6 u u 6 6 E 3 u 7 u 7 u 7 7 u 7 7 u u 7 7 E 3 u 8 u 8 u 8 8 u 8 8 u u 8 8 E 3 E E E 3 E E E 3 E 3 E 3 E 33 = 0 Sole E to E 3 -- The Essential Matrix Epipolar geometry example The Essential Matrix and Epipolar constraint The epipolar T constraint is homogenous in p, p and E It is bilinear in p and p. E.g., for a gien alue of p, it is linear in p and ice T ersa p p the direction of the image point or can be iewed as homogenous coordinates of 3D location of image points. Ep is the epipolar line associated with p E t p is the epipolar line associated with p Ee =0 and E T e=0 The eigenector of E corresponding to the zero eigenalue is the epipole e E is singular (determinant is zero can t be inerted E has two equal non-zero singular Projectie Transformation (Homography Under perspectie projection, the mapping from a plane to a plane is gien by a projectie transformation (aka homography, a linear transformation of homogenous coordinates.! U!! Transformation! 0 0 0! Rigid Transformation V = represented by represented by W intrinsic parameters extrinsic parameters Mapping from 3D to Image Coordinates p = H I Π p M E P X Y Z (x, y. u! x! y = H w Camera matrix u! U!! Transformation! 0 0 0! Rigid Transformation V = represented by represented by W intrinsic parameters extrinsic parameters Homography X Y Z Mapping from 3D to calibrated coordinates used with Essential Matrix p c = Π p M E P Mapping from calibrated coordinates to image coordinates p = H I p c Mapping from image coordinaes to calibrated coordinates p c = H I p

3 The Fundamental Matrix The epipolar constraint is gien by: where p and p are 3-D coordinates of the image coordinates of points in the two images. Gien a pair of images, transform both images so that epipolar lines are scan lines. Without calibration, we can still identify corresponding points in two images, but we can t conert to 3-D coordinates. Howeer, the relationship between the calibrated coordinates (p,p and uncalibrated coordinates (q,q can be expressed as p=aq, and p =A q e e Therefore, we can express the epipolar constraint as: (Aq T E(A q = q T (A T EA q = q T Fq = 0 where F is called the Fundamental Matrix. Can be soled using 8 point algorithm WITHOUT CALIBRATION Under perspectie projection, the mapping from a plane to a plane is gien by a projectie transformation (aka homography. Under perspectie projection, the mapping from a plane to a plane is gien by a projectie transformation (aka homography.. x L y L w L u L = H L L (u L, L (x L, y L x L y L w L u L = H L L (u L, L (x L, y L (u R, R (x R, y R x R y R w R u R = H R R Two images Two homographies e Image pair rectification Simplify stereo matching by warping the images Apply projectie transformation so that epipolar lines correspond to horizontal scanlines (u L, L e (x L, y L Gien a pair of images, transform both images so that epipolar lines are scan lines. H H should map epipole e to (,0,0, a point at infinity H should minimize image distortion Note that rectified images usually not rectangular See Text for complete method 0 = He 0 Input Images 3

4 Gien a pair of images, transform both images so that epipolar lines are scan lines. Rectified Images See Section for specific method Dense Correspondence: A Photometric constraint Features on same epipolar line Same world point has same intensity in both images (Constant Brightness Constraint Lambertian fronto-parallel Issues: Noise Specularity Foreshortening Truco Fig. 7.5 Using epipolar constant Brightness constraints for stereo matching Finding Correspondences For each epipolar line For each pixel in the left image compare with eery pixel on same epipolar line in right image pick pixel with minimum match cost This will neer work, so: Match windows CSE5, Winter 04 (Seitz W(pl Intro Computer Vision W(pr 4

5 Correspondence Search Algorithm? = Comparing Windows: f g u d For i = :nrows for j=:ncols best(i,j = - for k = mindisparity:maxdisparity c = Match_Metric(I (i,j,i (i,j+k,winsize if (c > best(i,j best(i,j = c disparities(i,j = k end end end end I I O(nrows * ncols * disparities * winx * winy (Camps Most popular For each window, match to closest window on epipolar line in other image. Match Metric Summary MATCH METRIC Normalized Cross-Correlation (NCC Sum of Squared Differences (SSD Normalized SSD Sum of Absolute Differences (SAD Zero Mean SAD Rank Census ( I( I ( I ( u + d, I ( I( I ( I ( u + d, I ( I ( I ( u + d, ( I( I ( I ( u + d, I ( ( ( ( + I I I u d, I I ( I ( u + d, ( I ( I ( I( u + d, I I k ( = I k ( m, n < I k ( m, n ( I ( I ( u + d, Ik ( = BITSTRINGm, n( Ik ( m, n < Ik ( HAMMING( I (, I ( u + d, DEFINITION These two are actually the same Stereo results Data from Uniersity of Tsukuba Scene Ground truth (Seitz Results with window correlation Results with better method Window-based matching (best window size (Seitz Ground truth Using global optimization Boyko et al., Fast Approximate Energy Minimization ia Graph Cuts, (Seitz Ground truth 5

6 Some Issues Ambiguity Some Issues A challenge: Multiple Interpretations Each feature on left epipolar line match one and only one feature on right epipolar line. Multiple Interpretations Multiple Interpretations Each feature on left epipolar line match one and only one feature on right epipolar line. Each feature on left epipolar line match one and only one feature on right epipolar line. 6

7 Multiple Interpretations Some Issues Each feature on left epipolar line match one and only one feature on right epipolar line. Window size Some Issues Effect of window size (Seitz W = 3 W = 0 Better results with adaptie window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptie Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 99. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 8(: 55-74, July 998 Window Shape and Forshortening Window Shape: Fronto-parallel Configuration W p U U W r W l 7

8 Some Issues Lighting Conditions (Photometric Variations W(P l W(P r Some Issues Half occluded regions Summary of Stereo Constraints CONSTRAINT BRIEF DESCRIPTION Stereo matching Monotonic Ordering Image Brightness Constancy Match Uniqueness Disparity Continuity Disparity Limit Fronto-Parallel Surfaces Feature Similarity -D Epipolar Search Arbitrary images of the same scene may be rectified based on epipolar geometry such that stereo matches lie along onedimensional scanlines. This reduces the computational complexity and also reduces the likelihood of false matches. Points along an epipolar scanline appear in the same order in both stereo images, assuming that all objects in the scene are approximately the same distance from the cameras. Assuming Lambertian surfaces, the brightness of corresponding points in stereo images are the same. For eery point in one stereo image, there is at most one corresponding point in the other image. Disparities ary smoothly (i.e. disparity gradient is small oer most of the image. This assumption is iolated at object boundaries. The search space may be reduced significantly by limiting the disparity range, reducing both computational complexity and the likelihood of false matches. The implicit assumption made by area-based matching is that objects hae fronto-parallel surfaces (i.e. depth is constant within the region of local support. This assumption is iolated by sloping and creased surfaces. Corresponding features must be similar (e.g. edges must hae roughly the same length and orientation. Similarity measure (SSD or NCC Optimal path (dynamic programming Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data Discontinuities (prior Structural Grouping Corresponding feature groupings and their connectiity must be consistent. (From Pollefeys (Cox et al. CVGIP 96; Koch 96; Falkenhagen 97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV 0 (From G. Hager 8

9 Variations on Binocular Stereo Trinocular Epipolar Constraints. Trinocular Stereopsis. Helmholtz Reciprocity Stereopsis These constraints are not independent! Helmholtz reciprocity Point Source Illumination ρ(θ in, φ in ; θ out, φ out = ρ(θ out, φ out ; θ in, φ in ^ n p ^ n p θ in, φ in ^ n θ out, φ out ^ n ^ l ^ r ^ l ^ r θ out, φ out o l o r o l o r θ in, φ in ˆn ˆ i l = ρ(ˆ r,ˆ l r o r p ˆn ˆ i r = ρ(ˆ l,ˆ r l o l p [Helmholtz, 90], [Minnaert, 94], [ Nicodemus et al, 977] = Disparity and Normal Field Experimental Setup 9

10 Helmholtz Stereopsis Experimental Aparatus Bulldog: Disparity Bulldog: Normal Field Plastic Baby Doll: Normal Field Second Generation Rig Plastic Baby Doll: Disparities 0

11 Surface after integrating normal field Renderings of Reconstruction Metric Reconstruction More on stereo The Middleburry Stereo Vision Research Page Established a long time ago, but many algorithms compared D. Scharstein and R. Szeliski. A Taxonomy and Ealuation of Dense Two- Frame Stereo Correspondence Algorithms. IJCV 47(//3:7-4, April-June 00. PDF file (.5 MB - includes current ealuation. Microsoft Research Technical Report MSR-TR-00-8, Noember 00. The KITTI Vision Benchmark Suite