Computer Vision I. Announcement. Stereo Vision Outline. Stereo II. CSE252A Lecture 15

Transcription

1 Announcement Stereo II CSE252A Lecture 15 HW3 assigned No class on Thursday 12/6 Extra class on Tuesday 12/4 at 6:30PM in WLH Room 2112 Mars Exploratory Rovers: Spirit and Opportunity Stereo Vision Outline Offline: Calibrate cameras & determine B epipolar geometry Online 1. Acquire stereo images C 2. Rectify images to convenient epipolar geometry D A 3. Establish correspondence 4. Estimate depth Baseline: d Focal length: f Disparity: (X L - X R ) BINOCULAR STEREO SYSTEM Estimating Depth Z (X,Z) Reconstruction: General 3-D case Given two image measurements p and p, estimate P. Z = (f/x L ) X Z= (f/x R ) (X-d) (f/x L ) X = (f/x R ) (X-d) X = (X L d) / (X L - X R ) X = d X L (X L - X R ) Z=f X L X R (0,0) (d,0) X Linear Method: find P such that Z = d f (X L - X R ) X L =f(x/z) X R =f((x-d)/z) (Adapted from Hager) Non-Linear Method: find Q minimizing where q=mq and q =M Q 1

2 Random Dot Stereograms Need for correspondence Epipolar Geometry Family of epipolar Planes Baseline Epipoles Epipolar Plane Epipolar Lines O O Family of planes π and lines l and l Intersection in e and e Skew Symmetric Matrix & Cross Product Epipolar Constraint: Calibrated Case The cross product a x b of two vectors a and b can be expressed a matrix vector product [a x ]b where[a x ] is the skew symmetric matrix: 0 a 3 a 2 [ a ] = a 3 0 a 1 a 2 a 1 0 A matrix S is skew symmetric iff S = -S T Essential Matrix (Longuet-Higgins, 1981) 2

3 Calibration The Eight-Point Algorithm (Longuet-Higgins, 1981) Much more on multi-view in CSE252B!! Here, F is the Essential Matrix Consider 8 points (u i,v i ), (u i,v i ) Set F 33 to 1 Solve for F 11 to F 32 Determine intrinsic parameters and extrinsic relation of two cameras For more than 8 points, solve using linear least squares The Eight-Point Algorithm (Longuet-Higgins, 1981) Much more on multi-view in CSE252B!! Epipolar geometry example Alternatively, view this as system of homogenous equations in F 11 to F 33 Solve as Eigenvector corresponding to the smallest Eigenvalue of matrix created from the image data. Equivalent to solving Minimize: under the constraint 2 F =1. T Properties of the Essential Matrix E p is the epipolar line associated with p. T E T p is the epipolar line associated with p. E e =0 and E T e=0. E is singular. E has two equal non-zero singular values (Huang and Faugeras, 1989). The Fundamental Matrix The epipolar constraint is given by: where p and p are 3-D coordinates of the image coordinates of points in the two images. Without calibration, we can still identify corresponding points in two images, but we can t convert to 3-D coordinates. However, the relationship between the calibrated coordinates (p,p ) and uncalibrated image coordinates (q,q ) can be expressed as p=aq, and p =A q Therefore, we can express the epipolar constraint as: (Aq) T E(A q ) = q T (A T EA )q = q T Fq = 0 where F is called the Fundamental Matrix. Can estimate F using 8 point algorithm WITHOUT CALIBRATION 3

4 Example: converging cameras Example: motion parallel with image plane (simple for stereo rectification) courtesy of Andrew Computer Zisserman Vision I courtesy of Andrew Computer Zisserman Vision I Example: forward motion Rectification Given a pair of images, transform both images so that epipolar lines are scan lines. e e courtesy of Andrew Computer Zisserman Vision I Image pair rectification simplify stereo matching by warping the images Rectification Apply projective transformation H so that epipolar lines correspond to horizontal scanlines e H e map epipole e to (1,0,0) try to minimize image distortion Note that rectified images usually not rectangular 4

5 Multiple Interpretations Multiple Interpretations Multiple Interpretations Multiple Interpretations The ordering constraint Correspondence: Photometric constraint Same world point has same intensity in both images (Constant Brightness Constraint) Lambertian fronto-parallel Issues: Noise Specularity Foreshortening The order of corresponding points along a pair of epipolar lines should be the same. Note: As seen in previous slides, doesn t always hold, but usually does and can vastly speed up computation 5

6 Using epipolar & constant Brightness constraints for stereo matching Slide Window to different disparities to find best match For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum match cost This will never work, so: Best Match amounts to minimizing (or maximizing some) Match Metric Improvement: match windows (Seitz) Finding Correspondences Comparing Windows: For each window, match to closest window on epipolar line in other image. W(pl) W(pr) (Camps) Correspondence Search Algorithm Match Metric Summary MATCH METRIC DEFINITION Normalized Cross-Correlation (NCC) For i = 1:nrows for j=1:ncols best(i,j) = -1 for k = mindisparity:maxdisparity c = Match_Metric(I1(i,j),I2(i,j+k),winsize) if (c > best(i,j)) best(i,j) = c disparities(i,j) = k O(nrows * ncols * disparities * winx * winy) Sum of Squared Differences (SSD) These two are actually the same Normalized SSD Sum of Absolute Differences (SAD) Zero Mean SAD Rank Census 6