Stereo matching Francesco Isgrò 3D Reconstruction and Stereo p.1/21
Structure of a stereo vision system Extract interesting point from each image Determine a set of matching points Compute the fundamental matrix Rectify the two images Compute a dense stereo matching Representation of the scene 3D Reconstruction and Stereo p.2/21
Corresponding points Given two sets {p i } i=1,,n and {p j} i=1,,m, establish correspondences between the two sets There are two schools of thought to find corresponding points Extract points in one image and then look for correspondences in the second image Extract points in both images independently and then match them 3D Reconstruction and Stereo p.3/21
Point matching Two major work in this class: correlation matching + relaxation SVD matching 3D Reconstruction and Stereo p.4/21
SVD matching 1. Build the matrix G, with g ij = C ij + 1 2 e p i p 2σ 2 j 2 2. C ij normalised cross correlation between points i and j 3. Compute the SVD of the matrix: G = UWV t 4. Replace W with E, with all 1 on the diagonal 5. Compute P = UEV t 6. If C ij > k and P ij is the largest element in its row and it column, match p i with p j 3D Reconstruction and Stereo p.5/21
SVD matching: example 3D Reconstruction and Stereo p.6/21
SVD matching: example 3D Reconstruction and Stereo p.6/21
Rectification P O l O r 3D Reconstruction and Stereo p.7/21
Given a pair of stereo images rectification determines two homographies H 1 and H 2 such that pairs of conjugate epipolar lines become collinear and parallel to one of the image axes Corresponding point between the two images are then on the same scan-line. The two rectified images can be regarded as obtained from a parallel stereo rig It corresponds to changing the epipolar geometry to a particular one. 3D Reconstruction and Stereo p.8/21
3D Reconstruction and Stereo p.9/21
3D Reconstruction and Stereo p.10/21
Correspondence problem Problem For each point in the left image find the correspnding point in the right image Assumptions most scene points are visible from both views Corresponding image regions are similar These assumptions holds if the distance of the fixation point is larger than the baseline. 3D Reconstruction and Stereo p.11/21
Dense stereo matching Output is a disparity map giving the relative displacement for each pixel The disparity is proportional to the inverse of the distance Disparity needed for 3D structure 3D Reconstruction and Stereo p.12/21
Dense stereo matching: constraints Epipolar constraint Similarity: image patches of corresponding pixels must be similar Order: images match, then matches of nearby points should maintain the same order Smoothness: disparities should change smoothly distances with the dsitances from the camera Uniqueness: each pixel cannot match more than one pixel in the other image 3D Reconstruction and Stereo p.13/21
What to do? We must make two choices. which image elements to match which similarity measure to adopt We stick to the following choices we match image windows use correlation based measures 3D Reconstruction and Stereo p.14/21
Correlation methods 1. for each pixel p in the left image consider a neighourhood N p of size n r n c 2. select a q on the epipolar line of p in the right image having the neighourhood N q most similar to N p 3D Reconstruction and Stereo p.15/21
Similarity measures C 1 (x, y, d) = ij I y+i,x+j I y+i,x+j+d 2 C 2 (x, y, d) = ij I y+i,x+j I y+i,x+j+d 2 P ij I2 y+i,x+j P ij I 2 y+i,x+j+d C 3 (x, y, d) = ij I y+i,x+j I y+i,x+j+d P ij I2 y+i,x+j P ij I 2 y+i,x+j+d C 4 (x, y, d) = ij (I y+i,x+j µ yx ) (I y+i,x+j+d µ y,x+d ) 2 σ yx σ y,x+d C 5 (x, y, d) = ij (I y+i,x+j µ yx I y+i,x+j+d µ y,x+d σ yx σ y,x+d 3D Reconstruction and Stereo p.16/21
Searching q over the whole epipolar line is not necessary In general the corresponding q is not so far from the position of p Normally the search is restricted to a segment around the position of p Remark: the correspondence problem is made more difficult by occlusions (i.e., point with no counterpart in the other image) 3D Reconstruction and Stereo p.17/21
Multi-window stereo matching 3D Reconstruction and Stereo p.18/21
Multi-window stereo matching 1. For each pixel in the left image, select a search region in the right image. 2. For each pixel in the search region, compute similarity for each of the nine windows 3. Return the position associated to the highest of the nine values returned as the valid left-to-right match. 4. Repeat steps 1-4 after swapping left and right images, yielding the best right-to-left set of matches. 5. Keep only the matching pairs which are found in both directions, and discard the others, leaving holes in the disparity maps. 3D Reconstruction and Stereo p.18/21
Results 3D Reconstruction and Stereo p.19/21
Results 3D Reconstruction and Stereo p.19/21
Results 3D Reconstruction and Stereo p.19/21
Results 3D Reconstruction and Stereo p.19/21
Large baseline stereo If the baseline is large the disparity range can be large Multi-resolution stereo matching can give speed-up and better accuracy We do stereo-matching at the coarsest level Result is used as prediction for disparity at finer level 3D Reconstruction and Stereo p.20/21
Results 3D Reconstruction and Stereo p.21/21
Results 3D Reconstruction and Stereo p.21/21