Announcements Stereo Vision Wrapup & Intro Introduction to Computer Vision CSE 152 Lecture 17 HW3 due date postpone to Thursday HW4 to posted by Thursday, due next Friday. Order of material we ll first cover recognition so that you re prepared for assignment. Then return to motion. Final Exam: Tuesday 6/7 8:00-11:00 AM, Here Rectification Rectification Given a pair of images, transform both images so that epipolar lines are scan lines. Using epipolar & constant Brightness constraints for stereo matching Correspondence Search Algorithm For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum match cost This will never work, so: Improvement: match windows For i = 1:nrows for j=1:ncols best(i,j) = -1 for k = mindisparity:maxdisparity c = Match_Metric(I 1 (i,j),i 2 (i,j+k),winsize) if (c > best(i,j)) best(i,j) = c disparities(i,j) = k O(nrows * ncols * disparities * winx * winy) 1
Match Metric Summary Stereo results MATCH METRIC Normalized Cross-Correlation (NCC) DEFINITION Data from University of Tsukuba Sum of Squared Differences (SSD) These two are actually the same Normalized SSD Sum of Absolute Differences (SAD) Zero Mean SAD Rank Census Scene Ground truth Disparity Map Results with window correlation Results with better method Window-based matching (best window size) Ground truth Near State of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. Ground truth Window size Window shape Lighting Ambiguity Ordering Some Issues Half occluded regions Effect of window size Window size W = 3 W = 20 Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): 155-174, July 1998 2
Window Shape and Forshortening Window Shape: Fronto-parallel Configuration W p U 1 U 2 W r W l Lighting Conditions (Photometric Variations) Ambiguity W(P l ) W(P r ) A challenge: Multiple Interpretations Multiple Interpretations 3
Multiple Interpretations Multiple Interpretations Problem of Occlusion CONSTRAINT Stereo Constraints BRIEF DESCRIPTION Monotonic Ordering 1-D Epipolar Search Arbitrary images of the same scene may be rectified based on epipolar geometry such that stereo matches lie along onedimensional scanlines. This reduces the computational complexity and also reduces the likelihood of false matches. Points along an epipolar scanline appear in the same order in both stereo images, assuming that all objects in the scene are approximately the same distance from the cameras. Image Brightness Constancy Match Uniqueness Disparity Continuity Disparity Limit Fronto-Parallel Surfaces Feature Similarity Structural Grouping Assuming Lambertian surfaces, the brightness of corresponding points in stereo images are the same. For every point in one stereo image, there is at most one corresponding point in the other image. Disparities vary smoothly (i.e. disparity gradient is small) over most of the image. This assumption is violated at object boundaries. The search space may be reduced significantly by limiting the disparity range, reducing both computational complexity and the likelihood of false matches. The implicit assumption made by area-based matching is that objects have fronto-parallel surfaces (i.e. depth is constant within the region of local support). This assumption is violated by sloping and creased surfaces. Corresponding features must be similar (e.g. edges must have roughly the same length and orientation). Corresponding feature groupings and their connectivity must be consistent. (From G. Hager) Stereo Matching using Dynamic Programming Stereo matching Similarity measure (SSD or NCC) Optimal path (dynamic programming ) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Ohta and Kanade, 1985) Reprinted from Stereo by Intra- and Intet-Scanline Search, by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine CSE152, Spring Intelligence, 2011 7(2):139-154 (1985). 1985 IEEE. (From Pollefeys) (Cox et al. CVGIP 96; Koch 96; Falkenhagen 97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV 02) 4
Some Challenges & Problems Photometric issues: specularities strongly non-lambertian BRDF s Variations on Binocular Stereo 1. Trinocular Stereopsis 2. Helmholtz Reciprocity Stereopsis Surface structure lack of texture repeating texture within horopter bracket Geometric ambiguities as surfaces turn away, difficult to get accurate reconstruction (affine approximate can help) at the occluding contour, likelihood of good match but incorrect reconstruction Trinocular Epipolar Constraints More on stereo The Middleburry Stereo Vision Research Page http://cat.middlebury.edu/stereo/ Recommed reading" D. Scharstein and R. Szeliski. " A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR-2001-81, November 2001." These constraints are not indepent! Myron Z. Brown, Darius Burschka, and Gregory D. Hager. Advances in Computational Stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(8): 993-1008, 2003. Announcements I Introduction to Computer Vision CSE 152 Lecture 17 HW 4 to be posted later today. It does not require a lot of coding, but does require understanding Order of material changed we ll first cover recognition so that you re prepared for assignment. Then return to motion. 5
Given a database of objects and an image determine what, if any of the objects are present in the image. Given a database of objects and an image determine what, if any of the objects are present in the image. Where are the coral heads and which ones are healthy and which are bleached? Given a database of objects and an image determine what, if any of the objects are present in the image. Bleached Input Image! Healthy Partially Bleached Segmented/labeled Image Object : The Problem Categories vs. Instances Camel Given: A database D of known objects and an image I: Mammal Face 1. Determine which (if any) objects in D appear in I 2. Determine the pose (rotation and translation) of the object Pose Est. (where is it 3D) Barbara Steele Two challenges: Segmentation (where is it 2D) Recognizing instances (what is it) Recognizing categories WHAT AND WHERE!!! 6
Challenges Within-class variability Different objects within the class have different shapes or different material characteristics Deformable Articulated Compositional Pose variability: 2-D Image transformation (translation, rotation, scale) 3-D Pose Variability (perspective, orthographic projection) Lighting Direction (multiple sources & type) Color Shadows Occlusion partial Clutter in background -> false positives Object Issues: How general is the problem? 2D vs. 3D range of viewing conditions available context segmentation cues What sort of data is best suited to the problem? Whole images Local 2D features (color, texture, 3D (range) data What information do we have in the database? Collection of images? 3-D models? Learned representation? Learned classifiers? How many objects are involved? small: brute force search large:?? A Rough Spectrum Appearance-Based (Eigenface, Fisherface) Geometric Invariants Shape Contexts Image Abstractions/ Volumetric Primitives Local Features + Spatial Relations 3-D Model-Based Bags of Features Function Increasing Generality 7