Stereo (Part 2) Introduction to Computer Vision CSE 152 Lecture 9

Transcription

1 Stereo (Part 2) CSE 152 Lectre 9

2 Annoncements Homework 3 is de May 9 11:59 PM Reading: Chapter 7: Stereopsis

3 Stereo Vision Otline Offline: Calibrate cameras & determine B epipolar geometry Online C D A 1. Acqire stereo images 2. Rectify images to conenient epipolar geometry 3. Establish correspondence 4. Estimate depth

4 Rectification Gien a pair of images transform both images so that epipolar lines are scan lines. e 2 e 1

5 Rectification Under perspectie projection the mapping from a plane to a plane is gien by a 2D projectie transformation (homography) x L y L w L L H L L 1 ( L L ) (x L y L )

6 Rectification Under perspectie projection the mapping from a plane to a plane is gien by a 2D projectie transformation (homography) x L y L w L L H L L 1 ( L L ) (x L y L ) ( R R ) (x R y R ) x R y R w R R H R R 1 Two images Two projectie transformations

7 Epipolar Rectification Create pair of irtal cameras Virtal cameras hae the same camera centers as real cameras Both irtal cameras hae the same: Camera rotation matrix R Camera calibration matrix K Rectification transformation matrices HK irtal R irtal R T real Kreal Camera centers Scene Real images

8 mage pair rectification Simplify stereo matching by warping the images e Apply projectie transformation so that epipolar lines correspond to horizontal scanlines ( L L ) e (x L y L ) H H shold map epipole e to (100) a point at infinity on the x-axis H shold minimize image distortion Note that rectified images are sally not rectanglar See textbook for complete method 1 0 He 0

9 Rectification Gien a pair of images transform both images so that epipolar lines are scan lines. npt mages

10 Rectification Gien a pair of images transform both images so that epipolar lines are scan lines. Rectified mages See Section for specific method epipolar lines rn parallel with the x-axis and are aligned between two iews (no y disparity)

11 Rectification Original Rectified

12 Epipolar lines Rectification

13 Rectification Original Rectified

14 Polar Rectification Homography-based Rectification Alternatie epipolar rectification method that minimizes pixel distortion Polar Rectification

15 Polar Rectification Epipoles are in images (white dot on ball) Homography-based rectification is not possible

16 Featres on same epipolar line Trco Fig. 7.5

17 Mobi: Stereo-based naigation

18 Epipolar correspondence

19 Symbolic Map

20 Using epipolar & constant Brightness constraints for stereo matching For each epipolar line For each pixel in the left image compare with eery pixel on same epipolar line in right image pick pixel with minimm match cost This will neer work so: match windows (Seitz)

21 Finding Correspondences W(p l ) W(p r )

22 Comparing Windows: =? f g Most poplar For each window match to closest window on epipolar line in other image. (Camps)

23 Correspondence Search Algorithm For i = 1:nrows for j=1:ncols best(ij) = -1 for k = mindisparity:maxdisparity c = Match_Metric( 1 (ij) 2 (ij+k)winsize) if (c > best(ij)) best(ij) = c disparities(ij) = k end end end end O(nrows * ncols * disparities * winx * winy) d 1 2

24 Match Metric Smmary d d d d d d 2 1 d ' 2 ' 1 m n k k k n m ' d HAMMNG ' 2 ' 1 m n BTSTRNG k k m n k ' MATCH METRC DEFNTON Normalized Cross-Correlation (NCC) Sm of Sqared Differences (SSD) Normalized SSD Sm of Absolte Differences (SAD) Zero Mean SAD Rank Censs These two are actally the same d _ 2 2 _ 1 1 ) ( ) (

25 Stereo reslts Data from Uniersity of Tskba Scene Grond trth (Seitz)

26 Reslts with window correlation Window-based matching (best window size) (Seitz) Grond trth

27 Reslts with better method Using global optimization Boyko et al. Fast Approximate Energy Minimization ia Graph Cts nternational Conference on Compter Vision September Grond trth (Seitz)

28 State of the Art Reslts Using neral networks Grond trth S. Droyer S. Becher M. Bilodea M. Moread and L. Sorbier. Sparse stereo disparity map densification sing hierarchical image segmentation. 13th nternational Symposim on Mathematical Morphology.

29 Some sses Epipolar ordering Ambigity Window size Window shape Lighting Half occlded regions

30 A challenge: Mltiple nterpretations Each featre on left epipolar line match one and only one featre on right epipolar line.

31 Mltiple nterpretations Each featre on left epipolar line match one and only one featre on right epipolar line.

32 Mltiple nterpretations Each featre on left epipolar line match one and only one featre on right epipolar line.

33 Mltiple nterpretations Each featre on left epipolar line match one and only one featre on right epipolar line.

34 Some sses Epipolar ordering Ambigity Window size Window shape Lighting Half occlded regions

35 Ambigity

36 Some sses Epipolar ordering Ambigity Window size Window shape Lighting Half occlded regions

37 Window size W = 3 W = 20 Effect of window size Better reslts with adaptie window T. Kanade and M. Oktomi A Stereo Matching Algorithm with an Adaptie Window: Theory and Experiment Proc. nternational Conference on Robotics and Atomation D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffsion. nternational Jornal of Compter Vision 28(2): Jly 1998 (Seitz)

38 Some sses Epipolar ordering Ambigity Window size Window shape Lighting Half occlded regions

39 Window Shape and Forshortening

40 Window Shape: Fronto-parallel Configration W p U 1 U 2 W l W r

41 Some sses Epipolar ordering Window size Ambigity Window shape Lighting Half occlded regions

42 Lighting Conditions (Photometric Variations) W(P l ) W(P r )

43 Some sses Epipolar ordering Ambigity Window size Window shape Lighting Half occlded regions

44 Half occlded regions

45 Smmary of Stereo Constraints CONSTRANT BREF DESCRPTON 1-D Epipolar Search Arbitrary images of the same scene may be rectified based on Monotonic Ordering epipolar geometry sch that stereo matches lie along onedimensional scanlines. This redces the comptational complexity and also redces the likelihood of false matches. Points along an epipolar scanline appear in the same order in both stereo images assming that all objects in the scene are approximately the same distance from the cameras. mage Brightness Assming Lambertian srfaces the brightness of corresponding points in stereo images are the same. Constancy Match Uniqeness For eery point in one stereo image there is at most one corresponding point in the other image. Disparity Continity Disparity Limit Fronto-Parallel Srfaces Featre Similarity Strctral Groping Disparities ary smoothly (i.e. disparity gradient is small) oer most of the image. This assmption is iolated at object bondaries. The search space may be redced significantly by limiting the disparity range redcing both comptational complexity and the likelihood of false matches. The implicit assmption made by area-based matching is that objects hae fronto-parallel srfaces (i.e. depth is constant within the region of local spport). This assmption is iolated by sloping and creased srfaces. Corresponding featres mst be similar (e.g. edges mst hae roghly the same length and orientation). Corresponding featre gropings and their connectiity mst be consistent. (From G. Hager)

46 Next Lectre Early ision: mltiple images Strctre from motion Reading: Chapter 8: Strctre from Motion