Efficient Large-Scale Stereo Matching

Transcription

1 Efficient Large-Scale Stereo Matching Andreas Geiger*, Martin Roser* and Raquel Urtasun** *KARLSRUHE INSTITUTE OF TECHNOLOGY **TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO KIT University of the State of Baden-Wuerttemberg and National Laboratory of the Helmholtz Association

2 Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: 2/24

3 Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: 3/24

4 Motivation Camera Laserscanner Efficient Large-Scale Stereo Matching Code: 4/24

5 Why is 3D from Stereo hard? Ambiguities Textureless regions Sensor saturation Non-Lambertian surfaces z grows quadratically Computational burden Efficient Large-Scale Stereo Matching Code: 5/24

6 Why is 3D from Stereo hard? Ambiguities Textureless regions Sensor saturation Non-Lambertian surfaces z grows quadratically Computational burden Efficient Large-Scale Stereo Matching Code: 5/24

7 Why is 3D from Stereo hard? Ambiguities Textureless regions Sensor saturation Non-Lambertian surfaces z grows quadratically Computational burden distance error focal length disparity error baseline Efficient Large-Scale Stereo Matching Code: 5/24

8 Related Work: Local Methods Local Methods Winner-takes-All Examples Block matching (Scharstein 02) Adaptive windows (Kanade 94, Yoon 06) Plane-sweep (Collins 96, Gallup 07) Problems Small matching ratios Border bleeding Efficient Large-Scale Stereo Matching Code: 6/24

9 Related Work: Global Methods Global Methods Minimize 1D/2D energy E(d) = E data (d) + λe smooth (d) Examples Graph cuts, Belief propagation (Kolmogorov 02, Felzenszwalb 06) Variational methods (Pock 07, Zach 09) Fusion moves (Woodford 08, Bleyer 10) Problems Computational and memory requirements Pairwise potentials can not model planarity Efficient Large-Scale Stereo Matching Code: 7/24

10 Related Work: Seed-and-Grow Seed-and-Grow Methods Grow disparity components from random seeds Examples (Cech 07) (Sara 03) Problems Slanted/textureless surfaces No dense disparity maps Efficient Large-Scale Stereo Matching Code: 8/24

11 Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: 9/24

12 Idea Assumption: rectified images Image pairs contain easy and hard correspondences Robustly match easy correspondences on regular grid Build prior on dense search space dense matching Efficient Large-Scale Stereo Matching Code: 10/24

13 Idea Easy Hard Assumption: rectified images Image pairs contain easy and hard correspondences Robustly match easy correspondences on regular grid Build prior on dense search space dense matching Efficient Large-Scale Stereo Matching Code: 10/24

14 Idea Robust Match Assumption: rectified images Image pairs contain easy and hard correspondences Robustly match easy correspondences on regular grid Build prior on dense search space dense matching Efficient Large-Scale Stereo Matching Code: 10/24

15 Idea Assumption: rectified images Image pairs contain easy and hard correspondences Robustly match easy correspondences on regular grid Build prior on dense search space dense matching Efficient Large-Scale Stereo Matching Code: 10/24

16 Efficient Large-Scale Stereo Notation Robust support points S = {s 1,..., s M } with s m = (u m v m d m ) T Disparity d n N Observations o n = (u n v n f n ) T Local image features f n Algorithm Split image domain into support points S and dense pixels Assume factorization of distribution over disparity, observations and support points into... Efficient Large-Scale Stereo Matching Code: 11/24

17 Efficient Large-Scale Stereo Notation Robust support points S = {s 1,..., s M } with s m = (u m v m d m ) T Disparity d n N Observations o n = (u n v n f n ) T Local image features f n Algorithm Split image domain into support points S and dense pixels Assume factorization of distribution over disparity, observations and support points into... Efficient Large-Scale Stereo Matching Code: 11/24

18 Efficient Large-Scale Stereo Notation Robust support points S = {s 1,..., s M } with s m = (u m v m d m ) T Disparity d n N Observations o n = (u n v n f n ) T Local image features f n Algorithm Split image domain into support points S and dense pixels Assume factorization of distribution over disparity, observations and support points into... Efficient Large-Scale Stereo Matching Code: 11/24

19 Model Left Image Right Image Graphical Model Support Points Support Points Observation in left image Observations p(d n, o (l) n, o (r) n, S) p(d n S, o (l) }{{} Prior n ) p(o (r) n o (l) n, d n ) }{{} Likelihood Efficient Large-Scale Stereo Matching Code: 12/24

20 Model Left Image Right Image Graphical Model Support Points Support Points Observation in left image Observations p(d n, o (l) n, o (r) n, S) p(d n S, o (l) }{{} Prior n ) p(o (r) n o (l) n, d n ) }{{} Likelihood Efficient Large-Scale Stereo Matching Code: 12/24

21 Prior and Likelihood Prior p(d n S, o (l) n ) Support pt. triangulation Piecew. linear manifold Local extrapolation Likelihood p(o (r) n o (l) n, d n ) Laplace distribution 5 5 block window 3 3 Sobel filter Efficient Large-Scale Stereo Matching Code: 13/24

22 Prior and Likelihood Prior p(d n S, o (l) n ) Support pt. triangulation Piecew. linear manifold Local extrapolation Likelihood p(o (r) n o (l) n, d n ) Laplace distribution 5 5 block window 3 3 Sobel filter Efficient Large-Scale Stereo Matching Code: 13/24

23 Sampling from the model Left image Sample mean Efficient Large-Scale Stereo Matching Code: 14/24

24 Sampling from the model Left image Sample mean Right image Efficient Large-Scale Stereo Matching Code: 14/24

25 Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: 15/24

26 Middlebury Benchmark 900 x 750 pixels, ground truth Efficient Large-Scale Stereo Matching Code: 16/24

27 Middlebury Benchmark 900 x 750 pixels, 0.4 seconds Efficient Large-Scale Stereo Matching Code: 16/24

28 Middlebury Benchmark 1300 x 1100 pixels, ground truth Efficient Large-Scale Stereo Matching Code: 16/24

29 Middlebury Benchmark 1300 x 1100 pixels, 1 second Efficient Large-Scale Stereo Matching Code: 16/24

30 Accuracy (on cones image pair) Error >2px [%] uniform prior Cech 07 Felzenszwalb 06 Kolmogorov 01 Kostkova 03 our method Image resolution [Megapixel] Efficient Large-Scale Stereo Matching Code: 17/24

31 Running times (on cones image pair) Kolmogorov 01 Kostkova 03 our method Felzenszwalb 06 Cech 07 uniform prior our method Running time [s] 1000 Running time [s] Image resolution [Megapixel] Image resolution [Megapixel] [For more details see: Geiger et al., ACCV 2010] Efficient Large-Scale Stereo Matching Code: 18/24

32 3D Reconstruction: Brussels 2 seconds [ Efficient Large-Scale Stereo Matching Code: 19/24

33 3D Face Reconstruction [ Efficient Large-Scale Stereo Matching Code: 20/24

34 Urban Scene Reconstruction Efficient Large-Scale Stereo Matching Code: 21/24

35 Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: 22/24

36 Summary Simple prior based on sparse feature matches Reduced ambiguities and run-time Takes into account slanted surfaces Real-time 3D reconstruction of static scenes on CPU C++ / MATLAB code available at Efficient Large-Scale Stereo Matching Code: 23/24

37 Summary Simple prior based on sparse feature matches Reduced ambiguities and run-time Takes into account slanted surfaces Real-time 3D reconstruction of static scenes on CPU C++ / MATLAB code available at Efficient Large-Scale Stereo Matching Code: 23/24

38 Summary Simple prior based on sparse feature matches Reduced ambiguities and run-time Takes into account slanted surfaces Real-time 3D reconstruction of static scenes on CPU C++ / MATLAB code available at Efficient Large-Scale Stereo Matching Code: 23/24

39 Summary Simple prior based on sparse feature matches Reduced ambiguities and run-time Takes into account slanted surfaces Real-time 3D reconstruction of static scenes on CPU C++ / MATLAB code available at Efficient Large-Scale Stereo Matching Code: 23/24

40 Summary Simple prior based on sparse feature matches Reduced ambiguities and run-time Takes into account slanted surfaces Real-time 3D reconstruction of static scenes on CPU C++ / MATLAB code available at Efficient Large-Scale Stereo Matching Code: 23/24

41 Future Work Develop better priors Incorporate segmentation / global reasoning on lines GPU implementation (goal: 20 fps at 1-2 megapixels) Employ as unitary potentials on global methods smaller label sets Thank you! Efficient Large-Scale Stereo Matching Code: 24/24

42 Future Work Develop better priors Incorporate segmentation / global reasoning on lines GPU implementation (goal: 20 fps at 1-2 megapixels) Employ as unitary potentials on global methods smaller label sets Thank you! Efficient Large-Scale Stereo Matching Code: 24/24

43 Future Work Develop better priors Incorporate segmentation / global reasoning on lines GPU implementation (goal: 20 fps at 1-2 megapixels) Employ as unitary potentials on global methods smaller label sets Thank you! Efficient Large-Scale Stereo Matching Code: 24/24

44 Future Work Develop better priors Incorporate segmentation / global reasoning on lines GPU implementation (goal: 20 fps at 1-2 megapixels) Employ as unitary potentials on global methods smaller label sets Thank you! Efficient Large-Scale Stereo Matching Code: 24/24