Efficient Large-Scale Stereo Matching

Efficient Large-Scale Stereo Matching Andreas Geiger*, Martin Roser* and Raquel Urtasun** *KARLSRUHE INSTITUTE OF TECHNOLOGY **TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO KIT University of the State of Baden-Wuerttemberg and National Laboratory of the Helmholtz Association www.kit.edu

Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 2/24

Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 3/24

Motivation Camera Laserscanner Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 4/24

Why is 3D from Stereo hard? Ambiguities Textureless regions Sensor saturation Non-Lambertian surfaces z grows quadratically Computational burden Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 5/24

Why is 3D from Stereo hard? Ambiguities Textureless regions Sensor saturation Non-Lambertian surfaces z grows quadratically Computational burden distance error focal length disparity error baseline Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 5/24

Related Work: Local Methods Local Methods Winner-takes-All Examples Block matching (Scharstein 02) Adaptive windows (Kanade 94, Yoon 06) Plane-sweep (Collins 96, Gallup 07) Problems Small matching ratios Border bleeding Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 6/24

Related Work: Global Methods Global Methods Minimize 1D/2D energy E(d) = E data (d) + λe smooth (d) Examples Graph cuts, Belief propagation (Kolmogorov 02, Felzenszwalb 06) Variational methods (Pock 07, Zach 09) Fusion moves (Woodford 08, Bleyer 10) Problems Computational and memory requirements Pairwise potentials can not model planarity Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 7/24

Related Work: Seed-and-Grow Seed-and-Grow Methods Grow disparity components from random seeds Examples (Cech 07) (Sara 03) Problems Slanted/textureless surfaces No dense disparity maps Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 8/24

Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 9/24

Idea Assumption: rectified images Image pairs contain easy and hard correspondences Robustly match easy correspondences on regular grid Build prior on dense search space dense matching Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 10/24

Idea Easy Hard Assumption: rectified images Image pairs contain easy and hard correspondences Robustly match easy correspondences on regular grid Build prior on dense search space dense matching Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 10/24

Idea Robust Match Assumption: rectified images Image pairs contain easy and hard correspondences Robustly match easy correspondences on regular grid Build prior on dense search space dense matching Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 10/24

Efficient Large-Scale Stereo Notation Robust support points S = {s 1,..., s M } with s m = (u m v m d m ) T Disparity d n N Observations o n = (u n v n f n ) T Local image features f n Algorithm Split image domain into support points S and dense pixels Assume factorization of distribution over disparity, observations and support points into... Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 11/24

Model Left Image Right Image Graphical Model Support Points Support Points Observation in left image Observations p(d n, o (l) n, o (r) n, S) p(d n S, o (l) }{{} Prior n ) p(o (r) n o (l) n, d n ) }{{} Likelihood Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 12/24

Prior and Likelihood Prior p(d n S, o (l) n ) Support pt. triangulation Piecew. linear manifold Local extrapolation Likelihood p(o (r) n o (l) n, d n ) Laplace distribution 5 5 block window 3 3 Sobel filter Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 13/24

Sampling from the model Left image Sample mean Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 14/24

Sampling from the model Left image Sample mean Right image Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 14/24

Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 15/24

Middlebury Benchmark 900 x 750 pixels, ground truth Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 16/24

Middlebury Benchmark 900 x 750 pixels, 0.4 seconds Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 16/24

Middlebury Benchmark 1300 x 1100 pixels, ground truth Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 16/24

Middlebury Benchmark 1300 x 1100 pixels, 1 second Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 16/24

Accuracy (on cones image pair) Error >2px [%] 20 15 10 uniform prior Cech 07 Felzenszwalb 06 Kolmogorov 01 Kostkova 03 our method 5 0 0 0.5 1 1.5 2 Image resolution [Megapixel] Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 17/24

Running times (on cones image pair) 2000 1500 Kolmogorov 01 Kostkova 03 our method 50 40 Felzenszwalb 06 Cech 07 uniform prior our method Running time [s] 1000 Running time [s] 30 20 500 10 0 0 0.5 1 1.5 2 Image resolution [Megapixel] 0 0 0.5 1 1.5 2 Image resolution [Megapixel] [For more details see: Geiger et al., ACCV 2010] Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 18/24

3D Reconstruction: Brussels 2 seconds [http://cvlab.epfl.ch/data/strechamvs/] Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 19/24

3D Face Reconstruction [http://www.fujifilm.com/products/3d] Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 20/24

Urban Scene Reconstruction Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 21/24

Contents 1 Motivation and Related Work 2 Efficient Large-Scale Stereo Matching 3 Experimental Evaluation 4 Summary and Future Work Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 22/24

Summary Simple prior based on sparse feature matches Reduced ambiguities and run-time Takes into account slanted surfaces Real-time 3D reconstruction of static scenes on CPU C++ / MATLAB code available at http://cvlibs.net Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 23/24

Future Work Develop better priors Incorporate segmentation / global reasoning on lines GPU implementation (goal: 20 fps at 1-2 megapixels) Employ as unitary potentials on global methods smaller label sets Thank you! Efficient Large-Scale Stereo Matching Code: http://cvlibs.net 24/24