3D reconstruction how accurate can it be?

Transcription

1 Performance Metrics for Correspondence Problems 3D reconstruction how accurate can it be? Pierre Moulon, Foxel CVPR 2015 Workshop Boston, USA (June 11, 2015)

2 We can capture large environments. But for real application we need accuracy!!! So, how accurate can it be?

3 Table of contents What is a 3D reconstruction? Reproducible research

4 What is a 3D reconstruction?

5 What is a 3D reconstruction? From 2d correspondences compute the scene structure & the camera locations.

6

7 3D reconstruction Image matching Pictures SfM: Structure from Motion Feature matching Which datasets are used? Multiple View Geometry

8 3D reconstruction Image matching Pictures Feature matching SfM: Structure from Motion Multiple View Geometry => Keypoints & matching accuracy. Which error models & datasets are used?

9 Multiple View Geometry How evaluate a matching accuracy? Residual Error Keypoint regions overlapping Need of Ground truth datasets

10 Feature matching VGG Oxford image dataset One image compared under a know homography matrix metric: residual error & % ellipses overlap K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir and L. Van Gool, A comparison of affine region detectors. In IJCV 65(1/2):43-72, PDF K. Mikolajczyk, C. Schmid, A performance evaluation of local descriptors. In PAMI 27(10): PDF

11 Feature matching Hannover high resolution image dataset use densely refined homography on High res. image Kai Cordes, Bodo Rosenhahn, and Jörn Ostermann: High-Resolution Feature Evaluation Benchmark, Computer Analysis of Images and Patterns (CAIP), Springer, 2013 Kai Cordes, Lukas Grundmann, and Jörn Ostermann: Feature Evaluation with High-Resolution Images, Computer Analysis of Images and Patterns (CAIP), Springer, 2015

12 Feature matching DTU Robot Image Data Sets Point Feature Data Set 2010 use 3d object with a moving camera robot metric: residual error & % ellipses overlap metric could take 3D depth/scale into account Henrik Aanæs, Anders Lindbjerg Dahl, and Kim Steenstrup Pedersen (2012): Interesting Interest Points. International Journal of Computer Vision, June, pdf bibtex

13 Feature matching What we have learned from those datasets? SIFT is still having very good performance (versatile) despite all the newcomers Residual error still in the [0.2;0.6] margin Interesting statistics about recall under viewing angle & distance

14 3D reconstruction Image matching Pictures Feature matching SfM: Structure from Motion Multiple View Geometry Camera pose accuracy evaluation. Which error models & datasets are used?

15 Multiple View Geometry How evaluate a 3d reconstruction accuracy? - Residual Error - # 3D Points Too much dependent of the keypoints & the matching steps.

16 Multiple View Geometry How evaluate a 3d reconstruction accuracy? - Residual Error - # 3D Points - Estimated parameters accuracy 3D poses location & orientations Need of Ground truth datasets

17 Multiple View Geometry Strecha MVS 6 dataset from 8 to 30 images. camera pose estimated from BA with points correspondences and LIDAR GT control points C. Strecha, W. von Hansen, L. Van Gool, P. Fua, U. Thoennessen On Benchmarking Camera Calibration and MultiView Stereo for High Resolution Imagery CVPR 2008 [pdf] 300 citations => only one new dataset released to the community since then

18 Multiple View Geometry DTU Robot Image Data Sets Point Feature Data Set 2010 use 3d object with a moving camera robot 60 scenes with 119 images Accurate positioning of the camera with a standard deviation of approximately 0.1 mm. Corresponds to a standard deviation of pixels if we projected a point onto the images. Henrik Aanæs, Anders Lindbjerg Dahl, and Kim Steenstrup Pedersen (2012): Interesting Interest Points. International Journal of Computer Vision, June, pdf bibtex

19 Multiple View Geometry What we have learned from those datasets? - Millimeter accuracy - reachable - Acquisition setup - crucial - Pipelines - sequential -> drift global -> best accuracy Moulon Pierre, Monasse Pascal and Marlet Renaud. Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion. ICCV 2013.

20 Multiple View Geometry What we have learned from those datasets? - Robust estimation - crucial Moulon Pierre, Monasse Pascal and Marlet Renaud. Adaptive Structure from Motion with a contrario model estimation. ACCV 2012.

21 Discussion Multiple View Geometry

22 Multiple View Geometry Those datasets are relatively small Today SFM community focus on handling large image collection Solution?

23 Multiple View Geometry Those datasets are relatively small Today SFM community focus on handling large image collection Solution? - use crowd sourced images (flickr) - find some geo tagged pictures - see how well the reconstruction fit "Discrete-Continuous Optimization for Large-Scale Structure from Motion," in CVPR 2011 (D. Crandall, A. Owens, N. Snavely, D. Huttenlocher) [pdf]

24 Multiple View Geometry Those datasets are relatively small Today SFM community focus on handling large image collection Solution? - use crowd sourced images (flickr) - find some geo tagged pictures - see how well the reconstruction fit => GPS is inaccurate! Does it is meaningful?

25 Can we do better? - GPS-rtk - centimeter accuracy - Laser scan of famous landmark - help picture registration - can allow estimation of the structure accuracy The 3D reconstruction community need it!

26 Reproducible research

27 Reproducible research How to enhance reproducible research?

28 Reproducible research How to enhance reproducible research? Providing open-source framework. Providing paper s corresponding implementation. OpenMVG

29 Reproducible research What is OpenMVG? OpenMVG is a list of libraries to solve MultiView Geometry problems, and SfM (Structure from Motion):

30 Reproducible research How to enhance reproducible research? OpenMVG provides 2 state of the art SfM pipelines & ready to use script to test accuracy Moulon Pierre, Monasse Pascal and Marlet Renaud. Adaptive Structure from Motion with a contrario model estimation. ACCV Moulon Pierre, Monasse Pascal and Marlet Renaud. Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion. ICCV 2013.

31 Questions Take home messages: Millimeters accuracy is reachable: check the scene to pixel ratio (resolution) ensure convergent view (stability) Community need accurate large scale GT datasets, please join the SfM force