STATE-OF-THE-ART in DENSE IMAGE MATCHING Fabio REMONDINO 3D Optical Metrology (3DOM) Bruno Kessler Foundation (FBK) Trento, Italy Email: remondino@fbk.eu http://3dom.fbk.eu Bruno Kessler Foundation (FBK) - Born in 1962 as public research center of the Autonomous Province of Trento (Italy) - Research primarily in the fields of ICT, Materials & Sensors - More than 350 researchers and developers, 7 laboratories, 26 research units, 11 spin-offs - Research units and labs: Biosensors, Bioelectronics, Optical sensors, Radiation detectors, Renewable energies, Intelligent interfaces, Predictive models, Technologies for Vision, 3D Optical Metrology, etc. - Hosting BSc, MSc and PhD students - Master and PhD programs jointly held with international universities 1
3D Optical Metrology (3DOM) unit 10 researchers Applied research in the fields of Photogrammetry, Laser scanning and Remote Sensing, 3D Mapping, Geomatics Hosting BSc, MSc and PhD students (2010: 4 guests; 2011: 9 guest; 2012: 7 guests; 2013: 9 guests, 2014: 4) Exchange agreement with int. universities for hosting student (ENSG Paris, CGT Siena, etc.) Funding from local (PAT, private sector, etc.) and international grants (EU, BMBF, private sector, etc.) Organization of int. events and summerschools for knowledge and technology transfer Collaborations with national and international universities and private companies 3D RECORDING & PROCESSING CHAIN SENSORS & PLATFORMS SOFTWARE & ALGORITHMS VISUALIZATION, SEGMENTATION, GIS, SHARING, etc DATA 2D-3D RESULTS 4 2
IMAGE-BASED PIPELINE Camera calibration & Image triangulation (Dense) 3D reconstruction Polygonal 3D model, maps, orthoimages, etc. 5 Establishment of image correspondences between 2 or more images Delivery of 3D coordinates from 2D image correspondences Automated vs interactive procedure IMAGE MATCHING O A (X0, Y0, Z0, w, f, k) O B (X0, Y0, Z0, w, f, k) Y Y X x A, y A X x B, y B image B image A X, Y, Z Z Y object space 6 Reference system X 3
IMAGE MATCHING Establishment of image correspondences between 2 or more images Delivery of 3D coordinates from 2D image correspondences Automated vs interactive procedure Natural features vs targets Single point measurement vs every pixel, i.e. sparse vs dense point cloud Statistical information of the matched 2D points and derived 3D coordinates 7 A LOOK AT THE PAST PHOTOGRAMMETRY 1950 s: Analogue image matching and stereo-plotter 1980 s: LSM & Multi-photo matching 1960 s: First digital crosscorrelation 1990 s: Digital stereo processing systems 2000 s: Close range photogrammetry (convergent images) http://www.mtzgeo.com/history.cfm COMPUTER VISION 1980: Stereo matching 1990 s: Multi-view stereo reconstruction 2000 s: Dense reconstruction 8 4
A LOOK AT THE PAST Automated 3D reconstruction of the Bamiyan Buddha from amateur images (2003) [Gruen, A., Remondino, F., Zhang, L., 2003: Image-based reconstruction of the Great Buddha of Bamiyan, Afghanistan. Proc. of Videometrics VII, Vol. 5013, pp. 129-13] Automated DSM generation from SPOT5 stereo-pair over Bamiyan, Afghanistan (2004) [Gruen, A., Remondino, F., Zhang, L., 2004: The Bamiyan Valley: Landscape Modeling for Cultural Heritage Visualization & Documentation. Int. Archives of PRS & SIS, Vol. XXXVI(5/W1)] 9 IMAGE MATCHING ALGORITHM Algorithm developments (1) Generally dense matching on the entire image (up 1:1) Semi-Global Matching, graph-cut, patch-based, optimal flow, least squares, etc. Multi-image vs stereo-pairs Image-based vs object-based (volumetric) Area-based vs feature-based Local vs global methods 10 5
IMAGE MATCHING ALGORITHM Algorithm developments (2) Open-source solutions: MicMac, PMVS Commercial solutions inside different tools: ERDAS-LPS (eate), Photoscan, Inpho Match-T, Pix4D, etc. Stand-alone commercial solution: SURE Comparable geometric results as laser scanning / LiDAR 11 Trento (Italy), Geo-Eye stereo-pair, 1m resolution DSM 12 6
Marseille, France: Vexcel Ultracam-X, 10 cm GSD 20 cm DSM 13 Paris, France: Oblique imagery, 10 cm GSD 20 cm DSM 14 7
Graz, Austria: Oblique imagery (Osprey camera), 10 cm GSD 15 UAV image block, avg GSD: 2-3 cm 16 8
Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.) ca 2 x 1.5 x 1 m ca 600 images avg GSD: 0.2 mm avg image scale: 1:25 17 Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.) Dense matching algorithm delivered ca 50 mil. punti 18 9
Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.) QUALITY CONTROL: intersection angles 19 Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.) QUALITY CONTROL: redundancy (numb. of images contributing to the 3D point) 20 10
Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.) QUALITY CONTROL: precision of the computed 3D points (avg: 0.06 mm) 21 Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.) 22 11
DENSE IMAGE MATCHING Comparisons (1) 23 DENSE IMAGE MATCHING Comparisons (2) 24 12
DENSE IMAGE MATCHING Open issues Repeated pattern Illumination and scale changes (oblique imagery) Noisy 3D data due to low quality images Mismatches Irrelevant points How to go from dense point clouds to more useful and structured 3D objects Large quantity of data (dense point clouds) to be managed 25 Acknowledgments THANK YOU 3D Optical Metrology unit, FBK Trento Fabio Remondino 3D Optical Metrology unit FBK Trento, Italy http://3dom.fbk.eu References: Remondino, F., Spera, M.G., Nocerino, E., Menna, F., Nex, F., 2014: State of the art in high density image matching. The Photogrammetric Record, Vol. 29(146), pp. 144-166 Rupnik, E., Nex, F., Remondino, F., 2014: Oblique multi-camera systems Orientation and dense matching issues. EuroCOW Proceedings Remondino, F., El-Hakim, S., Gruen, A., Zhang, L., 2008: Turning images into 3D models - Development and performance analysis of image matching for detailed surface reconstruction of heritage objects. IEEE Signal Processing Magazine, Vol. 25(4), pp. 55-65 26 13