Point Cloud Processing 2018 March 13th, 2018 TU Delft, the Netherlands POINT CLOUD ACQUISITION & STRUCTURING Fabio REMONDINO 3D Optical Metrology (3DOM) Bruno Kessler Foundation (FBK) Trento, Italy Email: remondino@fbk.eu http://3dom.fbk.eu with contributions from FBK-3DOM members: Isabella Toschi, Fabio Menna, Emre Oezdemir, Daniele Morabito, Elisa Farella, Erica Nocerino 1 POINT CLOUD ACQUISITION & STRUCTURING CONTENTS hardware / sensors dense image matching algorithms classification / segmentation mesh / polygonal model generation 2 1
WHERE DO WE USE POINT CLOUDS? generation / updating of 3D city models forest mapping / vegetation analytics monitoring corridor infrastructures volume computations (mining, landslides, etc.) heritage documentation and valorization Building Information Modeling (BIM) flood modeling change detection tunnel inspection monitoring coastal erosion dike monitoring etc. 3 WHERE DO WE USE POINT CLOUDS? 3D mapping / quantification of snow / ice lost on the Marmolada glacier Point cloud 2014 vs 2009 Point cloud 2014 2
WHERE DO WE USE POINT CLOUDS? 3D mapping of WWI fortifications: underground tunnels and under-forest trenches, classification of trench structures and gallery components (enrances, riflemen emplacements, barracks, etc, http://3dom.fbk.eu/repository/3dpointclouds/celva/index.html WHERE DO WE USE POINT CLOUDS? Brain anatomy with photogrammetric point clouds at 0.05 mm resolution to (i) study the white matter for an exhaustive understanding of the brain diseases and (ii) identify axons in the white matter responsible to transport information across the brain http://3dom.fbk.eu/repository/brain/index.html [Nocerino, E., Menna, F., Remondino, F., Sarubbo, S., De Benedictis, A., Chioffi, F., Petralia, V., Barbareschi, M., Olivetti, E., Avesani, P., 2017: Application of photogrammetry to brain anatomy. ISPRS Int. Archives of PRS&SIS, XLII-2-W4, pp. 213-219] 3
Object / Scene Complexity [points/object] 3/13/2018 TECHNIQUES FOR POINT CLOUD GENERATION 10 Mil 1 Mil 100 000 Close-range photogrammetry and UAV terrestrial laser scanners Aerial photogrammetry and Satellite 3D Remote Sensing 10 000 1 000 100 10 Tactile / CMM Hand Topography GNSS 1 measurements 0.1 m 1 m 10 m 100 m 1 km 10 km 100 km 1000 km Object / Scene Size 7 TECHNIQUES: IMAGING VS RANGING From 2000 there was a growing in popularity of /TLS sensors for the production of dense point clouds and photogrammetry could not efficiently deliver similar results Many researchers shifted their research interests to /TLS, resulting in a decline of new advancements / developments of new photogrammetric methods /TLS became the dominant technology for 3D recording and modelling, replacing photogrammetry in many application areas The bottleneck of photogrammetry was represented by massive manual data processing, high technical skills required, long processing time, etc. Over the past 10 years, improvements in hardware and software (primarily pushed from the Computer Vision community), have improved image-based tools and algorithms to the point that nowadays photogrammetry and /TLS can deliver comparable geometrical 3D results 8 4
9 PHOTOGRAMMETRY Origin born ca 1850 s born ca 1960 s Maturity Measurement principle Spectrum / Radiometry 60 s-70 s (BBA); 90 s digital sensors; 2000+ automated methods / SfM / DIM Triangulation Multispectral (VIS-NIR) 3D information To be derived direct Scale absent (to be provided) Implicit (1:1) Redundancy Multi-ray / Multi-view Single measure / Laser Scanning 2000 s with first commercial TLS and /ALS solutions TOF (long-range) and triangulation (short-range) Generally @ laser wavelength, rarely multispectral Dependency Light, geometry, texture Distance, material, object reflectance Statistics/ Quality parameters For each 3D TECHNIQUES: IMAGING VS RANGING Point density 10-100 pts/sqm 1-50 pts/sqm Generally one value for the entire cloud Precision/Accuracy XY most accurate than Z (depth) Z (depth) most accurate than XY Target detection Top-most surface (DSM) Multiple targets per pulse (DTM/DSM) TECHNIQUES: IMAGING VS RANGING Photogrammetry passive method (passive sensors) image-based method Scanning active method (active sensors) range-based method Image data acquisition Image pre-processing ACQUISITION Range data acquisition (dense 3D point cloud) Calibration and orientation PROCESSING Editing and alignment Dense 3D point cloud generation Surface generation, feature extraction and texture mapping STRUCTURING Surface generation, feature extraction and texture mapping Visualization, GIS products, Visualization, GIS products, VISUALIZATION replicas, inspection, virtual replicas, inspection, virtual 1:8 restoration, etc. restoration, etc. 1:10 10 5
POINT CLOUD ACQUISITION - SENSORS AERIAL TERRESTRIAL PASSIVE HYBRID ACTIVE Single frame cameras Multi-view cameras (oblique) DSLR cameras Panoramic cameras Single frame + Multi-view + Mobile Mapping systems Hand-held / backpack system RGB-D sensors Traditional linear Airborne Laser Scanning SPL/Geiger-mode Airborne Laser Scanning TOF laser scanner (long-range) Triangulation laser scanners (short-range) Structured light systems (shortrange) 11 POINT CLOUD ACQUISITION - SENSORS AERIAL TERRESTRIAL PASSIVE HYBRID ACTIVE Single frame cameras Multi-view cameras (oblique) DSLR cameras Panoramic cameras Single frame + Multi-view + Mobile Mapping systems Hand-held / backpack system RGB-D sensors Traditional linear Airborne Laser Scanning SPL/Geiger-mode Airborne Laser Scanning TOF laser scanner (long-range) Triangulation laser scanners (short-range) Structured light systems (shortrange) Midas Octoblique 12 Vexcel Ultracam Osprey 6
POINT CLOUD ACQUISITION - SENSORS PASSIVE HYBRID ACTIVE AERIAL Single frame cameras Multi-view cameras (oblique) Single frame + Multi-view + Traditional linear Airborne Laser Scanning SPL/Geiger-mode Airborne Laser Scanning TERRESTRIAL DSLR cameras Panoramic cameras Mobile Mapping systems Hand-held / backpack system RGB-D sensors TOF laser scanner (long-range) Triangulation laser scanners (short-range) Structured light systems (shortrange) Nadir camera Leica CityMapper 13 Oblique cameras Leica CountryMapper POINT CLOUD ACQUISITION - SENSORS PASSIVE HYBRID ACTIVE AERIAL Single frame cameras Multi-view cameras (oblique) Single frame + Multi-view + Traditional linear Airborne Laser Scanning SPL/Geiger-mode Airborne Laser Scanning TERRESTRIAL DSLR cameras Panoramic cameras Mobile Mapping systems Hand-held / backpack system RGB-D sensors TOF laser scanner (long-range) Triangulation laser scanners (short-range) Structured light systems (shortrange) Leica BackPack Pegasus Trimble Timms Siteco Road-Scanner Riegl VMX-2HA 14 7
POINT CLOUD ACQUISITION - SENSORS AERIAL TERRESTRIAL PASSIVE HYBRID ACTIVE Single frame cameras Multi-view cameras (oblique) DSLR cameras Panoramic cameras Single frame + Multi-view + Mobile Mapping systems Hand-held / backpack system RGB-D sensors Traditional linear Airborne Laser Scanning SPL/Geiger-mode Airborne Laser Scanning TOF laser scanner (long-range) Triangulation laser scanners (short-range) Structured light systems (shortrange) Laser output split in n x m array of laser beamlets (SigmaSpace / Leica: 10 x 10; Harris: 32 x 128) from 10 to 100 pts/sqm, up to 6 mil. pts/sec 5 times more effective than traditional Range of operation: 2000-4500 m AGL @ 200 knot speed DENSE IMAGE MATCHING (DIM) POINT CLOUD ACQUISITION IMAGE ALGORITHMS 1950 s: Analogue image matching and stereoplotter 1980 s: Least squares matching & Multi-photo matching 1990 s: Digital stereo processing systems http://www.mtzgeo.com/history.cfm Today: From feature matching to dense stereo 1960 s: First digital cross-correlation 2000 s: Close range photogrammetry, convergent images, MVS, SGM 8
POINT CLOUD ACQUISITION IMAGE ALGORITHMS Image rectification / epipolar Select a matching criteria (what and how do I match?) Use some assumptions (e.g. constant depth/disparity, continuity constraint, etc.) Apply local / global algorithms (aggregation) - iterative updating Apply optimization algorithms - energy (cost) formulation, Markov Random Fields, graph algorithms, least squares, etc. Consider multi-view stereo (MVS) Efficiency thru dynamic programming, GPU and FPGA implementations [Remondino, F., Spera, M.G., Nocerino, E., Menna, F., Nex, F., 2014: State of the art in high density image matching. The Photogrammetric Record, Vol. 29(146), pp. 144-166] 17 GEOMETRY and ATTRIBUTES not only geometry data (3D coordinates), but also attributes, e.g. PHOTOGRAMMETRY RGB reconstruction uncentainty redundancy intersection angles classes (post-processing) normals (post-processing) intensity returns time strip classes (post-processing) normals 18 9
GEOMETRY and ATTRIBUTES Photogrammetry attributes from bundle adjustment and dense image matching redundancy of 3D points (3-65) precision of 3D points (mean 0.06 mm) intersection angles (20-90 deg) [Menna, F., Nocerino, E., Remondino, F., Dellepiane, M., Callieri, M. and Scopigno, R., 2016: 3D Digitization of an Heritage Masterpiece - a Critical Analysis on Quality Assessment. ISPRS Int. Archives of PRS&SIS, Vol. XLI-B5, pp. 675-683] POINT CLOUD STRUCTURING Normally point clouds are unstructured 3D data (with few exceptions) Structuring can be seen as generation of an organized dataset (e.g. 2.5D grid DSM) generation of structured 3D data (e.g. mesh model, 3D building models, etc.) Bergamo (Italy) - AVT flight with Vexcel Osprey and dense point cloud @ 10 cm resolution Bergamo (Italy) - AVT flight with Vexcel Osprey and 3D semantic modeling of building based on primitive fitting [Toschi, I., Ramos, M.M., Nocerino, E., Menna, F., Remondino, F., Moe, K., Poli, D., Legat, K., Fassi, F., 2017: Oblique photogrammetry supporting 3D urban reconstruction of complex scenarios. ISPRS Int. Archives of PRS&SISI, Vol. XLII-1- W1, pp. 519-526] 10
POINT CLOUD STRUCTURING Normally point clouds are unstructured 3D data (with few exceptions) Structuring can be seen as generation of an organized dataset (e.g. 2.5D grid DSM) generation of structured 3D data (e.g. mesh model, 3D building models, etc.) Huge and unique mesh/polygonal model vs 3D modeling of each single building (LOD2) with parametric shapes fitted on the DSM Trento (Italy) - AVT flight with Vexcel Osprey @ 10 cm GSD POINT CLOUD STRUCTURING Normally point clouds are unstructured 3D data (with few exceptions) Structuring can be seen as generation of an organized dataset (e.g. 2.5D grid DSM) generation of structured 3D data (e.g. mesh model, 3D building models, etc.) generation of classified / segmented point clouds Dortmund (ISPRS benchmark) 3D dense point cloud and classification results using deep learning (supervised) methods 11
CONCLUSIONS & OPEN ISSUES Technology is super rapidly democratizing and point clouds are nowadays everywhere for many applications and at disposal of many end-users Point clouds are the real surveying product and have more added value than derived products as they keep details and they are not interpolated From a business point of view, probably, the money maker is not anymore the data acquisition part but the added value we can give to point clouds for domain-specific applications How do we enrich point clouds with attributes useful to end-users? Which are the attribute that should be (always) linked to geometry? How do we reliably and efficiently extract semantic information to support domain-experts? How do we store, visualize and transmit enriched point cloud data to non-specialist users? How do we provide analytics for multi-temporal point clouds? 12