Dense pointclouds fom combined nadi and oblique imagey by object-based semi-global multi-image matching Y X Thomas Luhmann, Folkma Bethmann & Heidi Hastedt Jade Univesity of Applied Sciences, Oldenbug, Gemany and Geoinfomatics Photogammetic Week Septembe 11-15, 2017 Univesity of Stuttgat 1
Content Intoduction, motivation Semi-global matching in object space Matching of nadi aeial images Combined matching of nadi and oblique images Summay and outlook 2
Intoduction Semi-Global Matching (SGM) [Hischmülle 2005, 2008] Common method fo dense steeo matching (1 XY point pe pixel) Usable fo vey diffeent kind of applications (eal-time applications, close-ange econstuction tasks, aeial image matching) Often used within multi-view steeo (MVS) appoaches fo complex 3D econstuction tasks P ' " P' y' y" k' px'=d k" x' P" x" Epipola steeo pai with x-paallaxe (dispaity) D O' b O" 3
Intoduction SGM: Minimization of a global enegy function E costs fo dispaity D matching costs tems fo adding the penalties P 1 and P 2 ( D) C( x', y', D) P1 [ D Dq 1] P2 x', y' qn qn x' y ' x' y ' [ D D q 1] Majo idea of SGM: Cost aggegation only in the diection of 1-dimensional paths ( semi-global solution, appoximation) 5 3 7 1 p' 2 k" D p p' 7 Dispaity space Path diections fo cost aggegation (e.g. fo 8 paths) 8 1 l 12 E(p,D) 4 6 Recusive computation sepaately fo each path with: k" L ( p', D) C( p', D) min[ L min L k ( p', D), L ( p', k) ( p', D 1) P, L 1 ( p', D 1) P,min L 1 i ( p', i) P 2 )] 4
Intoduction Advantages of SGM Robust esults in non- and weak-textued aeas (due to penalization of local dispaity changes) Good modelling of depth discontinuities in aeas with shap object boundaies (low smoothing) High esolution, detailed econstuction of even fine stuctues (dense matching) Good pefomance in pocessing speed Advanced softwae solutions: SURE, OpenCV and othes 5
Intoduction Limitations of SGM: Matching always in steeo images, no simultaneous multi-image matching commonly ectified images ae used fo matching in MVS appoaches evey image has to be esampled moe than one time, e.g. with thee images: Rectified image pai 2 Image 3 Rectified image pai 3 Example: Image bundle with15 images Numbe of all possible image pais: n (n-1) / 2 = 15 (15-1) / 2 = 105 image pais Resampling of n (n-1) = 210 images (wost case) Image 1 Image 2 Rectified image pai 1 Motivation Extension of SGM fo multi-image matching Convesion to object space 6
Semi-Global Matching in Object Space Ou pevious wok on object-based matching Facet Steeo Vision (FAST vision) by Wobel (1986), Weisensee (1991) and othes New implementation and applications by Wendt et al. (2004) Fist implementation of OSGM by Bethmann & Luhmann (2015) image intensity multi-image matching of ca mio foging template object intensity suface Y geometic model intensity model X 7
Semi-Global Matching in Object Space Modification Modified enegy of the function: enegy function E, tansfe fom image to object space: E ( D) C( x', y', D) P1 [ D Dq 1] P2 [ D Dq 1] Steeo SGM x', y' qn q p N p E ( ) C( X, Y, ) P1 [ q ] P2 X, Y qn qn X, Y X, Y [ q ] SGM in object space 8
Semi-Global Matching in Object Space Modified enegy function: E Pocessing steps 1. Sub-division of the object space (discetization) ( ) C( X, Y, ) P1 [ q ] P2 X, Y qn qn X, Y X, Y [ q ] Spatial esolution (ΔX, ΔY, Δ) is defined in object space (adapted to the GSD and spatial configuation of the cameas) ΔX Δ Y X ΔY 9
Semi-Global Matching in Object Space Modified enegy function: E ( ) C( X, Y, ) P1 [ q ] P2 X, Y qn qn X, Y Pocessing steps 1. Sub-division of the object space (discetization) 2. Calculation of matching costs fo each point / voxel X, Y [ q ] Image 1 Image 2 Image n No need fo image ectification Matching within pais, tiples and so on eal multi-image matching possible Y X 10
11 Y X paths Penalties effectuate smoothing with espect to a defined spatial axis (hee: -axis) Δ Pocessing steps 1. Sub-division of the object space (discetization) 2. Calculation of matching costs fo each point / voxel 3. Aggegation of matching costs (in object space instead of dispaity space) Modified enegy function: Y X Y X N q q q Y X N q P P Y X C E,, ] [ ] [ ),, ( ) ( 2, 1 Semi-Global Matching in Object Space ), ( min )] ), (,min ), (, ), ( ),, ( min[ ), ( ), ( 2 1 1 k v L P i v L P v L P v L v L v C v L k i
Semi-Global Matching in Object Space SEMI-GLOBAL MATCHING IN OBJECT SPACE Modified enegy function: E ( ) C( X, Y, ) P1 [ q ] P2 X, Y qn qn X, Y X, Y [ q ] Pocessing steps 1. Sub-division of the object space (discetization) 2. Calculation of matching costs fo each point / voxel 3. Aggegation of matching costs (in object space instead of dispaity space) 4. Summing up of path-wise aggegated matching costs, seach of minimum in S S ( X, Y, ) L ( X, Y, ) ( X, Y ) ag min S( X, Y, ) Result is a 2.5D point cloud instead of a dispaity map 12
Semi-Global Matching in Object Space Results fo close-ange applications: Bethmann & Luhmann 2015 Object: clay sculptue, 110mm x 70mm x 90mm Camea: Nikon D2x + 24mm Nikko lens, GSD=0.1mm 14
Semi-Global Matching in Object Space Results fo close-ange applications: Bethmann & Luhmann 2015 Object: clay sculptue, 110mm x 70mm x 90mm Camea: Nikon D2x + 24mm Nikko lens, GSD=0.1mm Image bundle with 38 images Refeence data, captued with finge pojection system (accuacy 20-50µm) Matching Resolution in object space ΔX = ΔY = Δ = 0.3mm (ca. 3x GSD) Diffeent efeence planes fo matching Median filte fo outlie emoval 110.000 object points 15
Semi-Global Matching in Object Space Results fo close-ange applications: Bethmann & Luhmann 2015 Object: clay sculptue, 110mm x 70mm x 90mm Camea: Nikon D2x + 24mm Nikko lens, GSD=0.1mm Image bundle with 38 images Refeence data, captued with finge pojection system (accuacy 20-50µm) Matching Resolution in object space ΔX = ΔY = Δ = 0.3mm (adapted to GSD) Diffeent efeence planes fo matching Median filte fo outlie emoval 110.000 object points 16
Semi-Global Matching in Object Space Results fo close-ange applications: Compaison to TIN deived fom finge pojection measuement [mm] Mean 3D deviations +0.098mm (pos) and -0.129mm (neg) Standad deviation: ±0.165mm 17
Matching of nadi aeial images Aeial images (EuoSDR benchmak Munich): Pat of EuoSDR dataset, inne city of Munich Set of 15 aeial images, 16 Bit PAN Camea: DMC II 230, 15552 x 14144 Pixel, c=91mm GSD 10cm 80% ovelap in flight and coss flight diection Uban aea, flat topogaphy but high buildings (up to 50m) 0.5km 0.25km 18
Matching of nadi aeial images Aeial images (EuoSDR benchmak Munich): Voxel esolution in object space ΔX= ΔY= Δ=10cm (adapted to GSD) Unfilteed point cloud (12 million points) TIN deived fom unfilteed point cloud 19
Matching of nadi aeial images Aeial images (EuoSDR benchmak Munich): Image section TIN deived fom unfilteed point cloud 20
Matching of nadi aeial images Aeial images (EuoSDR benchmak Munich): Voxel esolution in object space ΔX= ΔY= Δ=10cm (adapted to GSD) Image section TIN deived fom unfilteed point cloud 21
Matching of nadi aeial images Aeial images (EuoSDR benchmak Munich): Compaison to median DSM of the benchmak: Red : deviations < 0.3m Blue : deviations > -0.3m Range Numbe Points % all 11790368 100-0.1 to 0.1 8864297 75-0.2 bis 0.2 9864876 84-0.4 to 0.4 10829079 92 22
Matching of nadi aeial images Aeial images (EuoSDR benchmak Munich): Tue othoimage, based on DSM fom matching othoimage coesponding pespective view 23
Combined matching of nadi and oblique images Benefits of combined nadi and oblique image pocessing Closing gaps fom occlusions Impoved textue mapping fo vetical facades Convegent imaging angles fo bette intesection of ays Multi-image appoach fo highe accuacy Monoplotting fo simple height measuements ISPRS www.getmapping.com 24
Combined matching of nadi and oblique images 3D scenaios, oblique images X n2 I 3 I 4 I 5 I 6 Definition efeence images (hee e.g. in I 2, I 4 and I 6 ) Tansfomation of all othe exteio oientations into the coodinate system of each efeence image I 1 X n1 I 2 n1 n2 X n3 n3 I 7 Multi-image matching in each system Y w object Back tansfomation of esulting point clouds into wold coodinate system (X w, Y w, w ) X w Object point, matched in tempoay coodinate system I 2 Object point, matched in tempoay coodinate system I 4 Object point, matched in tempoay coodinate system I 6 25
Combined matching of nadi and oblique images Aeial and oblique images (benchmak dataset eche ollen): povided by ISPRS Scientific Initiative "Multi-platfom Vey High Resolution Photogammety" test aea: museum eche ollen industial buildings of diffeent complexity vey challenging fo matching algoithms due to fine object stuctues and occlusions GSD of nadi images: 10cm GSD of oblique images: 8cm 12cm Ovelap nadi: 75% (along tack) and 80% (acoss tack) Ovelap oblique: 80% (along tack) and 80% (acoss tack) 85 images (nadi and oblique) Gound "tuth" by ALS and TLS data PentaCam (IGI) 26
Combined matching of nadi and oblique images Aeial and oblique images (benchmak dataset eche ollen): unfilteed point clouds Matching with nadi images only Combined matching with nadi and oblique images unfilteed point clouds 27
Combined matching of nadi and oblique images Aeial and oblique images (benchmak dataset eche ollen) Matching with nadi images only Combined matching with nadi and oblique images unfilteed point clouds 28
Combined matching of nadi and oblique images Aeial and oblique images: Matching with nadi images only Combined matching with nadi and oblique images unfilteed point clouds 29
Combined matching of nadi and oblique images Compaison with diffeent softwae packages Facade Hedge Roof Road 30
Combined matching of nadi and oblique images Compaison SURE vs. OSGM Subset of 12 images 31
Combined matching of nadi and oblique images Compaison SURE vs. OSGM oof asphalt 32
Combined matching of nadi and oblique images Compaison SURE vs. OSGM gassland oad 33
Combined matching of nadi and oblique images Compaison SURE vs. OSGM without UAV imagey 34
Summay and outlook Advantages of the poposed appoach + instead of paiwise matching, multiple images can be used simultaneously + benefits of SGM ae maintained + (tue-)othoimage is geneated as a by-poduct of the matching pocedue + image ectification is not necessay any moe + fist esults on diffeent datasets ae vey pomising (without any point cloud filteing in postpocessing) Outlook + combined pocessing of aeial and UAV images + integation of existing 3D object data into matching pocedue + adaptive voxel esolution + optimized implementation 35
Thank you fo you attention! Acknowledgements: The eseach has been suppoted by the Lowe Saxony pogam fo Reseach Pofessos, 2013-2018. 36