OBLIQUE AERIAL IMAGERY A REVIEW FABIO REMONDINO 3D Optical Metrology unit Bruno Kessler Foundation (FKB) Trento, Italy http://3dom.fbk.eu Email: remondino@fbk.eu MARKUS GERKE EOS dept. ITC / Univ. Twente, Enschede, The Netherlands http://www.itc.nl Email: m.gerke@utwente.it Contents: History, concepts, sensors, data processing & applications EuroSDR questionnaire ISPRS / EuroSDR benchmark Conclusions & Outlook A look back..(1) First recorded aerial photo in the US (1860) by J.W. Black and S. King in Boston (USA) was an oblique shot from a balloon In 1906 G. R. Lawrence used between nine and seventeen large kites to lift a huge camera and take some oblique aerial images of San Francisco (USA) after the strong earthquake in the area Kites & balloon were then abandoned in favour of powered flight, which gained importance for military reconnaissance during World War I Source: Wikipedia Source: http://robroy.dyndns.info
A look back..(2) In the 1930 s USGS and militaries systematically captured oblique images During WWII oblique were used for inspection and reconnaissance In the analogue times too expensive technology Almost 10 years ago: Pictometry system Nowadays different systems available A returned technology / system January to May 2014 issues of the GIM Int. magazine USGC-9 Zeiss twin RMK Zeiss KS-153 General classification of oblique digital cameras single swiping camera Oblique aerial imagery and cameras VisionMap
Oblique aerial imagery and cameras General classification of oblique digital cameras single swiping camera 2 cameras IGI Dual Zeiss twin RMK DiMAC Oblique aerial imagery and cameras General classification of oblique digital cameras single swiping camera 2 cameras 3 cameras DLR-3K 3x Jena LMK 3x Fairchild K-17 Bagley 3-lens (1920)
Oblique aerial imagery and cameras General classification of oblique digital cameras single swiping camera 2 cameras 3 cameras 4 cameras NPO KSI Zeiss (1930) Rolleimetric AIC x4 Oblique aerial imagery and cameras General classification of oblique digital cameras single swiping camera 2 cameras 3 cameras 4 cameras 5 cameras Leica RCD30 Fairchild T-3A (1930 s) Microsoft/Vexcel Osprey
Oblique aerial imagery and cameras General classification of oblique digital cameras single swiping camera 2 cameras 3 cameras 4 cameras 5 cameras Multiple cameras NPO KSI Schiempflug (1904) 9-lens Fairchild Oblique aerial imagery and cameras Fan (static or sweeping) vs Maltese Cross configuration - IGI Dual Cam -A3 Visionmap - Track Air MIDAS - IGI Penta DigiCam - Hexagon/Leica RCD30 - Pictometry / BlomOblique - Microsoft/Vexcel Osprey
Maltese Cross configuration Oblique aerial imagery and cameras One vertically pointing camera + 4 oblique cameras pointing to the four cardinal directions Modular (i.e. varying angles) vs fixed Small vs medium vs large format camera sensors RGB + NIR (in the nadir) Wide vs narrow angle lens Leica RCD30 Microsoft/Vexcel VERTICAL: good observation of roofs, constant scale, traditional approach OBLIQUE Pros / Benefits: Visibility of roofs as well as vertical structures (feature extraction, texturing, etc.) Multiple views, including nadir Better interpretation (building footprints, number of floors, etc.) Higher redundancy & reliability 3D vs 2.5D point clouds More detailed 3D city models Possibility to automatically produce true-ortho A lot of hidden potential OBLIQUE Cons: Oblique imagery Pros & Cons More occlusions (mitigated through multiple views and overlap) Varying scale / GSD
Applications Manual inspection Monoplotting / Building height measurements Detailed 3D city modeling Dense Matching Texturing Damage assessment Scene classification and interpretation (trees, facades, roofs, streets, etc.) complexity Source: Pictometry online Source: Blom / EENA Source: FBK Trento Monoplotting / Building height measurements Applications Source: BlomUrbex
Interactive city modeling (Imagemodeler + Blomoblique) Applications Source: Xiao, 2013 Applications Dense matching Source: Gerke, 2013 Source: Fritsch et al., 2012 Source: FBK Trento, 2014
Applications Automatic generation of 3D textured models Source: EADS Streetfactory / Astrium Damage assessment Applications 2008 2009 Source: CGR / FBK
Applications Cadastral needs Number of floors Building footprints Data collection & processing State-of-the-art is 5 images for every acquisition position Large image block size (thousands of images) New issues / problems / challenges for flight planning Problems for traditional photogrammetric approaches: Convergent images Varying image scale / resolution (GCPs measured with different accuracy) Large perspective distortion (SIFT-like operator are less efficient) Long processing time
Simulated flight plan Data collection & processing - Simulations Flight 80/60, Maltese-cross camera with 30deg and 45deg oblique cameras Urban area with 20m height buildings, narrow (5m) and large (10m) streets Façades parallel to the flight direction High tilt angles (e.g. 45 deg) lead to inhomogeneous redundancy of points Smaller tilt angles (e.g. 30 deg) lead to fewer but more uniform redundancies To fully exploit advantages of oblique cameras, the street s width and the building s height play a major role in planning a successful urban survey campaign The taller the city architecture the lower the camera incidence angle should be A right compromise should be found between the camera tilt setting, given focal length, sensor size, overlap and geometry of the surveyed area [Rupnik, E., Nex, F., Toschi, I., Remondino, F., 2015: Aerial multi-camera systems: Accuracy and block triangulation issues. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 101, pp. 233 246] Simulated AT Data collection & processing - Simulations Different camera configuration tested (small, medium, large; 60/40, 80/60, 80/80, 90/90) Run AT and estimated std of objected coordinates The larger the overlap, the better the quality of object coordinates (and larger the redundancy) The higher the oblique angles, the better the quality of objected coordinates sigma [m] High tilt angles and very large overlap lead to very good object coordinates and large redundancy Cost-Benefit analysis [Rupnik, E., Nex, F., Toschi, I., Remondino, F., 2015: Aerial multi-camera systems: Accuracy and block triangulation issues. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 101, pp. 233 246]
Data collection & processing - AT Advanced workflow for AT Rely on GNSS/INS data Create a connectivity map/graph Use constraints like: Overlap, Look direction, Min numb of extracted tie points Exploit large observations redundancy [Rupnik, E., Nex, F., Remondino, F., 2013: Automatic orientation of large blocks of oblique images. ISPRS Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS Hannover 2013] Dense Image Matching Data collection & processing - DIM Use connectivity map Prefer images with same looking direction [Rupnik, E., Nex, F., and Remondino, F., 2014: Oblique multi-camera systems - orientation and dense matching issues. ISPRS Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-3/W1, pp. 107-114]
Data collection & processing - DIM Dense image matching Use connectivity map Prefer images with same looking direction Point cloud filtering to remove blunders in oblique views [Rupnik, E., Nex, F., and Remondino, F., 2014: Oblique multi-camera systems - orientation and dense matching issues. ISPRS Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-3/W1, pp. 107-114] Dense image matching Data collection & processing - DIM Use connectivity map Prefer images with same looking direction Point cloud filtering Merge multiple point clouds (from every looking direction) 1 view 2 views 4 views [Rupnik, E., Nex, F., and Remondino, F., 2014: Oblique multi-camera systems - orientation and dense matching issues. ISPRS Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-3/W1, pp. 107-114]
Dortmund dataset (IGI PentaCam) ISPRS/EuroSDR benchmark GSD 10cm > 1000 images Examples SURE dense point cloud Fabio Remondino - OBLIQUE SURE dense AERIAL point IMAGERY cloud A REVIEW
SURE true-orthophoto
Graz, Osprey camera, 10cm GSD Examples MicMac dense point cloud EuroSDR Questionnaire Questionnaire on the current status of oblique airborne imagery Run throughout 2014 200+ participants About 10 questions, divided in users vs hw/sw producers Publication in the GIM International, Vol28(12), 2014
Best application with oblique imagery? Users: identification / mapping tasks have priority EuroSDR Questionnaire Vendors: Visualization Texturing of city modeling Interpretation What can oblique do better? EuroSDR Questionnaire
EuroSDR Questionnaire GSD and Stereo Overlap 120 100 Q: What is the max allowed GSD at image centre (cm)? A: Mean 19cm, but majority in the 10cm range 80 60 40 20 0 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 50% 45% 40% 35% 30% 25% 20% 15% 10% Q: What is the desired min overlap? A: At least 60% (majority) 5% 0% at least 60% overlap (i.e. stereo) at least 70% at least 80% Multi-platform photogrammetry (initiated by F. Nex & M. Gerke) ISPRS Scientific Initiative + EuroSDR co-founded activity Benchmark http://www2.isprs.org/commissions/comm1/icwg15b/benchmark_main.html Aims: 1) Fully automatic and reliable co-registration of multi platform/perspective imagery 2) Dense image matching within/across platforms terrestrial image blocks UAV (nadir/oblique) conventional airborne (nadir/oblique) [F. Nex, F., Gerke, M., Remondino, F., Przybilla, H.-J., Bäumker, M., Zurhorst, A., 2015: ISPRS benchmark for multiplatform photogrammetry. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. II-3/W4, pp. 135-142. PIA15+HRIGI15 Joint ISPRS conference]
City center (Dortmund, Germany) OBLIQUE SYSTEM IGI PentaCam (80/80) GSD 10cm 1260 images (yellow area) UAV (3 selected buildings) oblique/nadir, GSD 1-2cm (red area) only rotary wing TERRESTRIAL (3 selected buildings) GSD < 1cm (red area) REFERENCE DATA GNSS, total station TLS, ALS Benchmark Area1 Zeche Zollern (Dortmund, Germany) OBLIQUE SYSTEM IGI PentaCam (80/80) GSD 10cm 910 images (yellow area) UAV (4 selected buildings) oblique/nadir, GSD 1-2cm (red area) rotary and wixed wing TERRESTRIAL (4 selected buildings) GSD < 1cm (red area) REFERENCE DATA GNSS, total station TLS, ALS Benchmark Area2
City center (Dortmund, Germany) Preliminary results (Pix4D & BLUH) Benchmark Area1: results Different configurations (overlap, GCP distribution, self calibration, etc.): Nadir-to-oblique pairs have 6times less tie points than nadirto-nadir pairs Only few matches across oblique viewing directions Self-calibration mandatory to reduce RMSE to 1 GSD (or better) In case of GCPs at the borders, always block deformation 80/80, in general, delivers better results than 60/60 Conclusions and outlook Oblique camera systems are definitely back and a promising solution for mapping purposes Different systems available but new systems might come out in the market soon Oblique airborne images will probably become a standard complementary to traditional large format nadir images Oblique will not replace traditional nadir acquisitions (maybe in urban areas?) Many possible applications: map update, 3D city modeling, inspection, interpretation, footprint identification, 3D cadastre, real estate, etc. NMCAs are slowly thinking to adjust their production pipeline to cope with obliques Oblique requires a new approach in the photogrammetric and production pipeline Data acquisition (flight plan) + Data processing (convergent, huge point clouds, etc.) + 3D restitution / Automated interpretation
Conclusions and outlook Additional costs of oblique flights (especially additional flight lines) might be compensated by additional outcomes and benefits: easier (or improved) object identification and readability of geographical information; generation of denser 3D point clouds, also on vertical elements; higher reliability with respect to traditional vertical acquisitions; measurement of heights, lengths and areas of features directly from the single images (improved monoplotting with 3D point cloud in background); quick generation of true-orthophotos; extension of traditional 2D GIS data; etc. Very dense, accurate and detailed point clouds but then? Conclusions and outlook New solutions to derive structured information out of unstructured point clouds Large polygonal models (e.g. GoogleEarth) are not useful if not only for visualization
Conclusions and outlook Some open research issues: scale and radiometric changes; correct and fast identification of homologues points, in particular across viewing directions; processing time; redundancy / overlap exploitation; fusion of point clouds coming from different viewing directions (and with different accuracy & resolution); automation in interpretation, especially for complex architectures etc. Concurrence / Competitor: UAV acquisitions? Conclusions and outlook BlomOblique Large area overview, details at 5-10 cm level on façades & buildings Point cloud (SURE) UAV Small area only, but details at cm level on façades & buildings and access unsafe areas
EuroSDR / ISPRS Workshop Ordnance Survey headquarter in Southampton (UK) Registration fee: free www.eurosdr.net/workshops/eurosdr-workshop-oblique-cameras-and-dense-image-matching THANK YOU! FABIO REMONDINO 3D Optical Metrology unit Bruno Kessler Foundation (FKB) Trento, Italy Email: remondino@fbk.eu MARKUS GERKE EOS dept. ITC / Univ. Twente, Enschede, The Netherlands Email: m.gerke@utwente.it Acknowledgments: Francesco Nex (ITC/Univ. Twente), Karsten Jacobsen (LU Hannover), Konrad Wenzel (nframes), Ewelina Rupnik (IGN France), Isabella Toschi (FBK Trento)