CURRENT DEVELOPMENTS IN AIRBORNE OBLIQUE IMAGING SYSTEMS AND AUTOMATED INTERPRETATION OF COMPLEX BUILDINGS. Markus Gerke

Transcription

1 CURRENT DEVELOPMENTS IN AIRBORNE OBLIQUE IMAGING SYSTEMS AND AUTOMATED INTERPRETATION OF COMPLEX BUILDINGS Markus Gerke

2 THE FIRST AIRBORNE PHOTOS WERE OBLIQUE First recorded aerial photograph in the US (Boston), by James Wallace Black, 1860, source Wikipedia

3 THE FIRST AIRBORNE PHOTOS WERE OBLIQUE Intuitively the operator shot from a slanted angle Recognition of buildings In the 1930 s: USGS and military systematically captured oblique images First recorded aerial photograph in the US (Boston), by James Wallace Black, 1860, source Wikipedia Fairchild T3-A camera, source Petrie (2009)

4 CONTENT 1. Properties, configurations and systems 2. Applications, products and research 3. Conclusions and Outlook

5 CONTENT 1. Properties, configurations and systems 2. Applications, products and research 3. Conclusions and Outlook

6 PROPERTIES SCENE OBSERVATION Vertical/nadir images: good observation of ground features and roof structures, assume constant scale Oblique images: observation of vertical structures, but occlusion more dominant, varying scale

7 PROPERTIES SCALE (symbols from Höhle (2008)) According to Shufelt (1999): Vertical images: t < 5 Low oblique: 5 < t <30 High oblique: t > 30 If (t+α) > 90 : horizon visible In this presentation: concentration on high oblique images which do not include the horizon

8 PROPERTIES OCCLUSION visible invisible Self- occlusion

9 PROPERTIES OCCLUSION visible invisible Self- occlusion Occlusion by other objects

10 PROPERTIES OCCLUSION Mitigation through multiple view and overlap visible invisible

11 PROPERTIES OCCLUSION - EXAMPLE All images Slagboom en Peeters All images Slagboom en Peeters

12 PROPERTIES OCCLUSION - EXAMPLE All images Slagboom en Peeters All images Slagboom en Peeters

13 PROPERTIES OCCLUSION - EXAMPLE All images Slagboom en Peeters

14 PROPERTIES OCCLUSION - EXAMPLE All images Slagboom en Peeters

15 PROPERTIES OCCLUSION - EXAMPLE All images Slagboom en Peeters

16 PROPERTIES CONFIGURATIONS According to Petrie (2009) we can distinguish between Fan and Maltese Cross configuration Some examples will follow

17 PROPERTIES FAN CONFIGURATION Fan configuration: increase swath width across/along track: more effective area coverage, reconnaissance applications/corridor mapping. Example: IGI Dual CAM Source: Homepage IGI

18 PROPERTIES FAN CONFIGURATION Visionmap A3: two stepping frame cameras: increasing the FOV through sweeping. Source: Homepage Visionmap Source: Pechatnikov et al., 2009

19 PROPERTIES MALTESE CROSS CONFIGURATION Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Source: Petrie, 2009 IGI Penta DigiCAM, Source: Homepage IGI

20 PROPERTIES MALTESE CROSS CONFIGURATION Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies offering such systems, eg: Track Air MIDAS IGI Penta DigiCam Hexagon/Leica 5 camera head, mid format camera RCD30 (multispectral) Leica Oblique 5-camera head, source homepage Leica

21 PROPERTIES MALTESE CROSS CONFIGURATION Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies using such a configuration, eg: Pictometry / BlomOblique Microsoft Bing maps

22 PROPERTIES MALTESE CROSS CONFIGURATION Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies using such a configuration, eg: Pictometry / BlomOblique Google Microsoft Bing maps Google maps

23 PROPERTIES MALTESE CROSS CONFIGURATION Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies using such a configuration, eg: Pictometry / BlomOblique Google Slagboom en Peeters (NL) Microsoft Bing maps Google maps

24 CONTENT 1. Properties, configurations and systems 2. Applications, products and research 3. Conclusions and Outlook

25 COMPLEXITY OF TASK VS. DEGREE OF AUTOMATION Low Complexity of task High 3D city modeling Monoplotting Manual inspection/visualisation Scene Interpretation Supervised classification Dense Matching/Meshing Low Degree of automation High

26 Complexity of task Manual inspection/visualisation Degree of automation

27 MANUAL INSPECTION/VISUALISATION Purpose: manually check images, also compare to existing (cadastre) maps Slagboom en Peeters GIS interface

28 MANUAL INSPECTION/VISUALISATION Purpose: manually check images, also compare to existing (cadastre) maps Pictometry ESRI interface

29 Complexity of task Monoplotting Manual inspection/visualisation Degree of automation

30 MONOPLOTTING Purpose: 2.5D mapping in images, e.g. for topographic maps, or height measurements, e.g. of houses, trees, lamp-posts etc. Tools e.g. Pictometry EFS (Electronic Field Study) or Pictometry online BlomDESKTOP Viewer or BlomWEB Viewer

31 MONOPLOTTING Principle of height measurement in oblique images: If a DTM is known, the X-Y location of T can be computed through ray intersection with the DTM. Fixing the X-Y location and measuring dr in the image leads to an estimate of dh.

32 MONOPLOTTING Tool: EFS Own assessment of EFS (Enschede images): 2D accuracy: RMSE = 0.5m height accuracy: RMSE = 0.6m Source: IFA Sydney Togarepi, 2009 Problem: DTM specs not disclosed, DTM not exchangeable Camera calibration and orientation parameters cannot be modified

33 Complexity of task 3D city modeling Monoplotting Manual inspection/visualisation Degree of automation

34 3D CITY MODELING/CAD-LIKE RECONSTRUCTION Purpose: interactive building modeling Texture mapping (Frueh et al., 2004, Wang et al., 2008) Pre-compiled collections like BlomUrbex 3D Interactive image-based modeling Eagleview

35 3D CITY MODELING/CAD-LIKE RECONSTRUCTION BlomUrbex 3D: Access to 400 city models in different LODs LOD 2, textures from Blom oblique images. Source: Homepage Blom.

36 3D CITY MODELING/CAD-LIKE RECONSTRUCTION Interactive image-based modeling Autodesk Imagemodeler. Image Blom.

37 3D CITY MODELING/CAD-LIKE RECONSTRUCTION Interactive image-based modeling Final model.

38 3D CITY MODELING/CAD-LIKE RECONSTRUCTION Roof/Wall report, e.g. from Eagleview/Pictometry Source:

39 Complexity of task 3D city modeling Monoplotting Dense Matching/Meshing Manual inspection/visualisation Degree of automation

40 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models Open source/research: Semi-Global matching (Hirschmüller 2008), PMVS2 (Furukawa and Ponce, 2010) Meshlab (meshlab.sourceforge.net) Commercial software: EADS Streetfactory (

41 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, SGM result from IFP Stuttgart (Fritsch et al., 2012)

42 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, SGM result from IFP Stuttgart (Fritsch et al., 2012)

43 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, PMVS2 point cloud

44 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, PMVS2 point cloud

45 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, PMVS2 point cloud, meshed (Poisson reconstruction)

46 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models Slagboom en Peeters, Enschede, PMVS2 point cloud, meshed

47 DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION Purpose: visualization of 3D structures, simple modeling, derivation of height models Commercial system: EADS Street Factory Source: EADS

48 Complexity of task 3D city modeling Monoplotting Supervised classification Dense Matching/Meshing Manual inspection/visualisation Degree of automation

49 SUPERVISED (3D) CLASSIFICATION Purpose: semi-automatically detect objects in the scene Especially vertical structures like walls interesting for object recognition Clear advantage over nadir images Example: supervised classification in object space: 3D points are classified instead of image pixels information fusion

50 SUPERVISED (3D) CLASSIFICATION Purpose: semi-automatically detect objects in the scene Disaster mapping (here: after 2010 Haiti earthquake) Facade (health) status important to assess building damage: advantage of oblique images over nadir images Vertical original images Pictometry

51 SUPERVISED (3D) CLASSIFICATION Purpose: semi-automatically detect objects in the scene Disaster mapping (here: after 2010 Haiti earthquake) Facade (health) status important to assess building damage: advantage of oblique images over nadir images Vertical East North original images Pictometry 3D point cloud from image matching (PMVS2)

52 SUPERVISED (3D) CLASSIFICATION Purpose: semi-automatically detect objects in the scene Disaster mapping (here: after 2010 Haiti earthquake) Classifying trees/vegetation, facades, intact roofs, destroyed roofs 3D points with label information Completeness reached in several experiments: Trees/Vegetation: 63% Facades: 82% Roof intact: 80% Roof destroyed: 75% source: Gerke 2011

53 SUPERVISED (3D) CLASSIFICATION Purpose: semi-automatically detect objects in the scene Same approach, but intact scene Enschede, overall accuracy ~80%

54 Complexity of task 3D city modeling Monoplotting Scene Interpretation Supervised classification Dense Matching/Meshing Manual inspection/visualisation Degree of automation

55 SCENE INTERPRETATION Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification.

56 SCENE INTERPRETATION: BUILDING DETECTION Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Step 1: façade detection see Xiao et al., 2012

57 SCENE INTERPRETATION: BUILDING DETECTION Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Step 2: outlining see: Xiao 2012

58 SCENE INTERPRETATION: BUILDING DETECTION Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Results: -detection(building level): correctness 95%, completeness: 70 to 90% -outlining: accuracy of outline (RMSE): 85cm 1.10m Quality of detection and delineation depend largely on image resolution and overlap source: PhD thesis Jing Xiao, unpublished yet

59 SCENE INTERPRETATION: BUILDING VERIFICATION Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building verification: Given a cadaster map the walls are verified using different evidence measures extracted from oblique images Eg: lines projected: if wall exists facade lines match in object space (blue) source: Nyaruhuma et al., 2012

60 SCENE INTERPRETATION: BUILDING VERIFICATION Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building verification: Given a cadastre map the walls are verified using different evidence measures extracted from oblique images Results: -correctness of decisions (building still existing/building demolished): % -potential to go for a per-wall check source: Nyaruhuma et al., 2012

61 CONTENT 1. Properties, configurations and systems 2. Applications, products and research 3. Conclusions and Outlook

62 CONCLUSIONS Oblique airborne photogrammetry in analogue times was too expensive, but with digital image evolution it comes back into focus Oblique airborne photogrammetry is becoming mature shift to mid format cameras and sophisticated integrated sensor orientation Applications in practice so far mainly visualization and (interactive) city modeling, automatic approaches (low level, high level) promising: Images implicitly contain 3 rd dimension; no need for additional data Facades are very important to identify buildings However: large image overlap preferred (mitigation of occlusion effects, multiple views per façade)

63 OUTLOOK What the future might bring (personal view) More airborne surveying companies will acquire cameras (see mid format heads from Leica/IGI) Oblique airborne images will become standard (soon in NL: capturing of entire country by AeroData and Cyclomedia in Spring Boost of usage, mostly for urban applications: map update, 3D modeling Automation: standard software needs to be extended, starting with bundle block adjustment/integrated sensor orientation

64 LITERATURE Fritsch, D., Kremer, J., Grimm, A A Case Study of Dense Image Matching Using Oblique Imagery Towards All-in-one Photogrammetry. GIM International. 26(4): Frueh, C., Sammon, R., Zakhor, A., Automated texture mapping of 3D city models with oblique aerial imagery. Proceedings of the 2nd International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'04), Furukawa, Y., and J. Ponce, Accurate, Dense, and Robust Multi-View Stereopsis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(8): Gerke, M., Supervised classification of multiple view images in object space for seismic damage assessment. Photogrammetric image analysis : ISPRS conference, PIA 2011, Lecture Notes in Computer Science Hirschmüller, H., Stereo Processing by Semi-Global Matching and Mutual Information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2): :

65 LITERATURE Höhle, J., Photogrammetric measurements in oblique aerial images. Photogrammetrie Fernerkundung Geoinformation, 2008(1): Nyaruhuma, A.P., Gerke, M., Vosselman, G., Mtalo. E.G., Verification of 2D building outlines using oblique airborne images. ISPRS Journal of Photogrammetry and Remote Sensing, 71: Petrie, G., Systematic oblique aerial photography using multi frame cameras. Photogrammetric Engineering & Remote Sensing,75(2): Shufelt, J.A., Performance evaluation and analysis of monocular building extraction from aerial imagery. IEEE Transactions on Pattern Analysis and Machine Intelligence 21 (4): Wang, M., Bai, H., Hu, F., Automatic texture acquisition for 3D model using oblique aerial images. The First International Conference on Intelligent Networks and Intelligent Systems (ICINIS '08),

66 LITERATURE Xiao, J., Gerke, M. and Vosselman, G., Building extraction from oblique airborne imagery based on robust facade detection. ISPRS Journal of Photogrammetry and Remote Sensing, 68: Xiao, J., Automatic building outlining from multi-view oblique images. XXII ISPRS Congress : Imaging a Sustainable Future, ISPRS Annals, I-3, 2012 ed. by M. Shortis, N. Paparoditis and C. Mallet,

67 Thank you for your attention. Questions?