Multi-modal, multi-temporal data analysis for urban remote sensing

Transcription

1 Multi-modal, multi-temporal data analysis for urban remote sensing Xiaoxiang Zhu

2 Uncontrolled Growth in Mumbai Slums Leads to Massive Fires and Floods

3 So2Sat: Big Data for 4D Global Urban Mapping 10^16 Bytes from Social Media to EO Satellites GUF: 2D binary map urban vs. non-urban So2Sat: 3D/4D urban models infrastructure type classification high resolution population density map Europe

4 Big Data Era of Earth Observation TerraSAR-X EnMap TanDEM-X Sentinel-2

5 Hyperspectral Sensor Radar Sensor Metal Roof Xiaoxiang Social Media Images Text Messages I m in the rooftop bar on 10th floor. Last day in Rio

6 Deep Learning in Remote Sensing Classical Neural Net mid 1980s Deep Neural Net (Google) since 2006/2012

7 What makes Deep Learning in Remote Sensing special? Retrieval of physical or bio-chemical quantities: High accuracy requirements Expert knowledge necessary using existing (traditional) models Multi-modal data: SAR, multi-/super-/hyperspectral, GIS Data and information fusion of complementary sources, even social media Data can be 5-dimensional (x-y-z-t-λ): Novel Deep Learning strategies for image time-series, high dimensional spectral images and complex valued data Very large and ever growing data volumes in RS Copernicus program with global applications

8 Spatiotemporal Scene Interpretation of Space Videos Data Workflow Results (Winner of IEEE GRSS Data Fusion Contest 2016): Spatiotemporal Scene Interpretation of Space Videos via Deep Neural Network and Tracklet Analysis, L. Mou, X. Zhu

9 Space Videos main challenges: - poor spatial resolution - parallax

10 Workflow Spatiotemporal Scene Interpretation of Space Videos via Deep Neural Network and Tracklet Analysis, L. Mou, X. Zhu

11 Spatial Analysis: Scene Labeling Map Spatiotemporal Scene Interpretation of Space Videos via Deep Neural Network and Tracklet Analysis, L. Mou, X. Zhu

12 Temporal Analysis: Activity Analysis Map Spatiotemporal Scene Interpretation of Space Videos via Deep Neural Network and Tracklet Analysis, L. Mou, X. Zhu

13 1 Temporal Analysis: Traffic Density Estimation 0 Spatiotemporal Scene Interpretation of Space Videos via Deep Neural Network and Tracklet Analysis, L. Mou, X. Zhu

14 Building Detection and Height Estimation from Motion

15 Building Instance Classification Using Street View Images Building instance classification based on façade information of street view images, J. Kang, M. Körner, Y. Wang, H. Taubenböck, X. Zhu

16 Chart 16

17 Building instances One area in Vancouver

18 The Whole City of Calgary Building instance classification based on façade information of street view images, J. Kang, M. Körner, Y. Wang, H. Taubenböck, X. Zhu

19 Unsupervised Spectral-Spatial Feature Learning via Deep Residual Conv-Deconv Network Network Architecture: Applications: I. Classification II. Free Object Detection Some neurons in the network own good description power for semantic visual patterns in the object level. Example: neurons # 52 and # 03 to precisely capture metal sheets and vegetative covers, respectively. Unsupervised Spectral-Spatial Feature Learning via Deep Residual Conv-Deconv Network, L. Mou, P. Ghamisi, X. Zhu

20 Matching of SAR and Optical Patches

21 CNN for the Identification of Corresponding Patches in SAR and Optical Imagery of Urban Scenes Network Architecture: A network with two separate, yet identical convolutional streams, which process the SAR patch and the optical patch in parallel, and resulting information fusion at a later decision stage. Results Overall accuracy of more than 93% with a false alarm rate of 0% A CNN for the Identification of Corresponding Patches in SAR and Optical Imagery of Urban Scenes, L. Mou, M. Schmitt, Y. Wang, X. Zhu

22 Urban Scene Classification Fusing PolSAR and HSI Data Network Architecture: FusioNet: A Two-Stream Convolutional Neural Network for Urban Scene Classification using PolSAR and Hyperspectral Data, J. Hu, L. Mou, X. Zhu

23 Optical image FusioNet FusionSVM PolSAR image HSINet HSISVM

24 Artificial Optical Image Generation from SAR Images using Adversarial Neural Nets SAR image Fake Optical Optical image new building and roads On the Possibility of Conditional Adversarial Networks for SAR Template Generation, N. Merkle, P. Fischer, S. Auer, R. Müller

25 Monitoring Annual Urban Dynamics on a Large Scale Data: Landsat images over 30 years from

26 A Deep Information Based Transfer Learning Framework H. Lyu et al., A deep information based transfer learning method to annual mapping of four cities in different continents from with Landsat data, RSE, (in revision)

27 Annual Urban Dynamics Overall Accuracy: 95.4% Beijing New York Melbourne Munich H. Lyu et al., A deep information based transfer learning method to annual mapping of four cities in different continents from with Landsat data, RSE, (in revision)

28 Example Urban Growth of Beijing Landsat Data Acquired over Beijing during Urban Expansion in Beijing in the Past 30 Years H. Lyu et al., A deep information based transfer learning method to annual mapping of four cities in different continents from with Landsat data, RSE, (in revision)

29 Example Munich Airport H. Lyu et al., A deep information based transfer learning method to annual mapping of four cities in different continents from with Landsat data, RSE, (in revision)

30 Open Issues - What are the further applications, other than classification and detection related tasks, in remote sensing which can potentially benefit from deep learning? - Is the transferability of deep nets sufficient to extract geo-information on a global scale? Complex light scattering mechanisms in natural objects, various atmospheric scattering conditions, intra-class variability, culture-dependent features and limited training samples - How to tackle problems raised by very limited annotated data in remote sensing? Is possible to learn deep hierarchical models for remote sensing image understanding in a weakly supervised, semi-supervised or even unsupervised way? - How to benchmark the fast-growing deep-learning algorithms in remote sensing applications? - Fusion of physics-based modeling and deep neural network is a promising direction.