DISTRIBUTION STATEMENT A APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED This work was supported by the United States Special Operations Command (USSOCOM). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of USSOCOM or the U.S. Government. This work was supported by the Intelligence Advanced Research Projects Activity (IARPA) via contract no. 2017-17032700004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA or the U.S. Government. Urban 3D Challenge & Future Directions 26 April 2018 Myron Brown Hirsh Goldberg Kevin Foster Andrea Leichtman Gordon Christie Sean Wang Shea Hagstrom Marc Bosch Scott Almes
USSOCOM Urban 3D Challenge (2017) Building footprint classification using satellite images and 3D data RGB Ortho Image Digital Surface Model Example Solution Ground Truth Source Data: Vricon 50cm Ortho, DSM, and DTM Products Acquired for IARPA CORE3D Solutions Assessed Using Building Outlines from Public HSIP 133 Cities Goldberg et al., Urban 3D Challenge: Building Footprint Detection Using Orthorectified Imagery and Digital Surface Models from Commercial Satellites, in Proc. SPIE Geospatial Informatics and Motion Imagery Analytics VIII, 2018 JHU Applied Physics Laboratory 23 April 2018 2
Public Challenge Data Set Jacksonville, FL Tampa, FL Richmond, VA Training and Provisional Testing Sequestered Testing City Area (km²) / Number of Tiles Number of Ground Truthed Buildings Jacksonville, Florida 117 52,675 Tampa, Florida 119 55,158 Richmond, Virginia 125 49,220 Total 361 157,053 JHU Applied Physics Laboratory 23 April 2018 3
Example 1km Tiles in Challenge Data Set Urban / Downtown Residential Commercial / Residential Mix Jacksonville, Florida, USA Tampa, Florida, USA Richmond, Virginia, USA JHU Applied Physics Laboratory 23 April 2018 4
Semi-Automated Ground Truth Production HSIP 133 Cities Building Outlines Instance Level Ground Truth Labels AnnoteGeo Software for Manual Annotation HSIP Data is available on AWS: s3://grid-uscities-data AnnoteGeo is available at: https://github.com/pubgeo/annotegeo JHU Applied Physics Laboratory 23 April 2018 5
Evaluation Metrics Each unique performer building region is assigned to at most one ground truth building region Match is declared if intersection-over-union (IoU) is > pre-defined threshold (0.45) True Positive (TP) Performer regions not matched to a GT region False Positive (FP) Unmatched ground truth building region False Negative (FN) Instance-Level Truth Sample Output Precision P = TP TP + FP Recall(R) = TP TP + FN F 1 = 2 P R P + R True Positive (TP) False Positive (FP) False Negative (FN) Final score is an average of F 1 scores across all tiles being evaluated JHU Applied Physics Laboratory 23 April 2018 6
Urban 3D Challenge on TopCoder Pre-registration: 25 Sep, 2017 Challenge Duration: 9 Oct 4 Dec, 2017 Final Results: 5 Jan, 2018 Participation - 217 registered - 54 actively participated Top 6 offered awards for open source JHU Applied Physics Laboratory 23 April 2018 7
Provisional and Sequestered Leaderboards JHU Applied Physics Laboratory 23 April 2018 8
Urban 3D Challenge Winning Approach Ensemble of Convolutional Neural Networks (CNNs) RGB Original U-Net is Shown for Illustration DSM ndsm DTM ndsm = DSM - DTM Semantic Segmentation Ensemble of CNNs with ResNet-34 Encoder 1 & U-Net Decoder 2 Watershed Algorithm 3 Instance Segmentation 1 He et al., Deep Residual Learning for Image Recognition, CVPR 2016 2 Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI, 2015 3 Credit: http://iacl.ece.jhu.edu/~prince/ws/ JHU Applied Physics Laboratory 23 April 2018 9
Is Urban 3D Challenge a Strong Baseline? Sequestered Test F1 Scores* Richmond Townhouse Retraining Before Retraining After Retraining San Fernando, Argentina Very Different & Not in Training Data * Top provisional score was 0.89 Sequestered testing suggests this solution generalizes well, so we re using it as a baseline for new research and industry solutions JHU Applied Physics Laboratory 23 April 2018 10
Can We Apply to Other Source Data? VRICON RGB True Ortho WorldView-2 Single Image Ortho Jacksonville, FL Tampa, FL Richmond, VA JHU Applied Physics Laboratory 23 April 2018 11
Sequestered Test Results Alternate Source Data for Training and Testing Retrained with Alternate Source Data Challenge Results Tested with Alternate Source Data (A) Vricon RGB and 3D (B) Vricon RGB and Lidar 3D (C) Single Image Ortho RGB and Lidar 3D (D) Retrained and Tested with (C) (E) Trained with (D) and Tested with Unedited Labels JHU Applied Physics Laboratory 23 April 2018 12
More Public Data to Enable Research Learn more at http://www.jhuapl.edu/pubgeo.html WorldView-2 Images to Complement the Urban 3D Challenge WV-3 Images for MVS M. Brown, H. Goldberg, K. Foster, A. Leichtman, S. Wang, S. Hagstrom, M. Bosch, and S. Almes, Large-Scale Public Lidar and Satellite Image Data Set for Urban Semantic Labeling, in Proc. SPIE Laser Radar Technology and Applications XXII, 2018 JHU Applied Physics Laboratory 23 April 2018 13
Public Data Sets and Baseline Algorithms Enabling Semantic 3D Reconstruction Multi-View Stereo 3D Reconstruction Semantic & Instance Segmentation Semantic 3D Reconstruction Notional Illustration Credit: Biljecki et al. IARPA MVS3DM Challenge (2016) USSOCOM Urban 3D Challenge (2017) IARPA CORE3D Public Data (2018) JHU Applied Physics Laboratory 23 April 2018 14