Synscapes A photorealistic syntehtic dataset for street scene parsing Jonas Unger Department of Science and Technology Linköpings Universitet.

Transcription

1 Synscapes A photorealistic syntehtic dataset for street scene parsing Jonas Unger Department of Science and Technology Linköpings Universitet 7D Labs VINNOVA

2 Photo-realistic image synthesis Light field imaging Multi-sensor imaging High dynamic range imaging f(i) Appearance capture and modelling Tone mapping and HDR video compression Medical visualization Glasses free 3D displays Sparse representations and compressive sensing for imaging Image synthesis for Machine learning HDR video compression

3 MACHINE LEARNING A Krizhevsky, I Sutskever, GE Hinton: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 2012

4 THE DATA CHALLENGE M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The Cityscapes Dataset for Semantic Urban Scene Understanding," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

5 SYNTHETIC DATA Why synthetic data? There is often a lack of high quality data for training and testing Expensive Manual annotation is difficult and prone to errors Synthetic data enables pixel accurate annotations Simulation of unrealistic non-physical views of data Data augmentation

6 SYNTHETIC DATA What is needed? High quality models and sensor simulation accuracy is highly important Full control over dataset variation

7 SYNTHETIC DATA What is needed? High quality models and sensor simulation accuracy is highly important Full control over dataset variation (long tail distributions)

8 SYNTHETIC DATA What is needed? High quality models and sensor simulation accuracy is highly important Full control over dataset variation Automatic generation is a must (procedural methods) Efficient handling of large image volumes is a must (rendering and storage)

9 REALISTIC SYNTHETIC DATA AT SCALE Experiment scope Experiment design Class selection Highly parallelized cloud computation and image synthesis Automatic procedural world modeling Photorealistic image synthesis with accurate segmentation and labeling

10 REALISTIC SYNTHETIC DATA AT SCALE

11 REALISTIC SYNTHETIC DATA AT SCALE S I M U L A T I O N T R A I N I N G T E S T I N G & V A L I D A T I O N A N N O T A T I O N D E S I G N N E U R A L N E T W O R K A N A L Y S I S

12 IMAGE SYNTHESIS

13 IMAGE SYNTHESIS

14 PROCEDURAL SCENARIO GENERATION

15 PROCEDURAL SCENARIO GENERATION

16 Procedural generation

17 REALISTIC SYNTHETIC DATA AT SCALE

18 REALISTIC SYNTHETIC DATA AT SCALE 2D BOUNDING BOXES

19 REALISTIC SYNTHETIC DATA AT SCALE 3D BOUNDING BOXES

20 REALISTIC SYNTHETIC DATA AT SCALE INSTANCE VISIBILITY

21 LIDAR

22 LIDAR

23 NON-PHYSICAL VIEWS

24

25

26 REALISTIC SYNTHETIC DATA AT SCALE CAN SYNTHETIC DATA REPRESENT THE REAL WORLD? THE TESTING PROBLEM Are my network s prediction on synthetic data reliable and actionable? THE TRAINING PROBLEM Can I train and/or augment a neural network using synthetic data? MAGNUS WRENNINGE AND JONAS UNGER, SYNSCAPES: A PHOTOREALISTIC SYNTHETIC DATASET FOR STREET SCENE PARSING, IN ARXIV E-PRINTS: , OCTOBER, 2018

27 Synthia GTA Our Data Road Building Fence Tr. light Veg. Sky Rider Truck Train Bicycle DEEPLAB_V3+ TRAINED ON CITYSCAPES INFERENCE ON SYNTHETIC DATA

28 Ros et al. (2016): The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 7D Labs REALISTIC SYNTHETIC DATA AT SCALE CITYSCAPES-TRAINED PREDICTING ON SYNTHIA

29 REALISTIC SYNTHETIC DATA AT SCALE CITYSCAPES-TRAINED PREDICTING ON GTA Richter, S. R. G., Vineet, V., Koltun, V. (2016): Playing for Data: Ground Truth from Computer Games. In Proceedings of European Conference on Computer Vision (ECCV), pp , Springer International Publishing

30 REALISTIC SYNTHETIC DATA AT SCALE CITYSCAPES-TRAINED PREDICTING ON OUR DATA

31 REALISTIC SYNTHETIC DATA AT SCALE USING SYNTHETIC DATA FOR ANALYSIS

32 DETECTION: FASTER R-CNN (TENSORFLOW)

33 DETECTION: FASTER R-CNN (TENSORFLOW)

34

35 SUMMARY - SYNTHETIC DATA I M A G E S L I D A R M E T A D A T A Custom camera models PSF modeling LDR & HDR images Class & instance segmentation Depth Custom lidar models Depth Surface reflectivity Class & instance segmentation Ego vehicle motion Sensor extrinsics & intrinsics 2D bounding boxes Oriented 3D bounding boxes Per-object occlusion & truncation

36 SUMMARY - SYNTHETIC DATA REA LISTIC SYN THETIC DATA AT SCA LE EVALUATION/APPLICATIONS TRAINING TESTING ANALYSIS FUTURE WORK SYN SCA PES CHARACTERIZE DOMAIN SHIFT A BEN CHM A RK SYN THETIC DATA SET ANALYZE DATA Realistic image synthesis CONTINUOUSLY process UPDATE TRAINING SET Rich annotations TRAINING and metadata STRATEGIES State-of-the-art results Contact us for access! synscapes@7dlabs.com

37 Thank you! 7D Labs