Perceiving the 3D World from Images and Videos. Yu Xiang Postdoctoral Researcher University of Washington

Transcription

1 Perceiving the 3D World from Images and Videos Yu Xiang Postdoctoral Researcher University of Washington 1

2 2

3 Act in the 3D World Sensing & Understanding Acting Intelligent System 3D World 3

4 Understand the 3D World Navigation Manipulation Geometry Free space Surface Semantics Objects Affordances 4

5 Recognize Objects in 3D Semantics 3D Location 3D Orientation 5

6 Lula Robotics 6

7 6D Object Pose Estimation camera coordinate object coordinate 7

8 Related Work: Feature-based methods Texture-less objects Symmetry objects Occlusion Lowe, ICCV 99 Rothganger et al., IJCV 06 Savarese & Fei-Fei, ICCV 07 Collet et al., IJRR 11 8

9 Related Work: Template-based methods Texture-less objects Symmetry objects Occlusion Gu & Ren, ECCV 10 Hinterstoisser et al., ACCV 12 Xiang & Savarese, CVPR 12 Cao et al., ICRA 16 9

10 PoseCNN An input image Texture-less objects Symmetry objects Occlusion 6D poses Y. Xiang, T. Schmidt, V. Narayanan and D. Fox. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. In arxiv:

11 Decouple 3D Translation and 3D Rotation 3D Translation 2D center camera coordinate Distance object coordinate 3D Rotation 11

12 PoseCNN: Semantic Labeling Feature Extraction Embedding Classification #classes Labels Convolution + ReLU Max Pooling Deconvolution Addition 12

13 PoseCNN: Object Center Voting 13

14 PoseCNN: 3D Translation Estimation Feature Extraction Embedding Classification / Regression #classes Labels Center direction X #classes Center direction Y Center distance Convolution + ReLU Max Pooling Deconvolution Addition Hough Voting 14

15 PoseCNN: Center Estimation with RANSAC Center hypothesis 15

16 PoseCNN: 3D Rotation Regression Feature Extraction Embedding Classification / Regression #class Labels Convolution + ReLU Max Pooling Deconvolution Addition Hough Voting RoI Pooling Fully Connected #class Center direction X RoIs Center direction Y 512 Center distance For each RoI #class 6D Poses 16

17 The YCB-Video Dataset 21 YCB Objects 92 Videos, 133,827 frames17

18 6D Pose Evaluation Metric Ground truth pose Average distance (non-symmetry) Average distance (symmetry) 3D model points Predicted pose 18

19 Results on the YCB-Video Dataset 19

20 Input Image Labeling Centers 6D Pose 20

21 Network Network + ICP Network + ICP + Multi-view 21

22 22

23 23

24 24

25 25 Y. Xiang, W. Choi, Y. Lin and S. Savarese. Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection. In WACV, Y. Xiang, W. Choi, Y. Lin and S. Savarese. Data-Driven 3D Voxel Patterns for Object Category Recognition. In CVPR, 2015 (Oral).

26 Online Multi-Object Tracking Y. Xiang, A. Alahi and S. Savarese. Learning to Track: Online Multi-Object Tracking by Decision Making. In ICCV, 2015 (Oral). Code available online 26

27 Semantic Mapping (Semantic SLAM) Geometry Semantics Camera poses 27

28 28

29 Semantic Mapping with Data Associated Recurrent Neural Networks (DA-RNNs) DA-RNN Code available online 29 Y. Xiang and D. Fox. DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks. In RSS, 2017.

30 Related Work: 3D Scene Reconstruction KinectFusion Geometry Data Association Semantics Newcombe et al., ISMAR 11 Henry et al., IJRR 12, 3DV 13 Whelan et al., RSS Workshop 12, RSS 15 Keller et al., 3DV 13 30

31 Related Work: Semantic Labeling Geometry Data Association Semantics Long et al., CVPR 12 Zheng et al., ICCV 15 Chen et al., ICLR 15 Badrinarayanan et al., CVPR 15 31

32 Related Work: Semantic Mapping SemanticFusion Geometry Data Association Semantics Salas-Moreno et al., CVPR 13 McCormac et al., ICRA 17 Bowman et al. ICRA 17 32

33 Our Contribution: DA-RNN Recurrent Neural Network RGB Images Data Association Semantic Labels Depth Images KinectFusion 3D Semantic Scene 33

34 Single Frame Labeling with FCNs Feature Extraction Embedding Classification Convolution + ReLU RGB Image Depth Image #classes Labels Max Pooling Concatenation Deconvolution Addition 34

35 Results on RGB-D Scene Dataset [1] [1] K. Lai, L. Bo and D. Fox. Unsupervised feature learning for 3D scene labeling. In ICRA 14 35

36 Video Semantic Labeling with DA-RNNs RGB Image Time t Depth Image RGB Image Time t+1 Labels data association Labels Convolution + ReLU Max Pooling Concatenation Deconvolution Addition Recurrent Units Depth Image 36

37 Data Association from KinectFusion Depth is available Assumption: static scenes Iterative Closest Point (ICP) 37

38 Data Associated Recurrent Units (DA-RUs) Data Association Hidden state h t i DA-RU i h t+1 Classification Time t Time t+1 Recurrent layer Input i x t+1 Weighted Moving Averaging with learnable parameters 38

39 Results on RGB-D Scene Dataset [1] FCN DA-RNN [1] K. Lai, L. Bo and D. Fox. Unsupervised feature learning for 3D scene labeling. In ICRA

40 Experiments: Datasets RGB-D Scene Dataset [1] 14 RGB-D videos of indoor scenes 9 object classes ShapeNet Scene Dataset [2] 100 RGB-D videos of virtual table-top scenes 7 object classes [1] K. Lai, L. Bo and D. Fox. Unsupervised feature learning for 3D scene labeling. In ICRA 14. [2] Chang et al., ShapeNet: an information-rich 3D model repository. arxiv preprint arxiv: ,

41 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Background 94.3 Bowl 78.6 Cap 61.2 Cereal Box 80.4 Coffee Mug 62.7 Coffee Table 93.6 Office Chair 67.3 Soda Can 73.5 Sofa 90.8 Table 84.2 MEAN 78.7 Metric: segmentation intersection over union (IoU) 41 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

42 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) 42 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

43 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Our GRU-RNN Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) 43 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

44 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Our GRU-RNN Our DA-RNN Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) 44 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

45 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Our GRU-RNN Our DA-RNN No Data Association Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR

46 RGB Image Our FCN Our DA-RNN 46

47 Experiments: Analysis on Network Inputs 47

48 RGB Images Depth Images Semantic Mapping 48

49 Conclusion 6D Object Pose Estimation from images and videos Semantic Mapping from RGB-D videos Deep neural networks with geometric representations 49

50 Future Work: Perception for Robotics 3D object recognition for robot manipulation 3D scene understanding for robot navigation Perception Combing perception, planning and control Planning Control Learning Reinforcement learning with deep neural networks 50

51 Acknowledgements Thank you! 51