3D Object Recognition and Scene Understanding from RGB-D Videos. Yu Xiang Postdoctoral Researcher University of Washington

Transcription

1 3D Object Recognition and Scene Understanding from RGB-D Videos Yu Xiang Postdoctoral Researcher University of Washington 1

2 2

3 Act in the 3D World Sensing & Understanding Acting Intelligent System 3D World 3

4 Understand the 3D World Navigation Manipulation Geometry Free space Surface Semantics Objects Affordances 4

5 Recognize Objects in 3D Semantics 3D Location 3D Orientation 5

6 Lula Robotics 6

7 6D Object Pose Estimation camera coordinate object coordinate 7

8 Related Work: Feature-based methods Texture-less objects Symmetry objects Occlusion Lowe, ICCV 99 Rothganger et al., IJCV 06 Savarese & Fei-Fei, ICCV 07 Collet et al., IJRR 11 8

9 Related Work: Template-based methods Texture-less objects Symmetry objects Occlusion Gu & Ren, ECCV 10 Hinterstoisser et al., ACCV 12 Xiang & Savarese, CVPR 12 Cao et al., ICRA 16 9

10 PoseCNN An input image Texture-less objects Symmetry objects Occlusion 6D poses Xiang et al. under review 10

11 Decouple 3D Translation and 3D Rotation 3D Translation 2D center camera coordinate Distance object coordinate 3D Rotation 11

12 PoseCNN: Semantic Labeling Feature Extraction Embedding Classification #classes Labels Convolution + ReLU Max Pooling Deconvolution Addition 12

13 PoseCNN: 3D Translation Estimation Feature Extraction Embedding Classification / Regression #classes Labels Center direction X #classes Center direction Y Center distance Convolution + ReLU Max Pooling Deconvolution Addition Hough Voting 13

14 PoseCNN: Center Estimation with RANSAC Center hypothesis 14

15 PoseCNN: 3D Rotation Regression Feature Extraction Embedding Classification / Regression #class Labels Convolution + ReLU Max Pooling Deconvolution Addition Hough Voting RoI Pooling Fully Connected #class Center direction X RoIs Center direction Y 512 Center distance For each RoI #class 6D Poses 15

16 The YCB-Video Dataset 21 YCB Objects 92 Videos, 133,827 frames16

17 6D Pose Evaluation Metric Ground truth pose Average distance (non-symmetry) Average distance (symmetry) 3D model points Predicted pose 17

18 Results on the YCB-Video Dataset 18

19 Input Image Labeling Centers 6D Pose 19

20 Network Network + ICP Network + ICP + Multi-view 20

21 Semantic Mapping (Semantic SLAM) Geometry Semantics Camera poses 21

22 22

23 Semantic Mapping with Data Associated Recurrent Neural Networks (DA-RNNs) DA-RNN Xiang & Fox. RSS 17 23

24 Related Work: 3D Scene Reconstruction KinectFusion Geometry Data Association Semantics Newcombe et al., ISMAR 11 Henry et al., IJRR 12, 3DV 13 Whelan et al., RSS Workshop 12, RSS 15 Keller et al., 3DV 13 24

25 Related Work: Semantic Labeling Geometry Data Association Semantics Long et al., CVPR 12 Zheng et al., ICCV 15 Chen et al., ICLR 15 Badrinarayanan et al., CVPR 15 25

26 Related Work: Semantic Mapping SemanticFusion Geometry Data Association Semantics Salas-Moreno et al., CVPR 13 McCormac et al., ICRA 17 Bowman et al. ICRA 17 26

27 Our Contribution: DA-RNN Recurrent Neural Network RGB Images Data Association Semantic Labels Depth Images KinectFusion 3D Semantic Scene 27

28 Single Frame Labeling with FCNs Feature Extraction Embedding Classification Convolution + ReLU RGB Image Depth Image #classes Labels Max Pooling Concatenation Deconvolution Addition 28

29 Results on RGB-D Scene Dataset [1] [1] K. Lai, L. Bo and D. Fox. Unsupervised feature learning for 3D scene labeling. In ICRA 14 29

30 Video Semantic Labeling with DA-RNNs RGB Image Time t Depth Image RGB Image Time t+1 Labels data association Labels Convolution + ReLU Max Pooling Concatenation Deconvolution Addition Recurrent Units Depth Image 30

31 Data Association from KinectFusion Depth is available Assumption: static scenes Iterative Closest Point (ICP) 31

32 Data Associated Recurrent Units (DA-RUs) Data Association Hidden state h t i DA-RU i h t+1 Classification Time t Time t+1 Recurrent layer Input i x t+1 Weighted Moving Averaging with learnable parameters 32

33 Results on RGB-D Scene Dataset [1] FCN DA-RNN [1] K. Lai, L. Bo and D. Fox. Unsupervised feature learning for 3D scene labeling. In ICRA

34 Experiments: Datasets RGB-D Scene Dataset [1] 14 RGB-D videos of indoor scenes 9 object classes ShapeNet Scene Dataset [2] 100 RGB-D videos of virtual table-top scenes 7 object classes [1] K. Lai, L. Bo and D. Fox. Unsupervised feature learning for 3D scene labeling. In ICRA 14. [2] Chang et al., ShapeNet: an information-rich 3D model repository. arxiv preprint arxiv: ,

35 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Background 94.3 Bowl 78.6 Cap 61.2 Cereal Box 80.4 Coffee Mug 62.7 Coffee Table 93.6 Office Chair 67.3 Soda Can 73.5 Sofa 90.8 Table 84.2 MEAN 78.7 Metric: segmentation intersection over union (IoU) 35 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

36 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) 36 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

37 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Our GRU-RNN Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) 37 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

38 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Our GRU-RNN Our DA-RNN Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) 38 [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR 15.

39 Experiments: Comparison on Network Architectures RGB-D Scenes Methods FCN [1] Our FCN Our GRU-RNN Our DA-RNN No Data Association Background Bowl Cap Cereal Box Coffee Mug Coffee Table Office Chair Soda Can Sofa Table MEAN Metric: segmentation intersection over union (IoU) [1] J. Long, E. Shelhamer and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR

40 RGB Image Our FCN Our DA-RNN 40

41 Experiments: Analysis on Network Inputs 41

42 RGB Images Depth Images Semantic Mapping 42

43 Conclusion & Future Work 6D Object Pose Estimation and Semantic Mapping from RGB-D videos Deep neural networks with geometric representations Perception Planning Control 43

44 Acknowledgements Thank you! 44