Local-Level 3D Deep Learning. Andy Zeng

Transcription

1 Local-Level 3D Deep Learning Andy Zeng 1

2 Matching 3D Data Image Credits: Song and Xiao, Tevs et al. 2

3 Matching 3D Data Reconstruction Image Credits: Song and Xiao, Tevs et al. 3

4 Matching 3D Data Reconstruction Image Credits: Song and Xiao, Tevs et al. Shape retrieval Object pose estimation 4

5 Matching 3D Data Reconstruction Image Credits: Song and Xiao, Tevs et al. Shape retrieval Object pose estimation Aligning deformable shapes 5

6 Matching 3D Data Establish 3D geometric correspondences 6

7 Matching 3D Data Establish 3D geometric correspondences Find interesting 3D features Match 3D features 7

8 Matching 3D Features in Scanning Data is Hard Partial and noisy scan data 8

9 Matching 3D Features in Scanning Data is Hard Partial and noisy scan data Viewpoint variance 9

10 Matching 3D Features in Scanning Data is Hard Partial and noisy scan data Viewpoint variance 10

11 Matching 3D Features in Scanning Data is Hard Partial and noisy scan data Viewpoint variance Traditional hand-crafted 3D feature descriptors do not work well! 11

12 Solution: Let the data speak for itself! 12

13 Solution: Let the data speak for itself! 3DMatch: 3D ConvNet that recognizes correspondences in 3D scan data

14 3D Data Representation Use truncated distance fields (TDF) Image Credits: Song and Xiao 14

15 3D Data Representation Use truncated distance fields (TDF) Intermediate 3D Representation Image Credits: Song and Xiao 15

16 3D Data Representation Use truncated distance fields (TDF) Intermediate 3D Representation Image Credits: Song and Xiao Enables 3D Convolution 16

17 3D Data Representation 17

18 3D Data Representation 18

19 Metric Network vs. L2 Distance 19

20 Metric Network vs. L2 Distance L2 contrastive loss 20

21 Metric Network vs. L2 Distance L2 contrastive loss Use metric network for accuracy, use L2 distance for speed 21

22 Generating Training Data Automatically Manually label geometric correspondences? Too much work! 22

23 Generating Training Data Automatically Manually label geometric correspondences? Too much work! Think of all those maps that we've built using large-scale SLAM and all those correspondences that these systems provide isn t that a clear path for building terascale image-image "association" datasets which should be able to help deep learning? The basic idea is that today's SLAM systems are large-scale correspondence engines which can be used to generate largescale datasets, precisely what needs to be fed into a deep ConvNet. Newcombe s Proposal: Use SLAM to help Deep Learning 23 Image Credits: Malisiewicz et al.

24 Generating Training Data Automatically Manually label geometric correspondences? Too much work! The basic idea is that today's SLAM systems are largescale correspondence engines which can be used to generate large-scale datasets, precisely what needs to be fed into a deep ConvNet. Newcombe s Proposal: Use SLAM to help Deep Learning Tomasz Malisiewicz s Computer Vision Blog ICCV s Future of Real-Time SLAM Workshop Solution: Use existing 3D reconstructions to fuel correspondence labels! 24 Image Credits: Malisiewicz et al.

25 Generating Training Data Automatically Solution: Use existing 3D reconstructions to fuel correspondence labels! 25 Image Credits: Shotton et al.

26 26

27 3DMatch for Reconstruction 27

28 3DMatch for Loop Closures 28

29 3DMatch for Loop Closures 29

30 3DMatch for Loop Closures 30

31 3DMatch for Loop Closures 31

32 3DMatch for 3D Reconstruction 32

33 3DMatch for Other Applications Shape retrieval Object pose estimation 33

34 Evaluation: 3DMatch vs. Others Correspondence 34

35 Evaluation: 3DMatch > Others Correspondence Geometric Registration 35

36 Keypoint Selection Does Not Matter 36

37 Keypoint Selection Does Not Matter The choice of keypoints do not matter much. 37

38 Conclusion 38

39 3DMatch 3D ConvNet that recognizes correspondences in 3D scan data 39