FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs

Transcription

1 FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs W. Nicholas Greene Robust Robotics Group, MIT CSAIL LPM Workshop IROS 2017 September 28, 2017 with Nicholas Roy 1

2 We want Autonomous Vision-Based Navigation 2

3 But Dense Monocular SLAM is Expensive Sparse Methods Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014) Efficiency Density

4 But Dense Monocular SLAM is Expensive Sparse Methods Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014) Efficiency Semi-Dense Methods Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016 Density

5 But Dense Monocular SLAM is Expensive Sparse Methods Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014) Efficiency Semi-Dense Methods Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016 Dense Methods DTAM (Newcombe et al., ICCV 2011) Graber et al. (ICCV 2011) MonoFusion (Pradeep et al., ISMAR 2013) REMODE (Pizzoli et al., ICRA 2014) Density

6 But Dense Monocular SLAM is Expensive

7 But Dense Monocular SLAM is Expensive Sparse Methods Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014)? Efficiency Semi-Dense Methods Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016 Dense Methods DTAM (Newcombe et al., ICCV 2011) Graber et al. (ICCV 2011) MonoFusion (Pradeep et al., ISMAR 2013) REMODE (Pizzoli et al., ICRA 2014) Density Can we more efficiently estimate dense geometry?

8 What Would a Dense Method Do? Image 8

9 Estimate Depth for Every Pixel Depthmap 9

10 Dense Methods Oversample Geometry Depthmap One depth estimate per pixel 100k - 1M pixels per image 10

11 Dense Methods Make Regularization Hard(er) Depthmap One depth estimate per pixel 100k - 1M pixels per image 11

12 Meshes Encode Geometry with Fewer Depths Depthmap 12

13 Meshes Encode Geometry with Fewer Depths Depthmap 13

14 Meshes Encode Geometry with Fewer Depths Depthmap 14

15 Meshes Encode Geometry with Fewer Depths Depthmap One depth estimate per vertex k vertices per mesh 15

16 Meshes Make Regularization Easy(er) Depthmap One depth estimate per vertex k vertices per mesh 16

17 FLaME: Fast Lightweight Mesh Estimation Fast (< 5 ms/frame) Lightweight (< 1 Intel i7 CPU core) Runs Onboard MAV Greene and Roy, ICCV

18 FLaME: Fast Lightweight Mesh Estimation Image and pose Feature Selection Inverse Depth Estimation Output Depthmap interpolation Variational Regularizer (NLTGV 2 -L1) Delaunay Triangulation 18

19 FLaME: Fast Lightweight Mesh Estimation Image and pose Feature Selection Inverse Depth Estimation Output Depthmap interpolation Variational Regularizer (NLTGV 2 -L1) Delaunay Triangulation 19

20 Select Easily Trackable Features - At each frame we sample trackable pixels or features over the image - These features will serve as potential vertices to add to the mesh 20

21 Select Easily Trackable Features - At each frame we sample trackable pixels or features over the image - These features will serve as potential vertices to add to the mesh 21

22 Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh 22

23 Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh 23

24 Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh 24

25 Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh Select pixel in each grid cell with maximum score: 25

26 FLaME: Fast Lightweight Mesh Estimation Image and pose Feature Selection Inverse Depth Estimation Output Depthmap interpolation Variational Regularizer (NLTGV 2 -L1) Delaunay Triangulation 26

27 Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 27

28 Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 28

29 Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 29

30 Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 30

31 Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 31

32 Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons - Each successful match generates an inverse depth measurement 32

33 Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons - Each successful match generates an inverse depth measurement - Measurements are fused over time using Bayesian update 33

34 FLaME: Fast Lightweight Mesh Estimation Image and pose Feature Selection Inverse Depth Estimation Output Depthmap interpolation Variational Regularizer (NLTGV 2 -L1) Delaunay Triangulation 34

35 Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations 35

36 Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations 36

37 Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations 37

38 Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations - We can project 2D mesh into 3D using the inverse depth for each vertex 38

39 Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations - We can project 2D mesh into 3D using the inverse depth for each vertex 39

40 FLaME: Fast Lightweight Mesh Estimation Image and pose Feature Selection Inverse Depth Estimation Output Depthmap interpolation Variational Regularizer (NLTGV 2 -L1) Delaunay Triangulation 40

41 Smooth Mesh Using Variational Regularization - Mesh inverse depths are noisy and prone to outliers - Need to spatially regularize/smooth inverse depths Raw 41

42 Smooth Mesh Using Variational Regularization - Mesh inverse depths are noisy and prone to outliers - Need to spatially regularize/smooth inverse depths - Will exploit graph structure to make optimization fast and incremental Raw Smoothed 42

43 Define Variational Cost over Inverse Depthmap Raw idepthmap 43

44 Define Variational Cost over Inverse Depthmap Smoothed idepthmap Raw idepthmap 44

45 Define Variational Cost over Inverse Depthmap Smoothed idepthmap Raw idepthmap NLTGV 2 stands for Non-Local Total Generalized Variation (Second Order) Ranftl et al. 2014, Pinies et al

46 Approximate Cost over Mesh - Reinterpret mesh as graph 46

47 Approximate Cost over Mesh - Reinterpret mesh as graph 47

48 Approximate Cost over Mesh 48

49 Approximate Cost over Mesh Raw idepthmap 49

50 Approximate Cost over Mesh 50

51 Approximate Cost over Mesh Raw mesh 51

52 Approximate Cost over Mesh Smoothed mesh Raw mesh 52

53 Approximate Cost over Mesh Smoothed mesh Raw mesh 53

54 Approximate Cost over Mesh Math Smoothed mesh Raw mesh 54

55 Approximate Cost over Mesh Math Smoothed mesh Raw mesh 55

56 Optimize using Primal-Dual on Graph 56

57 Optimize using Primal-Dual on Graph Math 57

58 Optimize using Primal-Dual on Graph Math 58

59 Optimize using Primal-Dual on Graph Math 59

60 Optimize using Primal-Dual on Graph Math Edges update using vertices 60

61 Optimize using Primal-Dual on Graph Math Vertices update using edges 61

62 Optimize using Primal-Dual on Graph Math Optimization steps are fast even without GPU (<< frame rate) 62

63 Optimize using Primal-Dual on Graph Math Optimization steps are fast even without GPU (<< frame rate) Optimization convergence is fast (~ frame rate) 63

64 Optimize using Primal-Dual on Graph Math Optimization steps are fast even without GPU (<< frame rate) Optimization convergence is fast (~ frame rate) Mesh can be augmented without restarting optimization 64

65 FLaME: Fast Lightweight Mesh Estimation Image and pose Feature Selection Inverse Depth Estimation Output Depthmap interpolation Variational Regularizer (NLTGV 2 -L1) Delaunay Triangulation 65

66 Benchmark Results Compared FLaME to two existing CPU-only approaches: - LSD-SLAM - MLM 66

67 Benchmark Results Compared FLaME to two existing CPU-only approaches: - LSD-SLAM - MLM Evaluated on two benchmark datasets (ground truth poses): - TUM RGBD SLAM 30 Hz) - EuRoC MAV 20 Hz) 67

68 Benchmark Results Compared FLaME to two existing CPU-only approaches: - LSD-SLAM - MLM Evaluated on two benchmark datasets (ground truth poses): - TUM RGBD SLAM 30 Hz) - EuRoC MAV 20 Hz) Desktop Intel i7 CPU only 68

69 Benchmark Results Lower Error 69

70 Benchmark Results More Dense 70

71 Benchmark Results Less Load 71

72 Benchmark Results Faster 72

73 Benchmark Results Faster 73

74 Benchmark Results 74

75 Flight Experiments DJI F450 frame Intel NUC flight computer 10 ms/frame runtime 1.5 core load Indoor Flight at 2.5 m/s Outdoor Flight at 3.5 m/s 75

76 Flight Experiments 76

77 FLaME Contributions Proposed a lightweight method for dense online monocular depth estimation Paper Code* * Coming soon! W. Nicholas Greene (wng@csail.mit.edu) 77

78 FLaME Contributions Proposed a lightweight method for dense online monocular depth estimation Formulated the reconstruction problem as a non-local variational optimization over a Delaunay graph, which allows for a fast, efficient approach to depth estimation Paper Code* * Coming soon! W. Nicholas Greene (wng@csail.mit.edu) 78

79 FLaME Contributions Proposed a lightweight method for dense online monocular depth estimation Formulated the reconstruction problem as a non-local variational optimization over a Delaunay graph, which allows for a fast, efficient approach to depth estimation Demonstrated improved depth accuracy and density on benchmark datasets using less than 1 Intel i7 CPU core Paper Code* * Coming soon! W. Nicholas Greene (wng@csail.mit.edu) 79

80 FLaME Contributions Proposed a lightweight method for dense online monocular depth estimation Formulated the reconstruction problem as a non-local variational optimization over a Delaunay graph, which allows for a fast, efficient approach to depth estimation Demonstrated improved depth accuracy and density on benchmark datasets using less than 1 Intel i7 CPU core Demonstrated real-world applicability with indoor and outdoor flight experiments running FLaME onboard, in-the-loop Paper Code* * Coming soon! W. Nicholas Greene (wng@csail.mit.edu) 80

81 Acknowledgments This material is based upon work supported by DARPA under Contract No. HR C-0110 and by the NSF Graduate Research Fellowship under Grant No Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. We also thank John Ware, John Carter, and the rest of the MIT-Draper FLA Team for continued development and testing support. Paper Code* * Coming soon! W. Nicholas Greene (wng@csail.mit.edu) 81