A Real-Time RGB-D Registration and Mapping Approach by Heuristically Switching Between Photometric And Geometric Information

Transcription

1 A Real-Time RGB-D Registration and Mapping Approach by Heuristically Switching Between Photometric And Geometric Information The 17th International Conference on Information Fusion (Fusion 2014) Khalid Yousif, Alireza Bab-Hadiashar, Reza Hoseinnezhad School of Aerospace, Mechanical, and Manufacturing Engineering RMIT University July, 2014 RMIT University SAMME 1

2 1 Introduction & Literature Review RMIT University SAMME 2

3 Dense 3D SLAM SLAM simultaneous estimation of camera pose and construction of an unknown environment 3D maps are very informative Allow improved path planning and navigation methods Provide enhanced functionality for robots Augmented reality applications RMIT University SAMME 3

4 RGB-D mapping- Literature Review Method Ransac + ICP refinement + Global optimization Authors Henry et al 2010, Endres et al.2012, Du et al Optical flow RGB-D SLAM Audras et al Dense ICP Newcombe et al. 2011, Whelan et al 2012 RGB-D SLAM + Monocular SLAM combination RGB-D SLAM in dynamic environments Use of both photometric and geometric information Hu et al Keller et. Al, 2013 Kerl et al. 2013, Yousif et al RMIT University SAMME 4

5 2 Methodology RMIT University SAMME 5

6 Selection Between Photometric and Geometric Features Matching photometric features is 5x faster than geometric features Photometric features are used as a default 3D features are used if number of photometric features are below threshold We selected the threshold that provided the best balance between accuracy and efficiency ffig 1. ORB features ffig 2. Proposed method (IS3D) RMIT University SAMME 6

7 Photometric Feature Extraction Extract ORB features from sequential frames. ORB features are based on FAST features ORB is 2x faster than SIFT Achieves similar accuracy. 3D Projecton using the standard pinhole camera model:, image coordinate of visual feature,, projected 3D coordinate are the focal lengths., is the 2D coordinate of the camera optical center. RMIT University SAMME 7

8 Pre-processing the pointcloud Remove points with no information (NaN). Remove points further than 5 metres away. Uniformly down sample the point. Assign a variable search radius to obtain around 4000 points Normal vector estimation: Fit a plane to a point and its neighbours using a LS method Use Large search radius ffig. 3 Normal estimation using small search raduis (left), large search raduis (right) RMIT University SAMME 8

9 Informatively Sampled Geometric 3D features Novel geometric feature extraction method (IS3D). Informative sampling choose best points for registration. A robust estimator for segmenting points into orientation groups (based on normal vectors). Selected keypoints are those not part of any dominant normal orientation group. Fig, 4 Uniformly sampled point cloud Fig. 5 Sampled points using IS3D RMIT University SAMME 9

10 Informatively Sampled Geometric 3D features Angle between normal vectors : cos. MSSE constraint: Where is the number of points included is the model dimension is a constant factor 2.5 is usually used to indicate an inclusion of around 99% of inliers based on a normal distribution). ffig. 6 MSSE segmentation. RMIT University SAMME 10

11 Feature Matching Photometric features are assigned BRIEF descriptors. Geometric features are assigned SHOT descriptors. BRIEF matching: Hamming distance. SHOT matching Nearest neighbour in descriptor space. Mutual consistency check Only pairs of corresponding points that are mutually matched to each other are considered as the initial matches ffig. 7 Initial matching RMIT University SAMME 11

12 Outlier Removal and Transformation Estimation using MSSE Least K-th order statistical model fitting (LKS) based on rank ordering statistics. Find the 6DOF transformation using the detected inliers. Relax fixed error threshold assumption used in RANSAC. The cost function to be minimized is: Modified Selective Statistical Estimator (MSSE) for estimating the scale: ffig. 8 Line segmentation using MSSE. 0 1 Works well with multiple structures ffig. 9 Initial matches (top), good matches (bottom) RMIT University SAMME 12

13 Global Pose Estimation and Mapping Previous steps estimate transformation between two frames. Concatenate all the transformation to obtain a global pose:,,, 0 1, Map obtained by transforming the points from the current frame the global reference frame using ffig. 10 Example of the constructed map of an office scene using the proposed registration method. RMIT University SAMME 13

14 3 Experimental Results RMIT University SAMME 14

15 Method 3 - Evaluation We used a publicly available RGB-D datasets. Evaluation metric is the absolute trajectory error between a sequence of camera poses,., and ground truth trajectory,, 1 Fig: Visualization of the absolute trajectory error (ATE) using: (a) freiburg3 nostructure texture near withloop sequence (b) freiburg3 structure texture far. RMIT University SAMME 15

16 Method 3 - Evaluation Texture vs. Structure Comparison with other methods Computational performance RMIT University SAMME 16

17 5 Conclusion RMIT University SAMME 17

18 Conclusion and Future Work We presented a method that uses both depth and visual information. Works well in low structure scenes as well as low texture. Method automatically switches between photometric and geometric features. Novel informative sampling method (IS3D) that selects only points carrying important information. Our method was evaluated using a publicly available RGB-D benchmark. Future work: Achieving global consistency by employing pose graph optimization or bundle adjustment. Mapping in dynamic environments, segmenting multiple motions and using camera motion only for registration. Possibly tracking the moving objects that are in the camera s field of view. RMIT University SAMME 18

19 Thank you RMIT University SAMME 19

20 References [1] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, A benchmark for the evaluation of rgb-d slam systems, in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2012, pp [2] H. Durrant-Whyte, D. Rye, and E. Nebot, Localization of autonomous guided vehicles, ROBOTICS RESEARCH-INTERNATIONAL SYMPOSIUM-, vol. 7, pp , [3] S. Thrun, Robotic mapping: A survey, Exploring artificial intelligence in the new millennium, pp. 1 35, [4] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, et al., Kinectfusion: realtime 3d reconstruction and interaction using a moving depth camera, in Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 2011, pp [5] P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, Rgb-d mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments, The International Journal of Robotics Research, vol. 31, no. 5, pp , [6] F. Endres, J. Hess, N. Engelhard, J. Sturm, D. Cremers, and W. Burgard, An evaluation of the rgb-d slam system, in Robotics and Automation (ICRA), 2012 IEEE International Conference on. IEEE, 2012, pp [7] A. Bab-Hadiashar and D. Suter, Robust segmentation of visual data using ranked unbiased scale estimate, Robotica, vol. 17, no. 6, pp , [8] E. Rosten and T. Drummond, Machine learning for high-speed corner detection, Computer Vision ECCV 2006, pp , [9] P. Besl and N. McKay, A method for registration of 3-d shapes, IEEE Transactions on pattern analysis and machine intelligence, vol. 14, no. 2, pp , [10] M. Lourakis and A. Argyros, Sba: A software package for generic sparse bundle adjustment, ACM Transactions on Mathematical Software (TOMS), vol. 36, no. 1, p. 2, [11] D. Lowe, Distinctive image features from scale-invariant keypoints, International journal of computer vision, vol. 60, no. 2, pp , [12] H. Bay, T. Tuytelaars, and L. Van Gool, Surf: Speeded up robust features, Computer Vision ECCV 2006, pp , [13] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, Orb: an efficient alternative to sift or surf, in Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 2011, pp [14] H. Du, P. Henry, X. Ren, M. Cheng, D. Goldman, S. Seitz, and D. Fox, Interactive 3d modeling of indoor environments with a consumer depth camera, in Proceedings of the 13th international conference on Ubiquitous computing. ACM, 2011, pp [15] C. Audras, A. Comport, M. Meilland, and P. Rives, Real-time dense appearance-based slam for rgb-d sensors, in Australasian Conf. on Robotics and Automation, [16] R. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, Kinectfusion: Real-time dense surface mapping and tracking, in Mixed and Augmented Reality (ISMAR), th IEEE International Symposium on. IEEE, 2011, pp [17] T. Whelan, M. Kaess, M. Fallon, H. Johannsson, J. Leonard, and J. McDonald, Kintinuous: Spatially extended kinectfusion, [18] A. Bachrach, S. Prentice, R. He, P. Henry, A. Huang, M. Krainin, D. Maturana, D. Fox, and N. Roy, Estimation, planning, and mapping for autonomous flight using an rgb-d camera in gps-denied environments, The International Journal of Robotics Research, vol. 31, no. 11, pp , [19] G. Hu, S. Huang, L. Zhao, A. Alempijevic, and G. Dissanayake, A robust rgb-d slam algorithm, in Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2012, pp [20] K. Yousif, A. Bab-Hadiashar, and R. Hoseinnezhad, 3d registration in dark environments using rgb-d cameras, in Digital Image Computing: Techniques and Applications (DICTA), 2013 International Conference on, 2013, pp [21], Real-time rgb-d registration and mapping in texture-less environments using ranked order statistics, in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)- In press. Source Software, 2009 RMIT University SAMME 20

21 References [22] C. Kerl, J. Sturm, and D. Cremers, Dense visual slam for rgb-d cameras, in Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on. IEEE, 2013, pp [23] L. Douadi, M.-J. Aldon, and A. Crosnier, Pair-wise registration of 3d/color data sets with icp, in Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on. IEEE, 2006, pp [24] S. Druon, M.-J. Aldon, and A. Crosnier, Color constrained icp for registration of large unstructured 3d color data sets, in Information Acquisition, 2006 IEEE International Conference on. IEEE, 2006, pp [25] F. Tombari, S. Salti, and L. Di Stefano, Unique signatures of histograms for local surface description, in Computer Vision ECCV Springer, 2010, pp [26] M. Calonder, V. Lepetit, C. Strecha, and P. Fua, Brief: Binary robust independent elementary features, Computer Vision ECCV 2010, pp , [27] R. B. Rusu, Semantic 3d object maps for everyday manipulation in human living environments, KI-K unstliche Intelligenz, vol. 24, no. 4, pp , [28] F. Fraundorfer and D. Scaramuzza, Visual odometry: Part ii: Matching, robustness, optimization, and applications, Robotics & Automation Magazine, IEEE, vol. 19, no. 2, pp , [29] M. Quigley, K. Conley, B. P. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, Ros: an open-source robot operating system, in ICRA Workshop on Open Source Software, 2009 RMIT University SAMME 21