PCL :: Features Michael Dixon and Radu B. Rusu July 1, 2011
Outline 1. Introduction 2. Feature Estimation 3. Keypoint Detection 4. Combining Keypoints and Features
Introduction In this session, we ll talk about features and keypoints in PCL. Local feature estimation Surface normal estimation Point Feature Histogram descriptors Keypoint detection 3D SIFT keypoint detection Global feature estimation Viewpoint Feature Histogram descriptors Download the code and data at http://dev.pointclouds.org/attachments/download/429/iros-pcltutorial-code.zip
Outline 1. Introduction 2. Feature Estimation 3. Keypoint Detection 4. Combining Keypoints and Features
What is a feature? Local Features What is a feature? In vision/perception, the word feature can mean many different things. In PCL, feature estimation means: computing a feature vector based on each points local neighborhood or sometimes, computing a single feature vector for the whole cloud
What is a feature? Local Features Feature vectors can be anything from simple surface normals to the complex feature descriptors need for registration or object detection. Today, we ll look at a couple of examples: Surface normal estimation (NormalEstimation) Point Feature Histogram estimation (FPFHEstimation) Viewpoint Feature Histogram estimation (VFHEstimation)
Local Features (1-2/5) Surface Normal Estimation. Theoretical Aspects: Basic Ingredients Given a point cloud with x,y,z 3D point coordinates
Local Features (1-2/5) Surface Normal Estimation. Theoretical Aspects: Basic Ingredients Given a point cloud with x,y,z 3D point coordinates Select each point s k-nearest neighbors, fit a local plane, and compute the plane normal
Surface Normal and Curvature Estimation Local Features (3/5) ξ i = C j = k i=1 ξ i (p i p j ) T (p i p j ), p = 1 k k i=1 p i e d 2 i µ 2, p i outlier 1, p i inlier σ p = λ 0 λ 0 +λ 1 +λ 2
Feature estimation basics Create the feature estimation object (Templated on the input and output point types) Set the search method to use for the underlying neighborhood searches Create a k-d Tree and pass it to the feature estimator Specify the size of the local neighborhood to use when computing the features setradiussearch (r) or setnearestksearch (k) Set the input points Takes a shared pointer to a pcl::pointcloud Compute Outputs into a reference to a pcl::pointcloud
NormalEstimation example Estimating surface normals using pcl::normalestimation http://docs.pointclouds.org/1.1.0/classpcl_1_1_normal_estimation.html SurfaceNormalsPtr estimatesurfacenormals (const PointCloudPtr & input, float radius) { pcl::normalestimation<pointt, NormalT> normal_estimation; normal_estimation.setsearchmethod (pcl::kdtreeflann<pointt>::ptr (new pcl::kdtreeflann<pointt>)); normal_estimation.setradiussearch (radius); normal_estimation.setinputcloud (input); SurfaceNormalsPtr normals (new SurfaceNormals); normal_estimation.compute (*normals); } return (normals);
NormalEstimation results $./test_feature_estimation robot.pcd -n 0.03
Now let s look at a more complex feature, like... Point Feature Histograms (PFH)
A 3D feature descriptor Point Feature Histograms (PFH) For every point pair (p s, n s ); (p t, n t ), let u = n s, v = (p t p s ) u, w = u v f 0 = v, n j f 1 = u, p j p i / p j p i f 2 = p j p i f 3 = atan( w, n j, u, n j ) i hist = x 3 x=0 f x d f x max fx min d x
A 3D feature descriptor Point Feature Histograms (PFH) f 0 = v, n j f 1 = u, p j p i / p j p i f 2 = p j p i f 3 = atan( w, n j, u, n j ) i hist = x 3 x=0 f x d f x max fx min d x
A 3D feature descriptor Point Feature Histograms (PFH) For every point pair (p s, n s ); (p t, n t ), let u = n s, v = (p t p s ) u, w = u v
Point Feature Histograms (PFH) Points lying on different geometric primitives
As an example, we ll compute Fast Point Feature Histgram descriptors (a faster approximation of PFH) Even though FPFH feature descriptors are more complex than surface normals, the code is very similar The main difference: we need to set input normals First use pcl::estimatenormals Then pass in a shared pointer to the resulting normals
FPHFEstimation example LocalDescriptorsPtr computelocaldescriptors (const PointCloudPtr & points, const SurfaceNo float feature_radius) { pcl::fpfhestimation<pointt, NormalT, LocalDescriptorT> fpfh_estimati fpfh_estimation.setsearchmethod (pcl::kdtreeflann<pointt>::ptr (new pcl::kdtreeflann<pointt>)); fpfh_estimation.setradiussearch (feature_radius); fpfh_estimation.setinputcloud (points); fpfh_estimation.setinputnormals (normals); LocalDescriptorsPtr local_descriptors (new LocalDescriptors); fpfh_estimation.compute (*local_descriptors); } return (local_descriptors);
Outline 1. Introduction 2. Feature Estimation 3. Keypoint Detection 4. Combining Keypoints and Features
What is a keypoint? Keypoints What is a keypoint? A keypoint (also known as an interest point ) is simply a point that has been identified as a relevant in some way. Whether any given point is considered a keypoint or not depends on the keypoint detector in question.
What is a keypoint? Keypoints There s no strict definition for what constitutes a keypoint detector, but a good keypoint detector will find points which have the following properties: Sparseness: Typically, only a small subset of the points in the scene are keypoints Repeatiblity: If a point was determined to be a keypoint in one point cloud, a keypoint should also be found at the corresponding location in a similar point cloud. (Such points are often called "stable".) Distinctiveness: The area surrounding each keypoint should have a unique shape or appearance that can be captured by some feature descriptor
Why find keypoints? Keypoints What are the benefits of keypoints? Some features are expensive to compute, and it would be prohibitive to compute them at every point. Keypoints identify a small number of locations where computing feature descriptors is likely to be most effective. When searching for corresponding points, features computed at non-descriptive points will lead to ambiguous feature corespondences. By ignoring non-keypoints, one can reduce error when matching points.
Keypoint detection in PCL Keypoints There are a couple of keypoint detectors in PCL so far. In this tutorial, we ll focus on pcl::siftkeypoint A 3D adaptation of David Lowe s SIFT keypoint detector
SIFT keypoint detection The SIFT keypoint detector finds local extrema in a difference-of-gaussians (DoG) scale-space. This scale space and its extrema are computed based on several parameters: the size of the smallest scale the number of times the scale doubles the number of scales in between each doubling the minimum absolute DoG value needed to qualify as a keypoint Now let s look at an example of how to compute 3D SIFT keypoints in PCL
SIFTKeypoints example Computing keypoints on a cloud of points PointCloudPtr detectkeypoints (const PointCloudPtr & points, const SurfaceNormalsPtr float min_scale, int nr_octaves, int nr_scales_per_oc { pcl::siftkeypoint<pointt, pcl::pointwithscale> sift_detect; sift_detect.setsearchmethod (pcl::kdtreeflann<pointt>::ptr (new pcl::kdtreeflann<pointt>)); sift_detect.setscales (min_scale, nr_octaves, nr_scales_per_octave); sift_detect.setminimumcontrast (min_contrast); sift_detect.setinputcloud (points); pcl::pointcloud<pcl::pointwithscale> keypoints_temp; sift_detect.compute (keypoints_temp); PointCloudPtr keypoints (new PointCloud); pcl::copypointcloud (keypoints_temp, *keypoints); } return (keypoints);
Outline 1. Introduction 2. Feature Estimation 3. Keypoint Detection 4. Combining Keypoints and Features
Combining keypoints and features Keypoints and feature descriptors go hand-in-hand. Let s look at how to compute features only at a given set of keypoints.
Estimating features at a subset of points Create the feature estimation object Set the search method to use for the underlying neighborhood searches Specify the size of the local neighborhood to use when computing the features Set the search surface These points are used when searching for the input points neighbors Set the input points These are the points for each feature is computed Compute
Keypoints and features Computing local feature descriptors for a set of keypoints LocalDescriptorsPtr computelocaldescriptors (const PointCloudPtr & points, const SurfaceNo const PointCloudPtr & keypoints, float featur { pcl::fpfhestimation<pointt, NormalT, LocalDescriptorT> fpfh_estimati fpfh_estimation.setsearchmethod (pcl::kdtreeflann<pointt>::ptr (new fpfh_estimation.setradiussearch (feature_radius); fpfh_estimation.setsearchsurface (points); fpfh_estimation.setinputnormals (normals); fpfh_estimation.setinputcloud (keypoints); LocalDescriptorsPtr local_descriptors (new LocalDescriptors); fpfh_estimation.compute (*local_descriptors); } return (local_descriptors);
Global feature descriptors Computing global feature descriptors A global feature descriptor is A single descriptor computed over the entire cloud. Used to compare clouds for object recognition later in this tutorial As an example, we ll be using Viewpoint Feature Histograms
VFHEstimation Computing a VFH feature descriptor GlobalDescriptorsPtr computeglobaldescriptor (const PointCloudPtr & points, const SurfaceNo { pcl::vfhestimation<pointt, NormalT, GlobalDescriptorT> vfh_estimatio vfh_estimation.setsearchmethod (pcl::kdtreeflann<pointt>::ptr (new p vfh_estimation.setinputcloud (points); vfh_estimation.setinputnormals (normals); GlobalDescriptorsPtr global_descriptor (new GlobalDescriptors); vfh_estimation.compute (*global_descriptor); } return (global_descriptor);
Testing Use test_feature_estimation to run your code on a PCD file. The options are: -n Estimate surface normals -k Detect keypoints -l Compute local descriptors
Questions?
example The slides after this point are random ones we didn t end up using...
PFH (5/5) Complexity Analysis Complexity is high: O(k 2 ). Optimizations to the rescue! Unordered Ordered
Basic Concepts Fast PFH (1/4) Re-formulate: FPFH(p) = SPF (p) + 1 k k i=1 1 ω k SPF(p k ) Point Feature Histograms (PFH) Fast Point Feature Histograms (FPFH)
Theoretical formulation Fast PFH (2/4)
Noise Analysis Fast PFH (3/4) Synthetically noiseless Synthetically noisified
Noise Analysis Fast PFH (4/4)
pcl::normalestimation<t> p; Features (1/4) p.setinputcloud (data); p.setradiussearch (0.01);
pcl::normalestimation<t> p; Features (2/4) p.setinputcloud (data); p.setradiussearch (0.01);
pcl::boundaryestimation<t,n> p; p.setinputcloud (data); p.setinputnormals (normals); p.setradiussearch (0.01); Features (3/4)
pcl::principalcurvaturesestimation<t,n> p; p.setinputcloud (data); p.setinputnormals (normals); p.setradiussearch (0.01); Features (4/4)
What scale to choose? Feature Persistence bad scale (too small) good scale Selecting the right scale (k-neighborhood) is problematic:
Consistent Normal Orientation Local Features (4-5/5) Before Extended Gaussian Image Orientation consistent for: 1. registration 2. feature estimation 3. surface representation normals on the Gaussian sphere should be in the same half-space
Consistent Normal Orientation Local Features (4-5/5) Before After (vsolet slookatsomecode : i or: propagate consistency through an EMST