Deformable Object Registration

Transcription

1 Aalto University Of Electrical Engineering AS Project in Automation and Systems Technology Deformable Object Registration Author: AnttiValkonen Instructor: Joni Pajarinen May 31, 2015

2 1 Introduction The problem of point set registration is to find a mapping between moving model point set M and a fixed scene set S. Mappings may represent either rigid or non-rigid transformations. Depending on the application, the point sets can be raw data from 3D-scanner or extracted feature points. Several different approaches have been developed for point set registration. Different methods are used depending on the type of input data and transformation model. If the two point sets have a similar structure, simple iterative methods exist for rigid point set registration. Often, however, the point sets are different in size, and one-to-one correspondence cannot be assumed. Probabilistic models have been proposed to allow for fuzzy correspondences between point sets. Sometimes the point sets are represented as a mixture of multivariate gaussian distributions, and the problem of matching discrete point sets becomes a problem of matching continuous distributions. The goal of this project is to register objects from point cloud data. A model point set from the object is constructed beforehand, and it is matched to a scene point set, in which the object is (partially) shown. The construction of point clouds is not part of the project. Three different algorithms are used for point set registration: Iterative Closest Point, Coherent Point Drift and Robust Point Set Registration Using Gaussian Mixture Models. Iterative Closest Point (ICP) is a simple iterative algorithm for point set registration. The classical ICP has shortcomings when dealing with imperfect data, but various modifications have been presented to overcome them. The two other algorithms model the point sets with mixtures gaussians and find the solution to problem by fitting the mixture distributions. As a result of this project, a c++ program is created to register objects. The actual implementation of the used algorithms is not part of this project, and the source code has been provided by the authors of [6]. 1

3 2 Background 2.1 Mixture Models Mixture models are probabilistic models to represent data. A mixture model assumes that all observations (or measurements) are generated from a mixture of probability distributions with unknown parameters θ. A probability density function of such model is where: p(x θ) = N α n p(x θ k ) (1) n=1 K = Number of mixture components N = Number of observations α = Vector of all mixture weights, sums to 1 α k = Weight parameter associated with micture component k θ i=1..k = Parameter of distribution associated with component i θ = Vector of all parameters θ p(x θ k ) = Probability density of distribution with parameters θ k Gaussian mixture model (GMM) is a mixture of K gaussian densities with parameters µ k, Σ k. With K gaussian distributions, p(x θ k ) is p(x µ x, Σ x ) = 1 det(2πσx ) exp{ 1 2 (x µ x) T Σ 1 (x µ x )} (2) Σ x = Covariance matrix of mixture component x µ x = Mean of mixture component x When fitting the distribution parameters θ to a given data set X, it is useful to have equation 1 as function of θ. Likelihood-function can be obtained by using component weights as priorprobabilities. The density of the likelihood of observations X given parameters θ is p(x θ) = The goal is to find θ which maximizes L, or N p(x i θ) = L(θ X ) (3) i=1 θ = arg max L(θ X ) (4) θ The difficulty of the problem depends on the form of p(θ X ). Often the log-likelihood is maximized, because it simplifies the computation without changing the maxima of the function. [4] Expectation Maximization algorithm Expectation Maximization (EM) is an iterative method for finding the maximum likelihood of θ given X. EM assumes that the data set X is incomplete, and a complete data Z = (X, Y) exists, with hidden data points Y. Also, a join density p(z θ) = p(x, y θ) = p(y x, θ)p(x θ) (5) is assumed. With equation 5, a likelihood function for complete data Z is defined as L(θ Z) = p(x, Y θ) (6) 2

4 Figure 1: Training a mixture of 10 Gaussian with EM algorithm. In (a), all parameters are set to have a large variances and a same mean, in (b) and (c), the means are starting to converge according to the training data. In (d) the converged solution is shown. [2, p. 406] EM algorithm first solves the expected value of a log likelihood with respect to unobserved data points, given the observed data X and parameters θ. This is called the E-step of the algorithm. where: Q(θ, θ (i 1 ) = E[log p(x, Y θ) X, θ (i 1 )] = log p(x, y θ)p(y X, θ (i 1) )dy y Y X = Known data points, constant θ (i 1) = Parameters θ from the previous iteration, constant Y = Unobserved data points, a random variable dictated by distribution p(y X, θ (i 1) ) θ = is the variable which is to be adjusted. (7) The second step is to maximize the previously computed expectation: θ = arg max Q(θ, θ (i 1) ) (8) θ When maximising the log-likelihood of a gaussian mixture model, the complete data log likelihood (7) gets form N H n=1 i=1 p old (i h){ (xn m i ) T Σ 1 i (x n m i ) 1 2 log det(2πσ i) + log p(i)} (9) and the computations needed for E and M steps are as shown below. E-step γ(z nk ) = α k N (x n µ k, Σ k ) K j=1 α jn (x n µ j, Σ j ) 3

5 M-step µ new k = 1 N k Σ new N γ(z nk )x n n=1 k = 1 γ(z nk )(x n µ new k )(x n µ new k ) T N k α new k = N k N E-step and M-step are iterated until the algorithm converges. [2, p ] 2.2 Transformation Models The point-set registration problem is to find a mapping T which maps the points S so that the error measure between point sets T (S) and M is minimized. The methods used in this project use the rigid transformation model and two types of non-rigid transformation models: thin plate splines and gaussian radial functions Rigid transformation A rigid transformation can be considered as a combination of translation, rotation, and uniform scaling. The resulting transformation has 7 parameters. Assuming 2 pairwise point sets M and S, translation can be obtained by computing centroid Center(X ) = N n=1 xi N from both point sets and computing X centred = X Center(X ) (10) To remove the scale difference, the scale of the (centred) point sets is measured by computing the average distance size(x ) = from the origin of point sets S to scale of M and computing X N S scaled = size(m) size(s) S (11) To remove the relative rotation, the rotation matrix R can be found by minimizing square distance of point-wise distance [1]. Usually, the point sets are non-pairwise, and scale factor s, translation T and rotation R have to be found differently. In section 3, non-pairwise point set registration is considered in more detail Non-Rigid Transformation In addition to the rigid transformation parameters, a non-rigid transformation models the deformation between the point sets. Non-Rigid transformations are often modelled by using radial basis functions (RBFs). An RBF maps a point x to location y using the function where: y = N w i φ(x, p i ) (12) i=1 w i = i-rh warping coefficient (d-dimensional vector) p i = i-th control point (d-dimensional vector) φ(r) = kernel function r = euclidean distance between the two points Common choices for kernel functions are thin plate splines (TPSs) and gaussian radial basis functions (GRBFs). A TPS has a kernel function φ(r) = r 2 log(r) in two dimensions, and φ(r) = r 4

6 Figure 2: The top row represents an unknown deformation of an image where the mapping of the control points in known, and the bottom row represents a thin plate spline deformation with the same control points in three dimensions. A Gaussian kernel is defined as exp( r2 2σ 2 ). Gaussian radial function has a free parameter σ, which represents the variance of the kernel. Given two point sets x and y, a control point set in point set x and a corresponding set in y, the deformation between the point sets can be computed. This is illustrated in Figure 2. In point set registration, however, the location of the control points is only known in one of the point sets, and the task is to find the mapping for each control point so that the error metric between the point sets is minimized. Therefore, the non-rigid transformation has d p additional parameters to non-rigid case, where d is the dimension of the point set, and p is the amount of the control points. An important aspect of the non-rigid mapping is the notation of bending energy. When computing non-rigid point set registration, the algorithms should prefer transformations with little deformation, and penalize those with a larger or less credible deformation. The bending energy of a non-linear mapping with radial functions is the following: Let Q be a matrix representing a set of p control points and kernel matrix K = {K ij } where K ij = φ(p i, p j ). Given the warping coefficients w, the bending energy of the deformation is E bending = trace(w T Kw) (13) To regularize a non-linear transformation, a penalty term λ 2 (wt Kw) can be added to the cost function. Lambda is a free parameter which dictates the amount of deformation allowed. 3 Algorithms Registration algorithms used in this project are described in this section: Iterative Closest Point[3], which is the most basic algorithm, is used because of its simplicity in implementation and computation. Methods based on gaussian mixture models, described in papers [6] and [7] are described in sections and

7 3.1 Iterative Closest Point Iterative Closest Point (ICP) [3] is one of the best-known algorithms used in point set registration. ICP automatically finds correspondences between two point sets, and uses these correspondences to estimate the transformation. Algorithm Outline Different ICP versions find the corresponding points differently. Early versions find points with shortest point-to-point -distance, but later versions use point-to-surface distance. A point-tosurface distance is defined as the distance between the point m and the surface plane that passes by corresponding point s in scene-set. The ICP implementation used in this project uses point-topoint -distances. Until a stopping criterion is met: 1. Find a set of corresponding points between the point sets. 2. Estimate the rigid transformation matrix T using a closed form solution described in [1]. 3. Apply the transformation T to the model point set 3.2 Point Set Registration With Gaussian Mixture Models To overcome the limitations of the Iterative Closest Point method, various probabilistic methods have been developed to solve the point set registration problem. Modelling point sets as mixtures of gaussians gives flexibility to the algorithms: rather than comparing individual points between the scene and the model, the alignment is conducted for probability distributions. This results in a better performance, especially in presence of noise and outliers. In the following two methods, the gaussian distributions are slightly simplified versions of those described in section 2.1. All the mixture weights are equal, and the covariances of all member distributions are σ 2 I Robust Point Set Registration Using Gaussian Mixture Models Jian et al. [6] presented a probabilistic framework for registering point sets with both rigid and different non-rigid transformation models. A gaussian mixture model is constructed from both scene- and model point set, and a numerical optimization is computed to minimize the L2-distance between the two GMMs. For non-rigid transformations, two different deformation models were used: thin plate splines and gaussian radial basis functions. Cost Function A similarity-measure L2-distance (d L2 ) is used to measure similarity between different mixture models. L2-distance has a closed-form expression for two gaussian mixtures, and also its gradient has a closed-form expression. This allows the use of quasi-newton algorithm to find d L 2. The L2-distance between the 2 GMMs generated from point sets is defined as d L2 (S, M, θ) = (g f) 2 dx (14) = f 2 dx 2 fgdx + g 2 dx where: f = gmm(t (M, θ)) = The probability density of the gaussian mixture generated from the model point set and undergone a transformation T with parameters θ g = gmm(s) = The probability density of the gaussian mixture generated from the scene point set 6

8 Since the scene point set is fixed during the optimization, only the two latter terms have an effect on the shape of the cost function. Algorithm Outline 1. Construct gaussian mixture models from point sets M, S, set initial parameters θ for transform T, and initial scale parameter σ. 2. Repeat phases 3 5 until stopping criteria is met 3. Minimize d L2 (S, M, θ) with quasi-newton optimization engine. A regularization term must be added for non-rigid transformation families. 4. Update the new parameter θ. 5. Decrease the scale parameter σ as annealing step Coherent Point Drift A similar point set registration algorithm called Coherent Point Drift, CPD was described in [7]. While the method in considers both point sets as mixtures of gaussians, CPD considers the model point set as a gaussian mixture, and the scene point set as a random sample drawn from the distribution generated from the model point set. In CPD, outliers are modelled as a uniform distribution, whose weight can be adjusted when running the algorithm. The method uses an Expectation Maximization framework to find a transformation T which maximizes the expectation of scene points being generated by the gaussian mixture, which is generated from point set T (M). The objective function (7) is a function of transformation parameters θ and variances σ, and the update equations depend on the type of transformation used. For non-rigid registration, gaussian radial basis functions were used. Therefore, additional free parameters for the non-rigid registration are the weight of the bending energy λ and the variance of the gaussian kernel. 4 Software One of the results of this project is the software created for point cloud registration. While the code for the registration algorithms is provided, the created software is needed to handle data conversion between different algorithms, and analysing and visualizing results. The software is written in C++ using Windows 8.1 platform and Microsoft Visual Studio Libraries Point Cloud Library (PCL) is an open-source library for point cloud processing. It contains algorithms for various computer vision tasks, and it has been written C++. In this project, the point sets are represented as PCL::PointCloud objects, and the visualization is done with functions within PCL. Dependencies for PCL, such as Boost, Eigen for linear algebra, VTK and QT for visualization. Vision Numerics Library(VNL) is a library for numerical calculation of algorithms in computer vision. gmmreg api is a library that computes point set registration with gaussian mixture models. The library implements the methods described in 3.2. The original sources can be found in [5]. The original version takes a text file and a registration method as input arguments. The input text file contains the configuration parameters for the methods and a path for the scene, model and 7

9 control point sets (which are also.txt files). The output of the original gmmreg api is a text file containing the transformed point set. The modified gmmreg api takes the input point sets, a control point set and the registration method as arguments, and returns the transformed point set as VNLMatrix. 4.2 Program Structure The structure of the program is the following: six classes were created to conduct the point set registration. Modified code for GMM registration was built as a static library and included to the project. RegistrationManager Class RegistrationManager has an instance of all the other classes as members. In the main function, a new RegistrationManager is created. Then, a member function RegistrationManager::InitalizePointClouds is called, which calls member function PointLoader::Load for the point clouds that were given as arguments for the program. Then, a member function Registration- Manager::InitalizeAlgorithm is called, which creates a RegistrationAlorithm object according to the input arguments passed to the program. Then, the algorithm is run with the function RegistrationManager::RunAlorithm, which runs the created algorithm for the point clouds that were loaded earlier. Finally, RegistrationManager::VisualizeResults is called to visualize the results of the registration. PointLoader Class PointLoader is responsible for the loading and the saving of the point clouds. PointLoader can load and save point clouds in file formats.pcd and.txt. RegistrationAlgorithm Class RegistrationAlgorithm is a base class for different types of RegistrationAlgorithm objects. It has member functions for centering and downsampling the point clouds, and abstract functions Run and PreProcessPointClouds, which depend on the type of RegistrationAlgorithm. GMM algorithm Class GMM algorithm represents the RegistrationAlgorithm for the methods in the gmmreg api (EM TPS, EM GRBF, rigid, TPS L2, and TPS GRBF). The implementation for the PreProcess- PointClouds is the following: point clouds are downsampled and centered, a set of control points is generated (unless the rigid method is used), and VNLMatrices are created from the PointCloudobjects. Member function Run simply calls the gmmreg api with the VNLmatrices, control points and the config-file that is associated with the method. ICP algorithm Class ICP algorithm represents the RegistrationAlgorithm for the method ICP. The member function PreProcessPointClouds downsamples and centers the point clouds, and the member function Run runs the registration algorithm (which is implemented in Point Cloud Library) and transforms the model cloud with the parameters that the ICP algorithm computed. Visualizer Class Visualizer takes care of visualizing the results by using the class pcl visualizer of Point- CloudLibrary. The member function SetPointClouds sets a vector of point clouds to the stage. Run simply runs the visualizer. The point clouds on stage can be toggled on and off by pressing the keyboard that corresponds to the point cloud (e.g. pressing button 1 toggles the first element of the vector passed to SetPointClouds). 8

10 Figure 3: Visualization of the model point sets m505, m503 and m506 Figure 4: Visualization of the scene point sets yellsmall mug, green mug and yellow mug 4.3 Instructions For Running The Software The program takes three input arguments, Scene, Model, and Method. The point sets can be in file format.txt or.pcd. Valid Methods are: ICP: Iterative Closest Point, described in [3]. rigid: Rigid registration with GMMs, described in [6], uses the error L2 distance as an error measure. Corresponding configuration file is parameters rigid.ini TPS L2: Non-rigid registration with GMMs, described in [6], uses thin-plate spline deformations and L2 distance. Corresponding configuration file is parameters TPS L2.ini GRBF L2: Non-rigid registration with GMMs, described in [6], uses gaussian radial basis function deformations and L2 distance. Corresponding configuration file is parameters GRBF L2.ini EM GRBF: Non-rigid registration with GMMs, described in [7], uses gaussian radial basis function deformations and EM-framework for transformation optimization. Corresponding configuration file is parameters EM GRBF.ini The input files, configuration files and the files containing the data sets should be in the same directory as the executable. The following example command executes the program and computes a non-rigid registration with L2-distance and thin-plate splines../deformable object registration scene.pcd model.pcd TPS L2 Deformable object registration.exe scene.pcd model.pcd TPS L2 (linux) (windows) 5 Experiments This section demonstrates the performance of the registration methods discussed in section 3. First, simple two-dimensional test cases are considered, and then, two test cases with real-world-data are considered. 9

11 Figure 5: The results of the Iterative Closest Point algorithm for rotated point sets. The results for a rotation of pi/8 are shown on the first row, and the results for a rotation of pi/4 are shown on the second row. 5.1 Generated Experiments In the first test for the algorithms, a two-dimensional point set was rotated, and the registration algorithms were used to find the transformation between the rotated and the original point set. Iterative Closest Point yielded good results as long as the rotation was sufficiently small. When rotation was pi/8, the result was accurate, but the algorithm converged to local minimum when the rotation was pi/4 or larger. The results are presented in Figure 5. All the GMM methods resulted in accurate results with any given angle. The second test was similar to the first one, except for some of the points which were removed from the scene point set. Iterative Closest Point worked in a similar fashion as in the first test: the correct rotation was found with the rotation angle of pi/8, and the algorithm converged to a local minimum when the rotation angle was larger. Rigid GMM method (L2-distance method) yielded accurate results with any rotation angle. The results from the non-rigid GMM method were dependent on the input parameters given to the algorithm. When the bending energy of the deformation was not penalized enough, the algorithms resulted in overfitted transformations. An overfitted transformation is presented in Figure Experiments with real-world data The dataset used in these experiments were created with Microsoft Kinect. Model point sets were created by moving the Kinect sensor around the object, and scene point sets were created by extracting the object from a single range image taken with Kinect. Model point sets were 10

12 Figure 6: The overfitted transformation, computed with L2-method and thin-plate splines. The registration yielded better results when the bending energy of the transformation was penalized. considerably larger by having around 10,000 50,000 points, while scene point sets have around 2,000 3,000 points. Examples of model point sets are shown in Figure 3, and examples of scene point sets are shown in Figure 4. When running the algorithms, the point sets were downsampled to roughly 1,000 points each. For the first test, coffee cups with similar shapes were chosen. m503 was chosen as the model point set, and yellsmall mug was chosen as the scene point set. The point sets are presented in Figures 3 and 4. Iterative closest point did not work at all. The algorithm was trapped to a local minimum, and the transformed model point set was really similar to the original one. The rigid GMM method found a reasonably good solution. The overall alignment was good, except for the handles of the mugs did not align. The authors of [6] point out that the local minima -issue exists when the point sets have symmetries or repeating patterns in their spatial structure. The non-rigid GMM methods worked as well as the rigid method when the regularization parameters were set to reasonable values. When the bending energy was not penalized, similar overfitting occurred as in Figure 6. For the second and final test, mugs with considerably different shape were chosen. m506 was chosen as the model point set, an red2 mug was used as scene point set. The results of the ICP were similar to the previous case: ICP could not handle the large number of outliers, and the local minimum to which the algorithm converged was close to the identity transform. The rigid GMM method did not work well in this case. The different shape of the coffee cups resulted in a solution where the bottom part of the model was fitted to the side of the scene point set. The results are Figure 7 The non-rigid GMM methods worked better with these point sets. With the resulting transformation, the model point set was stretched to align reasonably well with the scene point set, except for the handle. The results are shown in Figure 8 11

13 Figure 7: The rigid registration with GMMs was trapped in a local minimum when the data sets had considerably different shapes. Figure 8: The Non-rigid registration yielded good results when aligning two coffee cups of different shapes. 12

14 Work package Time reserved Time used 1. Get an overview of the different methods 20h 20h 2. Pre-study of the techniques 60h 50h 3. Compiling and running the code 50h 105h 4. Running with the datasets provided by school 20h 5h 5. Tweaking on the methods (parameters, approximations) 50h 30h 6. Documentation 16h 28h Total 216h 238h Table 1: A summary of time use: the total time used for the work packages For the non-rigid methods, a good set of parameters was found that worked reasonably well within the data set. Rather small set of control points ( 100) was found to be sufficient. Even though this value yielded good results within the data set provided, the size of the control point set depends on the nature of the deformation. Choosing the amount of data set is a trade-off between computing speed and accuracy. The deformation penalization coefficient λ yielded overfitted results when close to zero, and close to rigid results with large numbers. 6 Conclusions Clearly, by modelling point sets with gaussian mixture models, registration becomes robust compared to the basic ICP. Computing point set registration with discrete point sets is sensitive to outliers and noise, and both of these can be modelled with mixture models. Iterative Closest Point, however, is relatively simple to implement and it runs fast, and improved versions have been developed to overcome its problems. These methods are out of the scope of this project, and the results are based only on the basic implementation of ICP. Point set registration with mixtures of gaussians does have some of its own issues. All the methods require some tuning of free parameters, and while a set of parameters might work well with one data set, it may not work so well with another, different type of data set. Another issue with registration with gaussian mixtures is the speed: while the ICP computes a result relatively fast, the use of gaussian mixtures require extensive computation. A rigid GMM method had running times of 1 5 seconds and non-rigid methods seconds, when the data sets had around 1,000 points and around 100 control points were used. In order to make these methods faster, data sets can be down sampled as a pre-processing step, which can work well in some cases, but it adds another free tunable parameter. Also, some precision is lost during the down sampling of the data. Another consideration that was mentioned in [6] and [7] is the use of Fast Gaussian Transform during the computation of the methods. Two different deformation models were used to conduct non-rigid point set registration. Both GRBFs and TPSs provide a framework for modelling deformations via a set of control points. Personally, I could not see much difference in the results when comparing these two. GRBF has one more free parameter that has to be tuned, which in itself can make the tuning more difficult. 6.1 The Summary Of The Project Time management was quite optimistic during the whole project, and the work packages introduced in the project plan could have been more meaningful. Programming and tasks related to it, for example, took twice as much time as planned. The work package was called Compiling and running the [already provided] code, yet the final program consisted of six classes on top of the provided code. Table 6.1 summarizes the time use with the work packages. The total hours worked was recorded, but the division between the work packages is an estimation, since the work packages themselves were not defined accurately enough. The registration methods used in the project were chosen quite early on, and the two chosen 13

15 methods were quite similar. It could have been interesting to study and use methods more different from one another. On the other hand, the subject was quite difficult and the methods used a lot of machine learning and linear algebra. Understanding the ideas behind the chosen methods was already a great challenge. The project plan states the following about the goal of the project: The goal of the project is to make a robot recognize objects in its environment using state-of-the-art algorithms. Several approaches have been used to solve the problem, and in this project, different methods will be compared in regards of computing demands and performance. The methods were chosen, a piece of software was programmed and point sets were registered, so the project was successful in my opinion. 14

16 References [1] K Somani Arun, Thomas S Huang, and Steven D Blostein. Least-squares fitting of two 3-d point sets. Pattern Analysis and Machine Intelligence, IEEE Transactions on, (5): , [2] David Barber. Bayesian reasoning and machine learning. Cambridge University Press, [3] Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Robotics-DL tentative, pages International Society for Optics and Photonics, [4] Jeff A Bilmes et al. A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. International Computer Science Institute, 4(510):126, [5] Bing Jian and Baba C. Vemuri. Repository for the implementations of the robust point set registration algorithm. [6] Bing Jian and Baba C Vemuri. Robust point set registration using gaussian mixture models. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(8): , [7] Andriy Myronenko and Xubo Song. Point set registration: Coherent point drift. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(12): ,