3D Reconstruction on GPU: A Parallel Processing Approach

Transcription

1 3D Reconstruction on GPU: A Parallel Processing Approach Shubham Gupta, Siddharth Choudhary, and P.J. Narayanan Center for Visual Information Technology International Institute of Information Technology Hyderabad INDIA {shubham@students,siddharth_ch@students,pjn@}iiit.ac.in Abstract. This report presents the current state of art algorithm for fully automatic recognition and reconstruction of 3D objects in image databases and the scope of parallelization in them. The object recognition problem is posed as one of finding consistent matches between all images, subject to constraint that the images were taken from a perspective camera. We assume that the objects or scenes are rigid. For each image we associate a camera matrix, which is parameterised by rotation, translation and focal length. We use invariant local features to find matches between all image and the RANSAC algorithm to find those that are consistent with the fundamental matrix. Objects are recognised as subsets of matching images. We then solve for the structure and motion of each object, using a sparse bundle adjustment algorithm. We studied the above algorithm and are in the process of parallelizing the above algorithm. Key words: 3D Recognition and Reconstruction, Parallelization using CUDA SDK on GPU 1 Introduction A central goal of image-based rendering is to evoke a visceral sense of presence based on a collection of photographs of a scene. There has been significant research progress towards this goal ghrough view synthesis methods in the research community and in commercial products such as panorama tools. One of the dreams is that these approaches will one day allow virtual tourism of the world s interesting and important sites. Now a days, if we do a Google image search on Notre Dame Cathedral, it returns over 10,000 photos, capturing the scene from myriad viewpoints, levels of detail, lighting conditions, seasons, decades and so forth. The system for 3D Reconstruction, makes it easier for one to browse through the photos, along with a sparse 3D geometric representation of the scene. The current state of art has the total running time of a few hours for 120 photos to about two weeks in case of 2500 photos. The goal of this project is to find the

2 2 3D Reconstruction on GPU: A Parallel Processing Approach scope of parallelzation in the various modules explained below. The remainder of this report is structured as follows. In section 2 we describe the system overview. Section 3 describes the invariant feature extraction and matching scheme. Secton 4 describes the Fundamental Matrix Computation to find the correct image matches. Section 5 describes the auto calibration of the intrinsic camera parameters. Section 6 describes the sparse bundle adjustment algorithm used to solve jointly for the cameras and structures. 2 System Overview The system is divided into following modules: Feature Extraction and Matching Image Matching Fundamental Matrix Computation Self Calibration Bundle Adjustment 3 Feature Extraction and Matching The features used in this project are SIFT (Scale Invariant Feature Transfor) features. These locate interest points at maxima/minima of a difference of Gaussian function in scale-space. Each interest point has an associated orientation, which is the peak of a histogram of local orientations. This gives a similarity invariant frame in which a descriptor vector is sampled. Though a simple pixel resampling would be similarity invariant, the descriptor vector actually consists of spatially accumulated gradient measurements. This spatial accumulation is important for shift invariance, since the interest point locations are typically accurate in the 1-3 pixel range. Illumination invariance is achieved by using gradients and normalising the descriptor vector. We extract the SIFT features from all n images. Sincle multiple images may view the same point in the world, each feature is matched to k nearest neighbours (typically k = 4 ). Finding k nearest neighbours of all the features can be done using k-d tree using the ANN library. This section consists of SIFT Feature extraction which can be parallelized using SIFT GPU, which is already implemented and finding Approximate nearest neighbour is highly parallelizable, since all the queries (for all the features) run independent of each other. 3.1 KD Tree Construction Algorithm The SIFT feature vectors of each image is inserted into a KD Tree whose algorithm is as follows:

3 3D Reconstruction on GPU: A Parallel Processing Approach 3 function kdtree (list of points pointlist, int depth) { if pointlist is empty return nil; else { // Select axis based on depth so that axis cycles through all valid values var int axis := depth mod k; // Sort point list and choose median as pivot element select median by axis from pointlist; } } // Create node and construct subtrees var tree_node node; node.location := median; node.leftchild := kdtree(points in pointlist before median, depth+1); node.rightchild := kdtree(points in pointlist after median, depth+1); return node; 3.2 Feature Space Outlier Rejection The feature space outlier rejection is used to remove incorrect matches by comparing the distance of a potential match to the distance of the best incorrect match. In an ordered list of nearest-neigbour matches, we assume that the first n overlap -1 elements are potentially correct but that the n overlap element is an incorrect match. We denote the match distance of the n overlap element as e outlier, as it is the best matching outlier. In order to verifty a match, we compare the match distance of a potential correct match e match to the outlier distance, accepting the match if e match 0.8 x e outlier 4 Image Matching We used the k nearest neighbour of all the features (output of the previous step), to find the m nearest matching images for each images. This is done so that we need not compute the fundamental matrix for all pair of images ( which is quadratic in nature ). So, we have found it necessary to match each image only to a small number of neighbouring images in order to get good solutions for the camrea positions. We can find all the m nearest matching images of all images in parallel, and hence this module can be highly optimized.

4 4 3D Reconstruction on GPU: A Parallel Processing Approach 5 Fundamental Matrix Computation The fundamental matrix F is a 3 x 3 matrix of rank 2 which relates corresponding points in stereo images. In epipolar geometry, F y 1 describes a line of which the corresponding point y 2 on the other image must lie. That means, for all pairs of corresponding points with homogeneous image coordinates y 1 and y 2 holds T y 2 F y 1 = 0 Being of rank two and determined only up to scale, the fundamental matrix can be estimated given at least seven point correspondences. The application of the fundamental matrix makes it a useful tool for scene reconstruction.the rela- tion between corresponding image points which the fundamental matrix represent is referred to as epipolar constraint, matching constraint, or incidence relation. The method to solve for F without the problem of ambiguity or the need to solve cubic equations is the 8 point or least squares algorithm. This algorithm can also be easily extended to any larger number of points. Each point correspondence equation is written as a homogenous equation in 9 variables. All the equations are stacked into a matrix and the least squares optimal solution is found using eigen analysis. Unfortunately, this method does not enforce the rank constraint. F must be projected onto a rank two subspace. This can be done by performing singular value decomposition and setting the smallest singular value to zero. We use normalized coordinates to find F, which is unnormalized later on. 5.1 Eight Point Algorithm The eight point algorithm is described as follows: Given 8 correspondence points p11, p12...p18, and p21, p22... p28 1. Normalize the points for unit variance and centering the origin in the image q 1 = T 1 p 1 q 2 = T 2 p 2 2. Create the data matrix and express the epipolar constraint as Df = 0 where D is 8x9 Row i of D = [u u u v u v u v v v u v 1 ] where q 1 = [u v 1] T, q 2 = [u v 1] T And f = [G 11 G 12 G 13 G 21 G 22 G 23 G 31 G 32 G 33 ] T 3. Solve for f as the eigen vector corresponding to least eigen value in the form: Df = V 4. Ensure that G obtained has rank 2, by finding the nearest rank 2 matrix G. which minimizes the frobenius norm G G

5 3D Reconstruction on GPU: A Parallel Processing Approach 5 5. Denormalize to get F F = T T 2 G T RANSAC For a robust estimate of F, we have applied a ransac approach.the value of F decides the goodness of the essential matrix. It is an iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers. It is a non-deterministic algorithm in the sense that it produces a reasonable result only with a certain probability, with this probability increasing as more iterations are allowed. The algorithm was first published by Fischler and Bolles in The computation of all the Fundamental Matrices can be computed in parallel, since we can pass all the random samples in parallel, to compute all the fundamental matrices in one go, but to seperate inliers and outliers we need to compute it serially. RANSAC Algorithm The generic RANSAC algorithm, in pseudocode, works as follows: input: data - a set of observations model - a model that can be fitted to data n - the minimum number of data required to fit the model k - the maximum number of iterations allowed in the algorithm t - a threshold value for determining when a datum fits a model d - the number of close data values required to assert that a model fits well to data output: best_model - model parameters which best fit the data best_consensus_set - data point from which this model has been estimated best_error - the error of this model relative to the data iterations := 0 best_model := nil best_consensus_set := nil best_error := infinity while iterations < k maybe_inliers := n randomly selected values from data

6 6 3D Reconstruction on GPU: A Parallel Processing Approach maybe_model := model parameters fitted to maybe_inliers consensus_set := maybe_inliers for every point in data not in maybe_inliers if point fits maybe_model with an error smaller than t add point to consensus_set if the number of elements in consensus_set is > d better_model := model parameters fitted to all points in consensus_set this_error := a measure of how well better_model fits these points if this_error < best_error best_model := better_model best_consensus_set := consensus_set best_error := this_error increment iterations return best_model, best_consensus_set, best_error 6 Self Calibration 6.1 Self Calibration Camera Calibration is the process of finding the internal parameters of the camera. The camera off-line calibration procedure requires the euclidean distance of the various points on the object. It computes the camera matrix that minimizes the error between the image of the object and its reprojection. The key point of this technique is to have the prior knowledge of the euclidean geometry of the object. This may not be available all the time. Moreover if the camera internals are changed the camera has to be recaliberated. To circumvent the problem of not having the prior knowledge of the euclidean structure of the scene, self calibration is deployed to find out the internal parameters of the camera. This makes use of the correspondences of the points in the images taken at different rotations and the translations of the camera. The camera is sufficiently rotated and translated to avoid the epipoles meeting at the same point.the various assumptions that were employed are described in Section 2. In Section 3 we describe the various mathematical formulations.the pseudo code is explained in the Section Assumptions used in the Calibration The various assumptions employed during the calibration are listed as follows:

7 3D Reconstruction on GPU: A Parallel Processing Approach 7 The Skew parameter of the camera is assumed to zero. The aspect Ratio of the camera is assumed to be 1. The center is taken close to the actual center point of the image. ụ 6.3 Hartley s Equations and Cipolla s Method We use the Hartleys equations to find out the best focal length estimate of the camera. The Fundamental matrix used in the equation is derived from the point to point correspondences using RANSAC. With this focal length the internal parameters of the camera matrix are initialized. From the Fundamental Matrix obtained and the initial estimate of the camera matrix, the Essential matrix is calculated. The constraint that the two singular values of the Essential matrix should beequal forms the basis of the cost function which is used to find out the best estimate of the camera matrix such that it gives the two singular values of the Essential Matrix equal in magnitude. The focal length used to intialize the camera matrix and the cost function used are shown below: The focal length is derived from the equation: f= c T [e ] x IFC(c T F T c ) c T [e ] x IFIF T c I = c = ( u 0 v 0 1 ) T c = ( u 0 v 0 1 ) T This is the cost function used: C(K i,i=1,...,n) = n ij w ij Pnkl w kl 1 σij 2 σij 1 σij 6.4 Pseudo Code The algorithm is described as follows: Compute the fundamental Matrix using RANSAC for each pair of the images. Compute the best estimate of the focal length, which is used for initialization of K matrix.

8 8 3D Reconstruction on GPU: A Parallel Processing Approach Compute the Essential Matrix using E = K FK and find the singular values of E from SVD. Minimize the cost function of the two singular values using gradient Descent. After finding Fundamental Matrix, and Entrinsic parameters we can calculate the Essential Matrix. After finding Essential matrix, using triangulation we can calculate the relative Rotation and Translation matrices. 7 Bundle Adjustment 7.1 Introduction In 3D reconstruction from multiple views, point to point correspondences are computed using Sift algorithm. If there is error or noise in this point to point correspondence, this error will also reflect in the computation of Fundamental matrix, self calibration of camera and finally 3D point computation by triangulation method. So we have to some how minimize the total accumulated error at the end. We use bundle adjustment algorithm to rectify this error as the final step of 3D structure and viewing parameter estimate. It s name refers to the bundles of light rays originating from each 3D feature and converging on each camera centre, which are adjusted optimally with respect to both structure and viewing parameters. Bundle Adjustment amounts to minimizing the reprojection error between the observed and predicted image points, which is expressed as the sum of squares of a number of non-linear real-valued functions. Thus, the minimization is achieved using non-linear least squares algorithm, of which the Levenberg- Marquardt(LM) has proven to be the most successful due to its use of an effective damping strategy that lends it the ability to converge promptly from a wide range of initial guesses. By iteratively linearizing the function to be minimized in the neighborhoods of the current estimate, the LM algorithm involves the solution of linear system known as the normal equations. Considering that the normal equations are solved repeatedly in the course of the LM algorithm. Each computation of the solutions to a dense linear system has complexity O(N 3 ) in the number of parameters, it is clear that general purpose implementation of the LM algorithm are computationally very demanding when employed to minimize functions depending on many parameters. Fortunately, when solving minimization problems arising in Bundle Adjustment, the normal equations matrix has a sparse block structure owing to the lack of interaction among parameters for different 3D points and cameras. Therefore, considerable computational benefits can be gained by developing a tailored sparse variant of the LM algorithm which explicitly takes advantage of the normal equations zeros pattern. 7.2 Levenberg - Marquardt Algorithm The Levenberg-Marquardt (LM) algorithm is an iterative technique that locates the minimum of a multivariate function that is expressed as the sum of

9 3D Reconstruction on GPU: A Parallel Processing Approach 9 squares of non-linear real-valued functions. It has become a standard technique for non-linear least-squares problems, widely adopted in a broad spectrum of disciplines. LM can be thought of as a combination of steepest descent and the Gauss-Newton method. When the current solution is far from the correct one, the algorithm behaves like a steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Gauss-Newton method. The equation for Gauss-Newton method is given by LM equation is given by: J T Jδ f = J T ε (J T J+µI)δ f = J T ε The strategy of altering the diagonal elements of J T J is called damping and µ 0 is referred to as the damping term. If the updated parameter vector [p+δ f ] with [δ f ] computed from Eq 2 leads to a reduction in the error µ, the update is accepted and the process repeats with a decreased damping term. Otherwise, the damping term is increased, the augmented normal equations are solved again and the process iterates until a value of δ f that decreases error is found. The process of repeatedly solving Eq. 2 for different values of the damping term until an acceptable update to the parameter vector is found corresponds to one iteration of the LM algorithm. In LM, the damping term is adjusted at each iteration to assure a reduction in the error If the damping is set to a large value, matrix (J T J+µI) in Eq.2 is nearly diagonal and LM update step δ f is near the steepest descent direction. Moreover, the magnitude of δ f is reduced in this case. Damping also handles situations where the Jacobian is rank deficient and J T J is therefore singular. In this way, LM can defensively navigate a region of the parameter space in which the model is highly nonlinear. If the damping is small, the LM step approximates the exact quadratic step appropriate for a fully linear problem. LM is adaptive because it controls its own damping: it raises the damping if a step fails to reduce ; otherwise it reduces the damping. In this way LM is capable to alternate between a slow descent approach when being far from the minimum and a fast convergence when being at the minimum s neighborhood. The LM algorithm terminates when at least one of the following conditions is met: The magnitude of the gradient of ǫ T ǫ, i.e, J T ǫ in the right hand side of Eq. 1, drops below a threshold ǫ f The relative change in the magnitude of δ f drops below a threshold. The error ǫ T ǫ drops below a threshold ǫ 3 A maximum number of iteratopms k max is completed 7.3 Sparse Bundle Adjustment This section explains how can sparse variant of LM algorithm to deal efficiently with the problem of bundle adjustment. Assume that n 3D points are seen in

10 10 3D Reconstruction on GPU: A Parallel Processing Approach m views and let 000 be the projection of the 000 point on the image j. Bundle adjustment amounts to refining a set of parameters that most accurately predict the locations of the observed n points in the set of the m available images. More formally, assume that each camera j is parameterized by a vector 000 and each 3D point i by a vector 000. BA minimizes the reprojection error with respect to all 3D point of the camera parameters. n m min P,X i=1 j=1 d(q(p j, X j ), x ij ) 2 Q(P j,x j ) is the predicted projection of point i on image j. d(x,y) denotes the Euclidean distance between the inhomogeneous image points represented by x and y. Here P is the 3D point and M is the camera matrix. For a typical sequence of 20 views and 2000 points, a minimization problem in more than 6000 variables has to be solved. A straight-forward computation is obviously not feasible. However, the special structure of the problem can be exploited to solve the problem much more efficiently. The 3D points, camera points are independent of each other. This results in sparse matrices. Here, The J matrix is given by (for 3 camera views and 4 world points): Let A ij = δxij δa j Where x ij is the measured value of projection of ith 3D point on jth image and a j are the parameters of jth camera/image. A B A 12 0 B A 13 B A B A B J = 0 0 A 23 0 B A B A B A B 33 0 A B 41 0 A B A B 43 Here, A 11 etc. can be calculated parallely. For, calculation of J T J: Let: U j = 4 x=1 AT ij AT ij, V i= 3 j=1 BT ij B ij, W ij =A T ij B ij

11 3D Reconstruction on GPU: A Parallel Processing Approach 11 U W 11 W 21 W 31 W 41 0 U 2 0 W 12 W 22 W 32 W U 3 W 13 W 23 W 33 W 43 J T J = W11 T W 12 T W 13 T V W21 T W 22 T W 23 T 0 V W31 T W32 T W33 T 0 0 V 3 0 W41 T W42 T W43 T V 4 Note again that this matrix is build for 3 cameras and 4 world coordinates. The size of this square matrix is = 7*(no of cameras)+3*(no of Images) So for this particular case, the size of the matrix is = (7*3+3*4)x(3*4+7*4) After calculating the J T J parallely, we can solve for δ camera and δ images using shur complement and cholesky factorisation. And hence updating the damping parameter before each iteration. References 1. BROWN, M., AND LOWE, D Unsupervised 3d object recognition and reconstruction in unordered datasets. In Proc. Int. Conf. on 3D Digital Imaging and Modelling, Wei Wang, Hung Tat Tsui. A SVD Decomposition of Essential Matrix with Eight Solutions for the Relative Positions of Two Perspective Cameras 3. Noah Snavely, Steven M. Seitz, Richard Szeliski. Modeling the World from Internet Photo Collections 4. E Mouragnon, Mazime Lhuiller, Michel Dhome, Fabien Dekeyser, Patrick Sayd. 3D reconstruction of complex structures with bundle adjustment: an incremental approach 5. M Goesele, Noah Snavely. Multi-View Stereo for Community Photo Collections. 6. Noah Snavely, Steven M Seitz, Richard Szeliski. Photo Tourism- Exploring Photo Collection in 3D