Collinearity and Coplanarity Constraints for Structure from Motion

Transcription

1 Coinearity and Copanarity Constraints for Structure from Motion Gang Liu 1, Reinhard Kette 2, and Bodo Rosenhahn 3 1 Institute of Information Sciences and Technoogy, Massey University, New Zeaand, Department of Computer Science, The University of Auckand, New Zeaand 2 Max Panck Institute Saarbrücken, Germany Abstract. Structure from motion (SfM) comprises techniques for estimating 3D structures from uncaibrated 2D image sequences. This work focuses on two contributions: Firsty, a stabiity anaysis is performed and the error propagation of image noise is studied. Secondy, to stabiize SfM, we present two optimization schemes by using a priori knowedge about coinearity or copanarity of feature points in the scene. 1 Introduction Structure from motion (SfM) is an ongoing research topic in computer vision and photogrammetry, which has a number of appications in different areas, such as e-commerce, rea estate, games and specia effects. It aims at recovering 3D (shape) modes of (usuay rigid) objects from an (uncaibrated) sequence (or set) of 2D images. The origina approach [5] of SfM consists of the foowing steps: (1) extract corresponding points from pairs of images, (2) compute the fundamenta matrix, (3) specify the projection matrix, (4) generate a dense depth map, and (5) buid a 3D mode. A brief introduction of some of those steps wi be presented in Section 2. Errors are inevitabe to every highy compex procedure depending on reaword data, and this aso hods for SfM. To improve the stabiization of SfM, two optimizations are proposed using information from the 3D scene; see Section 3. Section 4 presents experimenta resuts, and Section 5 concudes the paper with a brief summary. 2 Modues of SfM This section gives a brief introduction for some of the SfM steps (and reated agorithms). For extracting correspondent points, we reca a method proposed in [14]. Then, three methods for computing the fundamenta matrix are briefy introduced. To specify a projection matrix from a fundamenta matrix, we describe two common methods based on [3, 4]. In this step we aso use the knowedge of intrinsic camera parameters, which can be obtained through Tsai caibration [12]; this caibration is performed before or after taking the pictures for the used

2 camera. It aows to specify the effective foca ength f, the size factors k u and k v of CCD ces (for cacuating the physica size of pixes), and the coordinates u 0 and v 0 of the principa point (i.e., center point) in the image pane. 2.1 Corresponding points We need a number of at east seven pairs of corresponding points to determine the geometric reationship between two images, caused by viewing the same object from different view points. One way to extract those points from a pair of images is as foows [14]: (i) extract candidate points by using the Harris corner detector [2], (ii) utiize a correation technique to find matching pairs, and (iii) remove outiers by using a LMedS (i.e., east-median-of-squares) method. Due to the poor performance of the Harris corner detector on specuar objects, this method is normay not suitabe. 2.2 Fundamenta and Essentia matrix A fundamenta matrix is an agebraic representation of epipoar geometry [13]. It can be cacuated if we have at east seven correspondences (i.e., pairs of corresponding points), for exampe using inear methods (such as the 8-Point Agorithm of [8]) or noninear methods (such as the RANSAC Agorithm of [1], or the LMedS Agorithm of [14]). In the case of a inear method, the fundamenta matrix is specified through soving an overdetermined system of inear equations utiizing the given correspondences. In the case of a noninear method, subsets (at east seven) of correspondences are chosen randomy and used to compute candidate fundamenta matrices, and then the best is seected, which causes the smaest error for a the detected correspondences. According to our experiments, inear methods have a more time efficient and provide reasonaby good resuts for arge (say more than 13) numbers of correspondences. Noninear methods are more time consuming, but ess sensibe to noise, especiay if correspondences aso contain outiers. For given intrinsic camera parameters K 1 and K 2, the Essentia matrix E can be derived from F by computing E = K T 2 F K Projection matrix A projection matrix P can be expressed as foows: P = K[R RT ] where K is a matrix of the intrinsic camera parameters, and R and T are the rotation matrix and transation vector (the extrinsic camera parameters). Since the

3 intrinsic parameters are specified by caibration, reative rotation and transation can be successfuy extracted from the fundamenta matrix F. When recovering the projection matrices in reference to the first camera position, the projection matrix of the first camera position is given as P 1 = K 1 [I 0], and the projection matrix of the second camera position is given as P 2 = K 2 [R RT ]. The method proposed by Hartey and Zisserman for computing rotation matrix R and transation vector T (from the essentia matrix E) is as foows [3]: 1. compute E by using E = K2 T F K 1, where K i = fk u 0 u 0 0 fk v v (note: K 1 = K 2 if we use the same camera at view points 1 and 2), 2. perform a singuar vaue decomposition (SVD) of E by foowing the tempate E = Udiag(1, 1, 0)V T, 3. compute R and T (for the second view point), where we have two options, namey R 1 = UW V T R 2 = UW T V T T 1 = u 3 T 2 = u 3 where u 3 is the third coumn of U and W = Another method for computing R and T from E (aso ony using eementary matrix operations) is given in [4], which eads to amost identica resuts as the method by Hartey and Zisserman. 2.4 Dense depth map At this point, the given correspondences aow ony a few points to be reconstructed in 3D. A satisfactory 3D mode of a pictured object requires a dense map of correspondences. The epipoar constraint (as cacuated above) aows that correspondence search can be restricted to one-dimensiona epipoar ines, it supports that images are at first rectified foowing the method in [10], and that correspondence matching is then done by searching aong a corresponding scan ine in the rectified image. We aso require a recovered base ine between both camera positions to cacuate a dense depth map. 3 Optimization with Prior Knowedge Since computations of fundamenta and projection matrix are sensitive to noise, it is necessary to appy a method for reducing the effect of noise (to stabiize SfM). We utiize information about the given 3D scene, such as knowedge about coinearity or copanarity.

4 3.1 Knowedge about coinearity It is not hard to detect coinear points on man-made objects, such as buidings or furniture. Assuming idea centra projection (i.e., no ens distortion or noise), then coinear points in object space are mapped onto one ine in the image pane. We assume that ens distortions are sma enough to be ignored. Linearizing points which are supposed to be coinear can then be seen as a way to remove noise. Least-square ine fitting (minimizing perpendicuar offsets) is used to identify the approximating ine for a set of noisy coinear points. Assume that we have such a set of points P = {(x i, y i ) i = 1,..., n} which determines a ine (α, β, γ) = αx + βy + γ. The coefficients α, β and γ are cacuated as foows [7]: α = µ xy µ 2 xy + (λ µ xx ) 2 λ µ xx β = µ 2 xy + (λ µ xx ) 2 γ = (αx + βy) where λ = 1 2 (µ xx + µ yy (µ xx µ yy ) 2 + 4µ xy ) µ xy = 1 n 1 µ xx = 1 n 1 n i=1 (x i x) 2, µ yy = 1 n 1 n (x i x)(y i y), x = 1 n i=1 n (y i y) 2 i=1 n x i and y = 1 n After specifying the ine, the points positions are modified through perpendicuar projection onto the ine. i=1 n i=1 y i 3.2 Knowedge about copanarity Copanar points can be expected on rigid structures such as on was or on a tabetop. For a set of points, a incident with the same pane, there is a 3 3 matrix H caed homography which defines a perspective transform of those points into the image pane [11]. Homography Consider we have an image sequence (generaizing the two-image situation from before) and p ki is the projection of 3D point P i into the kth image, i.e. P i is reated to p ki as foows: p ki = ω ki K k R k (P i T k ) (1)

5 where ω ki is an unknown scae factor, K k denotes the intrinsic matrix (for the used camera), and R k and T k are the rotation matrix and transation vector. Foowing Equation (1), P i can be expressed as foows: P i = ω 1 ki R 1 k Simiary, for point p i ying on the th image, we have P i = ω 1 i From Equations (2) and (3), we get p ki = ω ki K k R k (ω 1 i K 1 k p ki + T k (2) R 1 p i + T (3) R 1 p i + T T k ) (4) With R k = R k R 1 we define Hk = K k R k. We aso have epipoe e k = K k R k (T T k ). Equation (4) can then be simpified to p ki = ω ki ω 1 i (H k p i + ω i e k ) (5) H k is what we ca the homography which maps points at infinity (ω i = 0) from image to image k. Consider a point P i on pane n T P i d = 0. Then, from Equation (3), we have Then we have n T P i d = n T ω 1 i what can be rewritten as foows: R 1 p i + n T T d = 0 ω i = nt R 1 ω i = d 1 p i d n T T n T R 1 p i where d 1 = d n T T is the distance from the camera center (principa point) of the th image to the pane ( n, d). Substituting ω i into Equation (5), finay we have p ki = ω ki ω 1 i (Hk + d 1 e k n T R 1 )p i Let H = ω ki ω 1 i (H k + d 1 e k n T R 1 ) This means: points ying in the same pane have identica H which can be utiized as copanarity constraint; see [11]. Copanarity optimization Copanar points satisfy the reation described by homography. We use this reation for modifying noisy copanar points, using the foowing equation: p ki = H k p i Here, H k is the homography between kth and th image in the sequence, and p ki, p i are projections of point P i on the kth and th image, respectivey.

6 4 Experiments and Anaysis To anayze the infuence of noise, we perform SfM in a way as shown in Figure 1. At Step 1, Gaussian noise is introduced into coordinates of detected correspondences. At step 2, three different methods are compared to specify which one is the best to compute the fundamenta matrix. At Step 3, a quantitative error anaysis is performed. Fig. 1. Basic steps of SfM. This section shows at first experiments of the performance of different methods for computing the fundamenta matrix, and second the effect of those optimizations mentioned in the previous section. 4.1 Computation of fundamenta matrix Three agorithms (8-Point, RANSAC and LMedS) are compared with each other in this section. To specify the most stabe one in presence of noise, Gaussian Noise (with mean 0 and deviation δ = 1 pixe) and one outier are propagated to given correspondences. Performances of the three agorithms are characterized in Figure 2: due to the outier, the 8-Point Agorithm is more sensibe than the other two. 4.2 Optimizations To test the effect of the optimizations mentioned in the previous section, the resuts of spitting essentia matrices (rotation matrices and transation vectors)

7 Fig. 2. Performance of three agorithms in presence of noise. are utiized to compare with each other. Two images of a caibration object are used as test images (shown in Figure 3). The data got from caibration (intrinsic parameters and extrinsic parameters of camera) are used as the ground truth. Ro ange α, pitch ange β and yaw ange γ are used to compare the rotation matrices in a quantitative manner. These anges can be computed from a rotation matrix R by foowing equations [9]: r23 r13 α = atan2( sin(γ), sin(γ) ) r31 r32 β = atan2( sin(γ), sin(γ) ) p 2 + r2, r ) γ = atan2( r where rij is the eement of R at ith row and jth coumn, and atan( y ) (x > 0) y (πx atan( y )) (x < 0) x atan2(y, x) = y y π (y 6= 0, x = 0) y 2 undefined (y = 0, x = 0) Fig. 3. The first (eft) and second (right) candidate images.

8 Fig. 4. Errors in rotation matrices (eft) or transation vectors (right). First row: errors from non-noisy data. Second row: noisy data. Third or fourth row: errors from noisy data after optimization with coinearity or copanarity knowedge, respectivey.

9 Fig. 5. Epipoar ines resut from different data sets. The green ines (dash ines) from data without generated noise; the red ines (straight ines) are from noisy data; the bue ines (dash-dot ines) and yeow ines (dot ines) are from noisy data which has been optimized with coinearity and copanarity knowedge, respectivey. Since spitting the essentia matrix ony resuts in a transation vector up to a scae factor, a transation vectors (incude the ground true one) are transformed into a normaized vector (ength equa to one unit) to compare with each other in a quantitative manner. The comparison of rotation matrices and transation vectors are shown in Figure 4. The errors are mean error of ten times iteration when different number of correspondences are given. The noise propagated is Gaussian noise (with mean 0 and deviation δ = 1 pixe). The method used to compute the fundamenta matrix is the 8-Point Agorithm, which is more sensitive to noise than RANSAC and LMedS Agorithm. The method of Hartey and Zisserman is used to spit essentia matrix. According to the resuts shown in Figure 4, the copanarity knowedge gives a better optimization than coinearity knowedge. One possibe reason is that the coinearity optimization is performed on uncaibrated images, in which the true correation of coinear points are not stricty ying in a straight ine. For arbitrary images, the effect of optimizations can be seen from Figure 5 through ooking at reative positions of epipoar ines computed from different data sets. It shows that the two optimization strategies bring positive effects on

10 Fig. 6. Two different views of reconstructed points from the optimized SfM agorithm. Fig. 7. Trianguated surface mesh with textures reducing the infuence of noise, and the copanarity optimization performs better than the coinearity optimization. Figure 6 shows the reconstructed point coud of the CITR-buiding in Auckand and Figure 7 visuaizes the texture mapped surface mode. The main edges of the buiding are reconstructed with nearperfect 90 anges. Sight image noise (ess then 1 pixe) aready eads to anges between 20 and 140 which indicates the sensitivity of cassic SfM approaches. By incorporating the coinearity and copanarity constraints, the reconstruction quaity improved. 5 Summary Modues reating to structure from motion have been discussed in this paper. According to experiments, structure from motion is sensitive to noise and it is necessary to improve its stabiity. Two optimizations, using coinearity and copanarity knowedge, have been proposed, and the reating experiments show that the two proposed optimizations, especia the copanarity one, bring positive effects on reducing infuences of noise.

11 Acknowedgments The authors thank Danie Grest and Kevin Koeser from the University of Kie for the BIAS-Library and usefu hints. References 1. M. Fischer and R. Boes. Random sampe consensus: a paradigm for mode fitting with appications to image anaysis and automated cartography. Comm. ACM, 24: , C. Harris and M. Stephen. A combined corner and edge detector. In Proc. Avey Vision Conf., pages , R. Hartey and A. Zisserman. Mutipe View Geometry in Computer Vision. Cambridge University Press, New York, B. K. P. Horn. Recovering baseine and orientation from essentia matrix. ai.mit.edu/peope/bkph/papers/essentia.pdf, T. Huang. Motion and structure from feature correspondences: a review. Proc. IEEE, 82: , R. Kette, K. Schüns, and A. Koschan. Computer Vision Three-dimensiona Data from Images. Springer, Singapore, C. L. Lawson and R. J. Hanson. Soving Least Squares Probems. Prentice Ha, Engewood, H. Longuet-Higgins. A computer agorithm for reconstructing a scene from two projections. Nature, 293: , R. Murray, Z. Li, and S. Sastry. A Mathematica Introduction to Robotic Manipuation. CRC Press, Boca Raton, M. Poefeys, R. Koch, and L. van Goo. A simpe and efficient rectification method for genera motion. In Proc. Int. Conf. Computer Vision, pages , R. Szeiski and P. H. S. Torr. Geometricay constrained structure from motion: points on panes. In Proc. European Workshop 3D Structure Mutipe Images Large- Scae Environments, pages , R. Tsai. A versatie camera caibration technique for high-accuracy 3d machine vision metroogy using off-the-shef tv cameras and enses. IEEE J. Robotics and Automation, 3: , Z. Zhang. Determining the epipoar geometry and its uncertainty: a review. Int. J. Computer Vision, 27: , Z. Zhang, R. Deriche, O. Faugeras, and Q.-T. Luong. A robust technique for matching two uncaibrated images through the recovery of the unknown epipoar geometry. Artificia Inteigence J., 78:87 119, 1995.