3D Accuracy Improvement from an Image Evaluation and Viewpoint Dependency. Keisuke Kinoshita ATR Human Information Science Laboratories

Transcription

1 3D Accuracy Improvement from an Image Evaluation and Viewpoint Dependency Keisuke Kinoshita ATR Human Information Science Laboratories Abstract In this paper, we focus on a simple but important problem: given the coordinates of 3D points and their uncertainties, how much can we improve them provided there is a new set of corresponding image points. This is difficult because 3D points, image points and camera parameters, as well as their uncertainties, are closely coupled to each other. We fully model and update these uncertainties by utilizing constraints found in the projectivity between the 3D world and 2D image plane. The updated, improved uncertainties, represented as covariance matrices, can be used as a goodness measure of reconstructed 3D points. All systems that recover 3D scene information can profit from this approach. In particular, this measure is used for an active vision system that automatically moves the camera to the best viewpoint for reconstructing an object shape. Experimental results with synthesized and real data are shown. 1 Introduction Camera calibration and 3D reconstruction have been central interests for many computer vision researchers. This paper combines the two problems into a single framework and provides the details of how to estimate the camera parameters and to improve the 3D reconstruction together with their accuracy measures. As shown in Fig. 1, assume a set of 3D points and their uncertainties as well as a set of corresponding 2D points and their uncertainties. We update the coordinates and the uncertainties of the 3D points such that in order to obtain more accurate estimates. The importance of accuracy evaluation in computer vision was introduced by Kanatani[1]. He Hikaridai Seika-cho Soraku-gun Kyoto Kyoto Japan proposed using statistically optimal estimation methods and evaluating accuracy bounds for many problems, including 3D reconstruction. However, he did not consider the correlation between estimated 3D points and camera parameters. Although [2] and [3] discuss error analysis of 3D projective reconstruction, they do not discuss the relationship of two reconstructed 3D points. Regarding 3D reconstruction from image sequences, a single cycle of the Kalman filtering methods[4][5] is comparable to the approach in this paper. Also, these authors assume that camera parameters and estimated 3D points are statistically independent. Consequently, the assumption of uncorrelated input data makes the problem seem easier to analyze and faster to calculate. In fact, none of these authors consider the existing stochastic relations, which we consider important for precise 3D point re-estimation and error description. Extending the theory of Kanatani, we model and utilize all stochastic relations to re-estimate 3D points and their error description. 3D points, corresponding 2D points, and camera parameters, i.e. the three entities of the problem, are not indepen- 3D information + uncertainty + =? Uncalibrated image More accurate 3D information Figure 1: The objective of this paper is to find out how much we can improve the uncertainties of 3D point data when an uncalibrated image is provided. Initial 3D point coordinates and image point coordinates as well as their uncertainties (covariance matrices) are given. 1

2 dent of each other. The relation that constrains them is the projectivity between the 3D world and an image plane through a camera. We fully utilize this constraint to improve the accuracies 1.Reestimation is optimal and the optimality is inferred from Kanatani[1]. Our aim is to utilize this approach in the optimal camera pose prediction component of an active vision system. To verify the advantage of our proposed method, we estimate the best suitable camera pose for the reconstruction of the object shape, together with its improved 3D point estimations. From experiments, we found that the accuracy differs to the power of two between the best and the worst viewpoints. In section 2, after formulating the problem in a standard manner, the simultaneous estimation of 3D points and camera parameters under a strong correlation is shown. To maintain legibility, most of the equations are described in the appendix. Section 4 shows experiments on the method, and section 5 shows an application to an active vision system that seeks the best viewpoint for the 3D reconstruction task. Section 6 concludes the paper. 2 Problem Formulation Let N points in 3D space be X α, α =1,,N. A perspective camera projects a point X α onto an image point x α. Let the coordinates of X α and x α be X α =(X α,y α,z α, 1) T and x α =(x α,y α, 1) T,respectively. The coordinates X α and x α are related by the 3 4 camera projection matrix P =(p ij ): x α P X α. (1) Reformulating (1), we get a simple form of where α =1,,N, (a (i), u) =, (i =1,, 2N), (2) a (2α 1) = ( T X α T x α X α ) T, a (2α) = ( T X α T y α X α ) T, u = ( p 11 p 34 ) T and (, ) represents the inner product of two vectors. All subsequent calculations are derived from this basic constraint. Let the true coordinates of X α and x α be X α and x α, respectively. Observed data X α and x α,which are corrupted with noise, are given as where X α and x α are noise terms. A definition of the problem to be solved in this paper is stated as follows: Given observed data X α and covariance matrices V [X α, X β ], for α, β = 1,, N as well as independently estimated image points x α, optimally re-estimate ˆX α and their covariance matrices V [ ˆX α, ˆX β ]. Furthermore, estimate the optimal camera parameters û (see Fig. 2). It is important to note that noise X α and X β arecorrelatedevenwhenα β. Thisisparticularly true if our proposed method is iteratively applied. EveninthecaseofV [X α, X β ]=(α β), the updated estimates ˆX α and ˆX β become correlated when the uncertainties of the estimated camera parameters are considered. On the other hand, the image noise x α is assumed to be uncorrelated. Because methods that find feature points are usually based on local operators, therefore, correlations among the image points are negligible. In general cases, the noise contained in x α is not known. However, as the noise level of x α can be easily estimated from a by-product of other estimations, such as estimation of the fundamental matrix, we assume that the variance V [x α ]isknown. The problem stated above can be solved by extending Kanatani s optimal estimation theory [1] by applying it to (2). For the rest of the section, we analyze (2) instead of directly handling X α or x α. a (i) and their covariance matrices V [a (i), a (j) ]for i, j =1 2N, which can be derived from X α, x α and their covariance matrices, are the input data to be used in (2). Let the true camera parameters be u. a ( ) and u should satisfy (2). The problem can be restated, when using estimations a ( ) and their covariance matrices V [a (i), a (j) ], as finding â ( ) and û that strictly satisfy(2) as a pair and estimating their covariance matricesv [â (i), â (j) ]andv[û]. Estimation of covariance matrices V [â (i), â (j) ], which earlier work neglected, is novel in this paper. This problem can be interpreted as an optimization of the solution to (2). However, to make the structure of the problem clearer, we tentatively divide the problem into two parts: estimation of camera parameters and update of the points 3D positions. First, we estimate the camera parameter û and its covariance matrix V [û] that satisfy (2). It is known that minimization of (3) gives an optimal estimation. X α = X α + X α, x α = x α + x α, 1 More theoretical and general analysis is found in [6] J[u] = W (u) () (a (k), u)(a (l), u), (3) 2

3 3D data Camera pose Image data Figure 2: Camera parameters u are estimated from observed 3D point data X α and V [X α, X β ], image points and their covariance matrices. Simultaneously, the positions of the 3D data are updated such that the projection relation exactly holds where more accuracy is expected. Figure 3: Observed a (i), which is the sum of the true a (i) and the noise term a (i), is corrected to an updated â (i) such that it satisfies the projection relation. The covariance matrix V [â (i) ] of the updated â (i) is also calculated. where W (u) () is defined according to (W (u) () )=((u,v[a (k), a (l) ]u)). Now we impose the constraints (2) on the input data a ( ). That is, a ( ) should be corrected to exactly satisfy (2) together with û, as shown in Fig. 3. Let the correction be a (i) and the updated data be â (i). Then, â (i) = a (i) a (i), (4) a (i) = W () (a (l), û) V [a (i), a (k) ] û.(5) The pair â ( ) and û satisfy (2). After a long derivation, the covariance matrix of û is calculated as follows. V [û] =(M), M = W () a (k) a (l)t. In reality, we substitute W (), a ( ) with their optimal estimations W () = W (u) ()) and a ( ) = â ( ). The final step is to estimate covariance matrices V [â (i), â (j) ]. The details of the derivation are found in the appendix. Note that even when V [a (i), a (j) ]=(i j), V [â (i), â (j) ] becomes nonzero. This justifies our assumption that the correlation among the input data should not be omitted. 3 Experiments 3.1 Synthesized data Thirty-two points positioned in grid form are used for input 3D points. Image point data is synthe- sized from these 3D point coordinates and predefined camera parameters. Uncorrelated noise of the standard deviation of 1 pixel is added to each image point. The two images used in the experiments are shown in Fig. 4(a) and (b). For each X α, covariance matrix V [X α ]issetto V [X] = (6) Covariance matrices V [X α, X β ] between 3D points are preset to. The upper left 3 3 submatrix of the covariance matrix V [ ˆX α ] 3 3 corresponds to the uncertainty of the 3D point, namely, an error ellipsoid. The performance of the re-estimation can be measured by comparing the volume of the two error ellipsoids that correspond to V [X α ] 3 3 and V [ ˆX α ] 3 3. Re-estimation of V [X α ]ofα = 1, for example, withahelpoftheimageshowninfig.4(a)gives V [ ˆX 1 ] 3 3 = (7) The volume of the error ellipsoid that corresponds to V [ ˆX 1 ] 3 3 is only 21% of that of V [X 1 ] 3 3.For each point α =1,, 32, the s of the two ellipsoids are plotted in Fig. 5. The volume ratios are in the range of.1 to.3. The covariance matrices between two points, which were uncorrelated before, now become correlated. The correlation between points 1 and 2, 3

4 (a) (b) Figure 4: Images of 32 points used in the simulation. for example, is V [ ˆX 1, ˆX 2 ] 3 3 = (8) Next, to this correlated point data, we added another image depicted in Fig. 4(b) for further update. The variance matrix of point 1, for example, now becomes, V [ ˆX 1 ] 3 3 = (9) A comparison of (6), (7) and (9) shows that the error ellipsoid become.21 times smaller in the first update and then.61 times smaller in the second update,.13 times smaller in total. Again, Fig. 5 shows the error ratios of this second update procedure. To verify that the estimated covariance matrix is correct, 5 trials changing noise X 1 and x 1 according to their preset covariance matrices are plotted in Fig. 6. The figure reveals that all of the plotted points are inside the 3σ error ellipsoid. Therefore, the covariance matrix is estimated correctly. Note that the distribution of the updated 3D points is statistically unbiased with respect to the noise. Similar results can be obtained for any other point X α. These results empirically show that our estimation method is statistically optimal and unbiased as well. 3.2 Real data Data obtained from a vision system [7] is used. A box-shaped object with 31 points on its surface is used in the experiment. The 3D positions and their error ellipsoids are shown in Fig. 7(a). For simplicity, the covariance matrices between the points are not shown in the figure. Fig. 7(b) shows a set of corresponding image points used in the experiment. Fig. 7(c) shows the updated 3D points to- Figure 5: Volume ratios of the error ellipsoids that correspond to the covariance matrix of the 3D estimation. The s are.1 to.3 in the first update stage and.3 to.6 in the second update stage. Figure 6: Given true point and noise, updated 3D point ˆX 1 is estimated for 5 times. The results are superimposed on the 3σ error ellipsoid of the estimated covariance matrix V [ ˆX 1 ], which is centered at the true position X 1. Note that almost all trials of ˆX 1 are distributed inside the ellipsoid. gether with their error ellipsoids as well as the opticalaxisofthecamerashownwithanarrow. Aswe can be seen, the error ellipsoids remarkably shrink after the update procedure. Intuitively, the error ellipsoids should have larger errors along the optical axis of the camera in which depth information degenerates. However, the results show that for this experiment that assumption is not true. This is because the 3D points have different contributions to the estimation of the camera parameters as well as to their own updates. 4

5 + = (a) (b) (c) Figure 7: (a) 3D object points and their covariance matrices. (b) An image taken from an uncalibrated camera. This image is used to update the 3D data and to estimate the camera parameters. (c) Updated 3D points shown with their variance matrices. The volumes of the error ellipsoids are notably decreased. The optical axis of the camera is indicated by the arrow. 4 Application to an Active Vision System Our method can be applied to a system that seeks the best viewpoint for 3D reconstruction tasks. Iteratively moving the camera to the best pose should give 3D positions of the object points to be improved step by step. Since it is not realistic to find the camera pose that minimizes the covariance matrices analytically, we compute the covariance matrices of all possible poses and find their minimum. We place a virtual camera distributed around the object on a sphere with its radius set to the distance between the camera and the object and then compute the covariance matrices of the point accuracy for each pose. The criterion to be evaluated is the total volume of the error ellipsoids. Here, the possible camera poses are parameterized in spherical coordinates θ and φ with a range of 18 <θ<18 and 9 <θ<9 divided into 1 steps. We used two objects, a cube with 16 points and a L-shaped object with 16 points. The two object shapes have apparent symmetries, and this property may reveal the importance of the selection of viewpoints for the task. Fig. 9 and Fig. 1 show the viewpoint dependencies of the accuracy of the updated point positions. The small points on Fig. 9(a) and Fig. 1(a) show the object points with their uncertainties. Fig. 9(b) and Fig. 1(b) show the error volume reduction ratio with respect to the viewpoints. The ratios vary between.2 to.28 with respect to the viewpoints. The larger surface object in Fig. 9(a) is the projection of Fig. 9(b) regarding the viewpoints. This shows that when the viewpoints are from the corners, the error is smaller. When the viewpoints are perpendicular to the facets, the ratios are larger, which means it is not a good pose for camera placement. The same explanation applies to Fig. 1(a) with the L-shaped object. It is preferable to place the camera from the corners than from a pose parallel to the plane where the points lie. Fig. 9(c) and Fig. 1(c) show how much the error was reduced after updates from four consecutive best camera poses. The camera is placed at the best pose each time it observes the object. The total error ratios dropped from.2 to.1. In fact, the differences among the viewpoints are not larger than expected. However, to show the difference more distinctively, the errors between the best pose and the worst pose are compared. Fig. 11(a) and (b) show the viewpoint dependencies between the best three consecutive camera poses and the worst three consecutive camera poses. The errors ellipsoids grow two to three times in volume as depicted in Fig. 9(d) and Fig. 1(d). This demonstrates the importance of choosing a good viewpoint for object reconstruction. 5

6 Z φ X θ Y Figure 8: A virtual camera is placed on a sphere parameterized by θ and φ. For each camera pose, an image of the object is synthesized and used to update the 3D point accuracy. (a) "count.max.mat" 5 Conclusions In this paper, a method to update a set of 3D points by using a set of corresponding 2D points was proposed. The stochastic relations are fully modeled in our method: the correlation between the 3D points, the error propagation from the 3D points to the camera parameters, and that from the camera parameters back to the 3D points. The constraints of projectivity through the camera was utilized to obtain an optimal re-estimation of 3D points. This optimality was empirically shown in the experiments on synthesized data. The error ellipsoids can be applied to a vision system that requires an intelligent camera motion. For example, the camera can be guided to the pose where the volume of the error ellipsoids are minimum. This saves time and improves the accuracy of the 3D reconstruction (b) 6 (c) "count.max.mat" "count1.max.mat" "count2.max.mat" "count3.max.mat" Acknowledgements The author thanks Dr. Martin Tonko, who also worked at ATR Human Information Processing Research Labs., for fruitful discussions and Mr. Keiichi Sumita for helping with the experiments. The research reported here was supported in part by a contract with the Telecommunications Advancement Organization of Japan entitled, Research on Human Communication. (d) Figure 9: Cube experiments. (a) 16 points on a cube and a plot of viewpoint dependency of the 3D point accuracy. (b) Viewpoint dependency of the error. (c) Plots of error updated four times. (d) Error volume plot updated from three consecutive best camera poses. 6

7 "count4.max.mat" "count4.min.mat" (a) (a) "count.max.mat" "count4.max.mat" "count4.min.mat" (b) (b) "count.max.mat" "count1.max.mat" "count2.max.mat" "count3.max.mat" Figure 11: Comparison of plots of error volume after three consecutive best camera poses (upper plot) and those after three consecutive worst camera poses (lower plot). (a) Cube object. (b) L-shaped object. The accuracies differ more than 2 times in both cases. (c) (d) Figure 1: L-shaped object. Figures (a) to (d) correspond to those of Fig. 9 References [1] Kenichi Kanatani. Statistical Optimization for Geometric Computation: Theory and Practice. Elsevier Science, [2] Gabriella Csurka, Cyril Zeller, Zhengyou Zhang, and O. D. Faugeras. Characterizing the uncertainty of the fundamental matrix. Computer Vision and Image Understanding, 68(1):18 37, [3] N. Georgis, M. Petrou, and J. Kittler. Error guided design of a 3D vision system. IEEE Trans. Pattern Analysis and Machine Intelligence, 2(4): , [4] P. A. Beardsley, A. Zisserman, and D. W. Murray. Sequential updating of projective and affine structure from motion. International Journal of Computer Vision, 23(3): , [5] Q. T. Luong and O. D. Faugeras. Self-calibration of a moving camera from point correspondences and fundamental matrices. International Journal of Computer Vision, 22(3): , [6] Kenichi Kanatani and Keisuke Kinoshita. On covariance update by geometric constraints(in Japanese). In IPSJ SIG Notes Vol.CVIM , pages , 22. [7] Martin Tonko and Keisuke Kinoshita. On the integration of point feature acquisition, tracking and uncalibrated metric reconstruction. In SPIE Conference on Intelligent Robots and Computer Vision, pages ,

8 A Details of V [a (i), a (j) ] V [a (i), a (j) ] can be in eight forms depending on i and j. Here, we use V [b (2α 1), b (2β) ] as one example of them. If α β, V [a (2α 1), a (2β 1) ] V [Xα, X β] 4 x β V [X α, X β ] x αv [X α, X β ] 4 x αx β V [X α, X β ] and if α = β, V [a (2α 1), a (2α 1) ] V [Xα] x αv [X α] 4 x αv [X α] 4 x 2 αv [X α]+v [x α](x αx T α + V [X α]) 1 A, 1 A. The true values X α,x α are substituted with good estimations of X α and x α. The other combinations of i and j, V [a (2α 1), a (2β) ], V [a (2α), a (2β 1) ] and V [a (2α), a (2β) ], can be calculated similarly. B Details of V [â (i), â (j) ] The covariance matrices after the update, which are denoted V [â (i), â (j) ], can be rewritten as V [â (i), â (j) ]=E[(a (i) a (i) )(a (j) a (j) ) T ] = E[ a (i) a (j)t ] E[ a (i) a (j)t ] E[ a (i) a (j)t ]+E[ a (i) a (j)t ]. (1) To compute V [â (i), â (j) ], covariance matrices of a ( ) and a ( ) have to be determined. As a first step, we rewrite a ( ) using a ( ),whose covariance matrix is already known. a (i) = W () (a (l), û) V [a (i), a (k) ] û. (11) However, a ( ), a ( ) and also û have strong correlations among them. We substitute a (i) = a (i) + a (i), (12) û = u + u (13) into (11) and compute the first-order approximation with respect to a (i), u. Usingtherelation (a ( ), u) =, we get 2NX a (i) W () V [a (i), a (k) ] {(a (l), u)i 12 +u a (l)t } u 2NX + W () V [a (i), a (k) ] u u T a (l). Now, a (i) is described using u and a ( ),which are known. Furthermore, we get ([1](7.26)), u = M 2N W () M M a (k) û T a (l) = V [û] W () a (k) û T a (l). Finally, u is rewritten using a ( ).Let G (i) 1 = 2NX W () V [a (i), a (k) ]{(a (l), û)i 12 + u a (l)t }, G (i,l) 2 = { X k W () ( G (i) 1 V [û]a(k) + V [a (i), a (k) ]u)}u T then, a (i) can be rewritten in relation to a (l) as a (i) = G (i,l) 2 a (l). (14) l Substituting (14) into (1), V [â (i), â (j) ]=V [a (i), a (j) ] X l ( X n G (j,n) 2 V [a (n), a (i) ]) T + X l,n G (i,l) 2 V [a (l), a (j) ] G (i,l) 2 V [a (l), a (n) ]G (j,n) T 2, where we use V [a (l), a (j) ] = V [a (l), a (j) ], V [ a (l), a (n) ]=V [a (l), a (n) ]andsoon. To summarize, V [â (i), â (j) ] is computed by using V [a ( ), a ( ) ], V [û], a ( ),andu. a ( ), u is substituted by their optimal estimations â ( ), û. 8