A robust and convergent iterative approach for determining the dominant plane from two views without correspondence and calibration

Transcription

1 Proc. Computer Vision and Pattern Recognition (CVPR 97), pp , June 17-19, 1997, San Juan, Puerto Rico robust and convergent iterative approach for determining the dominant plane from two views without correspondence and calibration Pär Fornland Computational Vision and ctive Perception Num. nalysis and Comp. Science Royal Institute of Technology (KTH) S Stockholm, Sweden Christoph Schnörr B Kogs, FB Informatik Vogt-Koelln-Str. 30 Univ. Hamburg D Hamburg, Germany schnoerr@informatik.uni-hamburg.de bstract robust, iterative approach is introduced for finding the dominant plane in a scene using binocular vision. Neither camera calibration nor stereo correspondence is required. Recently Cohen formalized a framework guaranteeing (local) convergence of iterative two-step methods. In this paper, the framework is adopted, with a global step using tentative matches to estimate the planar projectivity, and a local step attempting to solve the stereo correspondence. detected point in the first image is matched to an auxiliary point in the second image, on the line joining the transformed first image point, and its closest detected second image point. Convergence is assured, while achieving robustness to both mismatching and non-coplanar points. 1. Introduction Since active robots change their visual attention by altering the camera parameters, they continuously require a recalibration. Therefore active robots cannot reconstruct any metric information about its surroundings, but it is still possible to extract useful, comparative information [6, 12], such as planarity. In fact, these measures can be more relevant for navigation than detailed metric information. In particular, coplanarity is important, since in-door environments abound with planar surfaces [14]. For example, advantageous to a robot is the capability of automatically detecting obstacles [2, 4, 5]; which means finding regions in the images with the dominant plane. The framework we consider is the use of binocular vision and point features. The fundamental matrix [10] relates points in the first image to corresponding points in the second image. In the context of planar surfaces, this relationship simplifies to a projective transformation with respect to the image coordinates [9, 11]. In [13] an approach was presented for finding planar regions in two images using point matches that are assumed to be found in a previous step. Five point matches are tested to be coplanar, and the planar projectivity between the two views is then calculated, which enables a prediction of positions from one image to the other. If sufficiently many actual and predicted points are close enough, an extended planar region is recursively found. In this paper, we address the problem of robustly finding points on the dominant plane from two images. The difference to previous work (e.g. [13]) is that we neither assume established correspondences nor camera calibration. Rather, the problems of matching corresponding points and estimating the corresponding planar projectivity are simultaneously addressed. In a related context, namely rigid registration of 3D points without given correspondence, iterative approaches have been reported [1, 15] that may be adopted to solve our task. However, since we also wish to apply robust estimators, we address the question of (local) convergence. Cohen [3] presented a general framework for a broad class of vision problems that has been solved by two-step methods. The key idea is to introduce auxiliary variables to guarantee at least local convergence. In the present paper, we adopt this framework to the problem of robustly estimating planar projectivities and point correspondences from stereo images. The method is a two-step iterative scheme. The first step predicts points from the first to the second image using the current estimate of the projective transformation. The closest point in the second image to each predicted point is then used to define auxiliary image points. Each such point is calculated by a linear interpolation between the closest point and the predicted point, and the weights depend on a

2 robust function of the distance between the two points. The second step returns an estimate of the projective transformation by a least squares approach fitting the points in the first image to the auxiliary points in the second image. The resulting transformation is then used for the first step again, until convergence is reached. The duality of points and lines in the projective plane implies that we can use the method presented here also for line coordinates, which is useful considering the abundance of straight lines in images of in-door environments. In the next section, we describe stereo geometry in sufficient detail, and how to estimate projectivities from matching points. In section 3 we describe properties of iterative two-step methods, and introduce the auxiliary variables. Section 4 is devoted to the experiments, testing convergence and stability properties of the method. We conclude the paper with a discussion. 2. Stereo geometry and projectivity estimation Two cameras at different positions generally assign different pixel coordinates to a fixed point in 3D space. The geometry of the camera set up causes various relationships between the pixel coordinates, such as the epipolar constraint [10]. In particular, each 3D plane gives arise to a projective transformation between the coordinates: x 2 = t 11 x 1 + t 12 y 1 + t 13 y 2 = t 21 x 1 + t 22 y 1 + t 23 (1) = t 31 x 1 + t 32 y 1 + t 33 where (x 1 ; y 1 ) and (x 2 ; y 2 ) are two points in the first and second images, respectively. Since we use homogeneous coordinates, any scalar 6= 0 is valid Estimating the projectivity By eliminating in (1), we obtain two equations, that are linear in elements of the projectivity matrix. t 11 x 1 +t 12 y 1 +t 13?t 31 x 1 x 2?t 32 y 1 x 2?t 33 x 2 = 0 t 21 x 1 +t 22 y 1 +t 23?t 31 x 1 y 2?t 32 y 1 y 2?t 33 y 2 = 0 Due to the linearity, we can estimate the matrix through multiple linear regression if we know the stereo correspondences. Since we can set the scale factor arbitrarily, we scale T to have unit Frobenius norm, that is, kt k F = 1, reducing the degrees of freedom to eight. full-rank estimation of these is possible using four matching points, but often we have many matching points, allowing us to solve the over-determined system by least square estimation. This is done by eigenvector analysis of the system matrix, using singular value decomposition. (2) 2.2. lgebraic versus Euclidean estimation Figure 1 depicts the performance of two different methods for estimating the projectivity from matching points. The first method applies linear regression to the equations in (2). The system of equations is solved by finding the smallest eigenvalue of the system matrix using singular value decomposition. The second method minimizes the average squared Euclidean distance (the prediction error) between the points in the second image, and the transformed points of the first image. This is a non linear problem, which however can be well approximated by iterating the linear method but using a weighted least squares approach instead. In the figure we show for both estimation procedures the prediction errors as a function of the standard deviation of the additive gaussian noise applied to the input point positions. The conclusion for this synthetic example is that the Euclidean estimation gives a better estimate of the projectivity for all noise levels. However, we also performed ad Figure 1. Prediction error plotted against the perturbation of input points, for the algebraic (solid) and Euclidean (dashed) distances. ditional experiments with projective transformations taken from real scenes, in which the projective transformation often is approximately affine. These experiments did not indicate a significant advantage for the Euclidean estimation over the algebraic estimation. The algebraic version is also favored by the computational savings that can be achieved, and is therefore used in the rest of this paper Sensitivity to projectivity perturbations Later in this paper we introduce methods that require an initial estimate of the projectivity matrix. Therefore it is important to consider how the projectivity changes qualitatively for small random perturbations of the matrix. If the transformed points move only moderately for slight perturbations, then we can allow minor errors in the initial matrix. The prediction error for perturbed projectivities was investigated by an experiment with a natural projectivity corresponding to the real scene in Fig. 7. Fifteen matching points on the floor were detected, and the projectivity matrix was

3 estimated from these. The points in the second image were then redefined to fit the estimated projectivity; this provided a convenient experimental set up with matching points in the two images and the projectivity perfectly transforming points between the images. Figure 2 shows how the prediction error increases with respect to perturbations of the projectivity matrix. For each noise level, we performed a series of 100 experiments. In each such experiment, the perturbation matrix was taken randomly, with a Frobenius norm equal to the desired noise level. The average prediction error was calculated as the sum of the squared prediction errors for all points, and in the figure we show the median value of all 100 average prediction errors as a function of the norm of the perturbation matrix. The experiment exhibits that even slight perturba Figure 2. Median prediction error plotted against projectivity perturbation. tions cause major prediction errors. For instance, if the allowed prediction error is 10 pixels, then the perturbation matrix should have a norm less than 5 10?6! The conclusion of this section is that small random changes of the projectivity matrix may considerably change the corresponding transformation. Therefore it is important to exploit available constraints to obtain a realistic initial projectivity estimate (see next section). Furthermore, this result emphasizes the necessity of being robust with respect to local mismatches, which is the objective of section Selecting a meaningful initial projectivity In the previous section we observed how small perturbations of the projectivity matrix may cause large changes of the transformation. By investigating the geometrical parameters defining the projectivity matrix, we can find realistic matrices by invoking constraints derived from the geometry. Often, we can assume that the interior calibration is equal in both cameras and approximated by a uniform scaling and a translation. The relative pose of the cameras can generally be modeled by a sideways translation and a rotation around the y-direction. The geometrical parameters thus defining the approximate projective matrix can be restricted to reasonable intervals by exploiting knowledge about the approximate camera set-up. These intervals can then be sampled at appropriate points, to find geometrically sound initial values of the projectivity matrix to be used in the method described later in this paper. 3. Convergent iterative joint-estimation of point correspondence and projectivity In the next section, we describe how to adopt the framework of current two-step iterative algorithms to our problem. The following section describes how we set up the joint-optimization problem in the present context of planar projectivities and point correspondences. In section 3.3 we extend the optimization problem by introducing auxiliary variables to preserve local convergence while including robust estimators. The impact of selected robust distances on the joint estimation procedure is discussed in section Iterative Closest Point Methods Many computer vision problems can be modeled as the joint minimization of an energy with respect to a number of unknown parameters. Several iterative algorithms has been presented in the literature that alternatively solve for some parameters, and then fix these parameters when solving for others. The convergence of the methods might not be guaranteed. typical example is the snake [8] algorithm which finds smooth curves enclosing regions in a grey level image by alternatively a) moving a number of reference points in the gradient direction and b) fitting a regularized curve that passes through the reference points. nother example is 3D registration where the aim is to find a transformation and a matching between two sets of 3D points. gain, this can be solved [1, 15] through a two-step algorithm. In this paper, we have a stereoscopic visual system, and the aim is to find the correspondence between points in two images, while simultaneously estimating the projective transformation between two sets of image points. We adopt the two-step procedure that we described above to this problem. The sets of points in the first and second images are defined as S 1 = fp 1 g and k S2 = fp 2 g, respectively. The k distance from any point p 0 in the second image to its closest point in the second set is referred to as d S 2 (p 0 ) and is defined through d S 2 (p 0 ) = d(p 0 ; S 2 ) = inf ; p 2 )g p 2 2S 2fd(p0 The method is initialized with a projective transformation H 0, and consists of two steps. Local step: Given a previous projectivity estimate H k, we define the points p 1 2 S 1 and p 2 2 S 2 to be matching, if p 2 is the closest point to the transformed point H k (p 1 ). Global step: The matrix representing the projectivity H k+1 is estimated from the matching done in the local step.

4 This two-step method is described in the following section as a joint energy minimization Energy minimization The iterative scheme that we presented above can be interpreted as the joint minimization of an energy with respect to both stereo matching and the projectivity. In principle, the performance can be improved by using robust estimators. But convergence is only assured if the distance measure used in the local step is the same as that induced by the norm used in the global estimation step, since then the same energy term is minimized in both steps. Therefore, convergence is not guaranteed [3] if a robust technique is used in one step. In the next section, we address this problem and apply the general framework developed in [3] to our stereoscopic setting. To enable the quantative evaluation of point matches in a variety of ways, a potential P (to be specified later on) is introduced for each point in the second image. The potential is a function of the distance d S 2 (p 0 ) defined above, that is, the potential is P (p 0 ) = f (d S 2 (p 0 )). Using these definitions, the energy that we wish to minimize jointly with respect to stereo correspondence and projective transformation can be defined as follows, E =X i P (H(p 1 i )) 3.3. uxiliary variable framework s we discussed above, there is no guarantee that the iterative methods converge except when using a least squares approach. Using ordinary least squares, however means that no distinction is made between mismatches and good matches, and that potential outliers are included. Therefore, we wish to incorporate robust estimation in the method. The question then is how to achieve convergence of the two-step iterative method. The key idea is to introduce points in the second image called auxiliary points, that might not coincide with any of the points in the second set. Following Cohen [3], energy E of the previous section has to be modified to include the auxiliary points as a second set of variables. It is shown that this energy can be minimized with respect to each of the two sets of variables alternately. The non-convex potential in the original formulation is transformed into a potential that is convex with respect to each set of variables. The auxiliary energy of the problem addressed in this paper reads: E aux = 1 2X i kh(p i )? ~p i k 2 +X i P 1 (~p i ) where we use the notation ~p for auxiliary variables. The new potential P 1 is associated to the potential P via conjugate functions (see [3]). The objective now is to minimize the auxiliary energy with respect to the projective transformation H and the auxiliary variables ~p i. For the special case of the original energy corresponding to least-squares, we choose P (p 0 ) = d 2 =2, where d = d(p 0 ; S 2 ). The point p 0 lies in the second image, as before. The corresponding potential P 1 is then P 1 (p 0 ) = 2(1?) d2 (p 0 ; S 2 ). Furthermore, the auxiliary variable ~p i is calculated by a linear interpolation with fixed weights between the predicted point and its closest point in the second set, that is, ~p i = (1? )p 0 i + p2 S 2 When we have a different potential in the original problem, we find the potential P 1 and the auxiliary variables ~p differently to satisfy convexity requirements. ccording to Theorem 5. in [3], we do not need to compute the auxiliary energy P 1 in order to find the auxiliary variable. In the following section, we derive the auxiliary variables corresponding to specific robust potentials Robust estimation The choice of the potential P (p) = f (d(p; S 2 )) determines where the auxiliary point lies on the line joining the predicted point and its closest point. bove we described that for least squares, the weight between the two points is fixed. However, if we want a robust estimation framework, or if we wish to include image features to aid the matching, then the potential is defined differently. Most robust functions are like least squares for low d-values, but for high values, the influence of the data decreases. The robust inffunction f (d) = inf(; d 2 =2) was used as an example in [3]; it assigns the same energy contribution to all d values above a threshold. From the theory in [3], we derive how to calculate the auxiliary point when we use the inf-function, ( (1? )p 0 + p 2 S 2 if d 2 < 2= ~p = p 0 otherwise Thus, it behaves like least squares up to a threshold, but when the distance is too large, the data in the second set is ignored. This difference between least squares and the inf-function is illustrated in Fig. 3. The Huber function [7] is well known in robust estimation; it is quadratic for small values on d, but linear above a threshold. The parameters are chosen such that the function and its derivative are both continuous, f (d) = ( d 2 =2 (d? =2) if 0 d if d > If the Huber function is used to define the potential P in the original energy, then we can derive the corresponding auxiliary points from the theory in [3] as ( (1? )p 0 + p 2 S 2 if 0 d ~p = (1? =d)p 0 + ( =d)p 2 S 2 if d >

5 Figure 3. Point traces of the predicted point (solid curve) and its corresponding auxiliary point (dashed) through the iterations. The two figures show the least squares approach (left) and the robust inf-function (right). Thus, using the Huber function in the auxiliary variable approach gives the least squares interpolation for low distances, but above a threshold, the weight of the predicted point increases with the distance. 4. Experiments Experiments are presented in this section that evaluate the robustness, stability and convergence of the auxiliary variables method. The performance for different robust functions, values on the parameters and different scenes are investigated for different levels of difficulty. Figure 4 shows the decrease of the auxiliary energy during the iterations, for two values of (0.3 and 0.6). The points are depicted in Fig. 5, and the robust inf-function was used. The residual decreases more rapidly for a higher -value (dashed curve), and the jumps in the graphs correspond to resolved matching ambiguities Figure 4. The energy E aux decreasing through the iterations, for two different -values. The prominent jumps correspond to resolved matching ambiguities. Figure 5 illustrates the advantage of using a robust potential in the auxiliary variable method. The figure shows the trace of predicted points during the iterations, and the points in the second set are marked with crosses. Each predicted point should converge to its corresponding cross. We defined twelve points on a circle and two other points close to the center of the circle. The same point sets are used in the two images, that is, the correct projectivity is the identity matrix. To test the robustness of the method against mismatches, we selected the initial projectivity matrix as a small rotation, scaling and translation. For instance, the point that corresponds to point E in the second image, is transformed by the initial matrix to be closer to the point D. lso, the point corresponding to point B in the second image is transformed to a point closer to point C. The least-squares method in the left part of Fig.5, with = 0:1, locks onto an incorrect minimum, due to mismatches between transformed points and their closest points in the second image. But when using the potential with the robust inf-function (right), it converges to the global minimum, and the initial mismatching, for instance between the two midpoints, is successfully resolved. B C D E Figure 5. Mismatching causes the least squares method (left) to lock onto an incorrect solution, but the robust method (right) successfully finds the global minimum. The aim of another experiment was to check if the method can segment out the coplanar points on the dominant plane. real projectivity from the stereo image in Fig. 7 was used to produce corresponding point sets in the two image planes. nother point was added that did not satisfy the projective transformation, simulating a noncoplanar point. In the left part of Fig. 6 the traces for the least squares approach is shown; it converges to an incorrect solution that is close the correct one. The closest point matching causes it to fail to notice that the point is too far from its corresponding point to be considered. However, since the point lies among the coplanar points, and not far away, it did not affect the estimated projectivity very much. It is clear that the robust method (right) using the inffunction, successfully finds the coplanar points, and converges to the correct projectivity estimate, while disregarding point as an outlier. Figure 7 shows the result for a real scene. Two of the 17 detected points are not on the floor, and they are segmented from the coplanar points. The traces of the predicted points converge (coplanar) or do not converge (not coplanar) to their corresponding points. We initilized the method with a B C D E

6 Figure 6. The coplanar points correctly converge with the robust method (right), but not with the least squares method (left). projectivity causing initial mismatches, and we observe how the robust method (using the inf-function) converges to the correct projectivity, despite the initial mismatches and perturbed predictions. Points that do not belong to the ground plane were disregarded. Figure 7. Real stereo image with detected points, and the traces of the predicted points in the right figure. Mismatching and outlier points are successfully handled. different real scene was also tested (Fig. 8) in which 18 coplanar points were detected, and five points not on the plane. The initial mismatching was resolved and the method converges correctly, disregarding the five outlier points. 5. Discussion We presented a robust iterative framework to detect the dominant plane in binocular vision, without requiring any calibration nor the matching between the images. The dominant plane is found by simultaneously addressing the problems of finding the planar projectivity between the images and the stereo matching, with an iterative closest point method. To assure its convergence, we employed the auxiliary variable framework developed by Cohen [3]. The theoretical framework was further elaborated to provide mechanisms for several robust estimators. Numerical experiments confirmed that the method converges as theoretically pre- Figure 8. nother real stereo image with detected points, and the traces of the predicted points in the right figure. Mismatching and outlier points are successfully handled. dicted, and that it successfully copes with mismatches and outlier points not belonging to the dominant plane. References [1] P. Besl and N. McKay. method for registration of 3-d shapes. IEEE Trans. PMI, 14(2): , February [2] S. Carlsson and J.-O. Eklundh. Object detection using model based prediction and motion parallax. In Proc. 1st ECCV, pages , pr [3] L. Cohen. uxiliary variables and two-step iterative algorithms in computer vision problems. JMIV, 6:59 83, [4] W. Enkelmann. Obstacle detection by evaluation of optical flow fields from image sequences. IVC, 9(3): , [5] P. Fornland. Direct obstacle detection and motion from spatio-temporal derivatives. In V. Hlaváč and R. Šara, editors, Proc. 6th CIP, pages , Prague, Czech Republic, Sep [6] R. Hartley, R. Gupta, and T. Chang. Stereo from uncalibrated cameras. In CVPR92, pages , [7] P. Huber. Robust Statistics. Wiley Series in Prob. and Math. Stat. John Wiley & Sons, Inc., [8] M. Kass,. P. Witkin, and D. Terzopoulos. Snakes: ctive contour models. IJCV, 1(4): , January [9] Q. Luong and O. Faugeras. Determining the fundamental matrix with planes: Instability and new algorithms. In Proc. CVPR, pages , [10] Q. Luong and O. Faugeras. The fundamental matrix: Theory, algorithms, and stability analysis. IJCV, 17(1):43 75, January [11] L. Robert and M. Hebert. Deriving orientation cues from stereo images. In Proc. 3rd ECCV, pages : , [12]. Shashua. Projective structure from uncalibrated images: Structure-from-motion and recognition. PMI, 16(8): , ugust [13] D. Sinclair and. Blake. Quantative planar region segmentation. IJCV, 18(1):77 91, [14] R. Tsai and T. Huang. Estimating three-dimensional motion parameters of a rigid planar patch ii: singular value decomposition. IEEE Trans. coustic, Speech and Signal Processing, 30, [15] Z. Zhang. Iterative point matching for registration of freeform curves and surfaces. IJCV, 13(2): , 1994.