is used in many dierent applications. We give some examples from robotics. Firstly, a robot equipped with a camera, giving visual information about th

Transcription

1 Geometry and Algebra of Multiple Projective Transformations Anders Heyden Dept of Mathematics, Lund University Box 8, S Lund, SWEDEN Supervisor: Gunnar Sparr Abstract In this thesis several dierent cases of reconstruction of 3D objects from a number of 2D images, obtained by projective transformations, are considered. Firstly, the case where the images are taken by uncalibrated cameras, making it possible to reconstruct the object up to projective transformations, is described. The minimal cases of two images of seven points and three images of six points are solved, giving threefold solutions in both cases. Then linear methods for the cases where more points or more images are available are given, using multilinear constraints, based on a canonical representation of the multiple view geometry. The case of a continuous stream of images is also treated, giving multilinear constraints on the image coordinates and their derivatives. Secondly, the algebraic properties of the multilinear functions and the ideals generated by them are investigated. The main result is that the ideal generated by the bilinearities for three views have a primary decomposition, where one component is the ideal generated by the trilinear functions and the other corresponds to the trifocal plane. Thirdly, reconstruction from calibrated cameras, making it possible to reconstruct the object up to similarity transformations, is presented. A new proof of the Kruppa- Demazure Theorem, stating that there are in general ten dierent solutions in the case of two images and ve points, is given. Fourthly, two dierent ways to make Euclidean reconstruction are described, when some information of the cameras is given. In the rst case, the images are obtained by perspective transformations. The second case is when the intrinsic parameters are the same for all images. Finally, the case of one image of an object, which is known to be piecewise planar, is investigated. Consistency conditions for such an image, called a line-drawing, to be correct are given together with two dierent correction methods. Background A central problem in computer vision is the reconstruction of 3D objects from a number of its 2D images under projections. This kind of information, obtained from cameras, AMS subject classication 5N5

2 is used in many dierent applications. We give some examples from robotics. Firstly, a robot equipped with a camera, giving visual information about the surrounding world, can use the information obtained from a reconstruction to avoid obstacles. Secondly, the reconstruction can be used to recognise dierent known object in the environment that the robot is looking for. Thirdly, the information can be used to determine the motion of the robot, if this information is not available from other sensors. Fourthly, by patching together dierent reconstructions, a map of the surroundings can be made. This map is useful when the robot is moving around. There exist several dierent formulations of the reconstruction problem, depending on the a priori knowledge about the image formation process (the camera) and about the object. In this thesis we will explore the geometry and algebra of the reconstruction problem in several dierent settings. In some applications, the motion of the camera is essential, and this information can usually be obtained from the reconstruction of the object. The motion of the camera may also be used as parameters, describing the multiple view geometry. Historically, this kind of reconstruction was rst done by photogrammetrists, cf. [] and the references therein. They in general used calibrated cameras, i.e. they assumed a number of properties of the imaging formation process to be known, e.g. the focal length and principal point of the camera. The emphasis was laid on accuracy, for example in order to build up maps from aerial photographs. Later on the computer vision community started to work on this problem from a dierent viewpoint, see [4] and [6]. The emphasis was now laid on fast and robust methods, in order to construct seeing systems that are able to work in real time applications. In order to calibrate the camera, several images of a calibration grid have to be analysed. Since this is a tedious procedure attention turned to reconstruction from uncalibrated cameras, i.e. without using the focal length etc. It was shown that in this case it is only possible to reconstruct the object up to projective transformations, see [20], [5], [7] and [9]. This area of research has grown a lot the last few years, e.g. in nding methods to make linear reconstruction, see [5], [9] and [0], and in nding representations of the camera matrices, see [5] and [2]. Since in the applications one often work with noisy images, robustness is a strong claim on the methods for computer vision. 2 Camera Models We start with an abstract denition of a camera. Denition 2.. By a camera, or pinhole camera, is meant a triple (C; ; P ). Here C denotes a point, called the focal point or centre of projection and denotes a plane, called the image plane, in ane 3-dimensional space, A 3, such that C =2. P denotes a perspective transformation that for each line L through C, not parallel to, maps every point on L onto the intersection of L and. The plane through the focal point, that is parallel to the image plane is called the focal plane. Using projective geometry, it is possible to give a meaning also to the perspective image of a point in the focal plane.

3 Assuming that the camera is situated in Euclidean 3-dimensional space, E 3, that is A 3 with a metric structure, then the distance between the image plane and the focal point is called the focal length and the orthogonal projection of the focal point onto the image plane is called the principal point. If the position of the focal point with respect to the image plane is known, in terms of the focal distance and the principal point, the camera is called calibrated. Then it is possible to use the scalar product to determine angles between dierent lines from the focal point to dierent image points. When the ambient space is only supposed to be ane, A 3, and the relative position of the image plane with respect to the focal point is unknown, the camera is called uncalibrated. Observe that these denitions are independent of coordinate representations. Introducing coordinate systems describing the camera, x and y, and the object, X, Y and Z, the calibrated camera can be written X 4 x y5 = K [ R j? Rt ] 6Y ; () where is the depth of the point, R denotes an orthogonal matrix, t a vector and 2 K = 4 3 x y x 0 0 y y 0 5 : (2) 0 0 s is called the skew and is needed when describing a camera where the image coordinate system is dened by light sensitive elements, and these do not form a rectangular array. The parameters x and y are called the magnications in the x- and y-directions and are needed when describing a camera where the array of light sensitive elements has dierent scales in dierent directions. The ratio = x = y is called the aspect ratio. The point (x 0 ; y 0 ) is again called the principal point and is the orthogonal projection of the focal point onto the image plane. These are called the intrinsic parameters and are assumed to be known in the calibrated case. The uncalibrated camera can be written X X 4 x y5 = P Z 6Y = K[ R j? Rt ] 6Y ; (3) Z where the intrinsic parameters in K are unknown. A 3 4 matrix P as in (3), relating camera coordinates to object coordinates is called a camera matrix. If the skew s = 0 and the aspect ratio = then we talk about a camera with Euclidean image plane. Let x = (x; y; z) 2 P 2 and X = (X; Y; Z; W ) 2 P 3, then the uncalibrated camera equation (3) can be interpreted as a projection from P 3 to P 2 ( meaning equality up to scale) Z x P X = [ B j? Bt ]X : (4)

4 3 Dierent Kinds of Reconstructions In this section we will describe the dierent kinds of reconstruction dealt with. Assumption 3.. It is assumed that we have a number of projective images of a common point conguration, called the object, and that it is known which points in the dierent images that correspond to the same point in the object. We will deal with the inverse problem, which in a general formulation reads: Problem 3.. Given a sequence of planar point congurations,? m Y i, with known point i= correspondences between the images. Determine all (object) congurations X that can be mapped onto each of the given point congurations by some perspective transformation. Denition 3.. Each such conguration X, in Problem 3., is called a reconstruction from the sequence? Y i m i=. Even if the intrinsic parameters are unknown and the images are taken by dierent cameras it is possible to specify a class of possible object congurations, when suciently many images and point correspondences are available. In this case, it can be proved that it is only possible to reconstruct the object up to a projective transformation. This is called uncalibrated reconstruction. When the intrinsic parameters are known for all the cameras, that have given the images, it is possible to reconstruct the object up to similarity transformations, if suciently many images and point correspondences are available. This is called calibrated reconstruction. In the case of cameras with Euclidean image planes it will be shown that it is possible to reconstruct the object up to similarity transformations, given at least four images. This is called reconstruction from Euclidean image planes and has not been considered before. One common situation is when all the images are taken by the same camera, and thus the intrinsic parameters are the same for all camera matrices. Then it is possible to reconstruct the object up to similarity transformations, given at least three images. This is called reconstruction from constant intrinsic parameters. When only one image is available, it is in general not possible to make a reconstruction, without some a priori knowledge about the object. Using the a priori knowledge that the object is piecewise planar, a line-drawing is obtained after an ideal edge detection. It is possible to give consistency condition on the correctness of such a line-drawing, and to describe in a precise way the class of possible reconstructions. 4 Uncalibrated Reconstruction - Minimal Cases The minimal cases of seven point matches in two images, were studied in [2] and [2]. The minimal case of six point matches in three images was independently solved in [8] and []. In this thesis these minimal cases are solved, using the same technique for both. The main idea is to use ane coordinate systems in each image and in the object, such that the rst three and four points respectively form an ane basis. It is shown that both these cases have three dierent solutions, obtained from a third degree polynomial equation. Often two

5 of these solutions are complex and can be discarded, leaving essentially one solution. It is furthermore shown that it is only possible to reconstruct the object up to projective transformations. The solutions are obtained by assigning a linear space, s(x ), called the shape of X, to every point conguration, X. It is shown that a reconstruction from m images of n ponts can be obtained from the so called kinetic depths, q i, describing ratios of depths of points in dierent images, obeying dim? s(y ) \ diag(q 2 )s(y 2 ) \ : : : \ diag(q m )s(y m ) = n? 4 ; (5) where the kinetic depths acts as coordinatewise multipliers on the linear spaces. Introducing ane coordinates in the images as well as in the object gives, after some calculations, the desired third order polynomials. Finally, the reconstruction, X, is obtained by the fact that s(x ) equals the intersection of the linear spaces in (5). 5 Uncalibrated Reconstruction - Linear Solutions When a large number of point matches is available it is possible to make reconstruction using fast linear methods. For two images this can be done using the fundamental matrix, see [5], and for three images by using the fundamental tensor, see [9], [9] and [0]. The fundamental matrix gives a so called bilinear constraint and the fundamental tensor gives so called trilinear constraints. It is also possible to use four images and quadrilinear constraints, see [23]. These are called multilinear constraints. These multilinear constraints are obtained by lining up the camera equations for the same point in m images, i x i = P i X; i = : : : m ; (6) by noting that they are linear in the unknown variables i and X, giving 2 3 X P x 0 0 : : : 0 P 2 0 x 2 0 : : : 0? 0 P x 3 : : : 0? ? 3 = 0 () Mu = 0 : (7) P m : : : x m 0? m This equation means that the rank of M must be m + 3. Picking out subdeterminants of M gives the so called multilinear constraints or multilinear forms. The multilinear constraints that only involves two of the image coordinates nontrivially is called the bilinear constraints and can be expressed using the fundamental matrix, F i;j (between images i and j), as x T i F i;jx j = 0. The entries in F i;j forms a second order tensor. In the same way it is possible to pick out trilinear and quadrilinear constraints by choosing subdeterminants involving three and four image coordinates nontrivially respectively. The coecients of these polynomial constraints form tensors of order three and four respectively.

6 Using these multilinear forms we give solutions to the cases of 2 images of 8 points, 3 images of 7 points and 4 images of 6 points using the bilinear constraints, the trilinear constraints and the quadrilinear constraints respectively, using only linear methods, i.e. singular value decompositions. In fact, a simplied version of the multilinear constraints are used, called reduced multilinear constraints, which results in less variables. This simplication is again obtained by chosing ane coordinate systems in the images and in the object. 6 Uncalibrated Reconstruction - Sequences and Streams Given a sequence of images, there exist a large number of multilinear constraints and complicated relations between them. The relations between dierent fundamental matrices were studied in [5]. A natural extension is to study continuous streams of images, instead of discrete time sequences. The extension of the bilinear constraint has been studied in [24]. We analyse both sequences and streams of images. Sequences are obtained when a sampled time scale is used and streams are obtained when a continuous time scale is used. In the discrete case there appear multilinear constraints and consistency conditions on these are given. The relations between the fundamental matrices and tensors in a sequence of images are explored. In the continuous case there appear a class of continuous constraints. It is shown that in the discrete case, the bilinear constraints are sucient to make a reconstruction and calculate the egomotion of the camera, whereas in the continuous case the third order constraints are needed. Furthermore, a canonical description of the camera matrices in a sequence are given. Using so called reduced ane coordinates the camera matrices can be written P i = [ D i j? D i t i ]; i = ; : : : ; m ; (8) where D i are diagonal matrices with D = I, t = 0, det D i = and jt 2 j =. 7 Uncalibrated Reconstruction - Algebraic Varieties The relations between dierent higher order multilinear constraints have been studied in [7], [8], [23] and [2]. The algebraic properties of the ideals generated by the multilinear constraints are investigated as well as the corresponding algebraic varieties. The main result is that the ideal, I b, generated by the bilinearities for three images have a primary decomposition, where one part is the ideal generated by the trilinearities, I t, and the other part corresponds to the trifocal plane, I tp, i.e. the plane dened by the three camera centres. This means that the bilinear variety, V b, dened by I b is reducible and its irreducible components are V t, dened by I t and V tp, dened by I tp, i.e. V b = V t [ V tp ; I b = I t \ I tp : (9) Another result is that for m 4 images, the ideal generated by all multilinear constraints can be generated by using only the bilinearities and that when m 5 even some bilinearities may be omitted.

7 8 Calibrated Reconstruction - The Kruppa Demazure Theorem The minimal case of 5 points in 2 images have been analysed in [3], [3], [4] and [6]. We give a new proof of the Kruppa-Demazure-Theorem, stating that there are in general 0 dierent solutions to this problem. One by-product is that the so called twisted pair solutions appear as the same solution, when using the kinetic depth instead of rotation matrices, to parametrise the problem. The theorem is proved by solving a special case and then using singularity theory to deduce the result for the general situation. 9 Euclidean Reconstructions Euclidean reconstruction from constant intrinsic parameters has been considered in [6] using the absolute conic and the epipolar transformations between images and i. Two dierent types of Euclidean reconstruction from a priori informations about the cameras are dealt with. The rst one, reconstruction from Euclidean image planes, where s = 0 and =, gives two quadratic constraints on 6 parameters, for all images, except the rst one, making it possible to obtain a solution for 4 images. The second case, reconstruction from constant intrinsic parameters, when the intrinsic parameters are constant, but unknown, gives also quadratic constraints on the intrinsic parameters, the Kruppa constraints. These constraints make it possible to obtain a solution for 3 images. 0 Line-Drawings This kind of reconstruction and consistency conditions have been studied in [22]. We analyse line-drawings, using the concept `weakly possible'. Consistency conditions are given using the so called S-matrix, together with two dierent correction methods. We also show how second order syzygies can be found and dealt with. References [] Buchanan, T., Photogrammetry and projective geometry - an historical survey, SPIE, Vol. 944, 993, pp [2] Chasles, M., Question 296., Nouv. Ann. Math., 4(50), 855. [3] Demazure, M., Sur Deux Problemes de Reconstruction, Technical Report, No 882, INRIA, Rocquencourt, France, 988. [4] Faugeras, O., D., Maybank S., Motion from Point Matches: Multiplicity of Solutions, Technical Report, No 57, INRIA, Rocquencourt, France, 990. [5] Faugeras, O., D., What can be seen in three dimensions with an uncalibrated stereo rig?, ECCV'92, Lecture notes in Computer Science, Vol 588. Ed. G. Sandini, Springer-Verlag, 992, pp [6] Faugeras, O., D., Luong, Q.-T., Maybank, S.,J., Camera Self-Calibration: Theory and Experiments, ECCV'92, Lecture notes in Computer Science, Vol 588. Ed. G. Sandini, Springer-Verlag, 992, pp

8 [7] Faugeras, O., D., Mourrain, B., On the geometry and algebra on the point and line correspondences between N images, Proc. ICCV'95, IEEE Computer Society Press, 995, pp [8] Faugeras, O., D., Mourrain, B., About the correspondences of points between N images, Proc. IEEE Workshop on Representation of Visual Scenes, 995. [9] Hartley, R., I., Projective Reconstruction and Invariants from Multiple Images, IEEE Trans. Pattern Anal. Machine Intell., vol. 6, no. 0, pp , 994. [0] Hartley, A linear method for reconstruction from lines and points, Proc. ICCV'95, IEEE Computer Society Press, 995, pp [] Heyden, A., Reconstruction and Prediction from Three Images of Uncalibrated Cameras, Proc. 9th Scandinavian Conference on Image Analysis, Ed. Gunilla Borgefors, Uppsala, Sweden, 995, pp Also in Theory & Applications of Image Processing II, World Scientic Publishing Co, Machine Perception Articial Intelligence, 995. [2] Heyden, A., Reconstruction from Image Sequences by means of Relative Depths, Proc. ICCV'95, IEEE Computer Society Press, 995, pp , Also to appear in IJCV, International Journal of Computer Vision. [3] Kruppa, E., Zur Ermittlung eines Objektes Zwei Perspektiven mit innerer Orientierung, Sitz-Ber. Akad. Wiss., Wien, math. naturw. Kl. Abt. IIa, 22, [4] Longuet-Higgins, H., C., A computer algorithm for reconstructing a scene from two projections, Nature, vol. 293, sept., 98, pp [5] Luong, Q.-T., Vieville, T., Canonic Representations for the Geometries of Multiple Projective Views, ECCV'94, Lecture notes in Computer Science, Vol 800. Ed. Jan- Olof Eklund, Springer-Verlag, 994, pp [6] Maybank, S., Theory of Reconstruction from Image Motion, Springer-Verlag, Berlin, Heidelberg, New York, 993. [7] Mohr, R., Arbogast, E., It can be done without camera calibration, Pattern Recognition Letters, vol. 2, no., 99, pp [8] Quan, L., Invariants of 6 Points from 3 Uncalibrated Images, ECCV'94, Lecture notes in Computer Science, Vol 80. Ed. J-O. Eklund, Springer-Verlag 994, pp [9] Shashua, A., Trilinearity in Visual Recognition by Alignment, ECCV'94, Lecture notes in Computer Science, Vol 800. Ed. Jan-Olof Eklund, Springer-Verlag, 994, pp [20] Sparr, G., An algebraic-analytic method for ane shapes of point congurations, proceedings 7th Scandinavian Conference on Image Analysis, 99, pp Also in Theory & Applications of Image Analysis I, pp , World Scientic Publishing Co, Machine Perception Articial Intelligence, 992. [2] Sparr, G., A Common Framework for Kinetic Depth, Reconstruction and Motion for Deformable Objects, ECCV'94, Lecture notes in Computer Science, Vol 80. Ed. J-O. Eklund, Springer-Verlag 994, pp [22] Sugihara, K., Machine Interpretation of Line Drawings, MIT Press, Cambridge, Massachusets, London, England, 986. [23] Triggs, B., Matching Constraints and the Joint Image, Proc. ICCV'95, IEEE Computer Society Press, 995, pp [24] Vieville, T., Faugeras, O., D., Motion analysis with a camera with unknown, and possibly varying intrinsic parameters, Proc. ICCV'95, IEEE Computer Society Press, 995, pp