Calibration of a fish eye lens with field of view larger than 180

Transcription

1 CENTER FOR MACHINE PERCEPTION CZECH TECHNICAL UNIVERSITY Calibration of a fish eye lens with field of view larger than 18 Hynek Bakstein and Tomáš Pajdla {bakstein, pajdla}@cmp.felk.cvut.cz REPRINT Hynek Bakstein and Tomáš Pajdla, Calibration of a fish eye lens with field of view larger than 18, in Proceedings of the CVWW 22, pages , February 22. Copyright: PRIP TU Wien Available at ftp://cmp.felk.cvut.cz/pub/cmp/articles/bakstein/bakstein-pajdla-cvww22a.pdf Center for Machine Perception, Department of Cybernetics Faculty of Electrical Engineering, Czech Technical University Technická 2, Prague 6, Czech Republic fax , phone , www:

2 Calibration of a fish eye lens with field of view larger than 18 Hynek Bakstein and Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Karlovo nám. 13, Prague , {bakstein,pajdla}@cmp.felk.cvut.cz Abstract We present a complete step-by-step approach to calibration of ultra wide angle fish-eye lenses with the angle of view larger than 18. Such a large field of view is necessary for some applications such as the 36 x 36 mosaicing. Recently, a Nikon FC-E8 fish eye lens converter with the field of view equal 183 become available. In this paper, we propose its model, suggest a calibration procedure, and demonstrate its use in a mosaicing application. First of all we propose a general model of a camera with field of view larger than 18. Then, we identify the structure and the parameters of the mapping between the incoming light rays and pixels for the Nikon FC-E8 converter. Finally, we present a complete camera calibration method from a known calibration target. Keywords: Camera calibration, wide angle lenses 1 Introduction Large field of view (FOV) is useful for some computer vision applications such as selfcalibration where it provides better conditioned views and less degenerate situations [11, 1]. Several ways to enlarge the FOV exist. Mirrors, lenses, moving parts, or a combination of the previous can be employed for this purpose. In this paper we focus on the use of a special lens, the Nikon FC-E8 fish eye converter [3], which provides FOV of 183. This FOV allows us to employ this lens in building of a 36 x 36 mosaic [7]. We mounted this lens on a Pulnix digital camera equipped with a standard 12.5mm lens as it is depicted in Figure 1. Our experiments also show that such a lens provides better results than mirrors, which were often used to build 36 x 36 mosaics [7]. Focusing of the lens is easier than focusing on the mirror and also the setup of the mosaicing camera is simpler. This work was supported by the following grants: MSM , GAČR 12/1/971, MŠMT KONTAKT 21/9. 1

3 Figure 1: Nikon FC-E8 fish eye converter mounted on a Pulnix digital camera with a standard 12.5mm lens. For many computer vision tasks, the relationship between the light rays entering the camera and pixels in the image has to be known. In order to find this relationship, the camera has to be calibrated. A suitable camera model has to be chosen for this task. It turns out that the pinhole camera model with a planar retina, see Figure 2, is not sufficient for sensors with large FOV [5]. Previous approaches used planar retina and pinhole model [1, 2, 12, 13]. In [9], a stereographic projection was employed but the experiments were evaluated on lenses with FOV smaller than 18. We introduce a method for calibration from a single image of one known 3D calibration target with iterative refinement of parameters of our camera model with a spherical retina, depicted in Figure 2. In the next section, we introduce a camera model with a spherical retina. Then we discuss various models describing the relationship between the light rays and pixels in Section 3. Section 4 is devoted to the determination of this model for the case of Nikon FC-E8 converter. A summary of the presented method is given in Section 5. Experimental results are presented in Section 6. 2 Camera Model The camera model describes how a 3D scene is transformed into a 2D image. It has to incorporate the orientation of the camera with respect to some scene coordinate system and also the way how the light rays in the camera centered coordinate system are projected into the image. The orientation is expressed by extrinsic camera parameters while the latter relationship is determined by intrinsic parameters of the camera. Intrinsic parameters can be divided into two groups. The first one includes the parameters of the mapping between the rays and ideal orthogonal square pixels. We will discuss these parameters in the next section. The second group contains the parameters describing the relationship between ideal orthogonal square pixels and the real pixels of image sensors. Let (u, v) denote coordinates of a point in the image measured in an orthogonal basis as shown in Figure 3. CCD chips often have a different spacing between pixels in the vertical and the horizontal direction. This results in images unequally scaled in the horizontal and vertical direction. This distortion causes circles to appear as ellipses in the image, as shown in 2

4 (u,v) (u,v) a) b) Figure 2: From image coordinates to light rays: (a) a directional and (b) an omnidirectional camera. Figure 3. Therefore, we introduce a parameter β representing the ratio between the scales of the horizontal respectively the vertical axis. A matrix expression of the distortion can be written in the following form: 1 u K 1 = β βv. (1) 1 This matrix is a simplified intrinsic calibration matrix of a pinhole camera [4]. The displacement of the center of the image is expressed by terms u and v, the skewness of the image axes is neglected in our case, because cameras usually have orthogonal pixels. u v K 1 u v Figure 3: A circle in the image plane is distorted due to a different length of the axes. Therefore we observe an ellipse instead of a circle in the image. 3 Projection Models Models of the projection between the light rays and the pixels are discussed in this section. Most commonly used approach is that these models are described by a radially symmetric function that maps the angle θ between the incoming light ray and the optical axis to some distance r from the image center, see Figures 7 and 7(b). This function typically has one parameter k. As 3

5 it was stated before, the perspective projection, which can be expressed as r = k tan θ, is not suitable for modeling cameras with large FOV. Several other projection models exist [5]: stereographic projection r = k tan θ 2, equidistant projection r = kθ, equisolid angle projection r = k sin θ 2, and sine law projection r = k sin θ. Figure 4 shows graphs of the above projection functions for angle θ varying from to 18 degrees. It can be noticed that perspective projection cannot cope with angles θ near 9. It can also be noticed that most of the models can be approximated with an equidistant projection for a smaller angle θ. However, when the FOV of the lens increases, the models differ significantly. In the next section we describe a procedure for selecting the appropriate model for Nikon FC-E8 converter Model function value Perspective Stereographic Sine law Equisolid angle Equidistant θ angle Figure 4: Values of projection models function for angle θ in range of and 18 degrees. 4 Model Determination In order to derive the projection model for Nikon FC-E8, we have investigated how light rays with constant increment in the angle θ are imaged on the image plane. We performed the following experiment. The camera was observing a cylinder with circles seen by light rays with known angle θ, as it is depicted in Figure 5(a). These circles correspond to an increment in the angle θ set to 5 for rays imaged to the peripheral parts of the image (θ = 9..7 ) and to 1 for the rays imaged to the central part of the image. Figure 5(b) show the grid which after 4

6 wrapping around a cylinder produced the circles. Figure 5(c) shows an image of this cylinder. It can be seen that circles are imaged to approximate circles and that constant increment in angles results in slowly increasing increment in radii of the circles in the image. Note that the circles at the border have angular distance 5, while the distance near the center is 1. (a) (b) (c) Figure 5: (a) Camera observing a cylinder with a calibration pattern (b) wrapped around the cylinder. Note that the lines corresponds to light rays with an increment in the angle θ set to 5 (the bottom 4 intervals) and 1 (the 5 upper intervals). (c) Image of circles with radii set to a tangent of a constantly incremented angle results in concentric circles with almost constant increment in radii in the image. We fitted all of the models mentioned in the previous section to detected projections of the light rays into the image. The stereographic projection with two parameters: r = a tan θ b provided the best fit but there was still a systematic error, see Figure 6. Therefore, we extended the model which resulted in a combination of the stereographic projection with the equisolid angle projection. This improved model is identified by four parameters, see Equation 3, and provides the best fit with no systematic error, as it is depicted in Figure 6. An initial fit of the parameters is discussed in the following section. 5 Complete Camera Model Under the above observations, we can formulate the model of the camera. Provided with a scene point X = (x, y, z) T, we are able to compute its coordinates X = (x, y, z) T in the camera centered coordinate system: X = RX + T, (2) where R represents a rotation and T stands for a translation. The standard rotation matrix R has three degrees of freedom and T is expressed by the vector T = (t 1, t 2, t 3 ) T. Then, the angle θ, see Figure 7(a), between the light ray through the point X and the optical axis can be computed. This angle determines the distance r of the pixel from the center of the image: r = a tan θ b + c sin θ d, (3) 5

7 1 Model fiting error [pixels] a tan(θ/b) a tan(θ/b) + c sin(θ/d) θ angle Figure 6: Model fit error for stereographic and combined stereographic and equisolid angle projection. Parameters a, b, c, and d were estimated in an optimization procedure. where a, b, c, and d are parameters of the projection model. z = optical axis (x,y,z ) (u,v ) ϕ θ y r ϕ (u,v ) v u x (a) (b) Figure 7: (a) Camera coordinate system and its relationship to the angles θ and ϕ (b) From polar coordinates (r, ϕ) to orthogonal coordinates (u, v ). Together with the angle ϕ between the light ray reprojected to xy plane and the x axis of the camera centered coordinate system, the distance r is sufficient to calculate the pixel coordinates u = (u, v, 1) in some orthogonal image coordinate system, see Figure 7(b), as u = r cos ϕ (4) v = r sin ϕ. (5) In this case the vector u does not represent a light ray from the camera center like in a pinhole camera model, instead it is just a vector augmented by 1 so that we can write an affine transform of the image points compactly by one matrix multiplication (6). Real pixel coordinates u = (u, v, 1) can then be obtained as u = Ku. (6) 6

8 The complete camera model parameters including extrinsic and intrinsic parameters can be recovered from measured coordinates of calibration points by minimizing an objective function J = N ũ u, (7) i=1 where... denotes the Euclidean norm, N is the number of points, ũ are coordinates of points measured in the image, and u are their coordinates reprojected by the camera model. A MAT- LAB implementation of the Levenberg-Marquardt [6] minimization was employed in order to minimize the objective function (7). The rotational matrix R has three degrees of freedom, as well as the vector of translation T, see (2). The image center, scale ratio of the image axes β, and the four parameters of the mapping between the light rays and pixels (3) give 7 intrinsic parameters. Therefore, our model is identified by 13 parameters. When minimizing the objective function (7), we set the image center to the center of the circle (ellipsis) surrounding the image, see Figure 5. This is possible because the Nikon FC- E8 lens is so called circular fish eye, where this circle is visible. Assuming that the mapping between the light rays and pixels (3) is radially symmetric, this center of the circle should be the image center. Parameters of the model were set to an ideal stereographic projection, which means that b = 2, c =, d = 1, and a was set using the ratio between the coordinates of points corresponding to the light rays with the angle θ equal to and 18 degrees. The value of the β parameters was set to 1. The camera position was set to be in the center of the scene coordinate system with the z axis coincident with the optical axis of the camera. 6 Experimental Results We performed two calibration experiments. In the first experiment, the calibration points were located on a cylinder around the optical axis and the camera was looking down into that cylinder, see Figure 5(a). The points had the same depth for the same value of θ. The second experiment employed a 3D calibration object with points located on a half cylinder. The object was realized such that a line of calibration points was rotated on a turntable, as it is depicted in Figure 8. Here, the points with the same angle θ had different depths. The first experimental setup was also used to determine the projection model, as it is described in Section 4. The total number of 72 points was manually detected. One half of them was used for the estimation of the parameters while the second half was used for the verification. The same approach was also used in the second experiment, where the number of calibration points was 285. Again, all points were detected manually. Figure 9(a) shows the reprojection of points, computed with parameters estimated during the calibration, compared with their coordinates detected in the image. The lines represent the errors between the respective points scaled 2 times to make the distances clearly visible. The same error is shown in Figure 9(b) for all the points. It can be noticed that the error does not show any significant systematic dependence. 7

9 (a) (b) (c) Figure 8: (a) Experimental setup for the half cylinder experiment. (b) One of the images with the calibration target in the middle of the image. (c) The calibration target is located 9 left from the camera, note the significant distortion Reprojection error (a) Calibration point number (b) Figure 9: (a) Reprojection of points for the cylinder experiment. The distances between the reprojected and the detected points are scaled 2 times. (b) Reprojection error for each point. Similar graphs illustrate the results from the second experiment. Figure 1 shows the comparison between the reprojected points and their coordinates detected in the image. Again, the lines representing the distance between these two sets of points are scaled 2 times. Figure 11(a) depicts this reprojection error for each calibration point. Note that the error is bigger for points in the corners of the image, which is natural, since the resolution here is higher and therefore one pixel corresponds to a smaller change in the angle θ. However, it can be seen in Figure 1 that the reprojection error is nearly random. To verify the randomness of the error, we performed the following test. Because the points in the image were detected manually, we suppose that the detection error has normal distribution in both image axes. Therefore, a sum of squares of these errors, normalized to unit variance, should be described by a χ 2 distribution [8]. Figure 11(b) shows a histogram of detection errors together with a graph of a χ 2 density. Note that χ 2 distribution describes well the calibration error distribution. 8

10 Figure 1: Reprojection of points for the half cylinder experiment. The distances between the reprojected and the detected points are scaled 2 times. Reprojection error [pixels] Calibration point number (a) Count Sum of squares of normalized detection errors (b) Figure 11: (a) Reprojection error for each point for the half cylinder experiment. (b) Histogram of sum of squares of normalized detection errors together with a χ 2 density marked by the curve. 7 Conclusion We have proposed a camera model for lenses with FOV larger than 18. The models is based on employment of a spherical retina and a radially symmetrical mapping between the incoming light rays and pixels in the image. We proposed a method for identification of the mapping function, which led to a combination of two mapping functions. A complete calibration procedure, involving a single image of a 3D calibration target, is then presented. Finally, we demonstrate the theory in two experiments, all using the Nikon FC-E8 fish eye converter. We believe that the ability to correctly describe and calibrate the Nikon FC-E8 fish eye lens converter opens a way to many new application of very wide angle lenses. 9

11 References [1] A. Basu and S. Licardie. Alternative models for fish-eye lenses. Pattern Recognition Letters, 16(4): , [2] S. S. Beauchemin, R. Bajcsy, and Givaty G. A unified procedure for calibrating intrinsic parameters of fish-eye lenses. In Vision Interface (VI 99), pages , May [3] Nikon Corp. Nikon www pages: 2. [4] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge, UK, 2. [5] Fleck M. M. Perspective projection: the wrong imaging model. Technical Report TR 95-1, Comp. Sci., U. Iowa, [6] J.J. Moré. The levenberg-marquardt algorithm: Implementation and theory. In G. A. Watson, editor, Numerical Analysis, Lecture Notes in Mathematics 63, pages Springer Verlag, [7] S. K. Nayar and A. Karmarkar. 36 x 36 mosaics. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR ), Hilton Head, South Carolina, volume 2, pages , June 2. [8] A. Papoulis. Probability and Statistics. Prentice-Hall, 199. [9] D. E. Stevenson and M. M. Fleck. Robot aerobics: Four easy steps to a more flexible calibration. In International Conference on Computer Vision, pages 34 39, [1] T. Svoboda, T. Pajdla, and V. Hlaváč. Epipolar geometry for panoramic cameras. In Hans Burkhardt and Neumann Bernd, editors, the fifth European Conference on Computer Vision, Freiburg, Germany, number 146 in Lecture Notes in Computer Science, pages , Berlin, Germany, June [11] Tomáš Svoboda, Tomáš Pajdla, and Václav Hlaváč. Motion estimation using central panoramic cameras. In Stefan Hahn, editor, IEEE International Conference on Intelligent Vehicles, pages , Stuttgart, Germany, October Causal Productions. [12] R. Swaminathan and S.K. Nayar. Non-metric calibration of wide-angle lenses. In DARPA Image Understanding Workshop, pages , [13] Y. Xiong and K. Turkowski. Creating image based vr using a self-calibrating fisheye lens. In IEEE Computer Vision and Pattern Recognition (CVPR97), pages ,