6 th nternational Symposium on Mobile Mapping Technology, Presidente Prudente, São Paulo, Brazil, July 1-4, 9 FAST CAMERA CABRATON FOR OW COST MOBE MAPPNG Sérgio Madeira 1, J A Gonçalves, uísa Bastos 1 Universidade de Trás-os-Montes e Alto Douro, 51-81, Vila Real, Portugal Universidade do Porto Faculdade de Ciências, Rua Campo Alegre, 687, 4169-7, Porto, Portugal smadeira@utadpt KEY WORDS: Mobile Mapping System, Surveying, Calibration, Camera, Optical, Bundle ABSTRACT: n a Mobile Mapping System based on CCD video cameras as remote sensors, the characteristics and behaviour of the video sensors are a key factor for the overall system performance, especially the robustness of the lenses and the capacity to keep the inherent distortion factors at low levels The lens system must have the possibility of iris and focal length fiing in order to allow keeping the internal characteristics practically unchanged, at least during a surveying session Furthermore, for application in a production environment, to have a quick, simple and reliable camera calibration method is mandatory This paper describes a method that allows for a fast calibration of CCD cameras, but maintaining sufficient accuracy t relies on using images taken of a regular pattern printed in a plane panel Several tests were carried out in order to determine which interior orientation parameters should be used For each camera several images must be obtained of a plane object with well defined points measured in a true-size object reference system Camera calibration is performed independently for each camera The initial approimations for the eterior parameters, three rotations and three translations for each image, in the object reference frame, are obtained with a process that relies on the collinearity equations; otherwise there are no required initial approimations, ecept for the focal distance The final calibration parameters are obtained in an iterative process using a bundle adjustment Results of several tests made in order to assess the overall performance of the method, based on re-projection from object to image space and vice-versa, are presented here 1 NTRODUCTON This article describes a camera calibration method in the scope of a Mobile Mapping System (MMS) that uses CCD video cameras as mapping sensors, developed by the authors The characteristics and behaviour of the video sensors are crucial in the quality of the data etracted from the MMS once the obtained absolute coordinates of objects are based on their image coordinates n that sense it is necessary to determine the internal characteristics of the lens system and to keep them practically unchanged, at least during a surveying session The internal parameters to determine are the interior orientation parameters, that include the position of the principal point, the focal distance and a vertical image scale factor, and some calibration parameters that account for systematic distortions or imperfections of the lenses For the implementation of a low cost terrestrial Mobile Mapping System (MMS) developed by the authors (Madeira et all, 7), based on two CCD video cameras as remote sensors, it was necessary to develop a reliable and simple camera calibration procedure This article addresses this particular aspect of the implemented MMS CABRATON MODE n order to develop the calibration method a mathematic model, based on the well known photogrammetric colinearity equations, had to be developed We begin with some definitions of the coordinates and reference systems in our model 1 Definitions For the acquired image two kinds of coordinates are considered (Figure 1) The acquired image, with discrete piels and the row ais downwards, is converted to the mage Plane system, in piels scale too, but continuous and with a vertical ais called z, upwards oriented The referential associated to the image space (Figure ) is a 3D etension of the previously defined mage Plane, by adding a y ais, forward, so defining a 3D direct referential system This particular definition of the mage Space Referential intends to approimate it to an earth fied referential, once the model was developed for terrestrial images, with optical ais nearly horizontal The origin of this referential is the low-left corner of the image plane t allows reconstructing the course of the light beams in the interior of the camera The focal point and the focal distance are represented respectively by and f in Figure 13 1 3 M-1 N-1 Digital mage (piels) mage Plane (piels) Discrete Continuous Figure 1: mage coordinates considered z
z a z a y The principal difference to the photogrammetric methods is that the calibration object is a flat surface, which facilitates the rigorous determination of the object coordinates and makes it possible to derive initial approimations from the images This type of calibration object requires, for the calibration procedure, more than one image, obtained from different angles The present method follows this approach and will be developed in the following section f z 4 Photogrammetric Model Figure : mage Space Referential For all the photogrammetric procedures the standard collinearity model is used (Wolf and Dewitt, ), with a difference in the letters: z instead of y n this situation the collinearity equations are written as follows: Calibration parameters The photo-image representation of an object always occurs with some kind of distortion, induced by systematic or random imperfections and misalignments of the lens components The parameters used to model it are well known from the literature and, basically, are divided in radial distortion parameters (RDP) and tangential distortion parameters (TDP) (Brown, 1966, 1971) The radial distortion parameters are used to model deformations related to the distance to the principal point, or radius The piel displacement depends only on the radius, r, and is modelled by the seven order polynomial in epression (1) The RDP are the coefficients k 1, k and k 3 Δ r k r 3 k r 5 k 7 (1) 1 3 r The tangential distortion parameters are used to model deformations related to misalignments of the lens components n this case the distortion suffered by a piel coordinate (,z) is represented by the epressions in equation () Δp p Δzp p The TDP are p 1 and p 3 Calibration approach [ r ( ) ] p( )( z z ) r ( z z ) p z z 1 [ ] 1( )( ) The two principal approaches in camera calibration comprise photogrammetric calibration and self-calibration methods (Fraser, 1997) n the photogrammetric calibration a 3D calibration object with rigorously measured point coordinates is used This method is normally carried out in laboratory, in the calibration of photogrammetric cameras, in which the position and attitude of the cameras can be measured Self-calibration methods derive from computer vision n these methods there are no object-point coordinates known or any kind of marks in the images The calibration parameters can be eplicit or implicit and are calculated from two or more images of a scene by imposing some restrictions (Fraser, 1997) Another approach, which is being more and more used, is the combination of the two approaches (Heikkila and Silven, 1997, Bouguet and Pernoa, 1998, Zhang, 1999), which allows the rigorous calibration of image sensors that are used intensively in mapping applications and whose internal parameters regularly change () z z Δ f r s Δz λ f q s where rm 11 (X-X )m 1 (Y-Y )m 13 (Z-Z ) sm 1 (X-X )m (Y-Y )m 3 (Z-Z ) qm 31 (X-X )m 3 (Y-Y )m 33 (Z-Z ) f focal length λ z coordinate scale factor, z measured image coordinates, z image coordinates of the principal point Δ, Δz corrections due to lens distortions as given in the epressions (1) and () and X, Y, Z coordinates of projection center X, Y, Z object coordinates in ground m ij elements of cosines matri rotation (include the three rotation angles ω, φ and κ) The system is non linear and has, for one camera, 11 variables n order to solve it, it can be linearised using Taylor s epansion and iterated upon initial approimations Two types of parameters are considered: 1 The fied parameters, which are the interior orientation parameters and the calibration parameters The variable parameters, which are the si eternal orientation parameters (camera position and attitude of each image) n Figure 3 a scheme of an image acquisition of the calibration panel is shown The Space Object Referential (X,Y,Z) is indicated in the Figure Camera T Tz z κ ω φ y Ty X Figure 3: Scheme of image acquisition for calibration n order to achieve a good consistency in the final calibration parameters, it is necessary to acquire a number of images of the calibration panel, varying as much as possible the attitude and Z Calibration panel Y (3)
position from where the images are acquired t is convenient to obtain some of the images with Y camera ais rotated near 9º For a calibration session, say with M images of the calibration object and with N known object-point coordinates, the following linear system can be obtained: X1 6 6 A1 1 A1 N N B1 X 6 6 1 A1 N A N B A X (4) : : : : : : : 6 6 A1M N N AM BM XM Where: A1 i : refers to the Jacobian matrices of each image in order to the 9 fied parameters, with N lines and 9 columns A i : refers to the Jacobian matrices of each image in order to the 6 eternal orientation parameters, with N lines and 6 columns N 6 : aren6 zero blocs X1: is a vector with corrections to the initial approimations of the 9 fied parameters X i : are vectors with corrections to the initial approimations of the 6 eternal orientation parameters for each image B i : are the collinearity equations, calculated for each point and each image, with the initial approimations Once the system has more observations than parameters, the solution will be obtained by the least squares method The corrections (actually the components of the solution vector) will be added to the initial estimates and the process will be repeated (or iterated) The process stops when the corrections are smaller than a pre-defined tolerance 5 Method to obtain initial approimations One fundamental aspect of the procedure presented is the definition of the values for the initial approimations to the parameters under determination These have to be close to the real values, or else the method may not converge to the correct solution nitial approimations to the internal parameters: For these parameters it is not difficult to achieve the initial estimates (table 1) The principal point is the centre of the image, and the vertical scale factor is 1, since these are construction conditions Parameter nitial approimations mage columns / z mage rows / F Manufacturer focal distance λ 1 k 1 k k 3 p 1 p Table 1 - Approimations to the internal parameters nitial approimations to the eternal parameters: These are the positions and orientation angles of the camera in the image acquisition instants, in Object Space, whose origin and aes coincide with the ones of the calibration panel Finding initial estimations to these parameters is not trivial We have the following process: Consider the collinearity equations (3), fully developed: m11( X X ) m1( Y Y ) m13( Z Z ) f m1( X X ) m( Y Y ) m3( Z Z ) m31( X X ) m3( Y Y) m33( Z Z ) z z f m1( X X ) m( Y Y ) m3( Z Z ) n the present case the Object Space Referential is defined by the calibration panel itself, as represented in Figure 3 When equations (5) are applied to the observation of known points in the calibration panel, the equations can be simplified as follows: m11x m13z f m1x m3z z z m31x m33z f m1x m3z [ m11x m1y m13z ] [ m1x my m3z ] [ m31x m3y m33z ] [ m X m Y m Z ] 1 since the Y coordinates of the panel points are always zero n equations (6) the unknowns are X, Y, Z and the three rotation angles which are implicit in the m ij parameters Defining parameters A to as follows: m1x my m3z m11 A m13 B m11x m1y m13z C m31 D m33 E m31x m3y m33z F m1 G m3 H The equations (6) can be rewritten as a projective transformation, as indicated in equations (8) f z z f AX BZ C GX HZ 1 DX EZ F GX HZ 1 The equations (8) are linear in the unknown parameters (A, B, C, D, E, F, G and H), which can be directly determined Finally the initial estimations can be obtained by the following epressions: G κ arctan κ [ 9º,9º] (9) A D ϕ arctan sin κ ϕ G [ 9º,9º] 3 (5) (6) (7) (8) (1)
cosϕ H ω arctan sinϕtanκ ϕ [ 9º,9º] cosκ E (11) m11 m1 m13 X C m m1 m m3 Y ; 11 A m31 m3 m33 Z F (1) This process of initial estimations determination does not need any etra user input, so greatly simplifying the calibration procedure 6 Calibration panel pattern The calibration pattern, shown in Figure 4, was designed in CAD software t follows the purposes of trouble-free image point determination, either in manual or auto mode, and to allow a quick perception of the panel orientation in the images, for which reason some reference marks were placed t was then printed at true scale over a rigid material ts dimensions are 11cm15cm Figure 5: Estimation of corners positions, based on 8 to 1 points (crosses) indicated by the user 3 TESTS AND RESUTS n order to assess its overall performance, the method was tested with two data sets corresponding to cameras with different sensor and lens characteristics The tests were repeated for each camera considering the following si sets of parameters: Set 1:, z, f Set :, z, f, κ 1 Set 3:, z, f, κ 1, κ Set 4:, z, f, λ, κ 1, κ Set 5:, z, f, λ, k 1, k, p 1, p Set 6:, z, f, λ, κ 1, κ, κ 3, p 1, p The meaning of the parameters is indicated in (1), () and (3) 31 Test 1 Figure 4: Calibration pattern 7 mage coordinate measurement The piel coordinates of the pattern corners are etracted automatically The process has two main stages Stage 1: The positions of the corners are estimated and enclosed in 11 piels image patches, as shown in Figure 5 n order to make this stage more robust it is requested that the user indicates the pattern corners plus four or si mid points and the number of horizontal and vertical squares Adapting second or third order polynomial to these points, it is possible to closely estimate the positions of points at the corners of the squares in the pattern The method was initially tested with a set of images, obtained with a camera whose lens were relatively narrow angle and without obvious geometric distortions Characteristics of the camera used (AVT Marlin): - ½ inch type CCD array - Cell size of 83μm 83μm - Array size of 78 58 piels Characteristics of the lens used: - Focal ength: 1mm - ris range: F14 Close - Minimum Object Distance:,1m - Field of view (Horizontal): 98º for a ½ inch CCD The images used to carry out this test are shown in Figure 6 Stage : in each image patch a corner detection algorithm is applied, based on the Sobel operator for edge detection (Sobel, 1978) Figure 6: mages used for camera calibration in Test 1 The parameters obtained for each test are indicated in Table, z and f are in piel units and the other parameters are in
normalized units The number of iterations needed is presented in the first row and the standard deviation of the parameters residuals, in normalized units, are in the last row Set 1 Set Set 3 Set 4 Set 5 Set 6 iter 5 5 5 6 6 8 49 497 497 449 3913 3899 z 918 963 96 971 95 968 f 154964 154896 15497 15511 15499 15484 λ 9995 9995 9995 κ1 34 374 373 343-634 κ -78-613 17 35 κ3-3446 p1 6 9 p -5-5 std 37 36 36 361 36 359 Table Estimations of the internal parameters The correlations between the parameters are presented in Table 3 This test is useful to assess if any set of parameters are highly correlated, which means that one of them is sufficient to include in the model n Table 3 the highest correlations found are shaded z f λ k1 k k3 p1 z -1 f - 3 λ 17-17 k1 5 1 4 6 k -6-1 -4-7 -97 k3 8 1 7 91-98 p1-9 -1-5 6-8 p -86-1 -1 1-1 -1 Table 3 Correlations obtained between the parameters Another test consisted in re-projecting the points of the panel to the image planes, using the calculated parameters and analysing the differences to the true positions The statistic parameters calculated were the maimum, minima and Root Mean Square of the errors (RMSE) in and z, in piel coordinates, and, also, the maima and minimum of the linear errors, in piels Results of this test are presented in Table 4 Set 1 3 4 5 6 Ma error 188 173 173 175 163 157 RMS error 43 43 43 43 4 4 Ma z error 115 11 11 97 1 19 RMS z error 9 8 8 8 8 8 Ma lin error 19 181 181 183 171 167 Mean lin error 41 4 4 39 4 39 Table 4 Results of re-projection to images (piels) 311 Analysis of the results of Test 1: The method performed stably, since the number of iterations was small Also, the fact that the z scale factor (λ) was very close to 1, indicates that this parameter could be avoided Results in Table 3 point to the fact that the principal point coordinates and decentering parameters, p1 and p, are almost dependent, once their correlation is high So p1 and p could be avoided Table 3 also shows also that the radial distortion parameters are highly correlated, so one would be sufficient to eplain the image radial distortion Finally, the inspection of Table 4 shows that, for this camera, little gain is achieved when calibration parameters were introduced besides the principal point coordinates and focal distance 3 Test The method was also tested with a second set of images, obtained with a camera whose lens were very wide angle and the geometric distortions were more obvious than that of test 1 One of the images obtained with this camera can be seen in Figure 5 Characteristics of the camera used: - ½ inch type CCD array - Cell size of 56μm 56μm - Array size of 64 48 piels Characteristics of the lens used: - Focal ength: Vario 18 36 mm - ris range: F16 Close - Minimum Object Distance:,m - Angle of view (Hor): 97º to 53º for a ¼ inch CCD Results obtained for this test are presented in the same table structure (Table 5, Table 6 and Table 7) Set 1 Set Set 3 Set 4 Set 5 Set 6 iter 7 6 6 6 6 6 33958 3179 3173 3166 3147 3144 z 5344 3569 3559 3566 367 3667 f 5316 584 597 58 58 553 λ 15 16 16 κ1 3541 3195 3194 3181 931 κ 1 1196 1175 374 κ3-4769 p1 3 3 p std 13496 3153 311 319 381 377 Table 5 Estimations of the internal parameters z f λ k1 k k3 p1 z 1 f -1 λ -4 4-14 k1-6 16 4 k -6-18 -5-96 k3 5 19 5 9-98 p1 11 4-6 -8 4 - p -4 1 1 7-9 9 3 Table 6 Correlations obtained between the parameters Set 1 3 4 5 6 Ma error 959 17 11 11 111 111 RMS error 144 33 33 33 33 33 Ma z error 75 1 15 17 94 94 RMS z error 16 7 6 6 5 5 Ma lin error 1 138 13 13 141 141 Mean lin error 138 35 35 35 34 34 Table 7 Results of re-projection onto images (piel units) Results in Table 5 reveal that the method performed well, once the number of iterations was small As before the z scale factor (λ) is very close to 1, indicating that with this camera it could also be dropped Also in this case, as seen in Table 6, the radial distortion parameters are highly correlated, so one parameter would be sufficient to eplain the image radial distortion
Finally, the inspection of Table 7 shows that, for this camera, little gain is achieved when calibration parameters were introduced besides the principal point coordinates, focal distance and κ 1 radial parameter So, in this case, the piel correction model could be given by the following equation: κ Δ r 1 3 f r (13) The division by f is necessary in order to convert normalized values to piel values, once f is the normalization factor Although the distortion reaches large values far from image centre it could be modelled only with k 1 Figure 7 shows the graphical representation of the radial correction radial correction (piels) 14 1 1 8 6 4 1 3 4 5 radius (piels) Figure 7: Radial correction in piel units 4 CONCUSONS The objective of the work presented in this paper was the development of a method to perform the calibration of cameras used in a Mobile Mapping System n this kind of environment it is regularly needed to change the image sensors or its internal parameters So the method has to be quick in its implementation, robust and accurate, since it conditions the final absolute accuracy of the MMS The method developed relies on the approach described by authors such as Heikkila and Silven (1997), Bouguet and Perona (1998) or Zhang (1999), which is a miture of photogrammetric calibration and self-calibration The main features of the developed method are the following: - The calibration object used is a flat panel with a printed pattern in which nearly 15 calibration-points could be rigorously obtained A flat calibration object is easier to produce than a tri-dimensional one - The user needs to obtain 1 images of the calibration panel, varying the place and the orientation of the camera - mage coordinates of the calibration points are etracted semi-automatically, with small user intervention - nitial approimations of the eternal orientation parameters of the camera are automatically obtained for each image - The final internal calibration parameters are obtained after an equation system adjustment based on the collinearity equations The results of the tests made, in order to assess the capabilities of the method, allow for some important conclusions The method is simple and quick in its operation The user only has to take the images and indicate, for each image, 8 corner points of the panel The method performs robustly once, in general, a small number of iterations are sufficient to obtain the results A small number of calibration parameters are enough to model the image distortions This is related to the low resolution of cameras used, as typical of MMS The tests show that the RMSE of the re-projection to the images is, in general, less than half a piel, which indicates high quality of the model obtained For this class of low-resolution cameras, there is little to improve in the calibration process The usefulness of the implemented method was confirmed 5 REFERENCES AND/OR SEECTED BBOGRAPHY Bouguet, J and Perona, P, 1998: Closed-Form Camera Calibration in Dual Space Geometry European Conference on Computer Vision, 1998, Freiburg, Germany Brown, D C, 1966: Decentering Distortion of enses Photogrammetric Engineering, Vol 3, nº3, pp 444-46 Brown, D C, 1971: Close Range Camera Calibration Photogrammetric Engineering, Vol 37, nº8, pp 855-866 Fraser, C, 1997 Digital camera self calibration, SPRS Journal of Photogrammetry & Remote Sensing, nº, pp 149-159 Heikkila, J and Silven, O, 1997: A Four Step Camera Calibration Procedure With mplicit mage Correction Procedures of the nd European Conference on Computer Vision and Pattern Recognition, 1997, pp 116-111, Puerto Rico Madeira, S, Gonçalves, J A, Bastos,, 7: mplementation of a ow Cost Mobile Mapping System Proceedings of The 5th nternational Symposium on Mobile Mapping Technology, Padua, taly Sobel,, 1978 Neighbourhood Coding of Binary images for Fast Contour Following a General Array Binary Processing, Computer Graphics and mage Processing, vol 8, pp 17-135 Wolf, P R and Dewitt, B A, Elements of photogrammetry, with applications in GS, McGraw-Hill, 3rd Edition, pp 33-59 Zhang, Z, 1999: Fleible Camera Calibration by Viewing a Plane From Unknown Orientations Proceedings of nternational Computer Vision Conference, 1999, Vol, pp 666-673