2 DETERMINING THE VANISHING POINT LOCA- TIONS

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE 1 Equidistant Fish-Eye Calibration and Rectiication by Vanishing Point Extraction Abstract In this paper we describe a method to photogrammetrically estimate the intrinsic and extrinsic parameters o ish-eye cameras using the properties o equidistance perspective, particularly vanishing point estimation, with the aim o providing a rectiied image or scene viewing applications. The estimated intrinsic parameters are the optical center and the ish-eye lensing parameter, and the extrinsic parameters are the rotations about the world axes relative to the checkerboard calibration diagram. Index Terms ish-eye, calibration, perspective. 1 INTRODUCTION The spherical mapping o most ish-eye lenses used in machine vision introduces symmetric radial displacement o points rom the ideal rectilinear points in an image around an optical center. The visual eect o this displacement in ish-eye optics is that the image will have a higher resolution in the central areas, with the resolution decreasing nonlinearly towards the peripheral areas o the image. Fish-eye cameras manuactured to ollow the equidistance mapping unction are designed such that the distance between a projected point and the optical center o the image is proportional to the incident angle o the projected ray, scaled only by the equidistance parameter, as described by the projection equation [1], []: r d = θ (1) where r d is the ish-eye radial distance o a projected point rom the center, and θ is the incident angle o a ray rom the 3D point being projected to the image plane. This is one o the more common mapping unctions that ish-eye cameras are designed to ollow, the others being the stereographic, equisolid and orthogonal mapping unctions. In the projection o a set o parallel lines to the image plane using the rectilinear pin-hole perspective model, straight lines will be imaged as straight lines and parallel sets o straight lines converge at a single vanishing point on the image plane. However, in equidistance projection perspective, the projection o a parallel set o lines in worlds system will be a set o curves that converge at two vanishing points, and the line that intersects the two vanishing points also intersects the optical center o the camera. This property was demonstrated explicitly in our earlier work [3]. In that work, it was also demonstrated how the vanishing points could be used to estimate the optical centre o the ish-eye camera, and it was shown that circles provide an accurate approximation o the projection o a straight line to the equidistance image plane. Hartley and Kang [] provide one o the more recently proposed parameter-ree methods o calibrating cameras and compensating or distortion in all types o lenses, rom standard low-distortion lenses through high distortion ish-eye lenses. Other important works on ish-eye calibration includes [5], [6]. While there has been some work done in the area o camera perspective with a view to calibration, there has been little work speciically on wide-angle camera perspective. Early work on calibration via perspective properties included [7], where a hexagonal calibration image was used to extract three co-planar vanishing points, and thus extract the camera parameters. Other work on the calibration o cameras using perspective principles and vanishing points has been done in, or example, [8], [9], [1], [11]. However, none o these publications address isheye cameras with strong optical displacement, i.e., these calibration methods assume that the cameras ollow the pin-hole projection mapping unction. In this paper, we demonstrate how equidistance vanishing points can be used to estimate a cameras intrinsic ish-eye parameters (both the optical center and the equidistance parameter ) and extrinsic orientation parameters (orientation about the world axes), using a standard checkerboard test diagram. Additionally, we compare the accuracy o using circle itting (as proposed in [3]) and the more general conic itting. In Section, we show how the vanishing points can be extracted rom a single ull-rame image o a checkerboard diagram under equidistance projection. To extract the vanishing points, the mapped lines o the test diagram are itted with both circles and the general conic section equation. While the itting o circles is a simpler solution to the calibration problem, we show that the itting o the general conic equation to the mapped lines is the more accurate solution. In Section 3, we show how intrinsic and extrinsic camera parameters can be estimated using the extracted vanishing points and in Section, we test and compare the accuracy o the proposed methods using a large set o synthetically created images, and a smaller set o real images. DETERMINING THE VANISHING POINT LOCA- TIONS In this section we will describe how the vanishing points can be extracted rom a ull-rame equidistance projected wide-angle image o a traditional checkerboard test diagram, such as that shown in Fig 1. The checkerboard diagram consists o two sets o parallel lines, each with a corresponding pair o vanishing points. We will denote the ish-eye vanishing point pair corresponding to the approximately horizontal direction as w X,1 and w X,, as this is the world axis system that these vanishing points are related to. Similarly we denote the vertical vanishing point pair as w Y,1 and w Y,. I the ish-eye

2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE Fig. 1. Fish-eye checkerboard test diagram. Fig. 3. The two sets o circles, convergent with the our vanishing points, that correspond to the checkerboard diagram in Fig. 1. Fig.. Extracted edges or use in calibration. (a) Vertical edges, and (b) horizontal edges. vanishing points w X,1, w X,, w Y,1 and w Y, are known, given the optical center and equidistance parameter, the corresponding rectilinear vanishing points v X and v Y can simply be determined by undistorting w X,1 and w Y,1 or w X, and w Y,..1 Using circles Here we describe a method o extracting the vanishing points by itting circles to the equidistance projection o checkerboard lines. Strand and Hayman [1], Barreto and Daniilidis [13] and Bräuer-Burchardt and Voss [1] suggest that the projection o straight lines in cameras exhibiting radial distortion results in circles on the image plane. Bräuer-Burchardt and Voss, in contrast to the others, explicitly use ish-eye images in their examples. All three methods assume that the camera system ollows the division model. However, in our previous work [3], we demonstrate the equidistance projection o straight lines can also be modelled as circles..1.1 Extracting the vanishing points The irst step to extract the vanishing points rom a isheye image o the checkerboard test diagram is to separate the curves into their vertical and horizontal sets. Here, or example, we used an optimized Sobel edge detector [15] to extract the edges. The gradient inormation rom the edge detection was used to separate the edges into their horizontal and vertical sets. Discontinuities in lines caused by corners in the grid are detected using a suitable corner detection method (or example, [16]), and thus corrected. Fig. shows edges extracted rom the checkerboard test image in Fig. 1. Schaalitzky and Zisserman [17] describe how the vanishing point or a non-ish-eye set o rectilinear projected concurrent lines can be estimated using the Maximum Likelihood Estimate (MLE), to provide a set o corrected lines that are exactly concurrent. The method described can be easily extended to support arcs o circles, and two vanishing points, rather than lines and single vanishing points. This is described in detail in [3]. Fig. 3 shows the two sets o itted circles with the MLE determined vanishing points.. Using conics It is proposed that any error in using circle itting can be reduced by allowing a it to a more general conic equation. This gives the it greater degrees o reedom, and thereore greater potential or accuracy. A conic on the image plane can be described as: Au + Buv + Cv + Du + Ev + F = () Zhang [18] describes several methods o itting conics to image data. Barreto and Araujo [19] also describe in detail methods o itting conics to projected lines, though in that case the itting is or paracatadioptric projections (i.e. projections involving a camera that consists o lens and mirror arrangements), rather than equidistance projections. In this case, we use a method similar to that o using circles, i.e., the edges are extracted in the same manner, and the MLE extraction o the vanishing point is used. However, instead o using a least-mean-squares (LMS) algorithm to it circles, we use a similar algorithm to it the general equation o a conic (which has 5 degrees o reedom, compared to the 3 degrees o reedom that the equation o a circle has). Other than using conic itting, the optimization and other steps are the same as or the circle it. 3 DETERMINING THE CAMERA PARAMETERS We now describe how, having estimated the locations o the vanishing points (by either circle or conic itting), the intrinsic ish-eye parameters and the orientation parameters o the camera can be extracted. In Section, we examine the accuracy o using circle and conic itting to determine the parameters.

3 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE 3 Fig.. Estimating the optical center and the equidistance parameter. The vanishing points in equidistance projection, w 1 and w, are given by [3]: [ ] where w 1 = [ [ w = [ D x D y D x D y D x D y D x D y ] ] ] E, ϕ D vp x +Dy ( E), ϕ D vp < x +Dy (E π), ϕ D vp x +Dy (π E), ϕ D vp < x +Dy (3) () E = arctan Dx + Dy (5) Where D = [D x, D y, D z ] is the vector that describes the direction o the parallel lines in the scene, and ϕ vp is the angle the line that intersects the vanishing point and the optical centre makes with the u-axis o the image plane ( π < ϕ vp π ). In our prior work [3], it was demonstrated that the line that joins the two vanishing points (the horizon line) also intersects the optical center. From the previous section, two pairs o vanishing points (with their two corresponding horizon lines) can be estimated rom a ish-eye image o a checkerboard test diagram. The intersection o these two horizon lines is, thereore, an estimate o the optical center (Fig. ). To estimate, we examine the distance between the vanishing points described by (3) and (), which is: w 1 w = (w 1,u w,u ) + (w 1,v w,v ) (6) It is straightorward to show that this reduces to: π w 1 w = DX + D D X + π Y DX + D D Y Y D z = π = π (7) Fig. 5. Rotation o the camera/world coordinate systems Thereore, given the vanishing points, can simply be determined as: = w 1 w (8) π As shown in Figs. 3 and, there are two distances d X and d Y that have corresponding equidistance parameters X and Y. The estimated value or is simply taken as the average o X and Y. Given the ish-eye parameters, the aim is to convert the ish-eye image to a rectilinear image. This is done by converting the points in the equistance image plane to their equivalent points on the rectilinear (pin-hole) image plane. Rectilinear projection is described as: r u = tan (θ) (9) The unction that describes the conversion o an equidistance projected point to a rectilinear projected point is obtained by combining (1) and (9): ( ) rd r u = tan (1) and its inverse: r d = arctan 3.1 Camera orientation ( ) ru (11) In previous sections, we assumed that the world coordinate system was centered on the center o projection o the camera. However, in order to take into account rotation between the camera coordinate system and the test diagram, it is more convenient or the world coordinates to be located at the test diagram, as described in Fig. 5, and rotated away rom the camera coordinate systems. Extending the rectilinear pinhole camera model to include translation and rotation gives x u = K[R T]X (1) where R is a 3 3 rotation matrix and T is a three element column vector describing the translational mapping rom world coordinates to camera coordinates. Gallagher [] describes how the location o the vanishing

4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE points is independent o the translation vector T, and is given by V = [v X, v Y, v Z ] = KR (13) where V is the matrix o vanishing point locations, made up rom the homogeneous column vectors v X, v Y and v Z that describe the vanishing points on the image plane related to each o the X-, Y - and Z-axes respectively. Under rotation about the X- and Y - axes only, the vanishing points become [] V XY = cos α sin α sin β sin α cos α sin β cos α cos β sin α sin α cos β cos α (1) This is equivalent to irst rotating about the Y -axis by angle β, and then rotating about the original X-axis by angle α. It is important to note that ater the homogeneous v Y is converted to image plane coordinates, it is constrained to all on the image plane v-axis. Adding urther rotation about the Z-axis by angle γ results in where R Z = V = R Z V XY (15) cos γ sin γ sin γ cos γ 1 (16) Thus, rotating about the Z-axis results in the vanishing points V XY being rotated by the same angle. The method or estimating v X and v Y was described in the previous section. Considering that v Y is constrained to all on the v-axis in image plane coordinates, the angle o rotation about the Z-axis can be determined by measuring the angle γ that v Y makes with the v- axis at the origin []. Given v X, v Y and γ, the other two orientation parameters can be determined rom the irst two column vectors in (15). Thus, given the three orietation parameters, the rotation matrix R can be constructed and used to rectiy a given image. RESULTS.1 Synthetic images To determine the accuracy o the parameter estimation, the algorithm was run on a set o synthetic images. The synthetic images were created with a set o known parameters, and a number o imperections were added (sharpness, noise, and imperect illumination). The method used to create the synthetic images is the same as that described in [3]. Fig. 6 shows two o the synthetically created calibration images. A total o 68 synthetic images were created, and the equidistance parameter, optical center and orientations were estimated using both circle itting and conic itting. Fig. 7 shows the distributions o the errors o the parameter estimates or the synthetic images, and Table 1 shows the mean and standard deviations o the errors. The errors are calculated as the dierence o the estimated parameters and the modeled parameters. It Fig. 6. Synthetic image samples. TABLE 1 ERRORS OF THE PARAMETER ESTIMATIONS FOR THE SYNTHETIC IMAGE SET Circle Fitting Conic Fitting µ σ µ σ c u c v α β γ The mean µ and the standard deviation σ o the errors o the camera parameter estimation using the synthetic images., c u and c v are in pixels; α, β and γ are in degrees. can be seen rom both Fig. 7 and Table 1 that, in general, both the average error and the deviation are smaller or the conic itting method than the circle it method. This is particularly true or the error in the estimation o or the circle itting method, which is considerably higher than or the conic itting method (Table 1). The error in estimating c v is larger than that or c u or both the circle and conic itting methods. This can be explained by the act that the estimation o c u is primarily determined by the horizontal horizon line. An error in the estimation o the vertical horizon line will have less o an impact on the estimation o c u than an error in the estimation o the horizontal horizon line. The opposite is true or the estimation o c v. Considering that the horizontal curves imaged in, or example, Fig. 6 have more data points (6 data points as opposed to 8), it is reasonable to assume that the circle/conic itting to these imaged curves will be more accurate than to the vertical curves. Thereore, the estimation o the horizontal horizon line will be more accurate than the vertical, and thus the estimation o c u tends to be more accurate than c v.. Real Images In this section, we examine the accuracy o the calibration algorithms using real images o the checkerboard calibration diagram captured using three dierent wide-angle cameras. For each camera, 15 images were captured and processed using the parameter estimation algorithms. Fig. 8 shows the results rom the parameter estimations or each image rom each camera. The results in

5 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE 5 estimation error (pixels) Synthetic image number estimation error (pixels) Synthetic image number v estimation error (pixels) u estimation error (pixels) u estimation error (pixels) v v estimation error (pixels) γ - γ - - α - β - α - β Fig. 7. Parameter estimation errors using synthetic images. (a) Distribution o the estimation errors o using circle itting, and (b) using conic itting. (c) Distribution o the optical center estimation errors using circle itting, and (d) using conic itting. (e) Distribution o the camera orientation estimations using circle itting (in degrees), and () using conic itting (in degrees). The resolution or all images was 6 8. general back up the synthetic image results. The estimation o using circle itting returns a higher value or than the conic itting method. While the ground truth data is unknown, the synthetic image results suggest that the result rom the conic itting is closer to the actual equidistance parameter o the camera. This is supported by the act that the distribution o the estimations using conic itting tends to be smaller than the estimations using circle itting. The results or the optical center estimation are similar or both cases, with the circle itting again showing a slightly larger distribution than the conic itting. These statements are supported by the data in Table, which shows the means and standard deviation o the sample real images or each camera. Note that Table gives absolute averages and standard deviations or the parameters o the real cameras, in contrast to Table 1 which gives averages and standard deviations or the errors in the parameter estimation. The orientation parameter estimation errors are more diicult to quantiy or the real images. The intrinsic parameters remain constant or each camera over the 15 images, while the extrinsic parameters, being the orientation o the camera against the checkerboard test diagram, will be dierent or each image due to the manual set up o the calibration rig. Table 3 shows the results or the orientation estimation or each o the 15 images rom one o the cameras. The results or both the circle and the conic itting methods are similar (never more than.5 o a degree in the dierence), which suggests that the error in using either method will be similar. Additionally, the results rom the synthetic images suggest that this error will be small. Fig. 9 shows a checkerboard image rom Camera 3, transormed to the rectilinear image using (11) with a back mapping method [1] and bilinear interpolation. Rotation is removed using the parameters rom both

6 6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE Camera Camera 3 (pixels) 3 3 Camera 1 (pixels) Camera 8 Camera Image number Camera Image number Camera 1 Camera Camera 1 Camera 3 u (pixels) 3 u (pixels) 3 1 Camera v (pixels) 1 Camera v (pixels) Fig. 8. Parameter estimation errors rom real images taken rom three cameras (15 images per camera). (a) Estimation o using the circle itting method or each image rom each camera; (b) using the conic itting method; (c) distributions o the optical center estimation using the circle itting method; (d) using the conic itting method. Camera 1 is a 17 ield-o-view equidistance ish-eye camera; Camera has a 163 ield-o-view; Camera 3 is a moderate wide-angle camera with a 13 ield-o-view. The sample resolution o all images was 6 8 pixels. TABLE INTRINSIC FISH-EYE PARAMETER ESTIMATIONS USING REAL IMAGES TABLE 3 EXTRINSIC ORIENTATION PARAMETER ESTIMATIONS USING REAL IMAGES Camera 1 Camera Camera3 µ σ µ σ µ σ Circle Fitting c u c v Conic Fitting c u c v The mean µ and the standard deviation σ o the errors o the camera parameter estimations o the three cameras (15 images per camera) used in the experiments. All values are in pixels. Camera 1 is a 17 ield-o-view equidistance ish-eye camera; Camera has a 163 ield-o-view; Camera 3 has a 13 ield-o-view. circle and conic itting methods, using the determined rotation matrix R and also using back mapping. It can be seen that using the parameters obtained rom the conic itting method returns an image with a greater degree o rectilinearity. This supports the results rom the synthetic images that the estimation o is more accurate using conic itting. Fig. 1 shows urther examples rom Camera. Circle Fitting Conic Fitting Im. # α β γ α β γ The estimated orientation parameters or the 15 calibration images o Camera 1. All values are in degrees. Similar results were obtained or Cameras and 3. 5 CONCLUSIONS In this paper we have demonstrated a method or extracting all the camera inormation required to provide

7 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE 7 Fig. 9. Rectiication o a checkerboard image rom Camera 3. (a) Original image; (b) calibrated using circle it parameters; (d) using conic it parameters; (e) rectiication using circle it parameters; () rectiication using conic it parameters. a rectilinear, rectiied image rom a ish-eye camera. We have presented two methods: circle-itting and conic itting. While the itting o circles is undoubtedly a simpler method than itting the general conic section equation to the project lines, we have shown that the itting o conics returns better results, particularly in the equidistance parameter estimation. The estimation o the opical center is a necessity. Hartley and Kang stated that (in a 6 8 image) the optical center can be as much as 3 pixels rom the image center []. However, we have shown here that it can be greater than that with some consumer cameras. Our results rom Camera 1 show a optical center that is almost 5 pixels rom the image center, and Camera is over pixels. These deviations are signiicant, and should not be ignored in ish-eye camera calibration. In our tests on real images using moderate wide-angle to ish-eye cameras, the optical center estimation appears to be at least as good as the results presented by Hartley and Kang []. However, Hartley and Kang use averaging to reduce the eect o noise in their optical center estimations, which we have ound is not necessary or our method. The estimation results o, particularly or the conic itting method, also show a low distribution. Future work in the area would involve the examination o similar perspective properties or other lens models, as well as the examination o the eect o radial distortion parameters or ish-eye lenses that do not ollow a particular model accurately. ACKNOWLEDGMENTS This research was co-unded by Enterprise Ireland and Valeo Vision Systems (ormerly Connaught Electronics Ltd.) under the Enterprise Ireland Innovation Partnerships Scheme. REFERENCES [1] K. Miyamoto, Fish eye lens, Journal o the Optical Society o America, vol. 5, no. 8, pp , Aug [] C. Slama, Ed., Manual o Photogrammetry, th ed. American Society o Photogrammetry, 198. [3] C. Hughes, R. McFeely, P. Denny, M. Glavin and E. Jones, Equidistant (θ) ish-eye perspective with application in distortion centre estimation, Image and Vision Computing, vol. 8, no. 3, pp , Mar. 1. [] R. Hartley and S. B. Kang, Parameter-ree radial distortion correction with center o distortion estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 9, no. 8, pp , Aug. 7. [5] S. Shah and J. K. Aggarwal, Intrinsic parameter calibration procedure or a (high-distortion) ish-eye lens camera with distortion model and accuracy estimation, Pattern Recognition, Vol. 9, no. 11, pp , Nov [6] F. Devernay and O. Faugeras, Straight lines have to be straight: automatic calibration and removal o distortion rom scenes o structured environments, International Journal o Machine Vision and Applications, vol. 13, no. 1, pp. 1, Aug. 1. [7] L. L. Wang and W.-H. Tsai, Camera calibration by vanishing lines or 3-d computer vision, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no., pp , Apr

8 8 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.??, NO.??, DATE Fig. 1. Further rectiication examples. (a) Checkerboard calibration diagram aligned with the rear wall o the room; (b) the extracted edges and corners; (c) original image; (d) using parameters rom the circle itting, and (e) conic itting; () rectiied using parameters rom the circle itting, and (g) conic itting. [8] B. Caprile and V. Torre, Using vanishing points or camera calibration, International Journal o Computer Vision, vol., no., pp , Mar [9] N. Avinash and S. Murali, Perspective geometry based single image camera calibration, Journal o Mathematical Imaging and Vision, vol. 3, no. 3, pp. 1 3, Mar. 8. [1] G. Wang, J. Wua, and Z. Jia, Single view based pose estimation rom circle or parallel lines, Pattern Recognition Letters, vol. 9, no. 7, pp , May 8. [11] Y. Wu, Y. Li, and Z. Hu, Detecting and handling unreliable points or camera parameter estimation, International Journal o Computer Vision, vol. 79, no., pp. 9 3, Aug. 8. [1] R. Strand and E. Hayman, Correcting radial distortion by circle itting, in Proceedings o the British Machine Vision Conerence, Sep. 5. [13] J. P. Barreto and K. Daniilidis, Fundamental matrix or cameras with radial distortion, in Proceedings o the IEEE International Conerence on Computer Vision, vol. 1, pp , Oct. 5. [1] C. Bräuer-Burchardt and K. Voss, A new algorithm to correct ish-eye- and strong wide-angle-lens-distortion rom single images, in International Conerence on Image Processing, vol. 1, pp. 5 8, Oct. 1. [15] B. Jähne, Digital Image Processing, 5th ed. Springer-Verlag,, ch. 1. [16] Z. Wang, W. Wu, X. Xu, and D. Xue, Recognition and location o the internal corners o planar checkerboard calibration pattern image, Applied Mathematics and Computation, vol. 185, no., pp , Feb. 7. [17] F. Schaalitzky and A. Zisserman, Planar grouping or automatic detection o vanishing lines and points, Image and Vision Computing, vol. 18, no. 9, pp , Jun.. [18] Z. Zhang, Parameter estimation techniques: a tutorial with application to conic itting, Image and Vision Computing, vol. 15, no. 1, pp , Jan [19] J. P. Barreto and H. Araujo, Fitting conics to paracatadioptric projections o lines, Computer Vision and Image Understanding, vol. 11, no. 3, pp , Mar. 6. [] A. C. Gallagher, Using vanishing points to correct camera rotation in images, in Proceedings o the Second Canadian Conerence on Computer and Robot Vision, May 5. [1] K. V. Asari, Design o an eicient VLSI architecture or non-linear spatial warping o wide-angle camera images, Journal o Systems Architecture, vol. 5, no. 1, pp , Dec..