Calibrating a Structured Light System Dr Alan M. McIvor Robert J. Valkenburg Machine Vision Team, Industrial Research Limited P.O. Box 2225, Auckland

Transcription

1 Calibrating a Structured Light System Dr Alan M. McIvor Robert J. Valkenburg Machine Vision Team, Industrial Research Limited P.O. Box 2225, Auckland New Zealand Tel: , Fax: a.mcivor@irl.cri.nz, b.valkenburg@irl.cri.nz Abstract Calibration of a structured light system is considered in this paper. The system consists of a projector whose output is controlled by a liquid crystal shutter, and a camera to capture the images. A sequence of patterns is projected to temporally encode 256 stripes on the visible surfaces. The transformation from 3D world coordinates to camera image plane coordinates and projector stripe ids are both modelled using perspective transformation matrices. The space of such matrices is characterised and a singular value decomposition procedure is described for estimating their elements. The accuracy of the calibration system so attained is investigated and improvements to the method are suggested. 1. Introduction The structured light system consists of a liquid crystal shutter controlling the output of a light source, and a CCD video camera to capture image of the scene. The liquid crystal shutter generates 8 dierent band images. The combination of these produces a temporal encoding of 256 dierent stripes. The raw output is thus an 8-bit stripe id for each pixel location in the image plane. Given the ray (in the world) that projects onto a particular image pixel, and the plane (in the world) projected by a stripe, the 3D coordinates of their intersection can be calculated. By this triangulation method, the raw sensor data is converted into 3D data. Calibration is the estimation of the parameters of a model that denes these rays and planes. The calibration method used is based on pin-hole optical models of the camera and projector. The transformation from 3D coordinates to camera image plane coordinates is commonly described by a 3 4 perspective transformation matrix (PTM). One of the contributions of this paper is to show that the projector can be similarly modelled by a 2 4 PTM describing the transformation from 3D coordinates to stripe ids. Another contribution is two theorems which describe the subset of 3 4 and 2 4 matrices which are PTMs.

2 2. System Model This section describes models of the camera and projector based on perspective transformation matrices. The physical interpretation of these models is also discussed. 2.1 Camera Model Given a homogeneous coordinate system, X w, in which world 3D positions are specied, and a homogeneous coordinate system, x c, in which pixel locations on the camera sensor are specied, the transformation from world coordinates to image coordinates (i.e., the \imaging" process) can be modelled by a 3 4 PTM [1]: The matrix C c has the following form: x c = C c X w (1) Denition 1 A 3 4 perspective transformation matrix is a matrix of the form C = f x kf y x o c 0 f y y o c h i 7 5 R t where R is a 3 3 rotation matrix, and, f x, f y are non-zero. (2) Physically, R and t dene the coordinate transformation between the world coordinate frame and a camera centric coordinate frame. The parameters f x and f y are the combination of camera focal length and pixel grid scaling factors, and x o c and y o c give the optic centre in image plane coordinates. The parameter k can be interpreted as a shear of the image plane coordinate system. The following theorem is useful in estimating a 3 4 PTM from observed data. Theorem 1 A 3 4 matrix is a 3 4 PTM i the submatrix formed by the rst three columns has rank 3. The condition being necessary follows because, from (2), the submatrix is the product of two matrices with non-zero determinant. Suciency follows from the observation that these two matrices form a QR decomposition of the 3 3 submatrix [2]. Further details of the proof are given in [3]. Given a PTM, there are 4 possible solutions for the decomposition in (2). 2.2 Projector Model The stripe ids can be thought of as a 1D coordinate frame on the projector plane. Then, the transformation from world coordinates to stripe ids (i.e., the inverse of the projection process) can be modelled by a 2 4 PTM between homogeneous world coordinates X w and homogeneous stripe ids x p : x p = C p X w (3) The matrix C p has the following form: Denition 2 A 2 4 perspective transformation matrix is a matrix of the form " # f C = p 0 x o h i p Q s where Q is a 3 3 rotation matrix, and and f p are non-zero. (4)

3 Figure 1: An image of the calibration target showing the fuducial marks and the outer band used for initial position estimation. Physically, Q and s dene the coordinate transformation between the world coordinate system and a projector centric one. The parameter f p is the projector focal length and x o p is the x-coordinate of the intersection of the optic axis and the liquid crystal shutter. Theorem 2 A 2 4 matrix is a 2 4 PTM i the submatrix consisting of the rst three columns is of rank 2. The proof of this theorem follows similarly to the camera case [3]. However, in this case, one parameter of the decomposition (s2) is indeterminant. 3. Implementation Issues Calibration involves estimating the PTMs of the camera and projector from a number of known world points and their corresponding image points and stripe ids. Physically, the world points are implemented as 72 circular fuducial marks (spots), 24 on each of three faces of a cube (Figure 1). The image sequence is processed to obtain the image location and stripe id of each spot center. 3.1 Estimating the Perspective Transformation Matrices We will only consider the task of estimating the camera PTM as the procedure for estimating the projector PTM is identical. Let (X i w ; xi c); i = 1 : : : n be a set of known world-image point pairs. We wish to nd an estimate of the PTM which is consistent with these data points in some optimal sense. Initially we assume that the data points come from a perfect camera which is noiseless and can be modelled exactly by a PTM.

4 Let x c = then where h x c y c 1 B = i T be an image point and ci be the i th row of a 3 4 matrix C, X w 0 0 X w?x c X w?y c X w x c = CX w, Bl = 0 (5) 3 T 7 5 h i T and l = c1 c 2 c 3 (6) If we associate the design matrix B i with the i th world-image point pair and dene A = h i T B T 1 : : : B T n we arrive at the following homogeneous equation. Al = 0 (7) It can be shown that in general rank(a) 11 and that, for n 6 suitably selected world points (non-coplanar and not lying on a twisted cubic passing through the origin of the camera coordinate system), rank(a) = 11 [1]. Assume that X i w ; i = 1 : : : n satisfy these conditions so nullity(a) = 1. Let ^l be any non-zero element of NS(A) and ^C be the associated 3 4 matrix. Let (X w ; x c ) be an arbitrary world-image point pair, B be the associated design matrix and h i T dene A = A T B T. It follows that B^l = 0 as nullity( A) = 1 and NS( A) NS(A). Equation (5) implies that x c = ^CX w and hence ^C is the required PTM. We know that the PTM, C, is only dened up to a scale factor. Hence, we may arbitrarily impose the constraint kck F = 1 (, klk 2 = 1 ) in order to make the problem of nding the PTM (nearly) well dened. Due to deciencies in the camera model, noise on the image coordinates, and noise (imprecision in measuring) on the world coordinates, the matrix A will in general be of full rank. Let S m denote the space generated by the singular vector associated with the smallest singular value of A. S m provides a generalisation of NS(A) in following the sense If l 2 S m then kalk kaqk for all q such that klk = kqk (8) In particular, if A has a non-trivial null-space then S m = NS(A). As previously mentioned the PTM, and hence l, is only required to a scale factor. However kalk depends on klk (i.e. l = kl o ) kalk = jkj kal o k) so it only makes sense to compare solution vectors of xed norm. Equation (8) shows that given a specied norm a vector in S m provides an optimal solution. The following equivalent formulation allows comparison with alternative techniques more readily. If we impose the constraint that klk = 1 then l 2 S m solves min l kalk such that klk = 1 (9) In the above we have ignored the structure of a PTM and just found a matrix which is consistent with the data. We actually want to nd the solution of min kalk such that klk = 1; det l 2 l1 l2 l3 6 4 l5 l6 l7 l9 l10 l = 0 (10) The condition for a matrix to be a PTM stated in Theorem 1 is weak in that the possibility of violating the constraint is very improbable (because the subset of matrices which satisfy

5 the constraint forms a sparse surface in R 12 ). Hence in practice we ignore the constraint, nd a matrix by solving (9) and then check that the matrix satises the determinant constraint and is therefore a PTM. If the solution of (9) satises the constraint it is obvious that it is also the solution of (10). 3.2 Localisation of marks Three types of information are required to perform calibration. It is necessary to know the world location, image location and stripe id of the fuducial spot centers. It is assumed that the cube has been measured and world location of the spots are known to sucient accuracy. The image location and stripe id of all spots are extracted from images taken of the cube. Currently we only use subpixel techniques to extract the image location of the spots and the centroid algorithm is used for this purpose [4]. The task of extracting the subpixel stripe id is more dicult and will be implemented in future versions. 4. Accuracy In [3], details are given of how the PTMs can be decomposed into the intrinsic and extrinsic parameters which describe the camera and projector transformations. The intrinsic parameters obtained in this way for the structured light system agree closely with those derived from the data sheets supplied by the component manufacturers. The PTM equations for the camera and the projector can be combined into TX w = " xc 0 # (11) where T = " c p 11? x pc p 21 c p 12? x pc p 22 C c c p 13? x pc p 23 c p 14? x pc p 24 # (12) and x p is the observed stripe id. Hence, the 3D position of the surface point can be determined by inverting T. Tests of this triangulation procedure show that the 3D data so generated from at surfaces has an RMS error of the order of 1mm and maximum errors of 4.6mm. This is for a variety of surface orientations within the 140mm140mm140mm working volume of the system. See [3] for details. 5. Discussion This paper has considered the calibration of a structured light system. The transformation from 3D to camera image plane coordinates is characterised by a 3 4 PTM and the transformation from 3D coordinates to projector stripe ids is characterised by a 24 PTM. After giving a denition of a PTM, the subset of matrices that are PTMs is determined. The components of the PTMs were determined using a singular value decomposition. It is noted that the subset of (suitably sized) matrices which are not PTMs is small. Therefore, the solution strategy is to solve over the space of all matrices, and check that the solution is indeed a PTM. One of the sources of errors in determining the PTMs is that the minimisation criteria used to determine the elements of the PTM does not have a good physical interpretation.

6 The selection of the PTM can be recast as a nonlinear least squares problem, where the observed image coordinates and stripe ids are compared with those generated by projecting the known 3D world coordinates, given the current PTM estimates. The objective would be to minimise the discrepancy between these. The calibration method described herein would be used as an initial estimate for the nonlinear optimisation scheme. This type of approach is currently the subject of on-going research. A major source of error is the fact that the linear pin-hole optics model does not accurately model an actual physical lens system. The latter suer from radial distortion, etc. Bundle Adjustment techniques exist for incorporating such factors into the optical models [5]. Applying these to structured light system calibration is not straight forward because the distortion model depends on components that are not directly observable for the projector 1. Accurate subpixel stripe estimates are also required. These areas will be considered in future work. References 1 Olivier Faugeras. Three-Dimensional Computer Vision. The MIT Press, Martin Armstrong, Andrew Zisserman, and Paul Beardsley. Euclidean structure from uncalibrated images. In Edwin Hancock, editor, BMVC94: Proceedings of the 5th British Machine Vision Conference, pages 509{518. BMVA Press, September Alan M. McIvor and Robert J. Valkenburg. Calibrating a Structured Light System. Report 362, Industrial Research Limited, February Robert J. Valkenburg, Alan M. McIvor, and P. Wayne Power. An evaluation of subpixel feature localisation methods for precision measurement. In International Symposium on Photonic Sensors and Control for Commercial Applications: Videometrics III. SPIE, November H. M. Karara, editor. Non-Topograhic Photogrammetry. American Society for Photogrammetry and Remote Sensing, 2 edition, The image plane coordinate parallel to the direction of the stripes is not measurable.