Self-recalibration of a structured light system via plane-based homography

Transcription

1 Pattern Recognition 4 (27) wwwelseviercom/locate/pr Self-recalibration of a structured light system via plane-based homography B Zhang, YF Li, YH Wu Department of Manufacturing Engineering Engineering Management City University of Hong Kong, Kowloon, Hong Kong Received 22 July 25; received in revised form 28 September 25; accepted 3 April 26 Abstract Self-recalibration of the relative pose in a vision system plays a very important role in many applications much research has been conducted on this issue over the years However, most existing methods require information of some points in general three-dimensional positions for the calibration, which is hard to be met in many practical applications In this paper, we present a new method for the selfrecalibration of a structured light system by a single image in the presence of a planar surface in the scene Assuming that the intrinsic parameters of the camera the projector are known from initial calibration, we show that their relative position orientation can be determined automatically from four projection correspondences between an image a projection plane In this method, analytical solutions are obtained from second order equations with a single variable the optimization process is very fast Another advantage is the enhanced robustness in implementation via the use of over constrained systems Computer simulations real data experiments are carried out to validate our method 26 Pattern Recognition Society Published by Elsevier Ltd All rights reserved Keywords: Self-recalibration; Plane-based homography; Eigenvalue decomposition; Relative pose; Cheirality constraint; Structured light system 1 Introduction Camera calibration 3D reconstruction have been studied for many years, but is still an active research topic in robot vision [1 3] This problem is related to structure from motion, stereo vision, pose determination so on, which results in its applications including object modeling, mobile robot navigation localization, environments building In general, the problem of camera calibration 3D reconstruction can be approached in three different ways When both the intrinsic extrinsic parameters of a vision system are known, the 3D reconstruction can be simply realized by traditional triangulation method When the parameters of the vision system are totally uncalibrated, the 3D structure can be reconstructed up to a projective transformation from two uncalibrated images, as concluded by Hartley [4] Faugeras [5] in their early work to address this problem Some efforts have been made to extend these results to Euclidean case by stratification method [6,7] Corresponding author Tel: ; fax: address: meyfli@cityueduhk (YF Li) In between the above two extreme cases, the vision system may be assumed to have some of its intrinsic extrinsic parameters calibrated while others unknown This is referred to as semi-calibrated vision system by some authors Usually, the intrinsic parameters are assumed to be known while the extrinsic parameters need to be calibrated [8 1] Nister s recent work [9] presented a method to solve for the relative pose problem using five non-coplanar points In the semi-calibrated case, some authors have noticed that the relative pose problem can also be solved from the correspondences between images of a scene plane The computational efficiency for the pose problem is of criticalimportance in robotic applications Planar surfaces are encountered frequently in some robotic tasks, eg in navigation of a mobile robot along ground plane wall climbing robot for cleaning, inspection maintenance of buildings Yet, the traditional calibration methods, such as the eight-point algorithm five-point algorithm, will fail or give poor performance in the planar or near planar environments since they require a pair of images from the three-dimensional scene Therefore, methods using only planar information need to be explored Hay [11] was the first to report the observation /$3 26 Pattern Recognition Society Published by Elsevier Ltd All rights reserved doi:1116/jpatcog2641

2 B Zhang et al / Pattern Recognition 4 (27) that two planar surfaces undergoing different motions could give rise to the same image motion Tsai [12] used the correspondences of at least four image points to determine the two interpretations of planar surfaces undergoing large motions, where a sixth-order polynomial of one variable was involved A year late, Tsai [13] approached the same problem by computing the singular value decomposition of a 3 3 matrix containing eight pure parameters Longuet- Higgins [14,15] showed that three dimensional interpretations were obtainable by diagonalizing the matrix, where the relative pose of the system normal vector of the planar surface could be achieved simultaneously by a secondorder polynomial Zhang [16] proposed a method for this problem from a case by case analysis of different geometric situations, where as many as six cases were considered Recently, Chen [17] also proposed a method for recalibrating the structured light system by using planar information However, there are two major differences between our work that of [17]: (1) Our work is based on Homography matrix, while fundamental matrix is used in [17] the minimum number of points required is 4 in our method but is 6 by Chen s; (2) Our method provides an analytic solution, while Chen s method does not since a fourth-order polynomial system with three variables is involved In summary, the existing methods can be divided into two categories One needs to solve high order equations [11 13,17] The other needs to discuss, respectively, many possible cases to obtain solutions from quadric equal-constraints on variables [14 16] In this paper, we will concentrate on the pose problem using planar information in a semi-calibrated vision system Specifically, we will develop a novel method for selfrecalibrating the relative pose parameters using plane-based Homography Here, self-recalibration refers to situations where the system has been initially calibrated but needs to be calibrated again due to changed relative pose Unlike the previous methods, we take two steps to solve the pose problem We will first obtain the translation vector then solve for the rotation matrix The advantages are that not only analytic solutions can be obtained but also an overconstraint system is constructed in each step to increase the robustness Besides, the normal vector of a scene plane can be computed simply after the translation rotation are obtained To solve the ambiguity of correspondence between feature points for environments with non-textured surfaces, we adopt a structured light system [18] referred to as an active vision system Here, the projector can generate many feature points on the surface of the scene by a predefined light pattern However, our method is not restricted to structured light systems It can be adopted for passive stereovision if a textured environment is encountered The remainder of this paper is organized as follows Section 2 describes the calibration task Section 3 presents the method for self-calibrating the relative pose of the vision system in details Section 4 gives some computer simulations real image experiments Finally, we conclude this paper in Section 5 2 Calibration task The structured light vision system here consists of a projector a camera (Fig 1) The projector is controlled by a computer to project onto the scene a light pattern The light pattern will be distorted by the surface of the scene These distortions are captured by the camera used for calibration of the system then reconstruction of the scene Throughout this paper, vectors matrices are denoted by bold case letters Scalars are denoted by Italic letters Superscript T in Greek style presents the transpose of a vector or matrix The bold case letter I always denotes the identity matrix The symbol [ ] represents skew symmetric matrix of a vector For the camera projector, we define a right-hed coordinate system with the origin at their optical centers respectively Let R t be the rotation matrix translation vector from the camera to the projector, the world coordinate system coincides with the camera coordinate system In this paper, the projector is regarded as a pseudo camera the camera is of pinhole model Then the intrinsic parameters of the projector camera can be given via the following two matrices [ f u s u ] K p = f v v (1) 1 [ ] fu s u K c = f v v, (2) 1 Camera Image Plane z c x c y c [R, t] Object y p Light Pattern z p x p Projector Fig 1 Geometrical relations in the vision system

3 137 B Zhang et al / Pattern Recognition 4 (27) where f u f v represent the focal length of the camera in pixel along u- v-axis respectively, (u v ) T is the principal point, s is a skew factor of the camera representing the cosine value of the angle subtended between u- v- axis Similar notations are defined for the projector For an arbitrary 3D point M =[X Y Z] T, its image in the camera the projector can be expressed as m c = αk c M (3) m p = βk p (RM + t), (4) where m c =[u v1] T m p =[u v 1] T are the projection points on the image plane the projector plane, α β are nonzero scale factors Let ] K p R = K p t = [ k1 k 2 k 3 ] k 2 k 3 [ k1 Then from (3) (4), we have four equations on the coordinates of point M AM = a, (5) where A = u k 3 k 1 v k 3 k 2 f u s u u f v v v k 1 k 3 u k a = 2 k 3 v According to (5), the 3D world point on the object surface can be determined by M = (A T A) 1 A T a (6) This formula describes the basic principle for 3D reconstruction using triangulation method Once the intrinsic extrinsic parameters of the camera the projector are obtained, we can compute the 3-D coordinates of object points by the above (5) (6) The whole calibration of the structured light system here consists of two parts The first part concerns the calibration of the intrinsic parameters (such as focal lengths optical centers) of the camera the projector, called static calibration The static calibration needs to be performed only once The second part deals with calibration of the extrinsic parameters of the relative pose in which there are six unknown parameters, three for the 3-axis rotation three for three-dimensional translation (Fig 1) To determine the relative pose between the camera the projector is the focus of this paper 3 Self-recalibration of the relative pose Before giving the method to solve the relative pose, two propositions are established below Proposition 1 Let g be any 3 1 nonzero vector G be a 3 3 nonzero symmetric matrix If [g] G[g] =, then the determinant of G is zero Proof Let g =[g 1 g 2 g 3 ] T [ ] G11 G 12 G 13 G = G 12 G 22 G 23 G 13 G 23 G 33 Since g is nonzero, without loss of generality, we let g 1 = to prove this proposition Exping [g] G[g] =, we have g1 2 G 22 + g2 2 G 11 2g 1 g 2 G 12 =, g1 2 G 33 + g3 2 G 11 2g 1 g 3 G 13 =, g1 2 G 23 + g 2 g 3 G 11 g 1 g 2 G 13 g 1 g 3 G 12 = from which G 22, G 33, G 23 are given as G 22 = g2 2 G g 1 g 2 G 12 /g1 2, G 33 = g3 2 G g 1 g 3 G 13 /g1 2, G 23 = g 2 g 3 G 11 + g 1 g 2 G 13 + g 1 g 3 G 12 /g1 2 (7) Then we substitute (7) into the expression of the determinant of G We obtain det(g) = The proposition can be proved similarly if g 2 = or g 3 = Proposition 2 Let f g be any two 3 1 non-zero vectors The three eigenvalues of the matrix I + fg T + gf T satisfy either (a) or (b): (a) The three eigenvalues are distinct from each other, the middle one is 1 (b) Two of the eigenvalues are both 1 while the third is not 1 Proof Denote I + fg T + gf T as Q let f =[f 1 f 2 f 3 ] T, g[g 1 g 2 g 3 ] T From the definition of characteristic function of Q, we have det(i + fg T + gf T δi) = (8) Exping (8) gives (1 δ)((1 δ) 2 + p(1 δ) + q) =, (9)

4 B Zhang et al / Pattern Recognition 4 (27) where p = 2(f 1 g 1 + f 2 g 2 + f 3 g 3 ) q = (f 1 g 2 f 2 g 1 ) 2 (f 1 g 3 f 3 g 1 ) 2 (f 2 g 3 f 3 g 2 ) 2 Therefore, one of the eigenvalues of Q is 1 The other two are the roots of (1 δ) 2 +p(1 δ)+q = Letting γ=1 δ changes this equation into γ 2 + pγ + q = (1) According to the expressions of p, we know that p = means that f is orthogonal to g From q =, we have f 1 /g 1 = f 2 /g 2 = f 3 /g 3, which indicates that vector f is parallel with g Therefore, p q cannot be zero simultaneously q So there are in total the following two cases: (a) If q =, then q< Thus, the two solutions of γ have different signs by (1) It follows that one of δ is larger than 1 the other is smaller than 1 (b) If q =, then p = By (1), we obtain γ=, γ= p =, from which we get δ = 1, δ = 1 + p = 1 In the following, we will introduce our method in details It should be noted that the camera the projector cannot be located at the same position Thus the translation between them is not zero Also the camera the projector lie on the same side of a scene plane 31 Computation of the plane-based homography Assume there is a plane π in the scene ahose images in the camera the projector are I c I p, respectively Let M be an arbitrary point on the plane Its corresponding projections between the image plane the projector plane are m c m p According to the projective geometry, there is a 3 3 transformation matrix H between I c I p satisfying m p = σhm c, (11) where σ is a nonzero scale factor In general, this matrix H is called plane-based homography Let [ ] h1 h 2 h 3 H = h 4 h 5 h 6 h 7 h 8 1 h = (h 1,h 2,h 3,h 4,h 5,h 6,h 7,h 8 ) T, From (11), each pair of corresponding points gives two constraints on the homography: (u, v, 1,,,, u u, u v)h = u, (,,,u,v,1, v u, v v)h = v (12) So given n(n 4) pairs of corresponding image points of the scene, we have the following 2n equations: Bh = b, (13) where u 1 v 1 1 u 1 u 1 u 1 v 1 u 1 v 1 1 v 1 x 1 v 1 v 1 B = u n u n 1 u n u n u n v n u n v n 1 v n u n v n v n b = (u 1 v 1 u n v n )T Then the homography can be determined up to a scale factor in a least squares sense according to h = (B T B) 1 B T b (14) 32 Constraints from the homography on translation Assuming that the equation of the plane π is n T M = 1, where n T is the normal vector of the plane From (4), we have m p = βk p (R + tn T )M (15) Combining (3) (15) produces m p = β α K p(r + tn T )K 1 c m c (16) By (11) (16), the explicit formula for the homography is λh = K p (R + tn T )K 1 c, (17) where λ is a scalar The equivalent form of (17) is λ H = λk 1 p HK c = R + tn T, (18) where H is the calibrated homography Since H, K p K c are known, H is known Let the translation be t =[t 1 t 2 t 3 ] T Its skew symmetric matrix is [ ] t3 t 2 [t] = t 3 t 1 t 2 t 1 The matrix [t] has some nice properties, for example, [t] t = [t] T = [t] Hence, multiplying matrix [t] on both sides of (18), we have λ[t] H =[t] R (19) It is well known that the right side of (19) is the essential matrix So this equation reveals the relationship between the calibrated homography the essential matrix As R is a rotation matrix, RR T = I From (19), we have λ 2 [t] HH T [t] =[t] [t] (2) Rearranging (2) gives [t] W[t] =, (21) where W = λ 2 HH T I is symmetric λ is an unknown scalar

5 1372 B Zhang et al / Pattern Recognition 4 (27) Determining the scale factor λ Since W is symmetric, according to Proposition 1, we have det(w) = det(λ 2 HH T I) =, (22) Eq (22) indicates that λ 2 is the inverse of one eigenvalue of matrix HH T There are three eigenvalues for HH T Next, we will discuss which eigenvalue would satisfy (2) by Proposition 2 From (18), we have λ 2 HH T = (R + tn T ) (R + tn T ) T (23) This can be equivalently changed into λ 2 HH T = I + Rtn T + tn T R T + n T ntt ( ) ( T ) = I + Rn + nt n 2 t t T + t n T R T + nt n 2 tt = I + st T + ts T, (24) where s = Rn + (n T n/2)t Because the camera the projector lie on the same side of the scene plane are located at different positions, both t s are nonzero Thus, according to Proposition 2, λ 2 HH T or I + st T + ts T, will have one eigenvalue as 1, which lies between the other two different eigenvalues of I + st T + ts T or which is as the eigenvalue with multiplicity two Since the eigenvalues of HH T are 1/λ 2 times of those of λ 2 HH T, we have the following conclusions: (a) If the three eigenvalues are distinct from each other, 1/λ 2 is the eigenvalue of HH T that lies between the other two eigenvalues of HH T ; (b) If one of the eigenvalues of HH T is mulplicity two, 1/λ 2 is this eigenvalue By these conclusions, we can determine the scalar λ that satisfies (2) 34 Determining the translation vector After λ is solved, from (21) there are six homogeneous constraints on the translation vector We assume t 3 = 1 The constraints on the translation vector are the following equations from (21) w 33 t 2 1 2w 13t 1 + w 11 =, w 33 t 2 2 2w 23t 2 + w 22 =, w 33 t 1 t 2 w 23 t 1 w 13 t 2 + w 12 =, w 13 t 2 2 w 23t 1 t 2 + w 22 t 1 w 12 t 2 =, w 23 t 2 1 w 13t 1 t 2 w 12 t 1 + w 11 t 2 =, w 22 t w 11t 2 2 2w 12t 1 t 2 =, (25) where w ij denotes the ij th element of matrix W Clearly, t 1, t 2 can be obtained analytically from the first two equations in (25), which satisfying the last four equations gives two solutions in general In case of noise data, these six equations then are used to do optimization 35 Determining the rotation matrix Since λ, t H have been determined in the previous sections, the left side of (19), say C = λ[t] H is known According to the first second columns of both sides of (19), we have r 21 t 2 r 31 = c 11, r 11 + t 1 r 31 = c 21, t 2 r 11 t 1 r 21 = c 31, r r r2 31 = 1, (26) r 22 t 2 r 32 = c 12, r 12 + t 1 r 32 = c 22, t 2 r 12 t 1 r 22 = c 32, r r r2 32 = 1, (27) where r ij c ij denote the ij th elements of matrix R C, respectively From (26) (27), the first second column vectors of matrix R can be determined analytically, the third column vector of R is then given by the cross product of these two columns 36 Implementation procedure Assuming that the intrinsic parameters of the camera the projector have been calibrated in static calibration stage, when the configuration of the system is changed, the procedure for self-recalibration of a structured light vision system is proposed as follows: Step 1: Computing the homography matrix between the camera plane projector plane according to (14); Step 2: Establishing constraints (21) on t from H determining the scale factor λ by the method in Section 33; Step 3: Calculating the translation vector from (25); Step 4: Calculating the rotation matrix from (26) (27); Step 5: Optionally, the results can be improved by bundle adjustment, after having obtained the relative pose Remark Once the rotation matrix, translation vector, H λ are obtained, the norm n of the scene plane can be linearly determined by the equations λ H R = tn T (28) The normal vector of the plane cannot be determined in this way if the translation vector is null, ie t= Fortunately, this case does not occur in our structured light system When it does happen in a passive vision, we can firstly perform 3D reconstruction using (6) to reconstruct some points on the plane, then compute its normal vector with these points

6 B Zhang et al / Pattern Recognition 4 (27) Experiments 41 Numerical simulation 411 Ambiguity of the solutions As to the ambiguity of the solutions, Tsai [12,13] Longuet-Higgins [14,15] showed how the two possible interpretations of the camera motion could be determined in a closed form from the correspondences in two images of a planar surface Negahdaripour [19,2] determined the relationship between the two solutions in a closed form In addition, the derivation showed the explicit relationship between the ambiguity associated with planar scenes that associated with curved surfaces Knowledge of the explicit relationship between the two interpretations permits the calculation of one solution directly from the other In this experiment, we will show that the results from our method coincide with their conclusions by simulations Here, we assume that the intrinsic parameters of both the camera the projector have been calibrated in the static calibration stage For each repeated experiment, the translation vector three rotation angles of the rotation matrix are selected romly the normal vector of the plane surface is also selected romly in order to cover all cases in practice Here, 1 rom simulations were performed to reveal the ambiguity of the solutions It should be noted that multiple solutions would be obtained by simply solving the given equations discarding the complex ones In order to determine which choice corresponds to the true configuration, the cheirality constraint [21] (the constraint that the scene points should be in front of the cameras) is imposed The final results are shown in Table 1 for distribution of solutions after imposing the cheirality constraint in these simulations Fig 2 shows the graphs of the distributions from Table 1 From these data, we can see that there can only be one or two solutions in most cases, with the probability of about 9565 percent Based on further observations, only one corresponds to the true configuration The other one corresponds to the reflections of the true configuration Here, a twisted pair is treated as two solutions However, the convention as to whether a twisted pair counts as one solution or two solutions varies from author to author (see p 21 in [22]) Therefore, the experimental results from our method coincide with their conclusions 412 Robustness of self-calibration In this test example, we assume that the rotation angles the translation vector between the projector the camera s coordinate system are r = (π/8, π/5, π/4) Table 1 Distributions of number of solutions No of solutions Frequency Frequencies Distribution of the number of solutions Fig 2 Distributions of number of solutions from Table 1 t = (8, 4, 1) respectively There is a plane in the scene whose equation is denoted as Z = X + 2Y + 1 The world coordinate system coincides with that of the camera The projector projects a virtual grid light pattern onto the scene while the camera captures the illuminated scene by an image By romly selecting n = 5 illuminated points on the scene plane, the correspondence on the camera plane projector plane is first identified used for calculating the homography matrix according to (14) Then the rotation matrix translation vector are obtained using the procedure given in Section 36 To test the robustness of the procedure, the residual error, which is the discrepancy between theoretical value computed result, is evaluated Here, different levels of Gaussian noise, ie N(, σ 2 ), were added to the projection points We varied the noise level from to 1 pixel For each noise level, we performed 5 trials calculated the average residual errors of the rotation angles translation vector Since there may exist multiple solutions, the minimum residual error for the rotation angles translation vector is defined as follows min i ( r i r i r r ) ( t i min i t i t t ) (29) where r i t i are the calculated solutions For comparison, we tested Higgins s method [15] under the same condition since this method also gives analytic solutions for the pose problem by analyzing plane-based Homography from two perspective views Here, the mean value stard deviation of minimum residual errors from experimental results are computed The results of the rotation angles translation vector are shown in Figs 3 4, where data1 data2 represent the results from our method Higgins s method respectively From these figures, we can see that both methods work well when the noise is less than 5 pixel When the noise level increases, the residual

7 1374 B Zhang et al / Pattern Recognition 4 (27) Mean value of Rotation: radian data 1 data 2 Mean value of Translation: mm data 1 data 2 (a) Gauss white noise: pixel (a) Gauss white noise: pixel Stard deviation of Rotation: radian data 1 data 2 Stard deviation of Translation: mm data 1 data 2 (b) Gauss white noise: pixel (b) Gauss white noise: pixel Fig 3 Rotation angles vs noise levels: (a) mean values of the residual errors; (b) stard deviation of the residual errors Fig 4 Translation vectors vs noise levels: (a) average results of the residual errors; (b) stard deviation of the residual errors errors increase too the graphs are somewhat bumpy This can be improved if we chose the points used more carefully (eg when the points are more evenly distributed) Furthermore, to reduce the bumpy effect, bundle adjustment can be used to improve the results obtained from the analytic solution On the whole, our method outperforms Higgins s method in the presence of noise data For example, when the noise level is 5 pixel, the average error residuals of our method are 141 ± 86 percent 241 ± 16 percent for the rotation angles translation vector, while the results from Higgins s method are 219 ± 311 percent 346±227 percent In our method, an over-constrained system can be easily constructed, eg six equations are used for solving two variables in (25) to increase the robustness Hence, the stard deviation for translation vector in our method is smaller, as seen in Fig 4b We notice that the estimates of the translation vector are a bit more sensitive to noise than the estimates of rotation This observation has already been discussed by other authors, eg [23] However, our method shows its robustness for the translation vector since an over-constrained system is employed In the simulations, we also compared our method with the classical 8-point algorithm [24] where the fundamental matrix is computed then decomposed directly into rotation matrix translation vector We observed that the residual errors from our method the 8-point algorithm are in a similar range, which validates our method In this paper, our motivation is to recalibrate the vision system using planar information, in which case the 8-point algorithm will fail as it requires points in general 3D positions Furthermore, the minimum number of points required in the 8-point algorithm is 8 while 4 points are sufficient in our method 42 Experiments with real image data In the real data experiments, the system setup consists of two major components, a PULNIX TMC-97 CCD camera a PLUS V131 DLP projector (Fig 5a) The relative pose

8 B Zhang et al / Pattern Recognition 4 (27) Extrinsic parameters (projector-centered) o p z p -1 x p -1-5 y p Fig 6 Ten positions of planar pattern for initial calibration Table 2 Intrinsic parameters obtained from static calibration Parameters f u f v u V Camera Projector Fig 5 The configuration of our structured light system: (a) layout of the experimental system; (b) color-encoded light pattern between the camera the projector can be changed freely If this occurs, the system will perform self-recalibration (instead of static calibration with manual operations) using the procedure proposed in Section 36 The projector is controlled by a computer to generate an illumination pattern (Fig 5b) The illumination pattern consists of many color-encoded grid blocks, which can be used to uniquely identify the correspondences between the projector plane the image plane [25] Here, seven different colors, ie red, green, blue; white, cyan, magenta; yellow, were used According to [25], we can have a matrix of Considering the requirement of our vision system, a4 51 sub-matrix, in which any 3 3 neighbor is unique, was selected The final light pattern is shown in Fig 5b The intrinsic parameters of the camera the projector were firstly calibrated by a planar pattern using Zhang s method [26] Theoretically, two positions of the pattern were sufficient for the calibration task [26] In our experiments, we placed the pattern at ten different positions as illustrated in Fig 6 to increase the calibration accuracy The results are shown in Table 2 This process was required to be performed only once in a static stage in our method When calibrating the extrinsic parameters, four or more point correspondences from a planar surface in the scene were chosen between the projector plane the camera image Then a linear system (13) was constructed for computing the Homography The final result was h = [144, 87, 4995, 791, 32, 39811, 1, 2] Steps 2 4 of the procedure were implemented for the relative pose The experimental results for the rotation matrix translation vector were R =[539, 9978, 378, 8318, 239, 5545, 5524, 613, 8313] t =[ 148, 368, 1], respectively After the system had been self-recalibrated, we performed 3-D object reconstruction using (6) to test the selfrecalibrated results qualitatively Fig 7a gives an image of a fan model, with the light pattern superimposed as illustrated in Fig 7b Fig 7c d show polygonized results of the reconstructed point clouds the CAD model of the clouds After the first experiment, we adjusted the relative pose between the camera the projector to test the system further The self-recalibration procedure was performed again to calibrate the new rotation matrix translation vector Then a phone hle was used for 3D reconstruction The experimental results were shown in Fig 8 It should be noted that in the current experiments, we do not make special efforts in the image processing the initial calibration, as we aim mainly at verifying the validity of our method With more sophisticated methods adopted in the image processing initial calibration, better results can be expected with the proposed method

9 1376 B Zhang et al / Pattern Recognition 4 (27) Our assumption of a planar surface can be satisfied in many practical applications In fact, planar or near planar scenes are encountered frequently, eg roadway or ground plane in mobile robot navigation, walls or ceilings for a climbing robot during cleaning, inspection maintenance of a building The traditional methods will fail or give a poor performance since they require a pair of images from the three-dimensional scene In these cases, our method provides a good solution Acknowledgement The work described in this paper was fully supported by a grant from the Research Grants Council of Hong Kong [Project no CityU126/4E] Fig 7 An experiment on the fan model: (a) a fan model used for the experiment; (b) image of model illuminated by the light pattern; (c) polygonized results of reconstructed point clouds; (d) CAD model of the reconstructed result Fig 8 Another experiment on a phone hle: (a) image of the phone hle illuminated by the light pattern; (b) polygonized results of reconstructed point clouds; (c) CAD model of the reconstructed results 5 Conclusions We have presented a new method for self-recalibration of a color-encoded structured light system When the relative pose between the camera the projector is changed, the system will recalibrate itself automatically, allowing immediate 3-D reconstruction to be followed This method can be used in a dynamic as well as a static application since only a single image is needed The experimental results show that the method is conceptually simple can be efficiently implemented with acceptable results When the scene is textureless, the projector will produce many feature points on the scene s surface by the light pattern These feature points will enable the computation of the Homography matrix for use in the on-line calibration by our method Our method is not restricted to structured light systems It can be adopted for passive stereo vision if a textured environment is present References [1] TS Huang, AN Netravali, Motion structure from feature correspondences: a review, Proc IEEE 82 (2) (1994) [2] Z Zhang, Q-T Luong, O Faugeras, Motion of an uncalibrated stereo rig: self-calibration metric reconstruction, IEEE Trans Robot Autom 12 (1) (1996) [3] A Zomet, L Wolf, A Shashua, Omni-rig: linear self-recalibration of a rig with varying internal external parameters, Proc 8th IEEE Int Conf Comput Vision 1 (21) [4] R Hartley, R Gupta, T Chang, Stereo from uncalibrated cameras, in: Proceedings of IEEE Conference on Computer Vision Pattern Recognition, Urbana, IL, 1992, pp [5] O Faugeras, What can be seen in three dimensions with an uncalibrated stereo rig?, in: L Santa Margherita (Ed), IEEE Proceedings of the European Conference on Computer Vision, 1992, pp [6] T Vieville, C Zeller, L Robert, Using collineations to compute motion structure in an uncalibrated image sequence, Int J Comput Vision 2 (3) (1996) [7] M Pollefeys, L Van Gool, A stratified approach to metric selfcalibration, in: IEEE Proceedings of the Conference on Computer Vision Pattern Recognition, Puerto Rico, 1997, pp [8] HC Longuet-Higgins, A computer algorithm for reconstructing a scene from two projections, Nature 293 (1) (1981) [9] D Nister, An efficient solution to the five-point relative pose problem, IEEE Trans Pattern Anal Mach Intell 26 (6) (24) [1] J Philip, Critical Point Configurations of the 5-, 6-, 7-, 8- point Algorithms for Relative Orientation, TRITA-MAT-1998-MA- 13, February, 1998 [11] JC Hay, Optical motion space perception: an extension of Gibson s analysis, Psychol Rev 73 (6) (1966) [12] R Tsai, T Huang, Estimating three-dimensional motion parameters of a rigid planar patch, IEEE Trans Acoust Speech Signal Process ASSP-29 (1981) [13] R Tsai, T Huang, W Zhu, Estimating three dimensional motion parameters of a rigid planar patch, II: singular value decomposition, IEEE Trans Acoust Speech Signal Process ASSP-3 (1982) [14] HC Longuet-Higgins, The visual ambiguity of a moving plane, Proc Roy Soc London Series B 223 (1231) (1984) [15] HC Longuet-Higgins, The reconstruction of a plane surface from two perspective projections, Proc Roy Soc London Series B 227 (1249) (1986)

10 B Zhang et al / Pattern Recognition 4 (27) [16] Z Zhang, AR Hanson, Scaled Euclidean 3D reconstruction based on externally uncalibrated cameras, in: IEEE International Symposium on Computer Vision, Coral Gables, FL, November 1995, pp [17] SY Chen, YF Li, Self-recalibration of a color-encoded light system for automated three-dimensional measurements, Meas Sci Technol 14 (1) (23) 33 4 [18] B Zhang, YF Li, Dynamic recalibration of an active vision system via generic homography, in: Proceedings of IEEE International Conference on Robotics Automation, Barcelona, Spain, April 25, pp [19] S Negahdaripour, Closed form relationship between the two interpretations of a moving plane, J Opt Soc Am 7 (2) (199) [2] S Negahdaripour, Multiple interpretations of the shape motion of objects from two perspective images, IEEE Trans Pattern Anal Mach Intell 12 (11) (199) [21] Richard Hartley, Chirality, Int J Comput Vision 26 (1) (1998) [22] S Maybank, Theory of Reconstruction from Image Motion, Springer, New York, Inc, Secaucus, NJ, 1992 [23] T Tian, C Tomasi, D Heeger, Comparison of approaches to egomotion computation, in: Proceedings of IEEE Conference on Computer Vision Pattern Recognition, San Francisco, June 1996, pp [24] RI Hartley, Estimation of Relative Camera Positions for Uncalibrated Cameras, ECCV, 1992 pp [25] PM Griffin, LS Narasimhan, SR Yee, Generation of uniquely encoded light patterns for range data acquisition, Pattern Recognition 25 (6) (1992) [26] Z Zhang, A flexible new technique for camera calibration, IEEE Trans Pattern Anal Mach Intell 22 (11) (2)