Accurate estimation of the boundaries of a structured light pattern

Transcription

1 954 J. Opt. Soc. Am. A / Vol. 28, No. 6 / June 2011 S. Lee and L. Q. Bui Accurate estimation of the boundaries of a structured light pattern Sukhan Lee* and Lam Quang Bui Intelligent Systems Research Center, Sungkyunkwan University, Suwon, South Korea *Corresponding author: lsh@ece.skku.ac.kr Received October 27, 2010; revised March 9, 2011; accepted March 16, 2011; posted March 16, 2011 (Doc. ID ); published May 5, 2011 Depth recovery based on structured light using stripe patterns, especially for a region-based codec, demands accurate estimation of the true boundary of a light pattern captured on a camera image. This is because the accuracy of the estimated boundary has a direct impact on the accuracy of the depth recovery. However, recovering the true boundary of a light pattern is considered difficult due to the deformation incurred primarily by the textureinduced variation of the light reflectance at surface locales. Especially for heavily textured surfaces, the deformation of pattern boundaries becomes rather severe. We present here a novel (to the best of our knowledge) method to estimate the true boundaries of a light pattern that are severely deformed due to the heavy textures involved. First, a general formula that models the deformation of the projected light pattern at the imaging end is presented, taking into account not only the light reflectance variation but also the blurring along the optical passages. The local reflectance indices are then estimated by applying the model to two specially chosen reference projections, all-bright and all-dark. The estimated reflectance indices are to transform the edge-deformed, captured pattern signal into the edge-corrected, canonical pattern signal. A canonical pattern implies the virtual pattern that would have resulted if there were neither the reflectance variation nor the blurring in imaging optics. Finally, we estimate the boundaries of a light pattern by intersecting the canonical form of a light pattern with that of its inverse pattern. The experimental results show that the proposed method results in significant improvements in the accuracy of the estimated boundaries under various adverse conditions Optical Society of America OCIS codes: , , , , INTRODUCTION The method of structured-light-based 3D reconstruction has been well established with rich literature available to date [1]. The advantage of structured-light-based 3D reconstruction over other approaches lies in its capability of acquiring a high depth accuracy of, as well as a high density of, reconstructed images within its dynamic range. For structured-light-based 3D reconstruction methods using stripe patterns as a regionbased codec, pattern boundaries may serve as a definite feature for defining the projector camera correspondence on a subpixel level. Therefore, a crucial step for ensuring high depth accuracy is the accurate estimation of the boundaries of a projected light pattern captured on an image. The conventional methods for estimating the boundaries of light stripes can be summarized into the following two (refer to Fig. 1 for illustration): 1. To threshold the captured image signal of a stripe pattern, using the average of the two reference image signals obtained from the projection of the two reference patterns, all-bright and all-dark, as the threshold [Fig. 1(a)]. 2. To obtain the zero crossings or intersections of the two image signals captured, respectively, from the two projected light stripe patterns, the original stripe pattern and its inverse stripe pattern [Fig. 1(b)]. Here, the inverse stripe pattern implies the pattern obtained by reversing the bright and dark signals of the original stripe pattern, respectively, to the dark and bright signals. For more details, refer to Trobina [2]. These conventional methods can accurately estimate the boundaries on the textureless surfaces where only optical blurring is involved in the deformation of captured stripe edge signals, and, thus, no asymmetric deformation is incurred, as illustrated by Fig. 2(a). However, they fail to estimate the correct boundaries when the textures at or around the boundaries of the light stripes incur a large variation in surface reflectance, leading to the severe asymmetric deformation of the captured stripe edge signals, as shown in Fig. 2(b). In Fig. 2(b), due to the abrupt variation of the surface reflectance at the boundary, the captured edge signal of the inverse stripe is deformed asymmetrically to that of the original stripe, causing the zero crossing or intersection of the two signals located far off from that of no asymmetric deformation, such as the one shown in Fig. 2(a), and leading to erroneous boundary estimation. As a matter of fact, the conventional methods are considered not firmly based on a, theoretically well-founded, formal model on the errors involved in boundary estimation. This paper presents a novel (to the best of our knowledge) method for estimating the true boundaries of light stripes under severe deformation due to heavily textured surfaces. Particular interest is given to the accurate estimation of boundaries even for the case of abrupt discontinuities in reflectance. To solve the problem based on a firm theoretical foundation, we formulate a formal mathematical model describing the deformation of the projected light pattern at the imaging end, taking the light reflectance variation, optical blurring, and sensor noise into account. Unlike conventional methods, we transform the edge-deformed, captured pattern signal into the edge-corrected, canonical pattern signal, i.e., a virtual pattern that would have been obtained if neither the /11/ $15.00/ Optical Society of America

2 S. Lee and L. Q. Bui Vol. 28, No. 6 / June 2011 / J. Opt. Soc. Am. A 955 Fig. 1. (Color online) Two boundary estimation methods: (a) to threshold the signal, using the average of the two, all-bright and all-dark, reference signals (shown as the top and bottom signals) as the threshold and (b) to use the zero crossing of the two signals, the signal (blue) captured from an original bright and dark pattern and its inverse signal (red) with the bright and dark pattern inversed from the original pattern. Fig. 2. (Color online) Illustration of the potential error in conventional zero-crossing-based boundary estimation due to the asymmetric deformation between the original (blue) and its inverse (red) signals incurred by the reflectance variation at/around the boundary: (a) no asymmetric deformation present at the boundary on a textureless surface and (b) asymmetric deformation present at the boundary on a textured surface. Fig. 3. Schematics of ray tracing from the pattern projection by a projector to the image captured by a camera.

3 956 J. Opt. Soc. Am. A / Vol. 28, No. 6 / June 2011 S. Lee and L. Q. Bui Fig. 4. Edge blurring as the result of the convolution between step function and Gaussian blur kernel. reflectance variation nor the blurring in imaging optics were present. This transformation is made feasible by the local reflectance indices estimated from the model. The key feature of the proposed method, which is also unique, is the introduction of the edge-corrected, canonical pattern that is independent of the variation of surface reflectance, thus, allowing the boundary estimation to be independent of surface reflectance. The true boundary is estimated by intersecting the canonical form of a captured light pattern with that of its inverse pattern. The originality of this paper is fourfold: (i) the formulation of a formal mathematical model for describing the deformation of a captured light pattern, (ii) the derivation of a new formula for estimating the reflectance indices from the model, (iii) the novelty in transforming an edge-deformed, captured light pattern into an edge-corrected, canonical pattern independent of surface reflectance, and (iv) the estimation of true boundaries using the canonical form of the light patterns. The remainder of the paper is organized as follows: in Section 2, we describe our boundary estimation method. The Fig. 5. Block diagram of the proposed boundary estimation: (a) process of estimating the boundaries with asymmetric deformation based on the zero crossing of canonically recovered signals and (b) overall process of estimating boundaries implemented by combining the two classes of boundaries. The boundaries with asymmetric deformation and with no asymmetric deformation. The conventional zero crossing is applied to the latter for computational efficiency.

4 S. Lee and L. Q. Bui Vol. 28, No. 6 / June 2011 / J. Opt. Soc. Am. A 957 Fig. 6. Deformed edges (blue in left) and their canonically recovered versions (middle) and after smoothing to eliminate noise (right). (Continues on next page) experimental results are provided in Section 3. Finally, Section 4 concludes the paper. 2. PROPOSED METHOD A. Modeling of a Captured Light Pattern Without losing generality, we assume that a projector camera pair is configured in such a way that the depth can be extracted along its epipolar lines independently [3]. This allows us to represent a light pattern in one-dimensional space along an epipolar line. Let sðxþ represent the ideal pattern along an epipolar line on the image, where ideal implies a virtual pattern on the image that would have been generated if no ambient light as well as no noise or disturbance were present along the entire optical passage. Because a projected light pattern is mostly of a binary form of dark and bright stripes, so is sðxþ. For further simplicity, here we represent sðxþ as a step function, where the step refers to an edge of stripes to be recovered, as follows: sðxþ ¼ H x 0 L x < 0 : In practice, sðxþ is subject to deformation and corruption: first, sðxþ is blurred from the optical passage through the projector lens, and then it is attenuated by the reflection from the object surface. Here we represent the attenuation due to the reflection at the local surface corresponding x by the reflectance index, RðxÞ, RðxÞ ½0; 1Š. Note that RðxÞ not only carries the variation of reflection coefficients due to the local differences in color and material, but it also carries the change in the amount of reflecting light due to the differences in the orientation of surface locals to the projector camera pair. In addition to the deformation of sðxþ due to the blurring and surface reflectance, it should be further corrupted from the additive signal, AðxÞ, reflected at the surface by ambient light. Finally, it is blurred once more from the optical passage through the camera lens, before it is contaminated by the noise, WðxÞ, associated with the imaging sensors. The whole process is described in Fig. 3. Taking all the aforementioned factors that deform and corrupt sðxþ into consideration, the captured light stripe signal, f s ðxþ, can be modeled as follows: f s ðxþ ¼ððsðxÞ g p ðx; σ p ÞÞRðxÞþAðxÞÞ g c ðx; σ c ÞþWðxÞ; ð1þ where represents a convolution operator. In Eq. (1), the two blurring processes associated with the projector and camera lenses are modeled as a convolution with the respective blur kernels, g p ðx; σ p Þ and g c ðx; σ c Þ, as illustrated in Fig. 4. The

5 958 J. Opt. Soc. Am. A / Vol. 28, No. 6 / June 2011 S. Lee and L. Q. Bui Fig. 6. (Color online) Fig. 6 Continued blur kernels, g p ðx; σ p Þ and g c ðx; σ c Þ, are chosen to be a normalized Gaussian function with ðx; σ p Þ and ðx; σ c Þ representing their respective (center, variance) pairs: where i is p or c. g i ðx; σ i Þ¼ p 1 ffiffiffiffiffi e 2π σ i B. Estimation of the Reflectance Indices Now, applying Eq. (1) to the two specially chosen, uniform light pattern projections, all-bright and all-dark, RðxÞ can be estimated. Let f 1 ðxþ and f 0 ðxþ represent the light patterns captured on the image in correspondence to the respective allbright and all-dark pattern projections [i.e., s 1 ðxþ ¼H and s 0 ðxþ ¼L]: x2 2σ 2 i ; f 1 ðxþ ¼ðH RðxÞþAðxÞÞ gðx; σ c ÞþW 1 ðxþ f 0 ðxþ ¼ðL RðxÞþAðxÞÞ gðx; σ c ÞþW 0 ðxþ: Assuming that the subtraction of sensor noise [W 1 ðxþ W 0 ðxþ], is small enough compared to that of captured light patterns, we have f 1 ðxþ f 0 ðxþ ððh LÞRðxÞÞ gðx; σ c Þ: The reflection index can now be estimated as follows: RðxÞ deconvlucyððf 1ðxÞ f 0 ðxþþ; σ c Þ ; ð2þ H L where deconvlucy represents the Richardson Lucy deconvolution operator [4,5] that we chose to use for the computation of deconvolution.

6 S. Lee and L. Q. Bui Vol. 28, No. 6 / June 2011 / J. Opt. Soc. Am. A 959 Fig. 7. (Color online) Profile of error in boundary estimation in terms of the degree of asymmetric signal deformation represented by dx: the relative pixel distance between the edge of a light stripe and the edge of a white black texture or of an abrupt change in reflection on an image. It shows that the smaller the dx is, or, equivalently, the more significant the asymmetric signal deformation becomes, the larger are the errors incurred by the conventional zero-crossing method (blue). On the other hand, the proposed zero-crossing method (red) based on canonically recovered signals is able to contain the errors within a small bound regardless of dx or, equivalently, of the significance of the asymmetric signal deformation. C. Canonical Representation of a Captured Light Pattern In order to be able to accurately estimate the boundary of a captured light pattern, we consider transforming the captured light pattern, f s ðxþ, into a canonical form, f c ðxþ, with which we can estimate the boundary independent of RðxÞ. The amount of light from the projector hitting the local surface corresponding to x, [SðxÞ gðx; σ p Þ L], before reflection works well as it eliminates RðxÞ as well as the ambient Fig. 9. (Color online) 3D point clouds of the calibration block using HOC-B (left) and HOC (right) in different views. (a) Full view of visible faces. (b) Front view and (c) top view of left face (plane X ¼ 0). (d) Front view and (e) top view of right face (plane Y ¼ 0). disturbance, AðxÞ, and the blurring in imaging optics. The transformation from f s ðxþ to f c ðxþ, f c ðxþ ¼ðSðxÞ gðx; σ p Þ LÞ, can be done as follows: The captured stripe pattern is subtracted by the reference data in correspondence to the all-dark pattern projection in order to remove the ambient light AðxÞ: f s ðxþ f 0 ðxþ ¼½ðSðxÞ gðx; σ p Þ LÞRðxÞŠ gðx; σ c ÞþðW s ðxþ W 0 ðxþþ: We can approximate as f s ðxþ f 0 ðxþ ½ðSðxÞ gðx; σ p Þ LÞRðxÞŠ gðx; σ c Þ: Thus the amount of light from the projector hitting the local surface corresponding to x is fcðxþ deconvlucyððf sðxþ f 0 ðxþþ; σ c Þ ; RðxÞ Fig. 8. (Color online) Performance of the proposed boundary estimation: (a) light pattern projected on a checker patterned calibration block to produce asymmetric signal deformation in experimentation, (b) boundaries of light stripes estimated by the proposed zero-crossing method with canonical signal representation, and (c) boundaries estimated by the conventional zero-crossing method. or fcðxþ deconvlucyððf sðxþ f 0 ðxþþ; σ c Þ ðh LÞ: deconvlucyððf 1 ðxþ f 0 ðxþþ; σ c Þ ð3þ

7 960 J. Opt. Soc. Am. A / Vol. 28, No. 6 / June 2011 S. Lee and L. Q. Bui Fig. 10. (Color online) Horizontal section of the 3D points of the calibration block. 3D points are reconstructed using (a) the HOC-B version and (b) with the original HOC version. The canonical form of the light pattern, f c ðxþ, computed by Eq. (3) is regarded as correcting the edges of f s ðxþ corrupted by RðxÞ, as well as by AðxÞ and gðx; σ c Þ, thus providing a means of recovering the true boundary embedded in sðxþ. D. Boundary Estimation The true boundary is now estimated by intersecting the canonical form of a captured light pattern with that of its inverse pattern. The block diagram of the proposed boundary estimation is described in Fig. 5(a). In practice, the proposed method is applied to cases of textured surfaces; in the case of a textureless surface, the method of projecting the additional inverse pattern and using the zero-crossing value as the threshold [2] is used. Figure 5(b) describes the block diagram of the general boundary estimation. In order to check the reflectivity of the surface, the reference data are used: f 1 ðxþ f 0 ðxþ ¼ððH LÞRðxÞÞ gðx; σ c ÞþðW 1 ðxþ W 0 ðxþþ ððh LÞRðxÞÞ gðx; σ c Þ: 1. If RðxÞ ¼R, the reflection index is a constant. From a property of the convolution: the convolution of a normalized Gaussian function with a constant A is A. We have f 1 ðxþ f 0 ðxþ ððh LÞRÞ gðx; σ c Þ¼ðH LÞR ¼ constant: Thus the first derivative is Table 1. ðf 1 ðxþ f 0 ðxþþ x ¼ 0: Errors of Reconstructed 3D Data Using HOC and HOC-B Standard Deviation of Error (mm) Plane X ¼ 0 Plane Y ¼ 0 Error Max (mm) Standard Deviation of Error (mm) Error Max (mm) HOC-B HOC ð4þ 2. If RðxÞ is not a constant, the absolute of the first derivative is ðf ðxþ f ðxþþ 1 0 x ¼ ðððh LÞRðxÞÞ gðx; σ c ÞÞ x constant: ð5þ These properties [Eqs. (4) and (5)] help us check the reflectivity of the surface. 3. EXPERIMENTAL RESULTS A. Deformed Edge Recovery When the stripe pattern is projected on a textured surface, depending on the direction and the relative distance between the stripe and the texture, the edge is deformed in different ways. To evaluate our proposed canonical form of the light pattern, a stripe pattern is projected on a textured surface that has a sharp change in reflectance. Then the direction and relative distance between the stripe and the texture are varied to measure the different deformations of the stripe edge as well as its recovered version computed by Eq. (3). The reference data in Fig. 6, especially when the pattern is all-bright, indicate the variation of the surface reflectance. In Figs. 6(a) 6(c), the left-side figures show the deformed edges when the edges and reflectance variation have the opposite direction, and the cases that they have the same direction are shown in the left-side figures of Figs. 6(d) 6(f). The middle figures illustrate the corrected edges using the canonical form of light pattern. However, the correction steps based on deconvolution may amplify the noise; thus, the corrected edges are simply smoothed to eliminate noise, as shown in the rightside figures. B. Evaluating the Proposed Boundary Estimation Method The model of the captured light stripe signal in Eq. (1) can be rewritten as f s ðxþ ¼ððsðxÞ g p ðx; σ p ÞÞRðxÞÞ g c ðx; σ c Þ þðaðxþ g c ðx; σ c ÞþWðxÞÞ:

8 S. Lee and L. Q. Bui Vol. 28, No. 6 / June 2011 / J. Opt. Soc. Am. A 961 In the case of a textureless surface, assuming that the reflectance is a constant, RðxÞ ¼R, thus f s ðxþ ¼ððsðxÞ g p ðx; σ p ÞÞRÞ g c ðx; σ c Þ þðaðxþ g c ðx; σ c ÞþWðxÞÞ ¼ RðsðxÞ g p ðx; σ p Þ g c ðx; σ c ÞÞ þðaðxþ g c ðx; σ c ÞþWðxÞÞ; where [AðxÞ g c ðx; σ c ÞþWðxÞ] is the ambient disturbance and noise, but it plays a role of lifting the ground of the captured signal; RðsðxÞ g p ðx; σ p Þ g c ðx; σ c ÞÞ is the purely captured light stripe. We can see that along the x axis, the stripe edge is only blurred by the optical effects of the projector and camera; thus, the boundary is preserved meaning that on a textureless surface, we can get the correct boundaries by using only the intersection between the stripe pattern and its inverse. To evaluate the proposed boundary estimation method, the relative distance between the stripe boundary and the change of surface texture, namely, dx, is manually varied, then at each position the boundary of the stripe is estimated using both the proposed method and the conventional method. In order to obtain the ground truth, at those same positions, the textured object is replaced by a white object that has the same shape. The stripe patterns are projected and captured, and then the stripe boundaries are estimated to use as references. Figure 7 shows the error between the estimated boundaries using the proposed method as well as the conventional method compared with the reference boundaries. As can be observed, the proposed boundary estimation method is much more accurate than the conventional one. C. Application of the Proposed Method in 3D Reconstruction In our experiments, we used a Canon LDP-3260K projector and a PGR Flea2 IEEE 1394 digital camera mounted with a TV LENS of 8 mm 1:1.3. The resolution of the projector was and of the camera was The position of the camera was about 30 cm on the right of the projector. The distance between the system and the objects was about 1 m. The original Hierarchical Orthogonal Code (HOC [6]) and the HOC with the boundary correction version (namely HOC- B) were implemented and evaluated. The system was calibrated using a calibration block with a coordinate attached on it, as shown in Fig. 8(a); thus, two front planes of the calibration block have the equation X ¼ 0 (left plane) and Y ¼ 0 (right plane), respectively. We also reconstructed 3D data of this calibration block to evaluate the HOC and HOC-B, because it has flat faces, which are easy for quantitative evaluations. Figure 8(a) shows the pattern of layer 4 of HOC on the calibration block. The boundaries of the four patterns estimated by the proposed method are shown in Fig. 8(b), and the ones estimated by the conventional method are in Fig. 8(c). Because the faces of the calibration block are flat, theoretically the detected boundaries of the light stripes should be straight lines. But the variant of the surface reflection deforms the edge of the pattern s stripe, thus the detected boundaries using conventional method are not on straight lines. As can be seen, the proposed method gives better estimation of the boundaries than the conventional method. Figure 9 illustrates the reconstructed 3D point clouds in various views of the calibration block using the HOC-B version on the left and the original HOC version on the right. Figure 10 shows the horizontal sections of the 3D point cloud of the HOC-B [Fig. 10(a)] and HOC [Fig. 10(b)]. Obviously, without boundary correction, the 3D point of the surface has much variation at the boundary of the black and white squares. Table 1 shows the quantitative measurement of the error in 3D data of only deformed edges. The error was defined as the distance between the reconstructed point to the corresponding plane (left plane, X ¼ 0 or right plane, Y ¼ 0). As can be seen, the max and standard deviation of the error were decreased when the boundary correction was applied to the HOC. 4. CONCLUSIONS We have presented a method to estimate the boundary of a structured light pattern on a textured object in order to improve the accuracy of the depth measurement. The idea of the proposed method is to estimate the incident light pattern hitting the object surface, which has the correct information of the boundary. In this paper, the proposed boundary estimator is implemented for HOC, but it also can be applied to other structured light codes, such as gray code and binary code. ACKNOWLEDGMENTS This research was performed for the Intelligent Robotics Development Program, one of the 21st Century Frontier R&D Programs (F ), and in part by the KORUS-Tech Program (KT-2008-SW-AP-FSO-0004) funded by the Korea Ministry of Knowledge Economy (MKE). This work was also partially supported by the Ministry of Education, Science and Technology, Korea, under the World Class University Program supervised by the Korea Science and Engineering Foundation (R ), and by MKE, Korea under ITRC NIPA-2010-(C ) [NTIS-2010-( )]. REFERENCES 1. J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, A state of the art in structured light patterns for surface profilometry, Patt. Recogn. 43, (2010). 2. M. Trobina, Error model of a coded-light range sensor, Technical report BIWI-TR-164 (Communication Technology Laboratory, 1995). 3. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University Press, 2004). 4. W. H. Richardson, Bayesian-based iterative method of image restoration, J. Opt. Soc. Am. A 62, (1972). 5. L. B. Lucy, An iterative technique for the rectification of observed distributions, Astron. J. 79, (1974). 6. L. Sukhan, C. Jongmoo, K. Daesik, J. Byungchan, N. Jaekeun, and K. Hoonmo, An active 3D robot camera for home environment, in Proceedings of IEEE Sensors (IEEE, 2004), Vol. 1, pp