Profilometry for the measurement of three-dimensional object shape using radial basis function, and multi-layer perceptron neural networks

Transcription

1 Profilometry for the measurement of three-dimensional object shape using radial basis function, and multi-layer perceptron neural networks Dinesh Ganotra, Joby Joseph, Kehar Singh* Photonics group, Physics Department, Indian Institute of Technology - Delhi, New Delhi , India Received 24 January 2002; received in revised form 10 June 2002; accepted 18 June 2002 Abstract Neural networks have been used to carryout calibration process in fringe projection profilometry for the measurement of three-dimensional object shape. The calibration procedure uses several calibration planes whose positions in space are known. Radial basis function based networks and multi-layer perceptron networks are investigated for the phase recovery. Preliminary studies are also presented for the direct reconstruction of the object without the use of the intermediate step of phase plane calculations. Experimental results are presented for diffuse objects. Keywords: Neural networks; Profilometry; 3-D shape measurement; Phase measurement 1. Introduction In manufacturing applications, non-contact, automated surface measurement is a desirable alternative to micrometers and calipers. Fringe projection techniques may be used to recover depth range data, other methods being Moire topography [1], laser triangulation measurement [2], stereo disparity [3], texture gradient [4], occlusion [5], etc. No one method seems clearly superior to the rest. Jarvis [6] and Marshall and Gilby [7] have discussed the performance of non-contact 3D data capture systems. All methods appear to have drawbacks which fall in one or more of the following categories: missing part problem, computational complexity, time-consuming in improvement of signal/noise ratio, limited indoor application, limited to highly textured or line structured scenes, limited surface orientation, and limited spatial resolution. The use of projection gratings for the measurement is an easy and efficient way to characterize three-dimensional information because the grating is phasesmodulated according to the topography of the object. The method has advantage of being contact free. It has applications in machine vision for automated manufacturing, component quality control, biomedicine, robotics, solid modeling, etc. In fringe projection profilometry a grating (Ronchi or sinusoidal) is projected onto the object

2 292 D. Ganotra et al. / Optics Communications 209 (2002) under study. It is observed from a direction different from that of the projector, and a phasemodulated fringe grating is recorded. There are several techniques to demodulate the phase from the images of the projected grating. It can be estimated using conventional demodulation techniques such as phase-shifting [8], phase-locked loop (PLL) [9], spatial synchronous detection [10], and Fourier transform profilometry [11]. The phase-to-depth conversion requires the physical parameters and optical aberrations of the system to be known. Calibration process in optical system often poses problems. Sometimes in dynamical robot vision systems, parameters such as focal distance of the video camera being used, reference plane position, grating frequency, distance and angle between the camera and the projector, are unknown, making the calibration task difficult. Some experimental factors arise when the object under analysis is located near the projectorcamera system. These factors include diverging fringe projection camera perspective distortion and the use of crossed-optical axes geometry. Problems like camera calibrations, perspective distortions and false matches, delineations, and geometrical consequences of substantially compressed images in photogrammetry are discussed in [12]. Cuevas et al. [13] have presented a method to calculate the object depth distribution from the demodulated phase by the use of radial basis functions (RBFs) based neural networks. Input to RBFs based neural network is the phase distribution from a set of planes inside the region where the depth data are being measured. The network operates as an interpolation function of phase values to the object height in places other than the calibrating planes. Cuevas et al. [14] in another application used multi-layer neural network for such calibrations, by sampling the fringe pattern with a 5x5 pixel window. Set of such patterns and height directional gradients (calculated from pixels within that window) were used as training set to the multi-layer perceptron neural network. They determined the depth of the object using path independent integration of the phase recovered using neural networks. In the present work we have investigated RBF [15], and multi-layer perceptron neural network models using Fourier transform and PLL phase recovery methods for object depth recovery. We have also shown that it is possible to recover the depth information by training the neural network with one spatial period of the projected grating over the object and a reference plane as the elements of the input vectors. 2. The fringe profilometry In fringe profilometry a grating pattern is projected on a 3-D surface. The deformed grating image intensity distribution modulated by height distribution is captured with a CCD camera. Let f be the spatial frequency of the projected Ronchi grating and / 0 and / respectively the phases of the reference plane and object. The images of the grating for the reference plane and object can be written [11,16] respectively as go(x,y) = r o (x,y) J^ A n exp{i«[2n/x + 4> 0 (x,y)]}, n= oo n=oo g{x,y) = r(x,y) J^ A n exp {in[2nfx + (f>(x,y)]}. n= oo (1) (2) Here rðx ; yþ, and r 0 ðx ; yþ are non-uniform distributions of reflectivity on the diffuse object and on the reference plane respectively and A n are the Fourier coefficients for the projected Ronchi grating Phase detection using Fourier transform profilometry (FTP) FTP can work for both Ronchi grating, and sinusoidal grating projection. The deformed fringe pattern is Fourier transformed and processed in its spatial frequency domain to demodulate the object shape from the fundamental frequency component in the Fourier spectra. With suitable filtering, the spectra are filtered to let only the fundamental component (n = 1) pass through. The reference and object planes after the filtering and inverse Fourier transform are written respectively as

3 D. Ganotra et al. / Optics Communications 209 (2002) g o (x,y) =A ir (x,y)exp{i[2nfx + 4> o (x,y)}}, (3) g(x,y) =A 1 r(x,y)exp{i[2nfx + <f>(x,y)}}. (4) The phase change due to the presence of the object with respect to the reference plane is thus y) = 4>(x,y)-U^y)- (5) This can be recovered using the above two filtered functions as y) = Im(log(g(x,y)-g* 0 (x,y))y (6) The depth change can be calculated from the phase change using the transformation (7) 2pfd 1 where d0 and d1 are the distances as shown in Fig. 1. The phase on point 'O' is same as on point 'A'. Point B and A are imaged by same sensor on CCD. Sensor plane and reference planes are conjugate to each other. If instead of a Ronchi grating a sinusoidal grating is projected on the object, then the modulated image of the grating as seen by the imaging system can be expressed as g(x,y) = a(x,y)+b(x,y) cos (<j)(x,y)), (9) where aðx ; yþ and bðx ; yþ are respectively the background intensity and fringe contrast obtained by the system. A quasi-sine grating can be generated from a Ronchi grating by projecting a defocused Ronchi grating on the object surface [17]. Two-dimensional Fourier transform and 2-D Hanning filtering can be applied to provide a better separation of the height information from noise when speckle-like structure and discontinuities exist in the fringe pattern. FTP cannot work well for measurements of steep object slopes, and step discontinuities. Also it needs high-resolution imaging systems Phase detection using phase-locked loop This technique is based on first order digital PLL. In phase-locked detection, the signal is passed through a low pass filter. The phase difference is assumed small enough to consider the linear approximation valid. Cuevas et al. [13] have used a reference plane object reference plane A..---" \ object (a) das (b) grating grating Fig. 1. (a) Experimental setup. (b) Schematic diagram showing points A, B, O and distances d 0 and d 1. PLL based phase recovery method using the equations below. A forward calculation along the x-axis from left to right is accomplished by the following updating rule: \,y) = <j>{x,y)-x[g{x+\,y) ð10þ A second iteration is required in a backward direction through the same line to recover incorrect phase values due to setting time in the dynamic system 4>(x,y) = 4>(x+l,y)-z[g(x+ \,y) ^(x+l,y)}. (11) In this technique an appropriate value of the gain loop parameter s is required, which occa-

4 294 D. Ganotra et al. / Optics Communications 209 (2002) sionally is difficult to determine with wide spectrum and noisy image of the grating Other techniques for phase detection Some of the other techniques for phase detection from the images of the projected grating are spatial synchronous detection, and discrete phaseshifting technique. In our simulations and experiments, we have restricted ourselves to the above two techniques, i.e. Fourier transform profilometry and phase detection using PPL Spatial synchronous detection In spatial synchronous detection [10] the phase can be retrieved assuming that the height information varies slowly as compared with the phase carrier. The phase is calculated by first multiplying the fringe pattern by the sine and then cosine of the carrier signal. Then a low pass filter is applied to both signals to remove the high frequency terms. The ratio of these two signals is then the tangent of the modulated phase. The detected phase given by this ratio is 2p wrapped.! [ h{x,y) * [g{x,y) sin(27i/x)] 1 4>(x,y) = tan h(x,y)* \g(x,y) cosð2pfxþ ; ð12þ where * denotes the convolution and hðx ; yþ transfer function for a low pass filter Discrete phase-shifting technique Upon translation of the original grating by a fraction 1=N of its period p, the phase of the pattern represented by Eq. (9) is shifted [6] by 2p=N. A new intensity function In ðx ; yþ is observed. Using N intensity functions (N = 3 ; 5 ;...), the phase / can be retrieved using <j){x,y) = tan -i ð13þ The selection of an odd number of phase is advantageous and should not be compromised under any circumstances. Here also the phase is wrapped in its principal value p to p. The slope of height variation of the measured object must be limited to prevent spectral components from overlapping when filtering in frequency domain. The phase-shifting phase-measuring profilometry gets its accuracy from the high spatial sampling rate of the evaluated profile. This sampling rate is equal to N times the number of pixels per frame of the high-resolution image of the projected and captured grating, where N is the number of phase steps used in an N-step phase-shifting algorithm. The N-step phase-shifting algorithm is inherently free of errors only if combined with sinusoidal patterns Phase unwrapping Most of the phase detecting formulae use arctan functions to calculate the phase (Eqs. (12) and (13)). The arctan function has a range between p and p. Even FTP gives the phase wrapped in this range. Thus the phases calculated are wrapped in this 2p range. The traditional approach to phase unwrapping is by removal of discontinuities. To reconstitute the continuous phase, the discontinuities are searched sequentially. Discontinuous fringe patterns are not handled even when phase is recovered using PLL properly, because the PLL may go out of the lock creating unwanted transients in the recovered phase [9]. Noise generated phase inconsistencies may be removed by iterative PLL [18]. Here the unknown phase, which modulates the fringe pattern is modified in order to demodulate the fringe pattern iteratively. For the successive iteration, the estimated phase found in the previous iteration is used as a new reference. After each iteration the constant s may be reduced to improve the signal-to-noise ratio in the estimated phase. For unwrapping extremely noisy phase maps, the regularized phase unwrapping technique presented by Servin et al. [19] can be used. Here two phase-shifted fringe patterns are obtained from the unwrapped phase. Phase and frequencies are sequentially adapted to optimise a cost function. It becomes almost impossible to distinguish between discontinuities, which are genuine surface discontinuities and the discontinuities due to the limited range of the arctan function. Other problems that lead to failure in

5 D. Ganotra et al. / Optics Communications 209 (2002) phase unwrapping are discontinuous phase jumps, high-frequency high amplitude noise of region under sampling of the signal, shadow, specklenoise, and height discontinuity. 3. p-phase shifting for reflectivity independent measurements In most of the profilometry experiments, the object is assumed to be a perfect diffuser and the changes in the reflectivity 'aðx ; yþ' are neglected. When the measured object is more complicated, with sharper rises and falls, the zero component resulting from aðx ; yþ cannot be neglected (see Eq. (14)). To eliminate the influence of aðx ; yþ, a grating p phase shift technique [20] is employed. In this technique the grating is moved half of the grating period along the x-axis while the other conditions remain unchanged. Thus the optical field is sampled twice. With the use of a spatial light modulator (SLM), by using a contrast reversed image of the grating, mechanical shifting to introduce p phase shifting can be avoided. Let the two images be gi(x,y) = a(x,y) + b(x,y) cos {Infox + <f>(x,y)), ð14þ gi{x,y) = a(x,y) + b(x,y) cos {2nf 0 x + 4>(x,y) + n). Subtracting Eq. (15) from Eq. (14), we get ð15þ g(x,y) =2b(x,y)cos(2nf 0 x+4>(x,y)). (16) Thus by using this p phase-shifting technique, gðx ; yþ becomes independent of aðx ; yþ and also the search for the first order spectrum in the Fourier transformation of the above becomes easier. 4. Experiment In our experiments we have used objects of size less than 6 mm in height and a laser has been used as the light source. An electrically addressed transmission type SLM (Jenoptik model SLM-M) has been used to project the grating. Though laser light source introduces speckle noise, it was satisfactory for comparison of one process/network from an- Neural Network Low Pass filter Fig. 2. Modified multi-layer perceptron network for direct object reconstruction, using fringes.

6 296 D. Ganotra et al. / Optics Communications 209 (2002) other. Pulnix TMC-76 camera was used to capture the images. A square wave Ronchi grating can be conveniently programmed and displayed in transmission type SLM. The grating periods as well as grating shifts are similarly accomplished by appropriate computer control. The CCD captures four pairs of images. The two images within a pair are p spatial phase shifted with respect to each other. Three pairs are of three reference planes and one pair for the object. The image within the pairs are subtracted from each other using Eqs. (14) and (15) to get reflectivity independent images (Eq. (16)). The phase difference of the reference planes and the object with respect to one of the above three planes are calculated using FTP and also using PLL method. The lowest plane was chosen as the reference plane and the rest of the planes' heights were calculated with respect to it. The differences between the heights of the planes were kept typically 1-2 mm. RBFs based neural network calculate the depth of the object by adjusting weights of a linear combination of Gaussian functions using the experimental reference planes fixing the centers of the RBFs over samples of detected phases from calibration planes. RBFs based network is comparatively less time consuming, as the learning process does not require any iteration. Two matrices are calculated and the inverse of one is multiplied with another. The details of the calculations are described by Cuevas et al. [13]. Though the network operates very fast (as it does not require any iteration) but numbers computed by such networks become very large in magnitude. This leads to a floating-point overflow and often to nearly singular matrices. Their inverse is needed for calculations. For training the RBFs based neural network, we have used the same equations as used by Cuevas et al. [13]. The difference being that their network uses PLL technique to calculate the phase change but we have included results using the FTP technique to calculate the phase change Multi-layer perceptron network Cuevas et al. [13] preferred a RBFs based neural network with Gaussian processor over a multi- 5. Neural network models used for phase to depth conversion A neural network is a multivariate nonlinear mapping. Once the network has been trained, then it can interpolate the intermediate vectors and even extrapolate to some extent. We have used RBFs based networks and multi-layer perceptron network to calculate the height when the phase information is given. y-pixel x-pixel Radial basis functions based network y-pix (b) Fig D reconstructed object using (a) RBF based neural networks and (b) multi-layer perceptron neural network. For both reconstruction same set of reference planes were used. Spatial period of projected fringes 13 pixels. 00

7 D. Ganotra et al. / Optics Communications 209 (2002) layer perceptron network with sigmoid processors and backpropagation training with gradient descent since the learning process requires very little time comparatively. We have included multi-layer perceptron network for comparison of one process with the other. Though these are just visual comparisons of the reconstructed 3-D objects, but they still carry useful information. Seventy-five phase values are sampled randomly from each phase plane as input to the network. Each sample is a three element input vector. Two elements are the indices of the matrix and the third the corresponding phase. The desired output for each sample is the mechanically set height/depth of the corresponding plane. Note that both inputs and outputs are re-scaled to the range ( 1, þ1þ. A three layer perceptron neural network with three input neurons, three hidden neurons and one output neuron is trained with the above set of input-output pairs. The output of the network for an arbitrarily shaped object is discussed in Section Multi-layer perceptron network for direct mapping from fringe patterns In this method for direct mapping of depth from fringe patterns using multi-layer perceptron network, the phases of the object or of the calibration 2 IT VXt v-pixel 0 Ci (a) y-\:i<&\ (b) Fig. 4. (a)-(c) Approximate profiles of the previously measured real objects used for 3-D reconstruction.

8 298 D. Ganotra et al. / Optics Communications 209 (2002) planes are not required to be calculated. Instead, the images were sampled horizontally for the number of pixels same as the period of the image of the grating. The computation of the phases from the image of the grating and then the height from the calculated phase is assumed to be a complicated nonlinear mapping, where neural networks seem to be a suitable choice. A neural network with number of input neurons equal to double the number of pixels in one period of the images of the grating is constructed. We have chosen one period of the projected grating as the inputs to the network. Two such periods from different planes carry sufficient information for the phase change. The neural networks do this mapping on their own. The first half elements of the input vector are the samples of one period length of the reference plane and the second half are the elements with the same indices but from a randomly chosen plane (see Fig. 2). The desired output to the neural network was chosen as the difference between the heights of the two planes. 6. Results and discussion Different diffuse objects like fold of paper, chalks and ground glasses were imaged. They were shaped to have slopes with different gradients within 3-6 mm height. Figs. 3(a) and (b) show 3-D reconstructed object using RBFs based network 400 y-pixel y-pixel x-pixel 400 y-pixel <-pixel Fig D reconstructed object using (a) RBF based neural networks, (b) multi-layer perceptron neural network and (c) reference planes used for the above reconstruction. Spatial period of the projected fringes 10 pixels.

9 D. Ganotra et al. / Optics Communications 209 (2002) and multi-layer perceptron network respectively. A profile of the previously measured real object used is shown in Fig. 4(a). Both figures are for the same object and have been reconstructed using same set of phase planes. It can be observed that RBFs based network gives discontinuities at the corners. These discontinuities were less severe in the multi-layer perceptron training based network. The mean square error between Figs. 3(a) and 4(a) is 0.39 whereas between Figs. 4(a) and 3(b) it is Fig. 5(a) is the reconstructed object using RBFs based network and Fig. 5(b) using multilayer perceptron network, for the profile of the previously measured real object shown in Fig. 4(b). The corresponding phase planes are shown in Fig. 5(c). There are three phase planes of the three reference planes shown in this figure along with the phases of the object. These figures are different from Figs. 3(a) and (b) in spatial frequency of the projected grating. The mean square error (mse) between Figs. 4(b) and 5(a) is 0.54 whereas between Figs. 4(b) and 5(b) it is A plot of standard deviation of the differences in depth values previously measured object and the measured values for different heights for the two networks is shown in Fig. 8. The lower curve corresponds to heights measured using multi-layer perceptron based networks and the upper curve to heights measured using RBF based network. We have tried with the spatial periods from 8 pixels to 16 pixels. Figs. 3(a) and (b) had the spatial period of the projected grating as 13 pixels and Figs. 5(a) and (b) of 10 pixels. A side view of the object Fig. 4(c), reconstructed using multi-layer perceptron algorithm, without filtering is shown in Fig. 6. It can be seen that though the complete profile of the object is not reconstructed, but variation in height can be 10r x-pixel Fig. 6. Side view of reconstructed object (Fig. 4(c)) by direct reconstruction with fringe images and modified multi-layer perceptron algorithm (without filtering).

10 300 D. Ganotra et al. / Optics Communications 209 (2002) seen clearly from the base plane corrupted by the high frequency noise. There are a large number of factors whose effects can be studied to optimize the result, such that coming to definite conclusions in neural networks is very difficult. We attempted to filter these frequencies using Fourier transformation but it worked only partially. The absolute height calculated using the neural network was thus lost during filtering. Since the object retains the shape, we can choose some plane/height/shape within the object and re-scale accordingly. Our future studies will concentrate on improving this reconstruction. Fig. 7(a) shows the same object reconstructed using RBFs based network and the y-pixel x-dixel 8-, 6-, 4-, ^ -4,, -6, y-pixel x-pixel Fig D reconstructed object using (a) RBF based neural network and (b) direct mapping of depth from fringe patterns and multilayer perceptron network.

11 D. Ganotra et al. / Optics Communications 209 (2002) without the use of the intermediate step of phase plane calculations also show that it is possible to get depth/height information directly. The results are found to be encouraging and require further investigations. Acknowledgements One of the authors, D. Ganotra, acknowledges financial support from the Council of Scientific and Industrial Research (CSIR) Government of India. Fig. 8. Plot of standard deviation between the depth values of the object and the measured values for various heights. The lower curve corresponds to multi-layer perceptron network calculated values and the upper curve for RBF based network. same phase planes. Fig. 7(b) is the high frequency filtered image of the reconstructed object. 7. Conclusion A multi-layer perceptron neural network seems to be promising tool in fringe projection profilometry. The calibration procedure using several planes whose positions in space are known can be used to train the neural network. RBFs based networks and multi-layer perceptron networks are studied for the phase recovery. Figs. 3(a) and (b) are reconstructed from the same set of fringe images. They belong to the same object whose sketch is shown in Fig. 4(a). The axes and viewpoints of all the three pseudo 3-D plots are kept same for comparison. It can be easily seen that Fig. 3(b), which is reconstructed using multi-layer perceptron neural network resembles more closely Fig. 4(a) as compared to Fig. 3(a). Similarly, Fig. 5(b), which is reconstructed using multi-layer perceptron neural network resembles more closely Fig. 4(b) as compared to Fig. 5(a). A comparison of the mse for these figures also confirms this. Thus we find that multi-layer perceptron neural networks perform better. Preliminary studies presented for the direct reconstruction of the object References [1] D.M. Meadows, W.O. Johnson, J.B. Allen, Appl. Opt. 9 (1970) 942. [2] X.-X. Cheng, X.-Y. Su, L.-R. Guo, Appl. Opt. 30 (1991) [3] M.D. Levine, D.A. O'handley, G.M. Yagi, Comput. Graph. Image Process. 2 (1973) 134. [4] R. Bajcsy, L. Lieberman, Comput. Graph. Image Process. 5 (1976) 52. [5] A. Rosenfeld, R.A. Hummel, S.W. Zucker, IEEE Trans. Syst. Man Cybern. SMC-6 (1976) 420. [6] R.A. Jarvis, IEEE Trans. PAMI-5 (1983) 122. [7] S.J. Marshall, J.H. Gilby, New opportunities in noncontact 3D measurement, in: Proceedings of the National Measurement Conference 2001, Harrogate, UK. [8] V. Srinivasan, H.C. Liu, M. Halioua, Appl. Opt. 23 (1984) [9] M. Servin, R. Rodriguez-Vera, J. Mod. Opt. 40 (1993) [10] S. Tang, Y.Y. Hung, Appl. Opt. 29 (1990) [11] M. Takeda, K. Mutoh, Appl. Opt. 22 (1983) [12] D.M. McKeown, I.J. Dowman, Proc. SPIE 2486 (1995). [13] F.J. Cuevas, M. Servin, R. Rodriguez-Vera, Opt. Commun. 163 (1999) 270. [14] F.J. Cuevas, M. Servin, O.N. Stavroudis, R. Rodriguez- Vera, Opt. Commun. 181 (2000) 239. [15] S. Haykin, Neural Networks; A Comprehensive Foundation, MacMillan, NJ, [16] X. Su, W. Chen, Opt. Lasers Eng. 35 (2001) 263. [17] X.-Y. Su, W.-S. Zhou, G. von Bally, D. Vukicevic, Opt. Commun. 94 (1992) 561. [18] M. Servin, R. Rodriguez-Vera, D. Malacara, Opt. Lasers Eng. 23 (1995) 355. [19] M. Servin, F.J. Cuevas, D. Malacara, J.L. Marroquin, R. Rodriguez-Vera, Appl. Opt. 38 (1999) [20] J. Li, X.-Y. Su, L.-R. Guo, Opt. Eng. 29 (1990) 1439.