3D Face Reconstruction Using the Stereo Camera and Neural Network Regression

Transcription

1 3D Face Reconstruction Using the Stereo amera and Neural Network Regression Wen-hang heng( 鄭文昌 ) hia-fan hang( 張加璠 ) Dep. of omputer Science and Dep. of omputer Science and nformation Engineering, nformation Engineering, haoang Universit of Technolog. haoang Universit of Technolog. wccheng@cut.edu.tw s94763@cut.edu.tw Abstract n this paper, two-view stereo reconstruction techniques combined with neural network regression method are studied to realize 3D face modeling recognition. Two-view stereo recognition techniques appl dual cameras to etract two images simulated left and right ees. Then searching the real points with world coordinates from the corresponding points in the two images computes depth information. With regard to the points without the corresponding points, depth information is computed b curve-fitting, a feature of neural network. According to the eperimental results, the correct and economical method we proposed reconstructs 3D face model effectivel. Kewords: computer vision, 3D reconstruction, neural network, regression function, face recognition. [0-4] in this paper is introduced to etract interest points to be the corresponding points from two images. n other words, b etracting the feature points, the great intensive points of the gradient between horizontal and vertical direction in the images and artificiall choosing the corresponding points, the vanishing depth information can be calculated. Finall, curve-fitting reconstructs 3D model smoothl. From the eperimental results, our stud can reconstruct 3D face model with complete facial profile, use in face recognition and reduce the cost of 3D scanners for reconstructing face model. Through our stud, face recognition has now matured into dail life and increased the level of data and life securit.. amera Model. ntroduction 3D face recognition techniques, in recent ears, are ver popular research topics. n the past 3D face model has to use 3D scanners; however, 3D scanners at least cost millions of NT dollars, a huge burden in practice. Therefore, stereo reconstruction techniques in computer vision field are the solution: emploing several images calculates the vanishing depth information to reconstruct 3D model [-5]. n previous research, man scholars alread brought up 3D reconstruction methods and applications, which are mainl applied in robot vision and industrial measurement field [6-9]. Two-view theor in computer vision field proposed in this paper calculates the vanishing depth information and reconstruct 3D face model; moreover, in contrast with etracting images of large buildings, camera coordinate calibration is easier. To realize camera coordinate calibration, etrinsic calibrated images are onl applied. Retrieving the etrinsic parameter issue thus is ecluded. n virtue of correctl calibrating camera coordinate, intrinsic parameters are computed b measuring eternal world coordinate data and appling camera geometr equation. n addition, to etract vanishing depth information, the corresponding points have to be found out from two images. Harris orner Detector Figure. Pinhole camera model Figure illustrated, a pinhole camera model, where is the lens center, O image plane center and f focal length. Via sub triangular, world coordinate (X, Y, Z) and image coordinate (, ), we get: X, and () f f Z Y, () f f Z where SF is scale factor. For the unit of an image is piel and space mm, direct computation causes ratio error. n order to convert between these two units, scale factor has to be obtained in advance. First a sequence of coordinates is chosen and based. Then the average movement distance on an image plane of a horizontal move of the reference coordinates etracted from two

2 images can be calculated. This is defined below: N [( ) ] n n, (3) N where N is point number. Scale factor on horizontal direction is: SF, (4) where space. S h n S h is the distance of horizontal movement in 3. Stereo amera Geometr etract the interest points from the great intensive points of the gradient between horizontal and vertical direction in the images. t can be computed as: [ ( ) ( )],, + u, + v, (8) W where u and v refer to and respectivel, (, )is an image intensit value on image coordinate (, ) and W is an image windows function. For simplifing the equation (7), Talor series epansion is emploed to get approimation:, u + uv + v, (9) when (u, v) is concerned with small shift variet,, can be defined as matri below: [ u, v] [ u v] T, M,, (0) where M is the b matri: M. () Let α and β be the two eigenvalues of M matri. When α and β are greater than the threshold, which means the greater intensive points of the gradient variet between horizontal and vertical direction. Figure. Two-view pinhole camera model For eample, figure is two view pinhole camera model simulating human being ees. We have two same cameras and coordinate sstem is calibrated. Where w is world coordinate and B distance between two lens centers. The world coordinates (, ) and (, ) are projected on two images: ( f Z ) X, and (5) f ( f Z ) X. (6) f Z coordinates of w are the same in two camera coordinate sstems; therefore, combining equation (4) with equation (5) can have the following result: f B Z f, (7) where Δ ( - ) is the piel deviation value, which is etracted and corresponded from the images of two cameras on the direction of w coordinates. From equation (6), for the sake of obtaining depth information, the corresponding points from two images have to be etracted. Hence, through our method, interest points from an image first are decided and then matched with feature points from the other image. The Harris orner Detector can (c) Figure 3. Stereo camera reconstruction the eample left camera image, right camera image, (c) reconstruction result. Figure 3 is an eample of stereo camera reconstruction. Two same cameras are used. Both of cameras coordinates sstem are correctl calibrated, and then let two lens centers in parallel have a fied distance to etract two different angular images, i.e. Figure 3 showed. The images are etracted from left and right cameras respectivel. Figure 3(c) shows the reconstruction result.

3 4. Neural Network Regression From figure 3(c), the corresponding point s position can quickl reconstruct depth information through the method we discussed above. t is still obvious that depth information cannot be computed if there is no corresponding point s position on a plane ou want to reconstruct. The better wa makes use of regression computation to get the depth information when no corresponding point s position is on a plane. Since most object surfaces are non-linear, in this paper neural network can realize non-linear regression computation. Figure 4. Architecture of a back-propagation neural network The learning process of neural network can be treated as a curve-fitting issue. Additionall, network input and output relationship is similar with a kind of none-linear input and output mapping. As a result, with the feature of curve-fitting, 3D model can be smoothl reconstructed. Back-propagation neural network [5], a kind of multilaer feedforward networks, in this paper is adopted. Figure 4 shows its architecture. The network is included one input laer, one or more hidden laers and one output laer. The signal inputted outside feedforward propagates from right to left and the output error signal with back-propagation from left to right. Back-propagation neural network training is included two stages: feedforward and back propagation. n feedforward stage, the input signal feedforward propagates to output laer through the hidden laer and network output value is computed. n back-propagation stage, according to error-correcting rule, network weight and bias can be corrected. B the correction, the network output value tends to the epected output value. Standard back-propagation algorithm is a kind of gradient descent algorithm. Weight value moves along negative gradient direction of the activation function. Each output of a neuron is included a non-linear and differential activation function. t is normall displaed with sigmoid function: f ( ). () + ep The learning purpose is to reduce the error between the network output value and the epected value; hence, the error function at the k time circulation can be defined as below: E [ di ( k) i ( k) ], (3) where ( k) i d i is the epected output and ( k) i the actual output. n learning process, that is process to have the error function minimized. The best solution of the error function can be computed b the steepest descent function. Each input data adjusts network weight value: ( ) Δ w k. (4) w According to the hain Rule: ( ) Δw k η, (5) v where η is the rate of learning, which value decides the convergence rates of the function, and v(k) net internal activit level of the neuron. Finall, whole network weight value is adjusted b the equation below: ( ) ( ) w k w k η. (6) v 5. Eperimental Results n our eperiments, two same cameras are used. Both of cameras coordinates sstem are correctl calibrated, and then let two lens centers in parallel have a fied distance to etract two different angular images. As Figure 5 shows, Figure 5 is the image etracted from the left camera and 5 from the right camera. t is however that most areas of human faces are smoother in the gradient variet of an image. To find an interest feature point is hard; as a result, in this eperiment the smooth areas are marked (as Figure 5) beforehand on the cheeks, forehead and profile respectivel to helpfull choose and match the feature points. Figure 5. Face image etraction left camera image, right camera image. Harris orner Detector is used to detect the interest points after two face images are marked. Figure 6 shows the results. The marked points are correctl detected on the obvious points of the gradient variet between horizontal and vertical direction. Net, 0 feature points with the smallest corresponding error choose from these interest points are adopted for reconstructing 3D face. Figure 7 shows the eperimental results. Figure 7 is the top view of 3D face model and 7 the lateral view.

4 Where the redder in the Figure 7, the greater in depth value. On the contrar, the bluer, the smallest. Where the unit for (X, Y, Z) coordinates is millimeter. From the eperimental results, the reconstructed data especiall conform to actual rate of human face. On the other hand, since 0 feature points are discontinuousl scattered on a face image, in order to get complete 3D face model, a method of interpolation with neural network is finall used. n other words, b the application of neural network regression method, the smoother 3D face model can be realized. Figure 8 shows the results of neural network regression. Where the Figure left is the lateral view of the original 3D face model and the Figure right is the effect on curve-fitting result through neural network regression. Where the blue points are the corresponding feature point positions on the face image and green points the approimate result b regression. The results are even close to actual 3D face model. n order to verif the accurac of the 3D face recognition we provide, we are going to proceed the mean squared error calculation with our result and the face model which recognized b the high accurac 3D scanner. First, find the top or the highest part of ever model and fi it as the center of a circle which normall located on the nose. Draw a circle b using r as the radius and record the depth b ever θ distance. Then, take the sample number and the depth of the center in the circle from the depth of the face, and normalize them, here is the method. ma( Z ) Zθ Dθ. (3) ma( Z ) where θ is the distance of the circle which r is the depth of the face, in this eperiment, we use the same sample of the 3D face model, using 4 degrees as the angle we measure and record the depth, and fi the radius as 0, 40 and 60 piel. The eperimental result is as shown in Table. etract two images simulating left and right ees. For having corresponding points part, through equations, images depth information can be computed. As for not having corresponding points part, we propose using neural network regression to solve. Finall, 3D face model can be reconstructed b neural network regression. This method, in the eperimental results, can quickl, efficientl and economicall compute 3D face depth information. As for future research, we are focusing on two directions. () n this paper, we first mark the positions on the face beforehand with detecting interest points to etract left and right corresponding points on images. n practice, it is impossible that a user is marked beforehand. Thus we emplo non-invasive method instead, for eample, illuminants and fringe projection or point light sources. () The reconstructed 3D face model can realize 3D face recognition sstem for improving the accurac of the traditional D face recognition. Figure 6. Feature detection on a face image left camera image, right camera image. 6. onclusion n this paper, two-view stereo reconstruction techniques with neural network regression method realize 3D face reconstruction. Two-view stereo reconstruction techniques appl two cameras to Figure 7. 3D face model reconstruction results top view, lateral view. Table. Mean squared error of 3D Scanner model and reconstruction model. r Avg

5 Figure 8. Lateral view of the 3D face model original model, regression result. Acknowledgement The authors would like to thank the National Science ouncil of the Republic of hina for financiall supporting this research under ontract No. 97--E Reference [] R. Hartle and A. Zisserman, Multiple View Geometr in omputer Vision, ambridge Universit Press, 003. [] D. A. Forsth and J. Ponce, omputer Vision: A Modern Approach, Prentice Hall, 003. [3] O. Faugeras, Three-Dimensional omputer Vision: A Geometric Viewpoint. MT Press, ambridge, 993. [4] M. Pollefes, R. Koch and L. V. Gool, Self-alibration and Metric Reconstruction in spite of Varing and Unknown nternal amera Parameters, nternational Journal of omputer Vision, 3(), pp. 7-5, 999. [5] M. Pollefes, L. V. Gool, M. Vergauwen, F. Verbiest, K. ornelis, J. Tops and R. Koch, Visual Modeling with a Hand-held amera, nternational Journal of omputer Vision, 59(3), pp. 07-3, 004. [6] S. Ganapath, Decomposition of Transformation Matrices for Robot Vision, Pattern Recognition Letters,, pp. 40-4, 984. [7] D. L. ardon, W. S. Fife, J. K. Archibald, and D. J. Lee, Fast 3D Reconstruction for Small Autonomous Robots, The EEE onference of 3st Annual ndustrial Electronics Societ, 6-0 Nov [8] V. arbon, M. arocci, E. Savio, G. Sansoni, and L. D. hiffre, ombination of Vision Sstem and a oordinate Measuring Machine for the Reverse Engineering of Freeform Surfaces, Advanced Manufacturing Technolog, 7, pp. 63-7, 00. [9] P. F. Luo, Y. J. hao and M. A. Sutton, Application of Stereo Vision to Threedimensional Deformation Analses in Fracture Eperiments, Optical Engineering, 33(3), , 994. [0]. Harris and M. Stephens, A ombined orner and Edge Detector, Proceedings of 4 th Alver Vision onference, Manchester: The Universit of Sheffield Printing Unit, pp. 47-5, 988. [] P. D. Kovesi, Phase ongruenc Detects orners and Edges, The Australian Pattern Recognition Societ onference, 003, pp [] M. Trajkovic, and M. Hedle, Fast orner Detection, mage and Vision omputing, 998, 6(): [3] K. Mikolajczk and. Schmid, ndeing Based on Scale nvariant nterest Points, n Proc. V, pages 55 53, 00. [4] K. Mikolajczk and. Schmid, Scale and Affine nvariant nterest Point Detectors, nt l J. omp. Vis., 60():63 86, 004. [5] P. Werbos, Beond Regression: New Tools for Prediction and Analsis in the Behavioral Sciences, Ph.D thesis, Harvard, ambridge, MA, 974.