3D FACE RECONSTRUCTION BASED ON EPIPOLAR GEOMETRY

Transcription

1 IJDW Volume 4 Number January-June 202 pp D FACE RECONSRUCION BASED ON EPIPOLAR GEOMERY aher Khadhraoui, Faouzi Benzarti 2 and Hamid Amiri 3,2,3 Signal, Image Processing and Patterns Recognition (SRIF) Laboratory, National School Engineering of unis (ENI), unisia khadhra.th@gmail.com Abstract: he aim of 3D reconstruction is to retrieve the 3D geometric representation of an object either from a single image, several images or a sequence images. It has been used in a number of application domains, with specific methods for buildings and towns or body parts. 3D face reconstruction is a technology used for reconstructing three-dimensional face geometry from media such as images and videos. his paper presents a global scheme for 3D face reconstruction into a limited number of analytical patches from stereo images. From a depth map, we generate a 3D model of the face. Keywords: component; Stereo Vision; 3D Face Reconstruction; Face Alignment. INRODUCION he recognition and the identification of faces play a fundamental role in our social interactions. Various applications in Computer Vision, Computer Graphics and Computational Geometry require a surface reconstruction from a 3D point cloud extracted by stereovision from a sequence of overlapping images [2]. he process of construction 3D facial models is an important topic in computer vision which has recently received attention within the research community. 3D face reconstruction from 2D images is very important for face recognition and facial analysis. he epipolar geometry between two views is essentially the geometry of the intersection of the image planes with the pencil of planes having the baseline as axis (the baseline is the line joining the camera centres). his geometry is usually motivated by considering the search for corresponding points in stereo matching, and we will start from that objective here. he notion of Epipolar Geometry is a strong tool for us to use in Computer Vision. he essential idea is that, for a stereo pair of cameras, the projection of 3D points on a camera screen will lie on a plane defined by the two camera centers and the 3D points. Furthermore, any such plane will always lie on a particular pair of points on both image screens, called the Epipoles. his idea means that, given knowledge of the relative position and orientation of a pair cameras and a given pixel in one image, we can constrain our search of the corresponding pixel in the other image to a single line, rather than the entire image. Stereovision is one of the effective methods to estimate depth and structural parameters of 3D objects from a pair of 2D images []. he recent advances in multi-view stereovision provide an exciting alternative for outdoor scenes reconstruction [2]. In this paper we are interested to recover the 3D face model based on Epipolar Geometry. his paper is organized as follows: In section 2, we review some previous related works. Homogeneous coordinates is given in section 3. Section 4 describes the stereoscopy method for 3D face reconstruction. he experimental results are given in section RELAED WORK In this section we provide a general overview of 3D face reconstruction methods reported in the literature. Kumar et al. [4] [2] proposed a reconstruction methodology for quadratic curves using non-digitized image planes extended problem with digitized and normalized image planes by considering various noisy cases and the analysis of the error in the reconstruction process. Zhang et al. [5] proposed a model based algorithm to fill the missing range information of a planar region in the depth map of an image obtained from a commercial stereo vision system. he morphable 3D face model

2 46 aher Khadhraoui, Faouzi Benzarti and Hamid Amiri proposed by Blanz and Vetter et al. [6] presented a 3D reconstruction algorithm to recover the shape and texture parameters based on a face image in arbitrary view. However, its speed cannot satisfy the requirements of practical face recognition systems. Pighin et al. [8] used a generic face model and multiple images to recover the 3D face model. It can estimate the depth information by multiple images. However, with the generic face model, it needs to specify many points to get accurate 3D model and cannot correct the mis-registration errors. Recently, Jiang et al. [7] presented an automatic 2D-to-3D integrated face reconstruction method to recover the 3D face model based on one frontal face image and it is much faster. However, Jiang s work can not accurately recover the depth information due to lack of the depth information. 3. HOMOGENEOUS COORDINAES In a normal Euclidean coordinate system, the length of the coordinate vector equals the number of dimensions of the space in which a point lies [9]. herefore, a 3 matrix should be sufficient to describe a point in 3D world. However, homogeneous coordinates are used for 3D reconstruction purposes, where is added to the end of a normal coordinate vector. he reason behind having this additional dimension is to obtain linear operations, which were actually non-linear in 3- dimensional space, e.g. translation. 3. Intrinsic and Extrinsic Calibration Stereo calibration involves determining the parameters of a system using corresponding 2D points. Intrinsic parameters of a camera include focal lengths measured in width and height of the pixels (f x, f y ), the skew (s) and the principal point (c x, c y ) [0]. herefore, the coordinates of the equation above should be scaled with these parameters as follows: x f s c x x x R y f f y y y R () Upper triangular matrix in this equation is called calibration matrix, and notation K is used for it. In addition, the pictures are never taken from the same camera position and angle. Rotation matrix R and the translation matrix t are the extrinsic camera parameters. We can combine the equation of the camera having intrinsic parameters with a specific position and orientation: X x fx s cx R R Y t y 0 fy fy 0 0 0, 0t Z m K R Rt M m ~ PM (2) he 3 4 matrix P is called camera projection matrix. 3.2 Uncalibrated Camera Problem Finding the depth information of a point becomes more difficult when cameras are uncalibrated, meaning that the intrinsic and extrinsic parameters of the cameras are unknown. herefore, it is necessary to find more matched points between two images for the uncalibrated case. When two cameras view a 3D scene from different positions, there are a number of geometric relations between the 3D points and their projections onto the 2D images. Epipolar geometry refers to the geometry of this stereovision []. For Epipolar geometry, the terms of epipole and epipolar line are important. 3.3 Epipolar Geometry and the Fundamental Matrix he epipolar geometry is the intrinsic projective geometry between two views. It is independent of scene structure, and only depends on the cameras internal parameters and relative pose Fundamental Matrix he fundamental matrix F is the mapping of image point x to its epipolar line l where l = F x is the epipolar line corresponding to x l = F x the epipolar line corresponding to x It can be shown that F [ e ] x P P (3) e 0 e e 3 2 e e 0 e 2 3 e e e 3 x (4) is the matrix representation of the cross product.

3 3D Face Reconstruction Based on Epipolar Geometry 47 Also, for any corresponding point pairs x, x x Fx 0 (5) F is a rank 2 homogeneous matrix with 7 degrees of freedom. F is invariant under projective transformation H on the world space. i.e., even if X HX, by letting P PH, F remains unchanged. here is a projective ambiguity in P. ( x x, x y, x, y x, y y, y, x, y, ) f 0 () Stacking n equations from n point correspondences gives linear system A f = 0, where A is an n 9 matrix. If rank A = 8 then the solution is unique (up to scale) but in reality we seek a least-squares solution with n 8. he LS solution is the last column of V in SVD of A = UDV (last column corresponds to the smallest singular value). Figure shows the epipolar geometry: Essential Matrix he fundamental matrix with the calibration matrices K, K removed. i.e., image points are normalized by ˆ x K x ˆ x K x (6) Letting E K FK xˆ E xˆ 0 (7) For normalized camera P = [I 0] and P = [R t], the following holds: E [ t] x R (8) E has 5 degrees of freedom: 3 rotation angles in R, 3 elements in t, but arbitrary scale. Given SVD of E = U diag(,, 0) V, and assuming first camera is P = [I 0], the second camera is one of the following: P [ UWV u ] or P [ UWV u3] 3 or P [ UW V u3] or P [ UW V u3] where W = [0 0; 0 0; 0 0 ] (9) Only one of these is physically possible (positive depth from both cameras) Computing the Fundamental Matrix Fundamental matrix can be estimated up to scale f f f 3 F f f f f [ f, f, f, f, f, f, f, f, f ] f f f (0) Each corresponding point pairs (x, y, ) and (x, y, ) gives an equation Figure : he Epipolar Geometry Representation 4. SEREOSCOPY MEHOD he algorithm for stereo matching employs epipolar geometry based face reconstruction to estimate the disparity map on a stereo pair. Stereo Epipolar requires two off-the-shelf digital cameras which are connected together and calibrated so that they focus the same object. he framework of stereo based face reconstruction is as follows:

4 48 aher Khadhraoui, Faouzi Benzarti and Hamid Amiri same row coordinates in the two images. It is a useful procedure in stereo vision, as the 2-D stereo correspondence problem is reduced to a -D problem when rectified image pairs are used. 4.3 Points Corresponding In stereo correspondence matching, since two images of the same scene are taken from slightly different viewpoints using two cameras, placed in the same lateral plane, so, for most pixels in the left image there is a corresponding pixel in the right image in the same horizontal line. he difference in the coordinates of the corresponding pixels is known as disparity. he basics of stereo correspondence matching are as follows: For each epipolar line For each pixel in the left image - compare with every pixel on same epipolar line in right image - pick pixel with minimum match cost Figure 2: Stereo 3D Reconstruction Process he process of 3D face reconstruction consists of the following stages: Perform uncalibrated stereo image rectification on a pair of stereo images, Match individual pixels along epipolar lines to compute the disparity map, he disparity map with the original stereo images to create a 3D reconstruction of the scene. We will see in the following main steps of the method stereo. 4. Stereo Image Pair In this step, the color stereo image pair is acquired and converted to the gray scale for the matching process. Using color images may provide some improvement in accuracy. 4.2 Rectify Stereo Images he rectification is the process of transforming stereo images, such that the corresponding points have the 4.4 Disparity Map he disparity can be defined by the following equation [3]: d = bf / z (2) Where z is the distance of the object point from the camera (the depth), b is the base distance between the left and right cameras, and f is the focal length of the camera lens. he disparity map and the knowledge of the relative distance between the two cameras are used to compute the depth map. here are some distinct advantages of using Epipolar geometry for 3D face reconstruction. It provides sufficient geometric information of the faces. In addition, 3D information has the potential to improve the performance of face recognition systems. 5. EXPERIMENAL RESULS For the implementation of our application, we successfully implemented the various features as a graphical user interface. he algorithm receives a pair of stereo images (left and right image) as an input, and outputs a disparity map (or the depth map). he algorithm comprises the following steps.

5 3D Face Reconstruction Based on Epipolar Geometry 49 Read image pair I L and I R. Set the parameters which will be used. Initialize O L and O R to be zeros. Select K points from the right image. For each point in the right image, a corresponding point in the left image is obtained via correlation using the epipolar geometry. Compute parameters of a model M from matched 2D points via triangulation. Estimate disparity according to Equation (2). Generate depth map. he 3D reconstruction algorithm must solve two basic problems: correspondence, which deals with finding an object in the left image that corresponds to an object in the right image, and reconstruction, which deals with finding the depth (i.e. the distance from the cameras which capture the stereo images) and structure of the corresponding point of interest. Experimental results are given to demonstrate the viability of the proposed 3D face reconstruction method. Figure 3 shows a pair of stereo images and Figure 4 shows the corresponding point using epipolar geometry. Figure 5 shows the corresponding depth map. 6. CONCLUSION Figure 5: Disparity (Depth) Map We have proposed an efficient 3D face reconstruction method using epipolar geometry with multiple images. he approach is definitely robust, simple, easy and fast to implement compared to other algorithms. It provides a practical solution to the reconstruction problem. Future work includes applying the 3D model to face animation and recognition, and using robust multi-view face alignment to automate the reconstruction. REFERENCES Figure 3: Stereo Pair Figure 4: Stereo Correspondence [] Gaurav Gupta, Balasubramanian Raman, and Rama Bhargava, Reconstruction of 3D Plane using Min- Max Approach, International Journal of Recent rends in Engineering, 2(), November [2] Nader Salman and Mariette Yvinec, Surface Reconstruction from Multi-View Stereo of Large-Scale Outdoor Scenes, he International Journal of Virtual Reality, 200, 5(3), -6. [3] M. Mozammel Hoque Chowdhury and Md. Al-Amin Bhuiyan, A New Approach For Disparity Map Determination, Daffodil International University Journal of Science and echnology, 4(), January [4] S. Kumar, N. Sukavanam, and R. Balasubramanian, Reconstruction of Quadratic Curves in 3D Using wo or More Perspective Views: Simulation Studies. Proc. SPIE 6066, 60660M (2006); doi:0.7/ [5] Zhang J. Chen, Jagath Samaranbandu, Planar Region Depth Filling Using Edge Detection with Embedded Confidence echnique and Hough ransform. In Proceedings of the International Conference on Multimedia and Expo, 2, pp , 2003.

6 50 aher Khadhraoui, Faouzi Benzarti and Hamid Amiri [6] S. Romdhani, V. Blanz, and. Vetter. Face Identification by Fitting a 3d Morphable Model Using Linear Shape and exture Error Functions. In Proceedings of the European Conference on Computer Vision, 4, pp. 3-9, [7] D. Jiang, Y. Hu, S. Yan, L. Zhang, H. Zhang, and W. Gao. Efficient 3d Reconstruction for Face Recognition. Pattern Recognition, 38, , [8] F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. Salesin. Synthesizing Realistic Facial Expressions from Photographs. In SIGGRAPH 98 Conference Proceedings, pp ACM, July 998. [9] om Davis, Homogeneous Coordinates and Computer Graphics, November 20, 200. [0] A. Akaydin, 3D Face Reconstruction from 2D Images for Effective Face Recognition, [] Makoto Kimura, Hideo Saito, 3D Reconstruction based on Epipolar Geometry, MVA2OOO IAPR Workshop on Machine Vision Applications, Nov , 2000, he University of okyo, Japan. [2] N. Sukavanam, R. Balasubramanian and S. Kumar, Error Estimation of Quadratic Curves in 3D Space, International Journal of Computer Mathematics, 84(), pp. 2-32, 2007.