Exploring Facial Expression Effects in 3D Face Recognition Using Partial ICP

Transcription

1 Exploring Facial Expression Effects in 3D Face Recognition Using Partial ICP Yueming Wang 1, Gang Pan 1,, Zhaohui Wu 1, and Yigang Wang 2 1 Dept. of Computer Science, Zhejiang University, Hangzhou, , China Tel: Virtual Reality Lab, Hangzhou Dianzi University, Hangzhou, , China {ymingwang, gpan}@zju.edu.cn Abstract. This paper investigates facial expression effects in face recognition from 3D shape using partial ICP. The partial ICP method could implicitly and dynamically extract the rigid parts of facial surface by selecting a part of nearest points pairs to calculate dissimilarity measure during registration of facial surfaces. The method is expected to be able to get much better performance than other methods in 3D face recognition under expression variation for its dynamic extraction of rigid parts of facial surface at the same time of matching. We also present an effective method for coarse alignment of facial shape, which is fully automatic. Experiments on 3D face database of 360 models with 40 subjects, 9 scans with four different kinds of expression for each subject, show partial ICP is very promising, compared with PCA baseline. 1 Introduction Automatic face recognition has been studied extensively over the past decade. Most efforts have been made for face recognition from 2D images[1] and a few approaches exploited 3D information [2, 3, 5, 6, 7, 8, 9, 10]. Although the 2D face recognition system has good performance under constrained conditions, since the 2D image essentially is a projection of the 3D human face, it is still challenged by changes in illumination, pose and expression [1, 17]. Utilizing 3D information can improve the system performance[17,7] due to its explicit representation of facial surface. However, facial expression is still a big challenge even using 3D data in face recognition because in fact facial surface is a non-rigid object. Some efforts have been made to conquer the problem. C.S.Chua et.al [5] extracted rigid parts of facial surface by a Gaussian model after registering the face range data with varying expression. These rigid parts were used to create a model library for indexing. After enrolling four scans for each subject, voting based on point signature was employed for recognition. It was reported that 100% recognition rate was obtained for total six persons. However, the model database is too small.. Furthermore, by Gaussian Distribution, The authors are grateful for the grants from the National Science Foundation of China ( , ) and Program for New Century Excellent Talents in University (NCET ). Corresponding author. P.J. Narayanan et al. (Eds.): ACCV 2006, LNCS 3851, pp , c Springer-Verlag Berlin Heidelberg 2006

2 582 Y. Wang et al. almost all extracted rigid parts of models only discarded mouth from full facial surface. Some expression may deform other areas of facial surface such as cheek. K.Chang et.al [11] proposed a local region approach to coping with expression variation in 3D face recognition. The algorithm is based on traditional ICP after finding nose area of facial surface. On a database with about 355 subjects and D models with seven different expressions, an average rank-1 rate 77.1% was obtained. The algorithm improved the recognition performance, compared with ICP-baseline method using complete facial shape. Their work treated nose area as the rigid region under varying expressions. But under certain expressions, parts of the nose still show some deformations. A.M.Bronstein et.al [12] reported a 3D face recognition approach based on a representation of the facial surface that was invariant to isometric deformations resulting from expression variation. However, geodesic distance is definitely variant when facial surface with a mouth open expression. There is a common assumption that though the face shape of the same person may change, sometimes greatly, due to various facial expressions, still there are regions which will keep their shape and position or be subjected to much less deformation among different expressions. If these regions can be identified, the 3D non-rigid face recognition problem can be reduced to the rigid case[5, 11]. However, there may be no large uniform subset of the face that is perfectly shape invariant across a broad range of normal expressions, and the deformation of facial surface of a certain person may be not as same as others with the similar expressions at all time. Figure 1(a) shows some deformation images of facial surface with three different expressions, smile, surprise and sad. The deformation image is obtained as follows. Registering neutral expression face with non-neutral face of same subject by nose area and subtracting the former from the latter along the depth value, we call the difference map deformation image, which indicates the deformation extent of surface region with certain expression relative to neutral facial surface. And the darker in the figure indicates more deformation and the lighter means less deformation. From the deformation images, it can be seen that: (1) For a subject, almost no fixed large parts of facial surface are invariant along three expressions. Shown in left four columns in Fig.1(a), smile expression leads to shape deformation of mouth and cheek, surprise affects mouth and sad even changes the shape of nose and forehead area slightly. (2) Comparing two subjects in Fig.1(a), the same expression of different subject affects different regions. Sad in left person changes shape of forehead, while affects mainly eyebrow in right person. Thus, just extracting and matching the same relatively rigid parts for all facial surfaces is only a choice to solve the expression problem and may not be perfect. In this paper, after analysis of iterative closest point(icp), we give the partial ICP for 3D facial shape recognition which can implicitly extract variant rigid regions of the face according to deformation extent under different expression during matching. The method does a proper selection of nearest points pairs

3 Exploring Facial Expression Effects in 3D Face Recognition 583 (a) (b) Fig. 1. (a) Deformation images for two subjects. The darker indicates the more deformation. (b) Discarded area in facial surface with different p-rate=0.9,0.7,0.2 (5th,6th,7th columns respectively). Regions in red indicate the removed parts. Removed areas are not fixed between facial surfaces under different expressions. to calculate RMS when using ICP to match two surfaces. When applied the method to three expressions, smile, surprise and sad in our experiments, 96.88% rank-one matching rate is obtained. We also implement the PCA-based 3D face recognition as a baseline algorithm. This paper is organized as follows: Sec. 2 analyzes the ICP algorithm and presents our method of implicitly extracting rigid parts of facial surface. Sec. 3 describes the data preparing. The experimental results and conclusions are in Sec. 4 and Sec. 5 respectively. 2 Analysis of Iterative Closest Point (ICP) The ICP algorithm, developed by Besl and Mckay [16], is used to register the point sets by an iterative procedure which is widely used in field of 3D rigid

4 584 Y. Wang et al. object registration. Let point set P 1 = {p 1 1,,p1 M } and point set P 2 = {p 2 1,,p 2 N }. The ICP algorithm is summarized as: 1. P 2 (0) = P 2, l=0 2. Do 3. For each point p 2 i in P 2 (l) 4. Find the closest point y i in P 1 5. End For 6. The closest points y i form a new point set Y(l) where 7. the pairs of points {(p 2 1,y 1 ),,(p 2 N,y N )} 8. describe the correspondences between P 1 and P 2 (l). 9. If registration error E between P 1 and 10. P 2 (l) istoolarge 11. Compute transformation T(l) between (P 2 (l), Y(l)), 12. including translation and rotation. 13. Apply transformation P 2 (l +1)=T (l) P 2 (l),l=l Else 15. Stop 16. End If 17. While P 2 (l +1) P 2 (l) >threshold where point y k in set Y(l) denotes the closest point in P 1 to the point p 2 k(l) in P 2 (l) and the registration error between P 1 and P 2 (l) is E = 1 N N y k p 2 k(l) 2 (1) k For convergence of ICP, a coarse registration step usually is carried out before the iterative process. Generally, in ICP-based 3D face recognition, two facial surfaces are registered by the above method, then the value of E computed in the last time of iterative steps is treated as dissimilarity measure of two faces. When matching two facial surfaces with different expressions, the difference between the pairs of nearest points may become large due to shape deformation which may have a large effect when performing least-squares minimization and E is no longer accurate as a dissimilarity metric. Thus, there is a significant performance drop by ICP-Based method in 3D face recognition when expression varies between gallery and probe, from average 93.6% to 61.1%, as reported by K.Chang [11]. If only those pairs of points with relatively less deformation can be selected as input of calculation of E, the registration error E may be still able to distinguish different subjects while remain small when matching models of same subjects with different expression. While the traditional ICP-based method uses all point pairs in computing transformation T (l) ande [11], we do it by selecting parts of the point pairs. After sorting the distances of pairs of points in increasing order, we reject the worst n% of pairs based on distance in each pair. That is, only first (1-n%) part of distances and corresponding point pairs from sorted distances are chosen

5 Exploring Facial Expression Effects in 3D Face Recognition 585 to compute transformation E and T (l). Considering the last E that is used as dissimilarity measure of matching, discarding n% of pairs means removing those points in non-rigid region of facial surface. Thus, it is a implicit method to extract points in rigid parts of facial surface to register and match and the rigid parts extracted are varied according to deformation of facial surface among different matching models. We denote it partial ICP for 3D face recognition approach and call (1-n%) p-rate. Figue 1(b) shows some deformation images in which the areas in red indicate those removed by setting different p-rate. From the removed area, it could be seen that red regions completely come from darker area in deformation images. When p-rate equals 0.7, 70 percent of face area is kept to match and most nonrigid parts are discarded. Thus, the method is expected to be able to get much better performance in 3D face recognition with expressions than other rigidparts-based methods for its dynamically extracting rigid areas of facial surface at the same time of matching. 3 Data Preparing Considering the convergence problem of partial ICP, we firstly transform all models into a canonical coordinate system by finding the symmetric plane of facial surface and detecting two fiducial points, nose tip and nose base. Then, facial regions for all models are well extracted by trimming face mesh models with a elliptical cylinder which coarsely extracts same facial regions for all models, as shown in Fig.1. After trimming face models, following two strategies are applied: (1) To compensate for the effect of resolution,we simplify trimmed models using mesh optimization [15]. Then, all facial surface meshes put into experiments have about 2000 vertices. (2) When finding nearest point pairs between two point face meshes in partial ICP, the nearest distance from point to surface is computed instead of nearest distance between vertices of meshes. The details of alignment and trimming are described as follows. 3.1 Transforming to the Canonical Coordinate System Suppose central profile passes through nose tip, nose base and is in the symmetric plane of the facial surface. From profile, we identify following information: nose tip p nt, nose base p nb and direction of symmetric plane d s. Obviously, six degrees of freedom of facial surface can be fixed by p nt, p nb and d s. After detecting these information for each facial surface, all models can be coarsely registered in a canonical coordinate frame. Finding Facial Central Profile. We apply our early work [13, 14] to detect the curve of the central profile of facial surface, as reviewed briefly as follows. Let S(p i ) denotes a point set of facial surface, where p i is a point in the set and S m (p m i ) denotes its mirror to some certain plane, where pm i is corresponding

6 586 Y. Wang et al. mirrored point of p i.whens m (p m i ) has been registered to S(p i), S m (p m i )is transformed into S m (p m i ). From Sm (p m i ) and S(p i), we can fit symmetric plane of facial surface from point set A(x i ), where each point x i obtained by: x i =(p i + p m i)/2 (2) We use the basic idea of the ICP to get a registration between facial surface and its mirror and find the symmetric plane by equation 2. Finally, we calculate the intersection of symmetric plane and the triangulated surface of S(p i )toget the central profile, shown in Fig.2. y x z (a) (b) (c) (d) (e) (f) Fig. 2. Symmetric plane detection and profile finding using ICP. (a) original model, (b) mirrored model, (c) detected symmetric plane, (e) profile and symmetric plane seen from another view, (f) determination of nose tip and nose base in profile. Locating Nose Tip and Nose Base. Since central profile curve passes through nose tip and nose base, we locate their positions based on the curve. Let C denotes the central profile curve extracted, l e denotes the line through both end points of the curve C, p nt and p nb denote nose tip point and nose base point respectively. Before location, we suppose following assumption hold up. (1) The nose tip p nt is a point on the curve C, with the maximum distance to the line l e. (2) The nose base p nb is a point on the profile curve C, and is the first distance extremum point to the line l e from p nt to forehead, as shown in Fig. 2(f). It can be formalized as: p nt = argmax p C dist(p, l e ) (3) L = {p p C, y p >y pnt,dist (p, l e )=0} (4) p nb = argmin p L (y p ) (5) where dist(, ) is the Euclidean distance function from a point to a line segment, y p is y-axis coordinate of point p, dist (, ) denotes first derivative of Euclidean distance to the point position at line l e extend to forehead. If facial surface is sampled only frontal view discarding hair and neck area, our assumption is appropriate so that p nb and p nt can be located accurately.

7 Exploring Facial Expression Effects in 3D Face Recognition 587 However, some certain samples of facial surface may be grotesque in shape which doesn t keep the assumption. To date we have never encountered a model on which failure happen in our experimental data set. Aligning Model. Given nose tip p nt, nose base p nb and normal direction v sp of symmetric plane for each model, a canonical coordinate system of all models can be determined. Subtracting p nb from p nt, we get unit vector v y after normalized. Taking p nt as the origin, v sp as x-axis, v y as y-axis, a new right-hand coordinate system is defined. Then all models are registered by transforming facial surface into the new coordinate system. Furthermore, we rotate the model 20 degree around x-axis counterclockwise in the new coordinate system for non-duplicate happened in projecting depth to x-y plane used in PCA-based face recognition. Some results are shown in Fig.3 (a). All our experiments are based on the aligned models. (a) (b) Fig. 3. (a) The canonical coordinate system for aligned models. Axis in red is z-axis, blue is y-axis, green is x-axis. (b) Face models acquired by InSpeck 3D MEGA Capturor DF. Each row shows 9 scans of one subject. The models in first row are rendering with texture. 3.2 Trimming Models Given aligned model mesh, facial regions can be extracted by removing those points and triangles of facial surfaces in the outer of following elliptical cylinder: (x x 1 ) 2 a 2 + (y y 1) 2 b 2 =1 (6) Since all models are in a canonical coordinate, the facial regions of all models produced by above equation are not only full frontal area, but also roughly same between models which is an important condition for our partial ICP method. For consistency, we set parameters as x 1 =0, y 1 =20, a=60 and b=80 which works well for all 360 models in our experiments. 4 Experiments 4.1 Data Acquisition Experiments use the 3D facial expression database ZJU-3DFED, collected by the authors. In ZJU-3DFED database, there are 40 different subjects, nine scans

8 588 Y. Wang et al. for each, and total 360 scans. Each subject has two scans with smile expression, two scans with surprise expression, 2 scans with sad expression and 3 scans with neutral expression. Facial surfaces of same subject with same expression are slightly different in extent. All face models are acquired by InSpeck 3D MEGA Capturor DF[18]. The facial models are in triangular mesh. We manually cut out the parts out of the face regions from the original model data and this is the only manual work in our whole works. Each facial mesh then have about points and triangles and the resolution is 0.04 mm. After mesh simplification [15], each scan has about 2000 points and about 4000 triangles. Figure 3(b) shows 3 subjects and 27 scans of face models. The face models have both shape and texture information, we only use shape information in the experiments. We put one neutral expression face model for each subject into gallery and the other 320 scans are classified into 4 probe sets. All 80 smile scans form smile-set, so do surprise scans, sad scans and neutral scans. A special probe set composed of the 320 scans is made for whole recognition results, called whole-set. 4.2 Results by Partial ICP with Different p-rate Twelve different values of p-rate { 0.05, 0.1, 0.2, 0.3, 0.4, 0.5,0.6, 0.7, 0.8, 0.9, 0.95, 1} are tested in our experiments. Additionally, we also consider a extreme instance that only a pair of nearest points is input into calculation of dissimilarity measure E after ICP process which use all points pair in iterative. The results are in Fig. 4. Rank-1 rate Smile Surprise Sad Neutral Whole p-rate (a) Rank-1 Rate Smile Surprise 0.85 Sad Neutral Whole p-rate (b) Fig. 4. (a) Rank-1 recognition rate of different p-rate with partial ICP method on five probe sets. (b) zooming out part of (a). From Fig.4, it can be seen that: (1) For three non-neutral probe sets, none of them has a rank-1 rate larger than 90% when p-rate equals 1(that is same as the traditional ICP-based matching method). But when setting p-rate value between 0.1 and 0.95, none of them has a rank-1 rate smaller than 90%. The largest improvements of rank-1 rate of three non-neutral sets are 7.5% for smile-set, 11.25% for surprise-set, 10% for sad set.

9 Exploring Facial Expression Effects in 3D Face Recognition 589 (2) Both highest rank-1 rates of smile-set and sad-set are 96.25% and obtained at p-rate =0.8 while that of surprise-set are 97.5% and obtained at p-rate= 0.1 or p-rate= 0.2. It is partly due to facial surface with surprise expression has a larger deformation area than the other expressions. (3) When setting p-rate as 0.1 which means 90% of facial surface is removed before matching using partial ICP method, an average rank-1 rate 94.17% is still reached on non-neutral probe sets. It is a cue indicating that small parts of facial surface still have enough information for recognition if nice extraction is performed. (4) As a whole, our method get an average rank-1 rate 95% on three nonneutral probe sets when p-rate=0.2 and 96.88% on whole-set. 4.3 PCA v.s. Partial ICP PCA-based method is implemented in our experiments for comparison. After models are trimmed, PCA-based method can easy be applied to 3D face recognition by projecting the trimmed models to x-y plane. We use the first 40 eigenvectors when test PCA-based method which hold 96.46% energy. We compare the performance between PCA-based method and partial ICP method on all five probe sets. The results are shown in Fig.5. Fig. 5. Rank-1 rate: PCA v.s. partial ICP The partial ICP method outperforms PCA-based-method on rank-1 performance among all probe sets. PCA-based-method is well known that is sensitive to noise. On all non-neutral expression probe set, PCA-based-method get average rank-1 rate 75.41% and the worst rank-1 rate 65% on surprise-set because shape deformation with different expressions act as a role of noise in a way. Between neutral and non-neutral expression probe sets, the rank-1 rate drop from 92.5% to an average 75.41% with PCA-Base method in recognition. The partial ICP with p-rate=1 gets a whole rank-1 recognition rate 89.69%. The partial ICP with best p-rate obtains an average rank-1 rate 96.88% on all probe sets. By well selecting the p-rate, partial ICP method is insensitive to expression variant in 3D face recognition.

10 590 Y. Wang et al. 5 Conclusion We propose a method, partial ICP method, which is capable of dynamically extracting rigid parts of facial surface. The extraction is completely dependent on the deformation extent of facial surface and extracted areas are varied between different expressions. Based on partial ICP, we perform several experiments for 3D face recognition on a database with 360 models. A rank-1 rate 96.88% demonstrate the high performance of our method in 3D face recognition with different expressions. The experimental results also show that our method significantly outperforms PCA-based method. References 1. W.Zhao, R.Chellappa, P.J.Phillips, A.Rosenfeld. Face recognition: a literature survey. ACM Computing Surveys, 35(4): , J.C.Lee, E.Milios. Matching range images of human faces. Proc. IEEE ICCV, p , G.G.Gordon. Face recognition from depth maps and surface curvature. SPIE Conf. on Geometric Methods in Computer Vision, 1570: , C.S.Chua, R.Jarvis. Point signatures: A new representation for 3D object recognition. IJCV, 25(1):63-85, C.S.Chua, F.Han, Y.K.Ho. 3D Human Face Recognition Using Point Signature. IEEE FG 00, pp , M.W.Lee, S.Ranganath. Pose-invariant face recognition using a 3D deformable model. Pattern Recognition, 36(8): , V.Blanz, S.Romdhani, T.Vetter. Face identification across different poses and illumination with a 3D morphable model. Int l Conf. on FG, p , C.Beumier, M.Acheroy. Automatic 3D face authentication. Image Vision Computing, 18(4): , W. Zhao, R. Chellappa. Illumination-insensitive face recognition using symmetric shape-form-shading. Proc. IEEE ICCV, 1: , G. Pan, Z. Wu, and Y. Pan, Automatic 3D face verification from range data, in Proc. IEEE ICASSP, vol.3, pp , K.Chang, K.Bowyer, P.Flynn. Effects on Facial Expression in 3D Face Recognition, Proc.oftheSPIE, Volume 5779, pp , A.M.Bronstein, M.M.Bronstein, R.Kimmel. Expression-invariant 3D face recognition. Proc. AVBPA 03, LNCS, vol.2688, 62-70, Yijun Wu, Gang Pan, Zhaohui Wu. Face Authentication based on Multiple Profiles Extracted from Range Data. Proc. AVBPA 03, LNCS, vol.2688, pp , Gang Pan, Zhaohui Wu, 3D Face Recognition from Range Data, Int l Journal of Image and Graphics, 5(3): , H.Hoppe, T.DeRose, T.Duchamp, J.McDonald and W. Stuetzle, Mesh optimization, Computer Graphics(SIGGRAPH 93 Proceedings), 27:19-26, Auguest, P.J.Besl, N.D.McKay, A method for registration of 3-D shapes, IEEE Trans.Pattern Anal.Mach.Intell. 14: , Face Recognition Vendor Test 2002, InSpeck Inc.,