DETC D FACE RECOGNITION UNDER ISOMETRIC EXPRESSION DEFORMATIONS

Proceedings of the ASME 2014 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference IDETC/CIE 2014 August 17-20, 2014, Buffalo, New York, USA DETC2014-34449 3D FACE RECOGNITION UNDER ISOMETRIC EXPRESSION DEFORMATIONS Javad Sovizi Department of Mechanical and Aerospace Engineering University at Buffalo Buffalo, New York 14260 Email: javadsov@buffalo.edu Rahul Rai Venkat Krovi Department of Mechanical and Aerospace Engineering University at Buffalo Buffalo, New York 14260 Email: rahulrai,vkrovi@buffalo.edu In this paper, 3D face recognition under isometric deformation (induced by facial expressions) is considered. The main objective is to employ the shape descriptors that are invariant to (isometric) deformations to provide an efficient face recognition algorithm. Two methods of the correspondence are utilized for automatic landmark assignment to the query face. One is based on the conventional iterative closest point (ICP) method and another is based upon the geometrical/topological features of the human face. The shape descriptor is chosen to be the well-known geodesic distance (GD) measure. The recognition task is performed on SHREC08 database for both correspondence methods and the effect of feature (GD) vector size as well as landmark positions on the recognition accuracy were argued. INTRODUCTION 3D objects recognition/retrieval has been a challenging problem in the computer vision studies. It has found several applications in robotics, manufacturing, video surveillance etc. A novel topic in this area is recognition of the 3D non-rigid shapes that are subjected to the deformations. Dealing with non-rigid 3D objects, two classes of the problems can be assumed, articulated and non-articulated deformations. The former, as the name suggests, concerns the objects that are piecewise rigid but can change their shape/configuration due to the articulations. A robotic arm or human hand movements are some illustrative examples of these deformations. However, the recognition task is Address all correspondence to this author. more problematic when the deforming object is not articulated. Change of the human face under different expressions (smile, laugh, anger etc.) can be categorized in this class of the problems that is the main focus of the current work. Although the 2D face recognition has been well addressed in the literature, however, the major works concerning 3D face recognition go back to last few years. A comprehensive survey on the existing approaches on 2D, 3D and 2D+3D face recognition has been reported in [1]. Using the depth component in the face recognition is significantly effective in the robustness of the algorithms against the facial expressions, lighting, make ups etc. that are negatively affecting the performance of the 2D image recognition schemes. Bronstein et al. [2] proposed an expression-invariant 3D face recognition scheme in which the face surface gradient field is used as the shape descriptor for the recognition purpose. Kakadiaris et al. [3] developed a deformable model through which the 3D data are projected on the 2D grid. Hence, the algorithm uses the descriptive information provided by 3D data and is computationally efficient as it works in 2D regular grid. In another effort, Mian et al. [4] utilized the spherical face representations to reject several candidate faces in the recognition task. In addition, they developed an automatic segmentation algorithm that subdivides the face to the areas sensitive and insensitive to the expressions. Many other sophisticated approaches have been utilized to tackle the 3D face recognition problem under facial expressions. For example, Chang et al. [5] addressed this problem by investigating multiple overlapping regions around the nose. Li et 1 Copyright c 2014 by ASME

al. [6] tackled the expression deformation by sparse representation technique applied to a set of geometrical features ranked based on their sensitivity to the expression. Al-Osaimi et al. [7] proposed a method that differentiates between deformations induced by expressions and interpersonal disparities. In their work, a key factor is learning the expression deformations in a PCA subspace. The main goals of this work include: 1) developing an automatic correspondence algorithm for landmark assignment; 2) using relatively simpler shape descriptors that are invariant to the isometric deformations (induced by facial expressions); and 3) investigation of the effect of feature vector size as well as landmark positions on the recognition accuracy. Geodesic distance (GD) vector is used as the shape descriptor that is invariant under the isometric deformations. The use of the GD vector in this study is motivated by the work of Smeets et al. [8]. They used the GD matrix as the shape representative for both articulated shapes retrieval and 3D face recognitions. GD matrix is a symmetric matrix whose elements (g i, j ) are the geodesic distance between the points (landmarks) i and j. However, in the current work, we use only the elements g i, j where i j to construct our shape descriptor vector. The rest of this paper is organized as follows. Next section describes the database (SHREC 2008 [9]) and the details of the available information along with preprocessing of the 3D faces mesh data. Afterward, automatic landmark assignment via two correspondence algorithms are discussed. One is based on the conventional ICP (iterative closest point) algorithm and another is based upon the geometrical/topological features of the human face. In the next section, the recognition based on the GD shape descriptor is performed. Effect of the GD vector cardinality as well as landmark positions on the recognition accuracy were investigated for both landmark assignment approaches. Finally, in the last section, a brief discussion and directions for future work are presented. FIGURE 1: 7 DIFFERENT 3D FACE SAMPLES OF A SINGLE SUBJECT; SHREC 2008 DATABASE [9] tip of the nose to all other mesh vertices are calculated. The vertices whose corresponding GD is more than predefined threshold are removed from the mesh. Consequently, a new mesh is generated using the processed point cloud. Fig. 2 shows the cropping procedure for one of the samples in the database. DATABASE AND PREPROCESSING Besides the complexity of the recognition problem, providing the realistic 3D face databases is also a tedious task. The database used in this study is obtained from internet repository SHREC 2008 [9]. The data is a subset of GavabDB: a 3D Face Database [10] in which different scans of 61 subjects are acquired with the Minolta VI-700 laser range scanner. In this work 3D face data corresponding to 50 subjects are used. There are 7 different samples per subject. 2 frontal view, 1 look up, 1 look down, 1 smile, 1 laugh and 1 random expressions. So, the entire database contains 350 3D face data. Fig. 1 shows 7 different realizations of a single subject. As Fig. 1 shows, the raw data include some extra parts that are not used in the recognition process, e.g. subjects ears, hairs, neck, shirt etc. To crop the main part of the mesh, GD s from the FIGURE 2: CROPPING EXAMPLE FOR ONE OF THE DATABASE SAMPLES AUTOMATIC LANDMARK ASSIGNMENT For recognition purpose, it is crucial to automatically assign the landmarks to different (predefined) spots on the query 2 Copyright c 2014 by ASME

face surface. The GD vector associated to these landmarks will then be used as the shape descriptor for recognition. Here, two methods, one based on the conventional ICP algorithm and another based on the geodesic distance and geometrical/topological features of the human face are utilized for correspondence and automatic landmark assignment. In both of these methods, 15 different landmarks are manually assigned to a reference face model and the goal is to find the landmarks corresponding point on the new face mesh vertices. Fig. 3 shows a reference face surface and 15 different landmarks that have been manually selected on the face surface. Different factors may be considered for choosing the landmark positions as feasibility of the automatic landmark assignment, sensitivity to the facial expressions and the discriminativity of the resulting feature vector. Most of the landmarks shown in Fig. 3 are approximately located in the local extrema of the surface height function. This facilitates the geometrical/topological landmark assignment that will be described shortly. In addition, we have set landmarks 0, 3, 4, 9 and 10 along the symmetry line of the face among which the position of landmarks 0, 3 and 4 are less sensitive to several facial expressions. Moreover, it is expected that the position of landmarks 9 and 10 is not significantly changing in many expressions with closed mouth. Other landmarks are distributed on the left and right sides of the face to provide sufficient and welldiscriminating information from the subject face. In this paper, we have not employed a sophisticated approach to determine the optimal choice of the landmark positions. However, existing literature on 3D face recognition as well as investigating the accuracy of the recognition corresponding to different choices of the landmark positions were effective in choosing the landmark positions illustrated in Fig3. FIGURE 3: 15 LANDMARKS POSITIONS ON THE REFER- ENCE FACE MODEL In the first method, using the ICP algorithm, 3D point cloud of the query object will be transformed such that the best alignment is achieved with the reference model point cloud. Euclidean distance is then used to find the closest point to each of the landmarks on the reference surface. Fig. 4 shows the procedure for landmark assignment based on this method. FIGURE 4: ICP LANDMARK ASSIGNMENT FIGURE 5: CANDIDATE POINTS AND REJECTION BASED ON THE LOCATION RELATIVE TO THE NOSE TIP In the second approach, to find the vertex corresponding to ith (i = 1,...,14) landmark, we first calculate the GD from nose tip (whose location is known) to that landmark on the reference model, denoting by g re f n,i. In the next step, an interval I i = [g re f n,i a i,g re f n,i +a i ] is considered (where a i is a scalar design parameter) and all vertices in the query mesh whose geodesic distance from the nose tip belongs to I i will be considered as candidate corresponding points. Next, several candidate points will be rejected based on the relative position of the landmarks respect to the nose tip. For example, to find the vertex corresponding to landmark 3 on the reference model, the candidate points (on the query) whose Y coordinates are less than nose tip will be rejected. Fig. 5 illustrates this procedure. The corresponding point is finally selected from the set of remaining candidate points based on a simple weight function assignment. Denoting the ith landmark weight function by W i, then the weight associated with candidate point j is W i j = α i X j X n +β i Y j Y n +γ i Z j Z n (1) where X j, Y j and Z j are the coordinates of the candidate point 3 Copyright c 2014 by ASME

constructed as G i 50 1 = [gi 0,1,...,gi 0,14,gi 4,1,...,gi 4,14,gi 7,1,...,gi 7,3,gi 7,5,...,g i 7,14,gi 8,1,...,gi 8,3,gi 8,5,gi 8,6,gi 8,8,...,gi 8,14 ]T (2) FIGURE 6: WEIGHT FUNTIONS ASSOCIATED WITH EACH LANDMARK and X n, Y n and Z n are the coordinates of the nose tip on the query surface. α i, β i and γ i are the constant parameters associated with the ith landmark. Fig. 6 shows the landmarks weight functions and their tuned parameters. As an example, landmark 3 is located between two eyebrows and is expected to have a very close X coordinate to nose tip, even under severe expressions. So, the weight function penalizes the absolute difference in the X components by α 3 = 4. For the landmarks 5 and 6, the candidates with higher difference in the Z components are more probable to be the correct corresponding vertex. So, the weighting function multiplies the Z coordinate difference by positive constants γ 5 = γ 6 = 3. The sign of the other constants are determined based on the same interpretation of the topological features of the human face and the weights are tuned such that the best accuracy is achieved. Note that, while defining the interval I i, a i has to be chosen such that the resulting candidate points are in the close neighborhood of the desired landmark (this can be tuned by, for example, visual inspection of the resulting candidate points corresponding to different values of a i ). Otherwise, selection based on the weight functions may be erroneous. Fig. 7 shows the result of the landmark assignment to some of the 3D faces in the database based on the proposed method. 3D FACE RECOGNITION Once the query 3D face is preprocessed (cropped) and the vertices corresponding to the landmarks are determined, the geodesic distance vector can be constructed. Then, different distance measures are utilized to obtain the similarity of a query face geodesic distance vector with those of the database 3D faces. The results are obtained corresponding to both landmark assignment schemes. It will be shown that recognition based on the geometrical/topological landmark assignment yields higher accuracy than that based on the ICP correspondence. The geodesic distance vector of the ith 3D face model is There are several choices for constructing G i that will ultimately result in different accuracies of the recognition. Although we have not performed a systematic survey on this problem here, however, we have chosen four landmarks (one on the nose tip, two on the cheeks and one on the forehead) whose locations are not drastically changing in facial expressions. In addition, their geodesic distance from other landmarks provide discriminative information (based on different trials results). To measure the similarity between the GD vectors the following distance measures are used [8]. Mean normalized max. norm distance Euclidean distance Mean normalized Manhattan distance χ 2 distance Mn = max 2 G i k G j k k G i k + G j k D i, j m D i, j Ec = (G i k G j k )2 k=1 D i, j m Mnh = 2 G i k G j k k=1 G i k + G j k G i k D i, j Cs = m k=1 2 G j k G i k + G j k To examine the recognition accuracy, the entire database is used as query in 7 trials. In each trial, one of the 7 expressions is chosen as a query and other 6 expressions are used as the available database. The GD vector associated to each subject is calculated by averaging over these 6 remaining expressions. So, 50 tests are performed per trial that is 350 tests after 7 trials. The recognition accuracy for different similarity measures are summarized in Tab. 1. It can be seen that Mean normalized Manhattan distance and χ 2 -distance outperform the Euclidean distance, and Euclidean distance outperforms the maximum norm distance measure. In addition, as the results show, comparing to ICP based landmark assignment, the geometrical/topological landmark assignment method yields higher accuracies, for all distance measures and in both first and second rank recognition rates. The ICP assignment may introduce larger errors in the cases of severe facial expressions while the geometrical/topological based method proves to be more robust to the shape deformations induced by facial expressions. Moreover, the impact of distance measure and landmark assignment method on the accuracy of the recognition can also be investigated from the results. From Tab. 1, it 4 Copyright c 2014 by ASME

FIGURE 7: LANDMARK ASSIGNMENT RESULTS BASED ON THE GEOMETRICAL/TOPOLOGICAL FEATURES TABLE 1: 3D FACE RECOGNITION ACCURACY OVER 350 TESTS; GD VECTOR IS 50 1 Maximum norm distance Euclidean distance Manhattan distance χ 2 -distance First rank accuracy Geometrical/topological land. ass. First rank accuracy ICP land. ass. Second rank accuracy Geometrical/topological land. ass. Second rank accuracy ICP land. ass. 34.86 % 58.86 % 62.86 % 62.86 % 30.57 % 46 % 55.71 % 55.71 % 47.71 % 73.43 % 75.14 % 75.14 % 40.57 % 55.43 % 67.14 % 67.14 % is clear that distance measure has a higher impact on the results. For example, while the increase of accuracy due to the geometrical/topological landmark assignment method (compared to ICPbased method) is 7.15 %, using χ 2 -distance instead of maximum norm distance increases the accuracy by 28 %. Fig. 8 shows the first and second rank recognition accuracies variations versus the dimension of the GD vector. It can be seen that the overall accuracy increases by populating the GD vector, however, at some steps adding more elements decreases the recognition accuracy. This implies the significance of the landmarks positions. For example, landmarks located in the areas with large expression displacements, e.g. those in the lower part of the face (for example, lip corners), may introduce error in the correspondence that in turn decrease the recognition accuracy once their corresponding geodesic distances are added to the GD vector. In addition, the results plotted in Fig. 8 convey information about the sensitivity of different distance measures to the GD vector size (cardinality). For example, the maximum norm distance measure shows the least sensitivity but also provides the minimum accuracy among other evaluated measures. Conversely, χ 2 -distance has maximum sensitivity but it also gives the maximum accuracy. Solid lines in Fig. 8 show the accuracies corresponding to the geometrical/topological landmark assignment and dashed lines are the results based on the ICP landmark assignment. It can be clearly seen that for GD vectors with more than 5 elements the geometrical/topological landmark assignment method yields higher accuracy than the ICP based correspondence method. DISCUSSION In this work 3D face recognition problem under facial expression deformations was considered. The geodesic distance measure, that is invariant under the isometric deformations, was 5 Copyright c 2014 by ASME

ICP based approach for all distance measures. Clearly, more experiments need to be performed with different available databases to robustly establish the accuracy and effectiveness of the proposed methods, that will be the subject of our future study. ACKNOWLEDGMENT This work was partially supported by the National Science Foundation awards IIS-1319084 and CNS-1314484. FIGURE 8: (a) First and (b) second rank accuracy versus number of GD vector elements used as the shape descriptor. 3D faces in the database were preprocessed and two different automatic landmark assignment methods were utilized. One based on the ICP algorithm and another based on the geometrical/topological features of the human face. Different distance metrics were examined to measure the similarity of the geodesic distance vectors of the 3D faces and the results of the recognition were obtained for both landmark assignment approaches. Clearly, the size of the feature vector affected the recognition accuracy. However, we were also able to quantitatively establish the sensitivity of different distance measures to the number of feature vector elements. For example, the maximum norm vector is the least sensitive but is also the least accurate. The χ 2 -distance and Manhattan distance perform the best followed by the Euclidean distance and the saturation effects, in these measures, start to be observed after about 50 GD vector elements. Moreover, the geometrical/topological landmark assignment approach turned out to be more accurate than a b REFERENCES [1] Bowyer, K. W., Chang, K., and Flynn, P., 2006. A Survey of Approaches and Challenges in 3D and Multi-Modal 3D+2D Face Recognition. Computer Vision and Image Understanding, 101(1), pp. 1 15. [2] Bronstein, A. M., Bronstein, M. M., and Kimmel, R., 2005. Three-Dimensional Face Recognition. International Journal of Computer Vision, 64(1), pp. 5 30. [3] Kakadiaris, I. A., Passalis, G., Toderici, G., Murtuza, M. N., Lu, Y., Karampatziakis, N., and Theoharis, T., 2007. Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach. IEEE Transactions on Pattern Analysis and Mchine Intelligence, 29(4), pp. 640 649. [4] Mian, A., Bennamoun, M., and Owens, R., 2006. Automatic 3d face detection, normalization and recognition. In 3D Data Processing, Visualization, and Transmission, pp. 735 742. [5] Chang, K. I., Bowyer, K. W., Flynn, P. J., and Member, S., 2006. Multiple Nose Region Matching for 3D Face Recognition under Varying Facial Expression. IEEE Transaction on Pattern Analysis and Machine Intelligence, 28, pp. 1695 1700. [6] Li, X., Jia, T., and Zhang, H., 2009. Expression-insensitive 3d face recognition using sparse representation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2575 2582. [7] Al-Osaimi, F., Bennamoun, M., and Mian, A., 2009. An Expression Deformation Approach to Non-Rigid 3D Face Recognition. International Journal of Computer Vision, 81(3), pp. 302 316. [8] Smeets, D., Hermans, J., Vandermeulen, D., and Suetens, P., 2012. Isometric Deformation Invariant 3D Shape Recognition. Pattern Recognition, 45(7), July, pp. 2817 2831. [9] SHREC 2008. http://give-lab.cs.uu.nl/shrec/shrec2008/. [10] Moreno, A. B., and Sanchez, A., 2004. Gavabdb: a 3d face database. In Workshop on Biometrics on the Internet, pp. 77 85. 6 Copyright c 2014 by ASME