Automatic Pose Correction for Local Feature-Based Face Authentication

Transcription

1 Automatic Pose Correction for Local Feature-Based Face Authentication Daniel González-Jiménez 1, Federico Sukno 2, José LuisAlba-Castro 1, and Alejandro Frangi 2 1 Departamento de Teoría de la Señal y Comunicaciones, Universidad de Vigo, Spain {danisub, jalba}@gts.tsc.uvigo.es 2 Departamento de Tecnología, Universidad Pompeu Fabra, Barcelona, Spain {federico.sukno, alejandro.frangi}@upf.edu Abstract. In this paper, we present an automatic face authentication system. Accurate segmentation of prominent facial features is accomplished by means of an extension of the Active Shape Model (ASM) approach, the so-called Active Shape Model with Invariant Optimal Features (IOF-ASM). Once the face has been segmented, a pose correction step is applied, so that frontal face images are synthesized. For the generation of these virtual images, we make use of a subset of the shape parameters extracted from a training dataset and Thin Plate Splines texture mapping. Afterwards, sets of local features are computed from these virtual images. The performance of the system is demonstrated on configurations I and II of the XM2VTS database. Keywords: Face Authentication, Automatic Segmentation, Pose Correction. 1 Introduction Although many algorithms have been proposed during the last decade, the general face recognition problem still remains unsolved because of several causes that affect the performance of face-based biometric approaches, such as illumination and pose variations, expression changes, etc [19]. Moreover, face recognition algorithms must be supplied with cropped images that ideally contain only face pixels, i.e. there must exist a previous step that locates the face (and perhaps a set of facial features) within the input image. Face authentication contests like [17] have shown that there is a general degradation in performance when changing between manual registration of faces and using automatic detection before authentication. In this paper, we address two aspects of the face authentication problem: automatic face modelling from still images and pose correction. One of the most popular approaches for statistical modelling are the active models of shape and appearance, introduced by Cootes et al. in [11,12]. These techniques allow for detailed modelling of a wide range of objects, as long as an appropriate training set is available. Their application to facial images has been previously exploited [16,15] to locate the main facial features (e.g. eyes, F.J. Perales and R.B. Fisher (Eds.): AMDO 26, LNCS 469, pp , 26. c Springer-Verlag Berlin Heidelberg 26

2 Automatic Pose Correction for Local Feature-Based Face Authentication 357 nose, lips) and recover shape and texture parameters. In this work we use the Active Shape Models with Invariant Optimal Features (IOF-ASM), an extension of Active Shape Models (ASM) that improves segmentation accuracy by means of a non-linear texture model based on local image structure [21]. As stated above, the presence of pose differences within the input images is one of the main factors that degrades the performance of face recognition systems. Up to now, the most practical and successful algorithms dealing with pose-invariant face recognition are those which make use of prior knowledge of the class of faces such as [1], where an individual eigenspace is constructed for each pose. Another approach is presented in [2], where from a single image of a subject and making use of face class information, virtual views facing different poses are synthesized, which are then used in a view-based recognizer. In [3], a morphable 3D face model was fitted to the input images. Among others, the parameters that account for pose are subject to modification, so that virtual images facing the adequate pose can be synthesized. The main drawbacks of this method are the need of a 3D face training database and the high computational complexity. Using a training dataset of face images, we built a Point Distribution Model and, from the main modes of variation, the parameters responsible for the pose of the face (namely the pose parameters) were identified. Using the segmentation results provided by the IOF-ASM approach, our system compensates for pose variations by normalizing these pose parameters and synthesizing virtual frontal images through texture mapping. Sets of local features are extracted from these virtual images by means of a two-stage approach. Experiments on the XM2VTS database showed how this simple strategy softens (moderated) pose effects, achieving error rates comparable to the state of the art. The paper is organized as follows: Section 2 presents the statistical modelling of the face and the approach used for segmenting facial features. In Section 3, the synthesis of pose corrected face images is addressed, while section 4 explains the two stages of feature extraction. In Section 5, we show our experimental results over the XM2VTS database [18]. Finally, conclusions are drawn in Section 6. 2 Statistical Face Modelling 2.1 A Point Distribution Model for Faces A Point Distribution Model (PDM) of a face is generated from a set of training examples. For each training image I i, N landmarks are located and their coordinates are stored, conforming a vector X i =(x 1i,x 2i,...,x Ni,y 1i,y 2i,...,y Ni ). The pair (x ji,y ji ) represents the coordinates of the j-th landmark in the i-th training image. After aligning all training examples, a Principal Components Analysis is performed in order to find the most important modes of shape variation. As a consequence, any training shape X i can be approximately reconstructed as: X i = X + Pb, (1) where X stands for the mean shape, P is a matrix whose columns are unit eigenvectors of the first t modes of variation found in the training set, and b is the

3 358 D. González-Jiménez et al Fig. 1. Effect of varying pose parameters. rotations in depth parameter (first row) and elevation parameter (second row). The middle column shows the average face shape, while the left and right columns are generated displacing the corresponding parameters by ±5 times the standard deviation of the training set. vector of parameters that define the actual shape of X i. Notice that the k-th component from b (b k,k =1, 2,...,t)weighsthek-th mode of variation. Examining the shapes generated by varying b k within suitable limits, we find those parameters responsible for pose, as indicated in figure 1. Note that, although a given eigenvector should not be assigned to an unique mode of facial variation, it is clear that the eigenvectors shown in this figure are mainly related to pose changes. Let b pose be the set of parameters which accounts for pose variation. Since P T P = I, then b = P T ( X i X ), (2) i.e. given any shape, it is possible to obtain its vector of parameters b and, in particular, we are able to find its pose (i.e. b pose ). We built a 62-point PDM using the set of manual annotated landmarks 1 from the training images shared by both configurations I and II [9] of the XM2VTS database[18]. 2.2 IOF-ASM When a new image containing a face is presented to the system, the vector of shape parameters that fits the data, b, should be computed automatically. Active Shape Models with Invariant Optimal Features (IOF-ASM) is a statistical modelling method specifically designed and tested to handle the complexities of facial images. The algorithm learns the shape statistics as in the original ASMs [11] but improves the local texture description by using a set of differential 1

4 Automatic Pose Correction for Local Feature-Based Face Authentication 359 Algorithm 1. IOF-ASM matching to a new image 1: Compute invariants for the whole image 2: T = Initial transformation guess for face position and size 3: X = X (modelshape = meanshape) 4: for i =1tonumberof iterations do 5: Project shape to image coordinates: Y = TX 6: for l =1tonumberof landmarks do 7: Sample invariants around l-th landmark 8: Determine best candidate point to place the landmark 9: if the best candidate is good enough then 1: Move the landmark to the best candidate point 11: else 12: Keep previous landmark position (do not move) 13: end if 14: end for 15: Let the shape with new positions be Y e 16: Update T and PDM parameters: b e = P T (T 1 Y e X) 17: Apply PDM constraints: b = PdmConstrain( b,β) e 18: Get new model shape: X = X + Pb 19: end for invariants combined with non-linear classifiers. As a result, IOF-ASM produces a more accurate segmentation of the facial features [21]. The matching procedure is summarized in Algorithm 1. In line 1 the image is preprocessed to obtain a set of differential invariants. These invariants are the core of the method and they consist on combinations of partial derivatives that result invariant to rigid transformations [22,2]. Moreover, IOF-ASM uses a minimal set of order K so that any other algebraic invariant up to order K can be reduced to a linear combination of elements of this minimal set [13]. The other key point of the algorithm is between lines 1 and 1. For each landmark, an image-driven search is performed to determine the best position for it to be placed. The process starts by sampling the invariants in a neighborhood of the landmark (line 1). In IOF-ASM this neighborhood is represented by a rectangular grid, whose dimensions are parameters of the model. A non-linear texture classifier analyzes the sampled data to determine if the local structure of the image is compatible with the one learnt during training for this landmark. A predefined number of displacements are allowed for the position of the landmark (perpendicularly to the boundary, as in [11]), so that the texture classifier analyzes several candidate positions. Once the best candidate is found, say (x B,y B ), the matching between its local image structure and the one learnt during training is verified (line 1) by means of a robust metric [14]. The applied metric consists on the evaluation of the sampled data grouped according to its distance perpendicularly to the shape boundary. Grouping this way, the samples can be organized in a one-dimensional profile of length l P. Based on the output from the texture classifier, each position on this profile will result as a supporting point or an outlier (the supporting points are those profile points suggesting

5 36 D. González-Jiménez et al. that (x B,y B ) is the best position for the landmark to be placed, while outliers indicate a different position and, therefore, suggest that (x B,y B ) is incorrect). If the supporting points are (at least) two thirds of l P, then the matching is considered accurate and the landmark is moved to the new position. Otherwise the matching is not trustworthy (i.e. the image structure does not clearly suggests a landmark) and the landmark position is kept unchanged (see [21] for details). The PDM constraints of line 1 ensure that the obtained shape is plausible according to the learnt statistics (i.e. it looks like a face). For this purpose, each component of b is limited so that b k β λ k, (1 k t); where t is the number of modes of variation of the PDM, λ k is the eigenvalue associated to the k-th mode and β is a constant, usually set between 1 and 3, that controls the degree of flexibility of the PDM (see [11]). 3 Correcting Pose Variations in Face Images Once the flexible shape model (with coordinates X) has been fitted to the face image I, the shape parameters b are extracted using equation (2). In particular, we are interested in the subset of parameters describing the pose (b pose ). In order to generate a frontal mesh, these parameters are set to zero 2. Hence, we obtain a new vector of parameters ˆb and, through equation 1, the frontal face mesh ˆX. Given the original face I, the coordinates of its respective fitted flexible shape model, X, and the new set of coordinates, ˆX, a virtual face image Î must be synthesized by warping the original face onto the new shape. For this purpose, we used a method developed in [4], based on thin plate splines. Provided the set of correspondences between X and ˆX, the original face I is allowed to be deformed so that the original landmarks are moved to fit the new shape. The full procedure of pose normalization is shown in figure 2. Test image X Pose X Normalization IOF ASM Fitting I I TPS Warping Fig. 2. Block diagram for pose normalization. TPS stands for Thin Plate Splines. 2 We will use the term frontal when referring to the pose of the mean shape of the PDM. However, the only requirement of the method is that all shapes can be mapped to a common view, then there is not a need for a strictly frontal mean-shape.

6 Automatic Pose Correction for Local Feature-Based Face Authentication Advantages over Warping onto a Mean Shape When warping an image onto the average shape ( X) of a training set, all shape parameters are set to zero. In other words, the fitted flexible shape model is forced to be moved to the coordinates of X. Holistic approaches such as PCA need all images to be embedded into a given reference frame (an average shape for instance), in order to represent these images as vectors of ordered pixels. The problem arises when the subject s shape differs enough from the average shape, as the warped image may appear geometrically distorted, and subject-specific information may be removed. Given that our method is not holistic but uses local features instead, the reference-frame constraint is avoided and the distortion is minimized by modifying only pose parameters rather than the whole shape. 4 Feature Extraction Once the normalization process has finished, we must proceed to extract features from the virtual frontal images Î. Up to now, most algorithms encoding local information have been based on localizing a pre-defined set of landmarks and extracting features from the regions surrounding those points. The key idea behind our approach relies on selecting an own and discriminative set of points per client, where features should be extracted. The choice of this set is accomplished through a two-layer strategy, whose stages are explained below. Layer I: Shape-driven selection and matching. In the first step, a preliminary selection of facial points is accomplished through the use of shape information [5]. Lines depicting face structure are extracted by thresholding the response Ridges & Valleys Thresholding Sampling A) Layer I B) Layers I+II Fig. 3. A) Layer I: A ridge and valley detector is applied to the original image (top left), and its response is shown on the right. Thresholding this representation leads to a set of lines depicting face structure (bottom left). The set of points P is obtained by sampling from these lines (bottom right). B) Layers I+II: Final set of points after layer II is applied.

7 362 D. González-Jiménez et al. of a ridge and valley detector, and a set of points P = {p 1, p 2,...,p n } is chosen automatically by sampling from these lines. Figure 3-A illustrates this procedure. Then, a set of multi-scale and multi-orientation Gabor features (so-called jet) is computed at each shape driven point. Let J pi be the jet obtained from point p i. Given the two faces to be compared, say Îtrain and Îtest, their respective sets of points are computed: P train = {p 1, p 2,...,p n } and P test = {q 1, q 2,...,q n }, and a shape matching algorithm based on shape contexts [6] is used to calculate the correspondences between the two sets of points, ξ (i) :p i = q ξ(i). Hence, jet J pi will be compared to J qξ(i). The comparison between J pi and J qξ(i) is given by the normalized dot product (< J pi, J qξ(i) >), but taking into account that only the moduli of jet coefficients are used. Layer II: Accuracy-based selection. Some previous approaches have been focusedonidentifyingwhichfeatureswere the most important for authentication purposes. Among others, [8], [7] have selected and weighted the nodes from a rectangular grid based on a Linear Discriminant Analysis (LDA). This kind of analysis is possible due to the fact that a given node represents the same facial region in every image. In our case, we can not assume this, so a different method is needed in order to select the most discriminative points. The problem can be formulated as follows. Given: a training image for client C, sayîtrain, } a set of images of the same client {Îc j,j =1,...,N c,and {Îim } a set of imposter images j,j =1,...,N im, we want to find which subset, P P train, is the most discriminative. As long as each p i from P train has a correspondent point in any other image, we evaluate the individual classification accuracy of its associated jet J pi, so that only the locations whose jets are good at discriminating between clients and imposters are preserved. With the set of images given above, we have N c client accesses and N im imposter trials for jet J pi to classify. We measure the False Acceptance Rate (FAR i ) and the False Rejection Rate (FRR i ) for this jet and, if the Total Error Rate (TER i =FAR i +FRR i ) exceeds a threshold τ, jetj pi will be discarded. Finally, only a subset of points, P, is chosen per image, and the score between Î train and Îtest is given by: S = f n { < Jpi, J qξ(i) >} p i P (3) where f n stands for a generic combination rule of the n dot products. Figure 3-B presents the set of points that was chosen after both layer selection. 5 Experimental Results on the XM2VTS Database The proposed method was tested using the XM2VTS database on configurations I and II of the Lausanne protocol [9]. The XM2VTS database contains image

8 Automatic Pose Correction for Local Feature-Based Face Authentication 363 Table 1. False Acceptance Rate (FAR), False Rejection Rate (FRR) and Total Error Rate (TER) over the test set for our method and automatic approaches from [17] Conf. I Conf. II FAR(%) FRR(%) TER(%) FAR(%) FRR(%) TER(%) UPV ± ±.71 UNIS-NC ± ± 1.15 IDIAP ± ±.71 Pose Corr.(Auto) ± ± 1.15 Pose Corr.(Manual) ± ± 1. No Pose Corr.(Auto) ± ± 1.28 No Pose Corr.(Manual) ± ± 1.28 data recorded on 295 subjects (2 clients, 25 evaluation imposters, and 7 test imposters).the database is divided into three sets: training, evaluation and test. The training set was used to build client models, the PDM, and the IOF- ASM 3, while the evaluation set was used to select the best features and estimate thresholds. Finally, the test set was employed to assess system performance. In all the experiments, n = 13 shape-driven points are computed for every image. However, only n 13 local scores are computed, because of the feature selection explained in Section 4. The median rule [1] was used to fuse these scores, i.e. f n median. Configurations I and II of the Lausanne protocol differ in the distribution of client training and client evaluation data, representing configuration II the most realistic case. In configuration I, there are 3 training images per client, while in configuration II, 4 training images are available. Hence, for a given test image, we get 3 and 4 scores respectively, which can be fused in order to obtain better results. Again, the median rule was used to combine these values, obtaining a final score ready for verification. Table 1 shows a comparison between the proposed method (Pose Corr.(Auto)) and a set of algorithms that entered the competition held in conjunction with the Audio- and Video-based Biometric Person Authentication (AVBPA) conference in 23 [17]. All these algorithms are automatic. In this table, and derived from the work in [24], 9% confidence intervals for the TER measures are also given. As we can see, our approach offers competitive error rates in both configurations (with no statistically significant differences between methods). Furthermore, the last three rows from this table show baseline results: Pose Corr.(Manual): The automatic segmentation provided by IOF-ASM is replaced by manual annotation of landmarks. No Pose Corr.(Auto): Automatic segmentation without pose correction (only in-plane rotations are corrected). No Pose Corr.(Manual): Manual segmentation without pose correction (only in-plane rotations are corrected). 3 The IOF-ASM was built with the same parameters detailed in [21].

9 364 D. González-Jiménez et al. It is clear that the use of IOF-ASM offers accurate results for our task, as the degradation between the error rates with manual and automatic segmentation is small. Moreover, the comparison between lines 4 and 6-7, shows that the use of pose-corrected images improves the performance of the system (even if manual landmarks are used to segment the original faces). 6 Conclusions We have presented an automatic face authentication system that reduces the effect of pose variations by synthesizing frontal face images. The segmentation of the face in the original image is accomplished by means of the IOF-ASM approach. A set of discriminative points and features is then selected in two steps: the shape-driven location stage and the accuracy-based selection step. The quality of the synthesized face (and thus, system performance) mainly depends on the segmentation accuracy, which is intimately related to the degree of pose variation in the input image and the dataset used for training. The achieved results on the XM2VTS database demonstrate the usefulness of the method in a limited range of pose variations, offering state-of-the-art error rates. As a main future research line, we plan to work on video-sequences in which facial features will be tracked in a frame-by-frame basis through the combination of IOF-ASM segmentation and a robust face tracker [25]. Acknowledgments This work is framed within the RAVIV project from Biosecure NoE, and has also been partially funded by grants TEC , TIC C2 and FIT from the Spanish Ministry of Science and Technology. FS is supported by a BSCH grant. AF holds a Ramón y Cajal Research Fellowship. References 1. Pentland, A. et al. View-based and Modular Eigenspaces for Face Recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 1994, pp Beymer, D.J. and Poggio, T. Face Recognition from One Example View. In Proc. International Conference on Computer Vision, 1995, pp Blanz, V. and Vetter, T. A Morphable model for the synthesis of 3D faces. In Proc. SIGGRAPH, 1999, pp Bookstein, Fred L. Principal Warps: Thin-Plate Splines and the Decomposition of Deformations. In IEEE Transactions on Pattern Analysis and Machine Intelligence 11, 6 (1989), González-Jiménez, D., Alba-Castro, J.L., Shape Contexts and Gabor Features for Face Description and Authentication, in Proc. IEEE ICIP 25, pp Belongie, S., Malik, J., Puzicha J. Shape Matching and Object Recognition Using Shape Contexts. In IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 24 (22),

10 Automatic Pose Correction for Local Feature-Based Face Authentication Duc, B., Fischer, S., and Bigun, S. Face authentication with sparse grid gabor information. In IEEE Proc. ICASSP, (Munich 1997), vol. 4, pp Argones-Rúa, E., Kittler, J., Alba-Castro, J.L., González-Jiménez, D. Information fusion for local Gabor features based frontal face verification. In Proc. International Conference on Biometrics (ICB), Hong Kong 26, (Springer), pp Luttin, J. and Maître, G. Evaluation protocol for the extended M2VTS database (XM2VTSDB). Technical report RR-21, IDIAP, Kittler, J., Hatef, M., Duin, R., and Matas, J. On Combining Classifiers. In IEEE Transactions on Pattern Analysis and Machine Intelligence 2, 3 (1998), Cootes, T., Taylor, C., Cooper, D., and Graham, J. Active shape models - their training and application. Computer Vision and Image Understanding 61, 1 (1995), Cootes, T., Edwards, G., and Taylor, C. Active appearance models. In Proc. European Conference on Computer Vision (Springer, 1998), vol. 2, pp Florack, L. The Syntactical Structure of Scalar Images. PhD thesis, Utrecht University, Utrecht, The Nedherlands, Huber, P. Robust Statistics. Wiley, New York, Kang, H., Cootes, T., and Taylor, C. A comparison of face verification algorithms using appearance models. In Proc. British Machine Vision Conference (Cardiff, UK, 22), vol. 2, pp Lanitis, A., Taylor, C., and Cootes, T. Automatic interpretation and coding of face images using flexible models. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 7 (1997), Messer, K., Kittler, J., Sadeghi, M., Marcel, S., Marcel, C., Bengio, S., Cardinaux, F., Sanderson, C., Czyz, J., Vandendorpe, L., and al. Face verification competition on the XM2VTS database. In Proc. 4th International Conference on Audio- and Video-based Biometric Person Authentication (AVBPA) Guildford, UK (23), pp Messer,K.,Matas,J.,Kittler,J.,Luettin,J.,andMaitre,G. XM2VTSDB:The extended M2VTS database. In Proc. International Conference on Audio- and Video-Based Person Authentication (1999), pp Philips, P., Moon, H., Rizvi, S., and Rauss, P. The FERET evaluation methodology for face recognition algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(1) (2), Schmid, C., and Mohr, R. Local greyvalue invariants for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(5) (1997), Sukno, F., Ordas, S., Butakoff, C., Cruz, S., and Frangi, A. Active shape models with invariant optimal features IOF-ASMs. In Proc. Audio- and Video-Based Biometric Person Authentication (New York, USA, 25), Springer, pp Walker, K., Cootes, T., and Taylor, C. J. Correspondence using distinct points based on image invariants. In British Machine Vision Conference (1997), vol. 1, pp Wiskott, L., Fellows, J.-M., Kruger, N., and von der Malsburg, C. Face recognition by elastic bunch graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 7 (1997), Bengio, S. Mariéthoz, J. A statistical significance test for person authentication. In Proc. Odyssey, 24, pp Baker, S. and Matthews, I. Equivalence and Efficiency of Image Alignment Algorithms. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 21, vol. 1, pp