Local Binary LDA for Face Recognition

Transcription

1 Local Binary LDA for Face Recognition Ivan Fratric 1, Slobodan Ribaric 1 1 Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, Zagreb, Croatia {ivan.fratric, slobodan.ribaric}@fer.hr Abstract. Extracting discriminatory features from images is a crucial task for biometric recognition. For this reason, we have developed a new method for the extraction of features from images that we have called local binary linear discriminant analysis (LBLDA), which combines the good characteristics of both LDA and local feature extraction methods. We demonstrated that binarizing the feature vector obtained by LBLDA significantly improves the recognition accuracy. The experimental results demonstrate the feasibility of the method for face recognition as follows: on XM2VTS face image database, a recognition accuracy of 96.44% is obtained using LBLDA, which is an improvement over LDA (94.41%). LBLDA can also outperform LDA in terms of computation speed. Keywords: Biometrics, Face recognition, Linear discriminant analysis, Local features 1 Introduction Biometrics is an emerging technology [1, 2] that is used to identify people by their physical and/or behavioral characteristics. The face is a biometric characteristic that contains a variety of features that are suitable for biometric recognition. However, extracting these discriminatory features from face images is a difficult task. The extracted features must provide good recognition accuracy and be robust to intra-class variations, which is a major problem due to variations in the position of the face, facial expressions, lighting, appearance caused by aging, etc. The most popular group of feature extraction methods for face recognition are the appearance-based methods, such as PCA [3] and LDA [4]. These methods observe entire images as a feature vector and then apply transformations that optimize some criterion function. Principal component analysis (PCA) finds the optimal transformation for the image representation; however, this transformation is not necessarily optimal for recognition, which is why linear discriminant analysis (LDA) usually gives better recognition accuracy. LDA finds a linear transformation that, when applied to a set of images, maximizes the between-class variance, while at the same time minimizing the within-class variance. Various modifications to the basic LDA approach have been proposed [5, 6, 7, 8].

2 Pentland et al. [9] used PCA to extract local features from rectangular patches placed on salient facial features: eyes, nose and mouth. Recently, even more attention has been given to local features, most notably the features extracted using a Gabor filter [10] and local binary patterns [11]. Local features are lighting-invariant and can give a better recognition accuracy than global features, such as those extracted using appearance-based approaches. Several approaches combine different local features or both local and global features for face recognition. Méndez-Vázquez et al. [12] combine local binary patterns and local discrete cosine transform (DCT) for face recognition. They first discard low-frequency DCT coefficients as a preprocessing step, and than apply local binary patterns to represent the facial features. Pan and Cao [13] combine local features obtained by applying 2D non-negative matrix factorization (NMF) in 2D DCT domain with the global features obtained by 2D PCA. We propose a new feature extraction method called local binary linear discriminant analysis (LBLDA) that combines the good characteristics of both appearance-based methods and methods based on local features. The general idea is to use LDA to locate and extract the discriminant local features. The images are divided into a set of possibly overlapping regions and then LDA is performed using the data for each region separately. In this way, we can extract the optimal local features in terms of the LDA criterion function. Based on this criterion function, we can also extract more features from the regions in the image that contain more discriminatory information. We take only the sign of the features and discard the magnitude in order to obtain a binary feature vector. Although it may appear that we are losing important discriminatory information by doing this, we demonstrate experimentally that using binary features significantly increases the recognition accuracy. There are several benefits of using binary features. In a recent paper, Sun and Tan [14] present several of the benefits of using ordinal measures for feature representation, but these benefits hold for binary features as well: (i) High-level measurements (i.e., measurements expressed as exact values) are sensitive to illumination changes, blur, noise, deformation, and other image degradations. Fine models of visual objects based on high-level measurements are useful for image detail preservation and image reconstruction, but are unnecessary for object recognition. (ii) Binary features are more compact and faster to process due to the simpler computations. (iii) Binary features are biologically plausible. For example, DeAngelis et al. [15] found that many striate cortical neurons visual responses saturate rapidly with the magnitude of the contrast as the input. This indicates that the determining factor of visual perception is not the absolute value of the contrast, but its polarity. The rest of the paper is organized as follows. In Section 2 we give a detailed description of the proposed method. In Section 3 we describe experiments that demonstrate the feasibility of the method for face recognition. The conclusions and suggestions for future work are given in Section 4.

3 2 LDA and LBLDA LDA, as commonly used in image-based biometrics [4, 16], involves using the information from the entire image. All the images in the training set are treated as n- dimensional vectors x i, i = 1, 2,, N, where N is the number of training images and n is the number of pixels in an image. LDA finds a transformation W LDA that transforms the original image vectors into a new space in which the between-class variance is maximized, while the within-class variance is minimized. LDA maximizes the criterion function T W S BW J ( W) (1) T W S W where S B is the between-class variance matrix and S W is the within-class variance matrix: W S S C B N i i 1 W N N C i 1 x i ( m i m)( m m) ( x m )( x m ) i i i T T (2) where N i is the number of samples in class ω i, m i is the mean sample of class ω i and m is the mean of all the samples. The solutions of the optimization problem (eq. 1) are the vectors w j, obtained as a solution of the generalized eigenvector problem j S W w j = S B w j, that correspond to the largest generalized eigenvalues λ j. The maximum dimensionality of the LDA space is C 1, where C is the number of classes. One problem that often arises with LDA in biometrics is that of the small sample size: if N < n C, the within-class scatter matrix S W is singular, and the computation of the LDA subspace becomes impossible by traditional means. The usual way of solving this problem is to first reduce the dimensionality of the training samples by means of PCA [4]. However, other solutions have also been proposed, such as direct LDA [5], regularized LDA [6] or discriminative common vectors [7]. In our approach, we use LDA to extract the local features. First, the image is divided into a set of N R possibly overlapping regions. In our implementation, we use square regions obtained by using a sliding-window approach. A window of size p x p pixels is positioned in the upper-left corner of the image. The first region is composed of all the pixels that fall inside the window. The window is then translated by t p pixels to the right and, when the window falls outside of the image, it is moved t pixels down and all the way to the left of the image. The process is concluded when the bottom-right corner of the window reaches the bottom-right corner of the image. Each window position defines one of the N R regions R r, r = 1, 2,..., N R, where each region consists of p x p pixels.

4 For each region R r and for each training image, we form a vector x r i, i = 1, 2,, N; r = 1, 2,, N R by arranging into a vector all the pixels from the image i that fall into the region R r. The size of each vector x r i is p x p. For each region R r : (i) We perform local PCA on the vectors x i r, i = 1, 2,, N and obtain a subspace W r PCA. We project each of the vectors x i r into this subspace and obtain the vectors z i r. The size of the vectors z i r is N PCA, N PCA min(n-1, p x p). (ii) We perform LDA on the vectors z i r, i = 1, 2,, N. In this process we obtain a subspace W r LDA. (iii) We obtain a final subspace for the region R r, W r PCA+LDA by multiplying the transformation matrices of W r PCA and W r LDA. This subspace is spanned by the local LDA basis vectors w j r, j = 1, 2,, N LDA. N LDA is the subspace dimensionality, N LDA = min(c-1, N PCA ). The size of each vector w j r is p x p. For each vector w j r, we also note the corresponding LDA eigenvalues λ j r, which give information about the goodness of the vector w j r in terms of the LDA criterion function (eq. 1). We now have a set of N LDA x N R vectors w j r and the corresponding eigenvalues λ j r. N LDA x N R can be quite large, for example, for 64x64 images with p = 16 and t = 8 we could obtain up to vectors w j r. In order to select the most discriminatory features, we sort the vectors w j r by the falling values of the LDA eigenvalues λ j r. By taking the first N LBLDA N R x N LDA vectors w j r we form the local feature space. In this way we can take more features from the image locations that are more discriminatory and fewer, or even no features, from the locations that do not contain significant discriminatory information. Finally, we organize the obtained optimal basis into a data structure that we call the local subspace. This local subspace consists of N LBLDA records, where each record contains a region index r and the basis vector w k, where w k is the k-th vector in the sorted sequence of vectors w j r. In the recognition phase, for an unknown image I, we can use this local subspace to extract a N LBLDA -dimensional feature vector y as follows. The image I is divided into N R regions in the same manner as used to obtain the local subspace. The k-th component of this feature vector y k is obtained by computing the scalar product of w k and a (p x p)-dimensional vector obtained by arranging the pixels of region R r of the image I into a vector, where w k and r are components of the k-th record of the local subspace. To obtain a binary feature vector b, we simply take only the signs of the components of y (b k = 1 for y k > 0 and b k = 0 otherwise). This binary feature vector is called a binary live template. The use of binary feature vectors has been shown to significantly increase the recognition accuracy in our experiments. By taking only the signs of the components of the feature vector y we in fact use only information about whether the correlation between the pixels of the region R r and the local LDA basis w k is positive or negative, while disregarding the exact extent of the correlation. An alternate way to view the obtained local features is to observe them as filter responses. Instead of using predefined filters, such as the Gabor filter, these filters are learned on the training data, separate for each image location, so that they emphasize the differences between the classes, while suppressing the within-class variances. We extract the features from each image region using the appropriate filter and take only

5 the binary response, in a similar manner the responses of the Gabor filters are encoded to form the iris code [17] and the palm code [18]. The classification is based on the Hamming distance between the binary live template and the binary templates stored in the database. 3 Experimental evaluation The proposed method was tested on the XM2VTS face image database [19]. The database consists of 2360 images of 295 individuals (8 images per person). The images were taken in four sessions with two images taken per session. Prior to the experiments on this database, all the images were normalized in such a way that the images contain only the face; the person s eyes were always in the same position, all the images were 64x64 pixels in size and a lighting normalization by histogram fitting [20] was performed. Four images of each person (images from the first two sessions) were used for the training, and the remaining four were used for the experiments. Fig. 1 shows several normalized images from the XM2VTS face image database. Fig.1. Several normalized images from the XM2VTS face image database. Images in the same column belong to the same person. The following experiments were performed. Firstly, we show the recognition results of our method on the described datasets for different parameter combinations (Experiment 1). Secondly, we examine the image regions from which the most features are taken by the method and compare the results to the results obtained when the regions of interest are manually placed on the visually salient facial features (Experiment 2). Thirdly, we compare the results of our method to the results obtained by classic LDA on the same databases (Experiment 3) and examine the effect of binarization on the performance of our method (Experiment 4). Finally, we evaluate and compare the computation time requirements of the methods. Experiment 1: Recognition results of our method for different parameter combinations There are four main parameters in our method: (i) p determines the local window width and height in pixels (ii) t determines how many pixels the local window is translated to define the next region. If t = p the regions do not overlap.

6 (iii) (iv) N PCA determines the dimensionality to which local samples are reduced prior to performing LDA. If N PCA = p x p, the reduction of the dimensionality is not necessary. N LDA determines the feature vector length. A series of recognition experiments was performed with different values of these parameters on our test dataset. For each combination of window size p and translation step t we marked the best score together with the corresponding N PCA and feature vector length N LDA. The experiments were performed using the 1-NN classifier with the Hamming distance. The results of the experiment are shown in Table 1. Table 1. Face recognition results for different parameter combinations Window size p Window translation step t N PCA for best recognition accuracy N LBLDA for best recognition accuracy Best recognition accuracy % % % % % % % % % % Several conclusions can be made based on these experiments. Firstly, the recognition results are better for the overlapping than for the non-overlapping regions. When t is decreased to p/2 or p/4 the recognition accuracy is improved as more discriminant features are added. However, in this case the feature vector length increases. In some cases, even better recognition results can be achieved with t = p/8, for example, when p = 16, but this leads to a dramatic increase in the binary feature vector length (for example, from 1500 to 7300; see Table 1). In most cases the best recognition results were achieved with input parameter N PCA = 100 or 150. An increase in N PCA beyond 150 usually results in a decrease of the recognition accuracy. The interpretation of these results is as follows. LDA, like all supervised learning methods, tends to give good results on the training set, but poor results on the unseen data, when given too many degrees of freedom. Often, it is better to limit the size of the vectors that are input into the LDA in order to achieve a better generalization. The optimal window size and the translation step for the database used in the experiments were p = 16 and t = 4. Although we cannot claim that these parameters

7 would also perform best on different databases, they pose a good estimate for the optimal values of the parameters. 3.1 Regions of interest LBLDA takes more features from the image regions that carry more discriminatory information. In this subsection we will show such regions for our database and compare the recognition accuracy to the one obtained using local binary features extracted from patches manually placed on the visually salient facial features. In Fig. 2 we visualize the number of features taken from each image region when LBLDA is learned on the face database. Several images are given, corresponding to the different total number of features (N LBLDA ). The lighter areas correspond to the image regions from which the larger number of features are taken and the black areas correspond to the image regions from which no features are taken. From Fig. 2 it is obvious that the most features are taken from the areas of the eyes, nose, mouth and eyebrows, which is consistent with the human perception of the distinctive features on faces. (a) N LBLDA = 1 N LBLDA = 50 N LBLDA = 100 N LBLDA = 500 N LBLDA = 1000 N LBLDA = 1500 (b) Fig. 2. (a) Mean face image from the database, (b) visualization of the number of features taken from different face image regions. The lighter areas correspond to image regions from which the larger number of features are taken and the black areas to the image regions from which no features are taken. Experiment 2: Comparison of the recognition results based on features extracted from regions that are located by our method and local binary features extracted from manually marked regions We compared the results of our method to the results obtained when local binary features are extracted from patches manually placed on the visually salient facial features. Fig. 3 shows a mean face image from the face database with manually marked overlapping regions of interest. Fig. 4. presents the recognition results of the experiment. The input parameters p = 16, t = 8 and N PCA = 100 are used in LBLDA.

8 It is clear from Fig. 4 that the selection of image regions by our method gives a better recognition accuracy. This suggests that, although the majority of discriminant features are located in the manually marked regions (these regions correspond to the lightest areas in Fig. 2), other areas of the image still contain discriminant features that may significantly improve the recognition accuracy. Fig. 3. mean face image with overlapping regions of interest marked manually 100 Recognition accuracy (%) Number of features Face - LBLDA Face - manual regions Fig. 4. Comparison of recognition results with regions located by our method and manually marked regions. 3.2 Comparison of recognition results of LBLDA and LDA Experiment 3: Comparison of LBLDA and classic LDA. In order to demonstrate the feasibility of our method, the recognition results obtained using LBLDA were compared to the results obtained using features extracted by classic LDA on the same database. We also wanted to test how global features extracted by the LDA perform if they are binarized in a similar way to the local features and the Hamming distance is used

9 to compare them. We will call this method global binary LDA (GBLDA) in the remainder of the text. Recognition experiments with all the feature extraction methods were performed using the 1-NN classifier. A normalized correlation was used as a matching measure for the LDA feature vectors, as it was demonstrated [21] that this performs better than the Euclidean distance. The results are shown in Fig. 5. The figure shows the recognition accuracy depending on the length of the feature vectors. For all the methods the parameters giving the highest recognition accuracy were used. The results show that LBLDA outperforms LDA and GBLDA in terms of recognition accuracy. LBLDA achieves better recognition accuracy with a larger number of features (above 1300), but it is important to note that LBLDA uses binary feature vectors, which are simple to store and process. 100 Accuracy (%) Number of components LDA GBLDA LBLDA Fig. 5. Recognition accuracy of PCA, LDA, GBLDA and LBLDA on the face database depending on the number of features. Experiment 4: Effect of binarization on local LDA and using different distance measures. We made an experiment showing the effect of binarization and different distance measures on the recognition accuracy, with local features extracted using local LDA. Fig. 6 shows the recognition accuracy for our method (LBLDA), and our method without binarization with the Euclidean distance and the normalized correlation for face recognition.

10 Accuracy (%) LBLDA + Hamming distance LBLDA without binarization + Normalized correlation LBLDA without binarization + Euclidean distance Number of features Fig. 6. Comparison of recognition accuracy obtained using the Hamming distance, the normalized correlation (without feature vector binarization) and the Euclidean distance (without feature vector binarization) on the face database, depending on the number of features. From Fig. 6 we can see that using binary features gives the best recognition accuracy, while the normalized correlation gives slightly better results than the Euclidean distance, as is the case with classic LDA. Table 2 gives a summary of the best recognition accuracies for the different feature extraction methods and the distance measures. Table 2. The best recognition accuracies for the different feature extraction methods and the distance measures Features Recognition accuracy LDA + Euclidean distance 90.00% LDA + Normalized correlation 94.41% Global binary LDA (GBLDA) + Hamming distance 82.03% LBLDA + Hamming distance 96.18% LBLDA without binarization + Euclidean distance 90.67% LBLDA without binarization + Normalized correlation 91.86% 3.3 Computation speed There are several steps that need to be performed in a biometric recognition system using LBLDA features. Here, we examine the time cost of each of them separately and compare them to the time cost of the same steps in LDA. Firstly, the transformations need to be learned, which is the most time-consuming task, but this task needs to be performed only once, during the training stage.

11 Secondly, features have to be extracted from the images. This task needs to be performed once per image. Thirdly, there is the time cost of computing the distance between two feature vectors. The number of comparisons depends on the number of feature vectors stored in the database during the enrollment. Table 3 shows the processing time for each of these steps for LDA and LBLDA on the face database. Both LDA and LBLDA were implemented in C++. The experiments were run on an Intel Core 2 Quad processor running at 2.4 GHz, using only a single core. LBLDA not only gives a better recognition accuracy, but, as shown in Table 3, it can also perform faster when compared to LDA. The speed increase in learning and feature extraction is obtained with LBLDA because it does not require computations on as large matrices as LDA does. The speed increase in the distance computation is obtained because the Hamming distance is much simpler to compute using binary operations and lookup tables than the normalized correlation used in the LDA. Table 3. Processing time for LDA and LBLDA on the face database LDA N LDA = 100 LBLDA p = 16, t = 8, N PCA = 100, N LBLDA = 1000 LBLDA p = 16, t = 4, N PCA = 150, N LBLDA = 1500 Learning time 233s 34s 139s Feature extraction time 0.67ms 0.59ms 1.40ms Distance computation time 0.43 s 0.16 s 0.22 s 4 Conclusion Extracting discriminatory features from images is a crucial task for biometric recognition based on the face features. We propose a new method of feature extraction from images, called local binary linear discriminant analysis (LBLDA), which combines the good characteristics of both LDA and local feature extraction methods. LBLDA uses LDA to extract a set of local features that carry the most discriminatory information. A feature vector is formed by projecting the corresponding image regions onto a subspace defined by the combination of basis vectors, which are obtained from different image regions and sorted by the descending order of their corresponding LDA eigenvalues. We demonstrated that binarizing the components of this feature vector significantly improves the recognition accuracy. Experiments performed on the face image databases suggest that the LBLDA outperforms classic LDA both in terms of recognition accuracy and speed. In the future we plan to apply LBLDA on different datasets to test the robustness of the method to lighting and facial expression.

12 References 1. Jain, A.K., Bolle, R., Pankanti, S.: Biometrics: Personal Identification in Networked Society. Kluwer Academic Publishers (1999) 2. Zhang D.: Automated Biometrics: Technologies & Systems. Kluwer Academic Publishers (2000). 3. Turk, M., Pentland, A.: Eigenfaces for Recognition. J. Cognitive Neurosicence 3(1), (1991) 4. Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE Trans. Pattern Analysis and Machine Intelligence 19(7), (1997) 5. Yu, H., Yang, J.: A Direct LDA Algorithm for High-Dimensional Data with Application to Face Recognition. Pattern Recognition 34(10), (2001) 6. Dai, D.Q., Yuen, P.C.: Regularized discriminant analysis and its application to face recognition. Pattern Recognition 36(3), (2003) 7. Cevikalp, H., Neamtu, M., Wilkes, M., Barkana, A.: Discriminativen common vectors for face recognition. IEEE Trans. Pattern Analysis and Machine Intelligence 27(1), (2005) 8. Kim, T.K., Kittler, J.: Locally linear discriminant analysis for multimodally distributed classes for face recognition with a single model image. IEEE Trans. Pattern Analysis and Machine Intelligence 27(3), (2005) 9. Pentland, A., Moghaddam, B., Starner, T.: View-based and modular eigenspaces for face recognition. In Proc. CVPR 1994, (1994) 10. Serrano, A., de Diego, I.M., Conde, C., Cabello, E.: Recent advances in face biometrics with Gabor wavelets: A review. Pattern Recognition Lett. 31(5), (2010) 11. Marcel, S., Rodriguez, Y., Heusch, G.: On the Recent Use of Local Binary Patterns for Face Authentication. Int'l J. on Image and Video Processing, Special Issue on Facial Image Processing, 1--9 (2007) 12. Méndez-Vázquez, H., García-Reyes, E., Condes-Molleda, Y.: A New Combination of Local Appearance Based Methods for Face Recognition under Varying Lighting Conditions. Lecture Notes in Computer Science 5197, (2008) 13. Pan, C., Cao, F.: Face Image recognition Combining Holistic and Local Features. Lecture Notes in Computer Science 5553, (2009) 14. Sun, Z., Tan, T.: Ordinal Measures for Iris Recognition, IEEE Trans. Pattern Analysis and Machine Intelligence 31(12), (2009) 15. DeAngelis, G.C., Ohzawa, I., Freeman, R.D.: Spatiotemporal Organization of Simple-Cell Receptive Fields in the Cat s Striate Cortex, I. General Characteristics and Postnatal Development. J. Neurophysiology 69(4), (1993) 16. Wu, X., Zhang, D., Wang, K.: Fisherpalms Based Palmprint Recognition. Pattern Recognition Lett. 24(15), (2003) 17. Daugman, J.: High confidence visual recognition of persons by a test of statistical independence. IEEE Trans. Pattern Analysis and Machine Intelligence 15(11), (1993) 18. Zhang, D., Kong, W.K., You, J., Wong, M.: Online Palm Print Identification, IEEE Trans. Pattern Analysis and Machine Intelligence 25(2), (2003) 19. Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTSDB: The extended M2VTS database. In Proc. AVBPA99, (1999) 20. Gonzales, R.C., Woods, R.E.: Digital Image Processing. Addison Wesley (1993) 21. Kittler, J., Li, Y.P., Matas, J.: On Matching Scores for LDA-based Face Verification. In Proc. British Machine Vision Conference 2000, (2000)