A new kernel Fisher discriminant algorithm with application to face recognition

Transcription

1 Neurocomputing 56 (2004) Letters A new kernel Fisher discriminant algorithm with application to face recognition Jian Yang a;b;, Alejandro F. Frangi a, Jing-yu Yang b a Computer Vision Group, Aragon Institute of Engineering Research, Universidad de Zaragoza, Zaragoza E-50018, Spain b Department of Computer Science, Nanjing University of Science and Technology, Nanjing , People s Republic of China Abstract Kernel-based methods have been of wide concern in the eld of machine learning and neurocomputing. In this paper, a new Kernel Fisher discriminant analysis (KFD) algorithm, called complete KFD (CKFD), is developed. CKFD has two advantages over the existing KFD algorithms. First, its implementation is divided into two phases, i.e., Kernel principal component analysis (KPCA) plus Fisher linear discriminant analysis (FLD), which makes it more transparent and simpler. Second, CKFD can make use of two categories of discriminant information, which makes it more powerful. The proposed algorithm was applied to face recognition and tested on a subset of the FERET database. The experimental results demonstrate that CKFD is signicantly better than the algorithms of Kernel Fisherface and Kernel Eigenface. c 2003 Elsevier B.V. All rights reserved. Keywords: Kernel-based methods; Principal component analysis (PCA); Fisher linear discriminant analysis (FLD or LDA); Feature extraction; Face recognition 1. Introduction Principal component analysis (PCA) and Fisher linear discriminant analysis (FLD) are two classical techniques for linear feature extraction. In recent years, the nonlinear feature extraction methods, such as Kernel principal component analysis (KPCA) and Corresponding author. Computer Vision Group, Aragon Institute of Engineering Research, Universidad de Zaragoza, Zaragoza E-50018, Spain, Tel.: ; fax: addresses: jyang@unizar.es (J. Yang), afrangi@unizar.es (A.F. Frangi), yangjy@mail.njust.edu.cn (J.-y. Yang) /$ - see front matter c 2003 Elsevier B.V. All rights reserved. doi: /s (03)

2 416 J. Yang et al. / Neurocomputing 56 (2004) Kernel Fisher discriminant analysis (KFD) have been of wide concern. KPCA was originally developed by Scholkopf [6], and KFD was subsequently proposed by Mika [3] and Baudat [1]. Mika s work is mainly focused on two-class problems, while Baudat s algorithm is applicable for multi-class problems. KFD turns out to be eective in many real-world applications due to its power of extracting the most discriminatory nonlinear features [1,3,7]. However, KFD always faces the ill-posed diculty in its application [3]. The reason is that KFD is implemented in the space spanned by all M mapped training samples. This means we are required to estimate an M M within-class covariance matrix using M samples; this covariance matrix is always singular (its rank is usually M c, where c is the number of classes). Regretfully, the present KFD algorithms (Mika [3], Baudat [1], and Yang [7]) all throw away the discriminant information contained in the null space of the within-class covariance matrix; this discriminant information turns out to be very important for face recognition [8,9]. In this paper, we show that KFD can be implemented in two phases, i.e., KPCA plus FLD. Based on this framework, a complete KFD algorithm is developed. This algorithm can make use of two kinds of discriminant information, that is, the discriminant information within the null space and that outside of it. The remainder of this paper is organized as follows. In Section 2, the KPCA plus FLD framework is derived and the complete KFD algorithm is developed. The proposed algorithm is tested on the FERET database in Section 3. Finally, a brief conclusion is oered in Section Complete KFD algorithm 2.1. Outline of the idea of KPCA and KFD For a given nonlinear mapping, the input data space R n can be mapped into the feature space F: : R n F x (x) (1) Correspondingly, a pattern in the original input space R n is mapped into a potentially much higher dimensional feature vector in the feature space F. An initial motivation of KPCA (or KFD) is to perform PCA (or FLD) in the feature space F. However, it is dicult to do so directly because it is computationally very intensive to compute the dot products in a high-dimensional feature space. Fortunately, kernel techniques can be introduced to avoid this diculty. The algorithm can be actually implemented in the input space by virtue of kernel tricks. The explicit mapping process is not required at all. Concerning the detailed algorithms of KPCA and KFD, please refer to [1,3,6,7]. For computational eciency, the polynomial kernel, k(x; y)=(x y +1) r, is adopted in this paper. And, the feature space is assumed to be an N-dimensional Euclidean

3 J. Yang et al. / Neurocomputing 56 (2004) space, i.e. F = R N. Actually, this assumption has been implicitly assumed in [7] A new KFD framework: KPCA plus FLD Based on the ideas of KPCA and KFD, the following equivalent relationships hold: KPCA in input space R n PCA in feature space R N ; (2) KFD in input space R n FLD in feature space R N : (3) In Euclidean space, the theoretical foundation of why FLD can be performed in the PCA transformed space has been laid in [9]. FLD was proven to be equivalent to PCA plus FLD for small sample size problems. In these problems, the number of training samples is less than the dimension of the feature vector, thereby the within-class scatter matrix is singular. Since real-world problems are always turned into small sample size problems by a nonlinear mapping, we can apply the result in [9] directly to the data in the mapped feature space R N. Thus, we have FLD in feature space R N PCA + FLD in feature space R N ; (4) PCA + FLD in feature space R N Phase 1: R N PCA R m ; Phase 2: R m FLD R d ; (5) where m = M 1; M is the number of the training samples, and d is the number of desired features. Now, let us consider the Phase 1 of the right side of Eq. (5). Since performing PCA in the feature space is equivalent to the implementation of KPCA in the input space (by Eq. (2)), we have PCA + FLD in feature space R N Phase 1: R nkpca R m ; Phase 2: R mfld R d : (6) From Eqs. (6), (4) and (3), it follows that KFD in input space R n Phase 1: R nkpca R m ; Phase 2: R mfld R d : (7) Eq. (7) tells us that KFD is equivalent to the two-phase algorithm: KPCA plus FLD. Thus, a new algorithm framework for KFD is built. Now, the remaining problem is how to implement FLD in the KPCA-transformed space. After all, the within-class scatter matrix is still singular in the KPCA-transformed

4 418 J. Yang et al. / Neurocomputing 56 (2004) space. We will address this problem in the following section. It should be emphasized that our approach will be to take advantage of this singularity rather than avoiding it Complete KFD algorithm In the KPCA-transformed space, let us dene the between-class and within-class scatter matrices, S b and S w : c S b = P(! i )(m i m 0 )(m i m 0 ) T ; (8) S w = i=1 c l i 1 P(! i ) (X ij m i )X ij m i ) T l i ; (9) i=1 j=1 where X ij denotes the jth training sample in class i; P(! i ) is the prior probability of class i; l i is the number of training samples in class i; m i is the mean vector of the training samples in class i; m 0 the mean vector across all training samples. The optimal FLD algorithm provided in [8] can be modied to implement the Phase 2 of Eq. (7). Its main idea is to split the within-class scatter matrix S w into two subspaces: the null space w and its orthogonal complement w. Both subspaces contain important discriminant information. We rst use the classical Fisher criterion [8,9] to extract the rst category of discriminant information from w and then use the between-class scatter criterion J b ( ) = T S b to extract the second category of discriminant information from w. The detailed algorithm is given as follows: CKFD algorithm. Step 1: Use KPCA [6] to transform the original data space (X-space) into an m-dimensional space R m (Y-space), where m = M 1, M is the number of the training samples. Step 2: In R m, construct the between-class and within-class scatter matrices S b and S w. Calculate S w s orthonormal eigenvectors 1 ;:::; m, assuming the rst q ones are corresponding to positive eigenvalues. Step 3: Extract the discriminant features of Category I: Let P 1 =( 1 ;:::; q ). Dene S b = P T 1 S bp 1 ; S w = P T 1 S wp 1, and calculate l generalized eigenvectors u 1 ;:::;u l of S b = S w using the algorithm in [2]. The discriminant features of Category I are Z j = (P 1 u j ) T Y; j =1;:::;l, where l = c 1; c is the number of classes. Step 4: Extract the discriminant features of Category II: Let P 2 =( q+1 ;:::; m ). Dene S b = P T 2 S bp 2, and calculate S b s rst d l largest eigenvalues and the corresponding eigenvectors v l+1 ;:::;v d. The discriminant features of Category II are Z j = (P 2 v j ) T Y; j = l +1;:::;d, where d is the number of desired features. From the CKFD algorithm, two categories of discriminant features are derived. The total number of discriminant features can be over c 1 and up to 2(c 1), where c is the number of classes. This is dierent from the existing KFD algorithms [1,7], which can yield c 1 discriminant features at most. Essentially, generalized discriminant analysis

5 J. Yang et al. / Neurocomputing 56 (2004) (GDA) [1] is equivalent to CKFD using only the discriminant features of Category I (from Step 1toStep 3). But, GDA is not as transparent and simple as CKFD. 3. Experiments The proposed algorithm was applied to face recognition and tested on a subset of the FERET face image database [4,5]. This subset includes 1400 images of 200 individuals (each individual has 7 images). It is composed of the images named with two-character strings: ba, bj, bk, be, bf, bd, and bg. These strings indicate the kind of imagery, see [4]. This subset involves variations in facial expression, illumination, and pose (±15 and ±25 ). In our experiment, the facial portion of each original image was cropped based on the location of eyes and, the cropped image was resized to pixels and pre-processed by histogram equalization. In our experiment, three images of each subject are randomly chosen for training, while the remaining images are used for testing. Thus, the total number of training samples is = 600 and the total number of testing samples is = 800. Kernel Eigenface [7], Kernel Fisherface [7], and the proposed CKFD algorithm are, respectively, used for feature extraction. Yang [7] has demonstrated that a second or third order polynomial kernel suces to achieve good results with less computation than other kernels. So, for consistency with Yang s studies, the polynomial kernel, k(x; y)=(x y+1) r, is adopted here (r =2). A minimum-distance classier is employed for classication. The above experiment is repeated 10 times. In each time, the training sample set is selected at random so that the training sample sets are dierent for each test. The experimental results are shown in Table 1. From Table 1, we can see that CKFD is superior to Kernel Eigenface and Kernel Fisherface when its two categories of discriminant features (199 features of Category I and 15 features of Category II) are combined. The performance of Kernel Fisherface is similar to that of CKFD using only the features ofcategory I. This result is reasonable since Kernel Fisherface can only utilize the rst category of discriminant information (that outside of the null space of the within-class scatter matrix). For CKFD, the performance is signicantly improved when the discriminant features of Category II are involved. This indicates that the second category of discriminant Table 1 The comparison of correct recognition rates (%) of Kernel Eigenface, Kernel Fisherface and the proposed CKFD over 10 tests Method Average Kernel Eigenface (200) Kernel Fisherface (199) CKFD, Category I (199) CKFD (214) Note: In Table 1, the values in the parentheses denotes the number of features, CKFD (214) refers to 199 features of Category I and 15 features of Category II being used.

6 420 J. Yang et al. / Neurocomputing 56 (2004) information (that within the null space of the within-class scatter matrix) is really critical for classication. 4. Conclusions A complete Kernel Fisher discriminant (CKFD) algorithm is developed in this paper. Compared to the existing KFD algorithms, CKFD has two advantages: rst, it is easier to be implemented due to the simplicity of the algorithm framework; second, it is capable of making use of two categories of discriminant information, i.e., the discriminant information within and outside of the null space of the within-class scatter matrix. Our experiments demonstrate that combination of these two categories of discriminant information can achieve the best results. Acknowledgements This work is supported by grants TIC C02 from the same Spanish Ministry of Science and Technology (MCyT). It is also partially supported by the National Science Foundation of China under Grant No AFF is also supported by a Ramon y Cajal Research Fellowship from MCyT. Finally, we would like to thank the anonymous reviewers for their constructive advice. References [1] G. Baudat, F. Anouar, Generalized discriminant analysis using a kernel approach, Neural Computation 12 (10) (2000) [2] G.H. Golub, C.F. Van Loan, Matrix Computations, 3rd Edition, The Johns Hopkins University Press, Baltimore and London, 1996, pp [3] S. Mika, G. Ratsch, J. Weston, B. Scholkopf, K.-R. Muller, Fisher discriminant analysis with kernels, IEEE International Workshop on Neural Networks for Signal Processing, Vol. IX, Madison, USA, August, 1999, pp [4] P.J. Phillips, The Facial Recognition Technology (FERET) Database, feret/feret master.html. [5] P.J. Phillips, H. Moon, S.A. Rizvi, P.J. Rauss, The FERET evaluation methodology for face-recognition algorithms, IEEE Trans. Pattern Anal. Mach. Intell. 22 (10) (2000) [6] B. Scholkopf, A. Smola, K.R. Muller, Nonlinear component analysis as a kernel eigenvalue problem, Neural Computation 10 (5) (1998) [7] M.H. Yang, Kernel Eigenfaces vs. kernel Fisherfaces: face recognition using kernel methods, Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (RGR 02), Washington DC, May, 2002, pp [8] J. Yang, J.-Y. Yang, Optimal FLD algorithm for facial feature extraction, SPIE Proceedings of the Intelligent Robots and Computer Vision XX: Algorithms, Techniques, and Active Vision, October, 2001, Vol. 4572, pp [9] J. Yang, J.-Y. Yang, Why can LDA be performed in PCA transformed space? Pattern Recognition 36 (2) (2003)

7 J. Yang et al. / Neurocomputing 56 (2004) Jian Yang was born in Jiangsu, China, on 3rd June He obtained his Bachelor of Science in Mathematics at the Xuzhou Normal University in China in He then continued on to complete the Masters of Science degree in Applied Mathematics at the Changsha Railway University in 1998 and his Ph.D. at the Nanjing University of Science and Technology in the Department of Computer Science on the subject of Pattern Recognition and Intelligence Systems in From March to September in 2002, he worked as a research assistant at department of computing, Hong Kong Polytechnic University. Now, he is a postdoctoral research fellow at the University of Zaragoza and is aliated with the Division of Bioengineering of the Aragon Institute of Engineering Research (I3A). He is the author of more than 20 scientic papers in the area of articial intelligence. His current research interests include pattern recognition, computer vision and machine learning. Alejandro F. Frangi obtained his M.Sc. degree in Telecommunication Engineering from the Universitat Politecnica de Catalunya, Barcelona in 1996 where he subsequently did research on Electrical Impedance Tomography for image reconstruction and noise characterization. In 2001 he obtained his Ph.D. at the Image Sciences Institute of the University Medical Center Utrecht on model-based cardiovascular image analysis. The same year, Dr. Frangi moved to Zaragoza, Spain. He is now Assistant Professor at the University of Zaragoza and is aliated with the Division of Biomedical Engineering of the Aragon Institute of Engineering Research (I3A), a multidisciplinary research institute of the University of Zaragoza. He is a member of the Computer Vision Group and his main research interests are in computer vision and medical image analysis with particular emphasis in model- and registration-based techniques, and statistical methods. Dr. Frangi has recently been awarded the Ramo y Cajal Research Fellowship, a national program of the Spanish Ministry of Science and Technology. Jing-yu Yang received the B.S. Degree in Computer Science from Nanjing University of Science and Technology (NUST), Nanjing, China. From 1982 to 1984 he was a visiting scientist at the Coordinated Science Laboratory, University of Illinois at Urbana-Champaign. From 1993 to 1994 he was a visiting professor at the Department of Computer Science, Missuria University. And in 1998, he acted as a visiting professor at Concordia University in Canada. He is currently a professor and Chairman in the department of Computer Science at NUST. He is the author of over 100 scientic papers in computer vision, pattern recognition, and articial intelligence. He has won more than 20 provincial awards and national awards. His current research interests are in the areas of pattern recognition, robot vision, image processing, data fusion, and articial intelligence.