Automatic Facial Expression Recognition Using Linear and Nonlinear Holistic Spatial Analysis

Transcription

1 Automatic Facial Expression Recognition Using Linear and onlinear Holistic Spatial Analysis Rui Ma 1, Jiaxin ang 1, 1 State Key Laboratory of Intelligent echnology and Systems, Department of Computer Science, singhua University, Beijing, , P.R.China mr02@mails.tsinghua.edu.cn Abstract. his paper is engaged in the holistic spatial analysis on facial expression images. e present a systematic comparison of machine learning methods applied to the problem of automatic facial expression recognition, including supervised and unsupervised subspace analysis, SVM classifier and their nonlinear versions. Image-based holistic spatial analysis is more adaptive to recognition task in that it automatically learns the inner structure of training samples and extracts the most pertinent features for classification. onlinear analysis methods which could extract higher order dependencies among input patterns are supposed to promote the performance of classification. Surprisingly, the linear classifiers outperformed their nonlinear versions in our experiments. e proposed a new feature selection method named the eighted Saliency Maps(SM). Compared to other feature selection schemes such as Adaboost and PCA, SM has the advantage of being simple, fast and flexible. 1 Introduction and Motivation 1.1 Background of Facial Expression Analysis Recent intelligent systems have devoted a great deal of efforts to the effective affective communication between human beings and mechanical entities in the virtual environment. he conveying and understanding of affective information via facial expression, voice, pose and gesture features largely in person-to-person communication. he purpose of affective communication is to incorporate such natural ways of communication in the person-to-machine interaction, and ultimately enlarging to the scope of natural machine-to-machine communication. One of the bases in affective communication is the automatic recognition of human emotional expressions. Human face can significantly reflect the emotional state of a person, and thus maybe one of the most natural means in human-machine interaction. Although the authentic emotions are affected by a variety of factors and might not be exactly revealed by the exterior visible facial clues, the explicit facial expression is

2 evidently the most direct indicator of a person s inner feelings. Human facial expression recognition is indispensable to a robust intelligent system as speech and gestures. Automatic facial expression recognition has been explored by researchers in a few decades and various methods were proposed in the relevant literature. Generally, facial expression recognition falls into two major categories: recognizing facial expression type and detecting visible facial actions. he former approach assigned each face image to be one of the seven primary emotions postulated by Ekman and Friesen in 1971 or other trained expression types. hese basic emotions which are ubiquitous across all human cultures comprise anger, disgust, fear, joy, neutral, sadness and surprise. In the latter case, it is more emphasized on the visible LOCAL facial features demonstrated in an image. Recognition methods fallen in this scenario attempt to detect and recognize the appearance of one single facial action unit or a combination of several facial action units descirbed in a FACS system, not necessarily assigning the face image as a specific type of expression. he pioneering work of facial expression analysis by Mase and Pentland employed optical flow to estimate the activity of 12 facial muscles. Other methods have been proposed to focus on the analysis of facial motions. However, these motion-based approaches ignored other important aspects of facial expression than the motion clues[1]. Via a set of measurable facial features, one of the earliest works to identify facial expressions concentrated on using a flexible face model and classification was made on these predefined features. Feature location and displacement were analyzed by optical flow or active appearance model. Facial actions or expressions were determined with neural networks or nonlinear embedding[11]. hese approaches drastically reduced the dimensionality of input features, but might lose vital information in the process. As an alternative to feature-based methods, the holistic analysis takes into account all the information contained in an image and focuses on discovering the intrinsic structural information via a set of sample images. Unsupervised holistic analyses such as principal component analysis(pca) and independent component analysis(ica) have been used to reveal the statistical dependencies among input features and to find the adaptive subspaces for classification[4,5]. Lyons applied the supervised Fisher linear discriminant analysis(fda) to learn an adequate linear subspace from class-specified training samples and the samples projected to this subspace can be best separated[9]. 1.2 Motivation and Proposed Approach However, both PCA and FDA address the linear projection. Representation of PCA and FDA projection encodes pattern information based on second order dependencies. In fact, the underlying features most useful for facial expression classification may lie within the higher order statistical dependencies among input features. Bartlett demonstrated that the ICA is superior to PCA in human face recognition in that ICA learns the higher-order dependencies in the input besides the correlations[1]. However, ICA seems to be time-consuming and whether the facial expression is composed of a set of independent components is not clear yet. People still try to seek out other solutions for this task. Our method is motivated by this concern and the nonlinear

3 holistic spatial analysis based on kernel methods would be investigated on the task of automatic facial expression recognition. Kernel trick is one of the vital reasons that make Support Vector Machine successful in solving nonlinear separable problems. Scholkopf combined kernel method with classical PCA and the extended Kernel PCA showed its good ability in extracting nonlinear features for efficient classification[10]. Likewise, Kernel FDA was also investigated and employed to many applications in computer vision. Existing works demonstrated the ability of Kernel FDA in seeking out the nonlinear discriminant features among the input patterns. In this paper, both the kernel based analyses will be applied to facial expression recognition. Another important aspect is about feature selection. Dailey et al. pointed out that local PCA carried out on random windows on face image can obtain better results in facial expression classification than performing PCA on a whole image[4]. hile Bartlett indicated that the holistic PCA and local PCA has the same effect, what contributes to this difference is the selection of random windows[1]. Lyons worked out a saliency map on human face that could facilitate facial expression recognition[9]. his map was actually a set of feature points such as eye corners sorted by their discriminating powers. Littlewort employed Adaboost to select features which were then fed into the SVM classifiers for recognition[8]. hese measures not only improve the classification accuracy but also speed up the recognition step which makes the realtime facial expression recognition possible. he idea of feature selection will also be incorporated in our system. e proposed a novel feature selection method named eighted Saliency Maps, see section onlinear Holistic Spatial Analysis he proposed approaches in this paper address nonlinear holistic spatial analysis on face images in extracting higher order statistical dependencies of training samples. wo kernel-based methods are examined: Kernel PCA and Kernel FDA. 2.1 Kernel Principal Component Analysis he idea of Kernel Principal Component Analysis is to find a feature space through kernel method, and then apply PCA in this space to seek the orthonormal components that denote the maximal variances of input features. Given a set of centered samples(zero mean, unit variance) { x1, x2,..., x }, n xk R, classical PCA finds a set of orthonormal vectors by solving the eigenvalue problem: λ v = Cv, where C is the covariance matrix of input xk, k = 1..., eigenvalues λ 0, and eigenvectors v R \{0}. n In Kernel PCA, each vector of x is projected from the input space to a higher feature space F. y = ( x), and y F. he dimensionality of F can be arbitrarily large. Applying PCA in F is equivalent to the eigenvalue problem: λ v = C v, where C is

4 the covariance matrix. It is often infeasible to compute C and work out the v in the high dimensional space F. Actually, kernel trick informs us that if we could represent any algorithm as the inner product of samples, we might easily construct the nonlinear version of it. In fact, all solutions of v with λ 0 must lie in the span of ( x1), ( x2),... ( x ), that is v = α( ) i 1 i x, where coefficients = i αi Ri, = 1,...,. Consider the following equation λ ( ( xk) v ) = ( ( xk) C v ), k = 1,..., If we define a matrix K by Kij = ( ( xi ) ( xj )) = k( xi, xj ), combining the above equations, we could get the eigenvalue problem of Kernel PCA: 2 λkα = K α λα = Kα Eigenvector α = [ α1, α2,..., α ]. hen normalize the each eigenvectors by their corresponding eigenvalues with 1 = λ( α α). 2.2 Kernel Fisher Discriminant Analysis Similarly, Kernel Fisher Discriminant Analysis first maps the input patterns to a higher dimensional feature space by a nonlinear mapping, and then applies Fisher Linear Discriminant Analysis to obtain a reduced space for better classifying the patterns. Given a set of centered samples(zero mean, unit variance) { x1, x2,..., x } n c labeled as c classes, xk R, and i samples are within class X i, =. In i= 1 i KFDA, the optimal projection w is the solution of following equation: ( w ) SBw w = arg max J( w ) = arg max. ( ) w w w S w S B and S is the within-class and between-class scatter matrix respectively. c S B = i( μ i μ )( μ i μ ), i= 1 c = ( ( k) μ i )( ( k) μ i ) i= 1 xk Xi, S x x where 1 μ = ( xk ), 1 i μi = ( xk), i = 1,..., c. k = 1 i k = 1 Likewise, all solutions of w must lie in the span of ( x1), ( x2),... ( x ), that is, there exists α [ α1, α2,..., α ] =, such that w = α ( ) i 1 i xi =Φα, = where Φ= [ ( x1 ),..., ( x )]. Project ( x k ) on w, we get ( ( x1) ( xk)) k( x1, xk) ( w ) ( xk) = α Φ ( xk) = α α M = M = α ξ. k ( ( x) ( xk)) k( x, xk)

5 c Denote KB = i( mi m)( mi m), K = ( ξk mi)( ξk mi) i= 1 c, i= 1 ξk Fi m i i = ξ, k = 1 k then, ( w ) S w = α K α and ( w ) S w = α K α. Kernel FDA is then equivalent to B α KBα solvingα = arg max J ( α) = arg max α α α K α. B 3 Facial Expression Classification he Japanese JAFFE expression database is employed in our experiments for the evaluation of our methods. It comprises 213 gray-level images of ten Japanese female subjects, each one expressing seven basic types of expressions. Several factors add difficulty to classification: the images have some variation in lighting, and some faces have slight in-plane and out-of-plane rotation. he original 256*256 pixels images were preprocessed to get the 70*50 pixels down-sampled aligned face images, with most of the background eliminated. he database was then enlarged to 426 images with the mirrored pictures. 3.1 Features and Feature Selection he features been used in the literature include the pixel intensity values of original images, diff.-images[5] and Gabor filtered images. 2D Gabor representation is obtained by filtering the images with a set of Gabor wavelets of different orientations and frequencies[4, 8, 9]. Since human face expressions differ with subtle changes in local areas, we would investigate how to capture these pertinent features to facilitate the classification. hus the aim of feature selection is to reduce the dimensionality of input patterns to make the computation feasible, meanwhile, to retain the most salient features that would reflect the expressional changes. In [4, 5] features are selected on the fixed grid called Gabor Jet. Littlewort et al. used Adaboost to iteratively select features. e here propose a new method named eighted Saliency Maps(SM) to select the appropriate features for facial expression recognition. he idea partly comes from selecting ICs in[1]. e compute the ratio of between-class variance and within-class variance of each feature across all the training samples. Denote an arbitrary feature as[ f 1, f 2,..., f ], is the number of training samples, then the ratio is: VarB σ k =, k = 1,..., n Var c where between-class variance 1 i 1 VarB = i 1 f j 1 j f = k 1 k, = i = c i i 1 within-class variance Var = f j fk i= 1 j= 1, i k= 1 n is the number of features. All the features are computed with above equations and sorted according to the obtained ratios(weights) in descending order. he first 500 2

6 Fig. 1. eighted Saliency Maps for seven basic expressions features sensitive to each of the seven basic facial expressions are shown in Fig.1. hey are marked in the image as grayscale points. he darkness of each pixel is proportional to its weight. Compared to the fixed feature points in Gabor Jet, feature selection methods based on learning are more promising to produce a better feature set. Adaboost iteratively selects a set of features based on their recognition errors respectively. PCA finds a subspace by solving an eigen problem. Input patterns project onto this space and transform to new features. In comparison with PCA, Adaboost and SM only investigate the classification ability of each feature solely, while PCA takes into account the whole distribution of training samples. onetheless, Adaboost and SM utilize the class label of each sample in learning the discriminant power of each feature, which to some extent would yield better result than PCA. Adaboost is an iterative process which may be not very fast to re-select pertinent features in a real-time application. SM is much faster than PCA and Adaboost, and it has the flexibility of employing different criteria to promote the performance. In our experiment, SM could remarkably reduce the dimensionality of input feature space and speed up the training process, with a little promotion of performance with certain classifiers. 3.2 Experiments e compared the performance of machine learning methods applied to the problem of automatic facial expression recognition, including supervised subspace analysis such as FDA and MDA, unsupervised subspace analysis such as PCA and ICA, SVM classifier and their nonlinear versions. e choose the polynomial kernel in KPCA, KFDA and RBF kernel in SVM. In the reduced space, recognition was performed based on the earest eighbor Classifier except in SVM approach. Generalization performance was tested using the leave-one-subject-out cross-validation. Since FDA and SVM make binary decisions, the one-against-the-rest scheme was adopted. In other words, in each round of training, we computed seven projection matrices in subspace-based approach, and seven SVM classifiers in SVM-based approach. e first compare the performance of FDA, KFDA, linear SVM and SVM with RBF kernels under different feature selection schemes(fig.2). here are 3500 possible features in the grayscale images. e can see that with PCA feature selection or no feature selection, the performances for each of the four methods are the same. ith SM feature selection, the performances promote a little in linear SVM but drop in FDA.

7 Recognition Feature selection Rate (%) one PCA SM FDA KFDA SVM (linear) SVM (RBF) Fig. 2. Leave-one-out generalization performance of FDA, KFDA, linear SVM and SVM with RBF kernels(70*50 pixels images). hey are compared with no feature selection, with feature selection by PCA, and with feature selection by SM e then compare the leave-one-out generalization performance of PCA, FDA, MDA, ICA, SVM, KPCA, and KFDA on facial expression recognition(fig. 3). he best performance was gained with linear SVM and 1900 SM features. e obtained the following conclusions from the experiments: 1) Supervised vs. unsupervised learning. Supervised learning methods showed a better performance than unsupervised ones, such as SVM and FDA outperformed PCA and ICA, and so did their nonlinear versions. he class labels of training samples were utilized in supervised learning, thus facilitated the classification. 2) onlinear vs. linear methods. he linear methods outperformed their kernel-based nonlinear versions in our experiments, as illustrated in linear SVM vs. SVM with RBF kernels, PCA vs. KPCA, and FDA vs. KFDA with polynomial kernels. 3) Binary decision vs. multiple decision. he multiple decision classifiers MDA and ICA yielded the recognition rate of 69.0% and 63.0%, respectively. Overall, the binary decision classifiers outperformed multiple decision classifiers. 4 Conclusions e generally make the comprehensive comparison and discussion on facial expression classification by using the holistic spatial analysis methods, including supervised and unsupervised subspace analysis, SVM and their nonlinear versions. Compared with model-based and feature-based approach to facial expression recognition, image-based holistic spatial analysis is more adaptive to different recognition task. Features relevant to expression classification or the most discriminant subspaces are learned directly from training images. e proposed a novel feature selection method named eighted Saliency Maps(SM). Compared with other feature selection schemes, as Adaboost randomly select features by testing their exclusive performance on a weak classifier iteratively and PCA tries to solve an eigenvalue problem to produce an orthonormal set for projection, SM has the advantage of being simple, fast and flexible. he features are extracted with supervised learning, which makes them different and superior to

8 Method Recognition Rate (%) SVM 91.4 (linear) FDA 89.5 KFDA 87.7 SVM 85.7 (RBF) PCA 85.0 KPCA 81.9 MDA 69.0 ICA 63.0 Fig. 3. Leave-one-out generalization performance of linear SVM, FDA, KFDA, SVM with RBF kernels, PCA, KPCA, MDA and ICA (70*50 pixels images) those predefined features in feature-based approach. References 1. M.S.Bartlett: Face Image Analysis by Unsupervised Learning. Boston: Kluwer Academic Publishers M.S.Bartlett, J.R.Movellan and.j.sejnowski: Face Recognition by Independent Component Analysis. IEEE ransaction on eural etworks. 13(2002): P..Belhumeur, J.P.Hespanha and D.J.Kriegman: Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE ransaction on Pattern Analysis and Machine Intelligence. 19(1997): M..Dailey and G..Cottrell. PCA = Gabor for Expression Recognition. UCSD Computer Science and Engineering echnical Report CS-629, October G.Donato, M.S.Bartlett et al.: Classifying Facial Actions. IEEE ransaction on Pattern Analysis and Machine Intelligence. 21(1999): R.O.Duda, P.E.Hart and D.G.Stork: Pattern Classification. Second Edition. ew York: iley, B.Fasel and J.Luettin: Automatic Facial Expression Analysis: A Survey. Pattern Recognition. 36(2003): G.Littlewort, M.S.Bartlett,I.Fasel,J.Susskind and J.Movellan: Dynamics of Facial Expression Extracted Automatically from Video. In IEEE Conference on Computer Vision and Pattern Recognition. orkshop on Face Processing in Video M.J.Lyons, J.Budynek and S.Akamatsu: Automatic Classification of Single Facial Images. IEEE ransaction on Pattern Analysis and Machine Intelligence. 21(1999): B. Scholkopf, A.Smola, and K.R.Muller: onlinear Component Analysis as A Kernel Eigenvalue Problem. eural Computation, 10(1998): Ya Chang, Chang Hu, and Matthew urk. Probabilistic Expression Analysis on Manifolds. Proc. IEEE Conference on Computer Vision and Pattern Recognition. 2(2004):