Intensity Rank Estimation of Facial Expressions Based on A Single Image

Transcription

1 2013 IEEE International Conference on Systems, Man, and Cybernetics Intensity Rank Estimation of Facial Expressions Based on A Single Image Kuang-Yu Chang ;, Chu-Song Chen : and Yi-Ping Hung ; Institute of Information Science, Academia Sinica, Taipei, Taiwan : Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan ; Dept. of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan song@iis.sinica.edu.tw Abstract In this paper, we propose a framework that estimates the discrete intensity rank of a facial expression based on a single image. For most people, judging whether an expression is more intense than others is easier than determining its realvalued intensity degree, and hence the relative order of two expressions is more distinguishable than the exact difference between them. We utilize the relative order to construct an image-based ranking approach for inferring the discrete ranks. The challenge of image-based approaches is to conduct a representation for subtle expression changes. We employ an efficient descriptor, scattering transform, which is translation invariant and can linearize deformations. This scattering representation recovers the lost high frequencies and retains discrimination under invariant property. Our experimental results demonstrate that the proposed framework with scattering transform outperforms other compared feature descriptors and algorithms. Index Terms expression intensity estimation; ordinal ranking; scattering transform; (a) Anger (b) Disgust (c) Fear (d) Happiness (e) Sadness (f) Surprise Fig. 1: Illustration of the six basic categories of facial expressions. Fig. 2: Image sequence of surprise expression from neutral to apex. I. INTRODUCTION Facial expression reflects the affection of an individual, which is essential to nonverbal communications. How to recognize facial expressions automatically is therefore important to human-computer interaction and affective computing systems, which has been a significant research topic over the past few decades. Most of the automatic facial-expression recognition (AFR) researches [1 5] focus on recognizing the six universal expressions, categorized as anger, disgust, happiness, fear, surprise and sadness, as illustrated in Fig. 1. Although the categorical information is fundamental to AFR, it is sometimes insufficient for real-world applications when we need to know further how strong an expression is. For example, Fig. 2 illustrates an image sequence of the surprise expression from neutral to apex. Although these expressions belong to the same category, their degree of strongness (or strength) are different. On the other hand, the expression intensity information would be critical for choosing an appropriate reaction strategy for human-computer interaction. A. Expression Intensity Estimation In this paper, instead of classifying a face image into one of the facial expression categories, we focus on the problem of estimating the intensity of facial expression. Recently, there has been a growing interest to this problem. In the past, some early studies [6 8] simply extend the outcomes of expressioncategory classification for obtaining the intensity-estimation results. In the training phase, these approaches employed only the category-labeled data (which lack of the intensity labels) for learning a classifier. Then, the expression intensity is directly estimated as a value proportional to the distance between the query example and the classification boundary. For example, Littlewort et al. [6] applied SVM for facial expression recognition, and the distances to the SVM hyperplanes are used to estimate the expression intensities. Chang et al. [7] employed a discriminative manifold learning approach for facial components, and estimated the probability of an expression according to the distances to the manifold. However, such a naive estimation method causes easily inaccurate results because distances to the classification boundary in the feature space do not necessarily reflect how strong an expression is. Later studies [9 14] employed the training data with known intensity labels to conduct more accurate expression-intensity estimation results. The intensity labels can be represented as either continuous values [12, 15] or discrete ranks [9, 11]. Then, learning mechanisms such as nonlinear (ordinal) regression can be applied to learn an estimator from the training data with intensity labels. Since the labels of intensities have been /13 $ IEEE DOI /SMC

2 Fig. 3: Relative order between two intensities of facial expressions. employed, the expression intensity degree can be predicted and validated more accurately. However, these approaches require the employment of an image sequence or multiple images to estimate the facial expression intensities. In this way, an advantage is that motion-based features can be extracted to reflect the temporal changes of facial expressions. Nevertheless, it has the limitation that the estimation can be done only when sufficient images are available in a sequence-based approach. B. Motivation of Our Approach In this paper, we introduce a single-image-based approach for facial expression intensity estimation. Our approach is more flexible since it can infer the intensity degree when only one image is available. In addition, it can also serve as a basic module for a sequence-based approach. To avoid the difficulty of normalizing the intensities for different individuals in the continuous-valued representation (also note that it is easier for human to distinguish the intensity ranks than to infer the exact values), we divide the intensity into three discrete ranks (or levels), low, medium, and high. Such descriptive levels are more intuitive for representation than continuous values. The principle employed to infer the intensity rank is as follows. When representing the intensity labels as ranks, only the relative order between labels are considered. That is, we do not concern what the exact value of the intensity difference is, but care only about which expression is more intense. For example, in Fig. 3, we employ only the relative-order information that right-image expression is more intense than left-image expression. Besides, we adopt a feature descriptor that is translational invariant but stable to local deformation for expression intensity estimation. This descriptor can tolerate local changes caused by inexact facial alignments across individuals without losing discriminating abilities for subtle expression variations. The study of Delannoy et al. [11] would be the most relevant one to ours. In this work, an image-based approach was developed for the three-levels (low, medium, and high) expression intensity estimation. They regard the problem as a multi-class classification problem and employ one-against-all SVMs for training a classifier. However, since a multi-class approach assumes that the intensity levels are independent to each other, the ordinal relationship among labels (i.e., high medium low ) has not been well employed for performance enhancement. In addition, although a facialexpression image can be classified into one of the three Fig. 4: The intensity output of classification and ranking approaches. intensity levels, the action units (AUs) must still be extracted from an image sequence as a feature representation in this approach. We compare the performance of our approach with [11] in the experiments. This paper is organized as follows: In the next section, we describe the ranking framework of expression intensity estimation. In Section III, we review the scattering transform. Experimental results are presented in Section IV. Section V gives the conclusions. II. EXPRESSION INTENSITY RANKING FRAMEWORK In this paper, we treat the single-image-based expression intensity estimation as a ranking problem. Ranking was widely used in information retrieval, where many algorithms have been proposed in the past. As discussed above, multi-class classification approaches overlook the ordinal consistency of the predicted intensities. Besides, regression approaches suffer from the difficulty of gathering the ground-truth values for the continuous intensity labels. Therefore, learning-to-rank (or ordinal regression) is a better choice for discrete intensity-level estimation. Among the learning-to-rank approaches, the thresholds model is a simple and intuitive method for employing the ordinal information, which can produce consistent and nonambiguous ranking results. Shashua and Levin [16] proposed a ranking model that applies the large margin principle with the thresholds model. In this approach, there are a set of parallel hyperplanes that divide the data monotonically. Li and Lin [17] extended the parallel-hyperplanes approach [16] by considering the cost-sensitive property to improve the performance, and reduces ordinal ranking to binary classifications [17]. In this work, we employ the RED-SVM developed in [17] in our intensity-level estimation framework. Given the training examples tx i,y i i 1,..., N u, where x i P R D is the i-th feature vector and y i Pt1,..., Ku is the intensity ranks. How to extract the feature vector x i from a face image will be introduced in Section III. Note that y i is a well-ordered set, i.e., intensity ranks K K , where means larger than and denotes the relative order between different intensities. Details of RED-SVM is presented in Algorithm

3 Algorithm 1 RED-SVM. Input: training examples tx i,y i i 1,..., N u; Output: 1: For each k where 1 k K, divide the original training data into two sets X k X k tpx i,y pkq i q y i ku tpx i,y pkq i q y i ku, (1) where y pkq i 1 if y i k and y pkq i 1 otherwise. 2: For all k, the binary classifier gpx i,kqsignp w, Φpx i q b θ k q, (2) can be obtained by solving min w,θ,b,ξ s.t. }w} 2 }θ}2 λ C Ņ K 1 i1 k1 c pkq i ξ pkq i (3) y pkq i p w, Φpx i q b θ k q 1 ξ pkq i, ξ pkq i 0, for i 1,..., N, and k 1,...K 1, where θ is a set of well-ordered thresholds, i.e., θ 1 θ 2... θ K 1, w and b are the hyperplane parameters in the kernel space, Φpq is a mapping to the kernel space, c pkq i y i k is the absolute cost suggested in [17], C and λ are the weights of loss and regularization, respectively. 3: A ranking rule r constructed from g is used to collect all the relative orders and predict the expression intensity of x: K 1 rpxq 1 k1 vgpx, kq 0w. (4) where vw is 1 if the inner condition is true, and 0 otherwise. 4: return rpxq; An important characteristic of RED-SVM is rankmonotonic, i.e., gpx, 1q gpx, 2q... gpx, K 1q for every x. Note that the rank-monotonic function g can provide consistent ranking results. The consistent property is important to design a ranking rule r that there are no conflicts among the binary classifiers vgpx, kq 0w, 1 k K 1. In summary, RED-SVM constructs parallel hyperplanes to divide the data in a kernel space specified by the implicit mapping Φ. In our work, the radial basis function (RBF) kernels are used. The parameters of C and RBF kernel are selected using 5-fold cross validation in the training dataset, and the default value of λ is 1 in RED-SVM. III. SCATTERING TRANSFORM Facial expression intensity is a subtle variation and thus identifying the facial changes is important. For a video-based approach, the variation of expression intensity can be predicted by adjacent frames, and the divergence among different persons can be mitigated by tracking the facial points. For an image-based approach, it is more critical to have a powerful feature representation because there are no other reference images of the same subject. Besides describing the subtle variation of expressions, a common problem encountered in face-recognition related task is to align the facial images in both the training and testing phases. An intuitive method is to use the location of a set of facial landmark points extracted, such as Active Appearance Models (AAMs) [18]. However, such an alignment requirement would be too demanding in practice because automatically detecting the facial points is still unstable in general situations. A tradeoff is to align only two eyes of face images. Although this setting causes some inexact facial alignments among individuals, our feature representation introduced below can still tolerate the misalignment and achieve better performance. Brunna and Mallat [19, 20] proposed a local descriptor, called scattering transform, which is translation invariant and linearizes deformations. In this paper, we employ the scattering transform for extracting the subtle expression changes and mitigating local changes across different persons. Given an input signal z P R 2 for scattering transform, we construct the directional wavelet filter ψ j,γ by scaling a filter ψ with 2 j and rotating z by an angle γ. ψ j,γ pzq 2 2j ψp2 j R γ zq, (5) where R γ z indicates the rotation of z. Leung et al. [21] have proved that complex Gabor function for ψ is effective for texture analysis. We use complex Gabor function for ψ in our experiments. The resulted wavelet transform of f at a position z is a filter bank defined by f ψ j,r pzq W J f pzq, (6) f φ J pzq j J, γpγ where φ J pzq 2 2J φp2 J zq is a low-pass filter that carries the low frequency information. In this paper, we use Gaussian filter as the low-pass filter φ J pzq. Some descriptors achieve invariance property by averaging, such as Scale-Invariant Feature Transform (SIFT) [22] and Histogram of Oriented Gradients (HOG) [23]. The wavelet coefficient amplitude of f are averaged by φ J pzq in the following f ψ j,r φ J pzq. (7) These averaged wavelet coefficients are almost invariant to translation or deformation. Convoluting with low pass filter can increase the invariant ability, but this sacrifices high frequencies simultaneously. The lost high frequency information in (7) can be restored again by finer scales wavelet coefficients as f ψ j1,r 1 ψ j2,r 2, where j 1 j 2,r 1,r 2 P Γ, (8) where j 2 is the adjacent finer scale of j 1. In order to preserve invariance and keep discriminating information of (8), it is

4 (a) Fig. 5: A scattering transform is a cascade of wavelet decompositions. required to reduce the variability of these coefficients. The high frequencies are averaged with low-pass filtering: f ψ j1,r 1 ψ j2,r 2 φ J pzq. (9) In Fig. 5, the scattering coefficients are computed with a cascade of wavelet decompositions. The input signal can be decomposed into finer scales wavelet coefficients of different orientations. In theory, the high frequencies can be kept restoring and averaging iteratively. In real application, the number of levels can be set to a constant because the distinctiveness decreases with finer scales. The results in [20] suggest that three levels are enough for most cases. In sum, high frequencies appear to be instable to deformation but are discriminative. Conventional feature descriptors may overlook the high frequencies when extracting locally invariant representations. Scattering transform is computed with a cascade of wavelet decompositions and modulus operators, which recovers the lost high frequencies. This transform is also proved to be Lipschitz continuous [19] and the resulted representation guarantees translation and/or rotational invariance. SIFT [22] or Daisy [24] type of descriptors can be approximated by the first layer of the cascade of scattering transform. Because of the above advantages, scattering transform has been applied to audio and image pattern recognition [20, 25, 26] and achieves good results. In addition, recent studies on face recognition in the wild (such as [27]) demonstrated that scattering transform outperforms other features, e.g., Local Binary Pattern, HOG and Gabor wavelet. We adopt the scattering transform as a translation-invariant and discriminative feature representation for single-imagebased expression intensity estimation. According to the suggestion in [28], we divide a face image into several components by applying a mask in Fig. 6(a). There are totally eight components including forehead, middle of eye brow, left eye, right eye, nose, mouth and chin, left cheek, and right cheek as shown in Fig. 6(b). In addition to facial components, holistic faces aligned with two eyes are used for feature extraction as in [29]. We combine both scattering features of facial components and holistic face, and perform PCA on the extracted features to reduce the feature vectors to D-dimensions that preserves 98% energy. For comparison, we also apply two widely used feature descriptors, AAMs [18] and Gabor wavelet in our experiment. (b) Fig. 6: (a) Facial component mask (b) eight facial components used in our method. AAMs describe both shape and appearance variation of facial images, and have been widely used as a feature descriptor in expression recognition [30]. The main limitation of AAMs is the requirement of 68 landmarks on faces. Gabor wavelet is a also a powerful and popular feature descriptor for face recognition. In [31], Bartlett et al. utilized Gabor wavelets for facial expression category recognition and we evaluate its performance for intensity estimation in this work. IV. EXPERIMENTAL RESULTS A. Database Our experiments are conducted on the extended Cohn- Kanade (CK+) dataset [30], which is a large and publicly available dataset for facial expression recognition. CK+ dataset is the extension of original Cohn-Kanade (CK) dataset and contains more subjects, video sequences and expressions. In CK+ dataset, there are 593 image sequences from 123 subjects of seven prototypic emotions, but the dataset provides expression labels for only some portion of data. Like [11], we use the six universal expressions: Anger, Disgust, Fear, Happiness, Sadness, and Surprise for performance comparison. In [11], the authors selected only 68 images from 10 subjects in the CK dataset, and use 10-fold cross validation in their experiment. We report their results directly from [11] in the first row of Table I. Since the subjects chosen in the experiments of [11] would be too few to reflect the performance, we employ more data from CK+ to validate our approach. In our experiments, all the data with labels in CK+ are used, where there are 4,043 images from 91 subjects in 222 image sequences, which is a severer experimental setting than that in [11]. The image sequences are manually divided into K=3 levels associated with the intensity levels from neutral to apex. B. Features and Experiment Setup We evaluate the performance of two widely used feature descriptors, AAMs and Gabor wavelets, in comparison with the scattering transform. In AAMs, each image is aligned with the given 68 labeled landmark points, and both texture and shape information are extracted for facial images. Gabor wavelets are set with 3 scales and 6 orientations. Both the Gabor and scattering transforms align the input images with only two

5 TABLE I: Mean error rate and MAEs of the compared approaches in CK+ database. Feature Type Error on low Error on medium Error on high Mean Error MAEs One-against-all SVMs [11] AUs SVR Scattering Transform Our framework Scattering Transform SVR AAMs Our framework AAMs SVR Gabor Our framework Gabor eyes. The feature vectors of AAMs and Gabor are obtained by preserving 98% energy by PCA, too. We apply 10-fold cross validation to evaluate the performance. In each fold, the training set contains different subjects to the testing set. We apply a 5-fold cross validation procedure only on the training set for parameter selection, and report the average performance of the testing set. We also employ a standard nonlinear regression method, Support Vector Regression (SVR), for comparison in the experiments. The RBF kernel function is adopted and the associated parameters are selected by five-fold cross validation in training dataset. In [11], Delannoy et al. focus on the discrimination of binary classifications and evaluate the area under curve (AUC). In order to comparing with SVR, we replace AUC with mean error rate. We report the mean error rate 1 N test N test i1 and mean absolute error (MAEs) N test i1 Ipy i y i q (10) y i y i {N test (11) for expression intensity estimation, where N test indicates the number of test images, and y i is the predicted intensity level. Because the outputs of SVR are continuous values, the predictions of SVR are rounded when calculating the mean error rate. Mean error rate only focuses on the classification accuracy. For a more accurate comparison, we employ MAEs to measure the absolute difference between prediction and ground truth. MAEs has been used to evaluate expression intensity estimation [9, 32], which measure how close the predictions are to the ground truth. Therefore, MAEs could be a more appropriate measurement for expression intensity estimation. We report both mean error and MAEs in the experimental results. C. Results and Discussion There are two parts of our experimental results. At first, we focus on the comparison between different approaches for estimating discrete intensity ranks. Table I shows the mean error rate and MAEs of the compared approaches under different feature representations. Error on low, medium and high reveal the error rates on different intensity ranks. The error rats on medium are generally higher than the other two intensities, which indicates that medium is more difficult to be separated from the others. The results also demonstrate that SVR and our framework using scattering transform perform better under the measurement of mean error rate than the other compared approaches. It is worth noting that although the mean error rate of SVR and our framework are close, the MAEs of our framework are significantly lower than SVR. It reveals that even when the classification accuracy of SVR and our framework are approximate, the misclassified ranking labels of SVR are farther from the ground-truth than those obtained by our framework. Our approach performs better than conventional regression SVR and the compared approach [11] on both measurements. The proposed ranking framework for expression intensity estimation performs consistently better than SVR under different types of feature representations. The second part is the comparison between feature descriptors. Bartlett et al. [31] utilized Gabor wavelets for expression category recognitions. From our results, the Gabor wavelets perform worse on expression intensity estimation than the others. AAMs is a popular feature extraction approach on face recognition, e.g., facial expression recognition [5] and age estimation [33]. The restrictions is that AAMs require 68 landmarks to align the facial images. In the experiment, we only align two eyes for the scattering transform. In Table I, the results also show that the performance of scattering transform is better than AAMs (68 landmarks) and Gabor wavelets. In brief, scattering transform is a powerful descriptor and can achieve better performance for expression intensity estimation based on face images. V. CONCLUSION In this paper, we propose a ranking framework to estimate discrete intensity ranks based on a single image. Discrete intensity ranks are closer to human s experience than continuous intensity values. We employ a parallel-hyperplanes ranker in our approach for ordinal regression. Although image-based approaches ignore temporal clues, it can be more flexibility employed than video-based approaches. We have shown that the feature representation of scattering transform performs

6 well. Experimental studies reveal that our framework outperforms the other competitive methods. ACKNOWLEDGMENT This work was supported in part by the National Science Council, Taiwan, under the grants NSC E REFERENCES [1] Z. Zeng, J. Tu, B. Pianfetti, and T. Huang, Audio visual affective expression recognition through multistream fused hmm, IEEE Transactions on Multimedia, vol. 10, no. 4, pp , [2] P. Yang, Q. Liu, X. Cui, and D. Metaxas, Facial expression recognition based on dynamic binary patterns, in IEEE Conference on Computer Vision and Pattern Recognition, [3] Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, A survey of affect recognition methods: Audio, visual, and spontaneous expressions, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp , [4] P. Yang, Q. Liu, and D. N. Metaxas, Exploring facial expressions with compositional features, in IEEE Conference on Computer Vision and Pattern Recognition, [5] T. Pfister, X. Li, G. Zhao, and M. Pietikainen, Recognising spontaneous facial micro-expressions, in IEEE International Conference on Computer Vision, [6] G. Littlewort, M. S. Bartlett, I. Fasel, J. Susskind, and J. Movellan, Dynamics of facial expression extracted automatically from video, Image and Vision Computing, vol. 24, no. 6, pp , [7] W.-Y. Chang, C.-S. Chen, and Y.-P. Hung, Analyzing facial expression by fusing manifolds, in Asian Conference on Computer Vision, [8] S. Koelstra and M. Pantic, Non-rigid registration using freeform deformations for recognition of facial actions and their temporal dynamics, in IEEE International Conference on Automatic Face & Gesture Recognition, [9] M. Kim and V. Pavlovic, Structured output ordinal regression for dynamic facial emotion intensity prediction, in European Conference on Computer Vision, [10] M. F. Valstar and M. Pantic, Fully automatic recognition of the temporal phases of facial actions, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 42, no. 1, pp , [11] J. Delannoy and J. McDonald, Automatic estimation of the dynamics of facial expression using a three-level model of intensity, in IEEE International Conference on Automatic Face & Gesture Recognition, [12] C. Liao, H. Chuang, and S. Lai, Learning expression kernels for facial expression intensity estimation, in IEEE International Conference on Acoustics, Speech, and Signal Processing, [13] A. Dhall and R. Goecke, Group expression intensity estimation in videos via gaussian processes, in International Conference on Pattern Recognition, [14] A. Savran, B. Sankur, and M. Taha Bilge, Regression-based intensity estimation of facial action units, Image and Vision Computing, vol. 30, no. 10, pp , [15] K. Song and S. Chien, Facial expression recognition based on mixture of basic expressions and intensities, in IEEE International Conference on Systems, Man, and Cybernetics, [16] A. Shashua and A. Levin, Ranking with large margin principle: Two approaches, in Advances in Neural Information Processing Systems, [17] L. Li and H.-T. Lin, Ordinal regression by extended binary classification, in Advances in Neural Information Processing Systems, [18] T. Cootes, G. Edwards, and C. Taylor, Active appearance models, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp , [19] S. Mallat, Group invariant scattering, Communications in Pure and Applied Mathematics, vol. 65, no. 10, pp , [20] J. Bruna and S. Mallat, Classification with scattering operators, in IEEE Conference on Computer Vision and Pattern Recognition, [21] T. Leung and J. Malik, Representing and recognizing the visual appearance of materials using three-dimensional textons, International Journal of Computer Vision, vol. 43, no. 1, pp , [22] D. G. Lowe, Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, vol. 60, no. 2, pp , [23] N. Dalal and B. Triggs, Histograms of oriented gradients for human detection, in IEEE Conference on Computer Vision and Pattern Recognition, [24] E. Tola, V. Lepetit, and P. Fua, Daisy: An efficient dense descriptor applied to wide-baseline stereo, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 5, pp , [25] J. Andén and S. Mallat, Multiscale scattering for audio classification, in The International Society for Music Information Retrieval, [26] L. Sifre and S. Mallat, Combined scattering for rotation invariant texture analysis, in European Symposium on Artificial Neural Networks, [27] K.-Y. Chang, C.-F. Lin, C.-S. Chen, and Y.-P. Hung, Applying scattering operators for face recognition: A comparative study, in International Conference on Pattern Recognition, [28] N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar, Attribute and simile classifiers for face verification, in IEEE Conference on Computer Vision and Pattern Recognition, [29] M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, Recognizing facial expression: machine learning and application to spontaneous behavior, in IEEE Conference on Computer Vision and Pattern Recognition, [30] P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression, in IEEE Conference on Computer Vision and Pattern Recognition, [31] M. S. Bartlett, J. R. Movellan, G. Littlewort, B. Braathen, M. G. Frank, and T. J. Sejnowski, Towards automatic recognition of spontaneous facial actions, What the Face Reveals, pp , [32] O. Rudovic, V. Pavlovic, and M. Pantic, Multi-output laplacian dynamic ordinal regression for facial expression recognition and intensity estimation, in IEEE Conference on Computer Vision and Pattern Recognition, [33] A. Lanitis, C. Draganova, and C. Christodoulou, Comparing different classifiers for automatic age estimation, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 34, no. 1, pp ,