A comparison of alternative classifiers for detecting occurrence and intensity in spontaneous facial expression of infants with their mothers

Transcription

1 A comparison of alternative classifiers for detecting occurrence and intensity in spontaneous facial expression of infants with their mothers Nazanin Zaker 1, Mohammad H.Mahoor 1, Whitney I. Mattson 2, Daniel S. Messinger 2, and Jeffrey F. Cohn 3 1 Department of Electrical and Computer Engineering, University of Denver 2 Department of Psychology, University of Miami 3 Department of Psychology, University of Pittsburgh and Carnegie Mellon University s:( Nazanin.ZakerHabibabadi, mmahoor)@du.edu, (dmessinger, w.mattson)@miami.edu, jeffcohn@pitt.edu Abstract To model the dynamics of social interaction, it is necessary both to detect specific Action Units (AUs) and variation in their intensity and coordination over time. An automated method that performs well when detecting occurrence may or may not perform well for intensity measurements. We compared two dimensionality reduction approaches Principal Components Analysis with Large Margin Nearest Neighbor (PCA+LMNN) and Laplacian Eigenmap and two classifiers, SVM and K-Nearest Neighbor. Twelve infants were video-recorded during face-to-face interactions with their mothers. AUs related to positive and negative affect were manually coded from the video by certified FACS coders. Facial features were tracked using Active Appearance Models (AAM) and registered to a canonical view before extracting Histogram of Oriented Gradients (HOG) features. All possible combinations of dimensionality reduction approaches and classifiers were tested using a leave-onesubject-out cross-validation. For detecting consistency (i.e. reliability as measured by ICC), PCA+LMNN and SVM classifiers gave best results. Keywords-Facial Expressions; Action Unit; Histogram of Oriented Gradients, Laplacian Eigenmap, Structural SVM. 1. INTRODUCTION Face-to-face communication has attracted much attention in behavioral science and developmental psychology, and facial expressions are an important channel for nonverbal facial communication [1]. Efficient measurement of emotional responses via facial expressions is necessary to understand the functionality of face-to-face communication. The Facial Action Coding System (FACS) is the most comprehensive method to annotate facial expression [2]. FACS and BabyFACS [3] (the application of FACS to infants) provide a description of all possible visually detectable facial variations in terms of different Action Units (AUs) For example, AU 12 (lip corner puller) codes contraction of orbicularis oculi, AU 6 (cheek raiser) codes oblique action of zygomaticus major, and AU 20 (lip stretcher) codes action of risorius. Psychological studies have employed manual coding of facial expressions for many applications, such as the evaluation of the temporal structure of face-to-face communication between mothers and infants [4] and individual differences in facial expression [5]. Because manual coding is labor-intensive and difficult to standardize within and between research groups, automated computer systems that detect AUs and measure their intensity would contribute importantly to behavioral science. Some success has been achieved in automatic measurement of facial expression and facial action units under controlled conditions [6, 7, and 8]; however, automated measurement is still quite limited for real-world applications. Research psychologists are interested in analyzing the time-varying interactive dynamics of emotional expressions between partners and precise measurement of expression intensity over time. In order to measure the intensity of facial AUs, six intensity levels from absent to maximal appearance are usually considered. The intensities are denoted with labels 0 to 5 where label 0 shows the absent of an AU and labels 1-5 correspond to five intensity levels defined by FACS. Many works have been done on automatic action unit detection and measurement [6, 7, 9, 10] with concentration on posed facial expression data (Cohn-Kanade database [22]). Less work has been done on non-posed expressions [11, 12]. In automatic facial expressions measurement, many challenges and difficulties arise, such as head motion and individual differences in face shape and appearance. In addition, AU intensity and timing differ across applications, especially between posed and non-posed, facial actions [11]. Since the non-posed expressions are more representative of facial behaviors in real life, they have gained more attention very recently [11]. In addition to the difficulties mentioned above, unique challenges are considered in measuring the intensities of AUs for infants. With few exceptions, almost all research in automated facial expression analysis has been limited to adult participants, most of whom are young. Faces change over the lifespan. Infant faces in particular have different shape and proportions than adults, they have less facial texture, and their head pose, movement, and expressive

2 behaviors differ markedly. Sudden and rapid head rotation is common, some unique facial actions occur (See Baby FACS), and the likelihood of particular AU combinations differs (e.g., less AU 12+15). Diversity in participant characteristics such as these is critical to the development of automated facial expression analysis. This characteristic of non-posed expressions leads to the demand for more sophisticated automated systems being able to measure the intensity of action units more accurately. Many methods have been devised to detect the AUs and their intensity variation. Efforts have identified promising features and classifiers. Leading features include HOG [13, 14], and Local Binary Pattern (LBP) Histogram are used [14]. Both appear relatively robust to the small to moderate registration error. Classifiers include Support Vector Machines (SVMs) and Hidden Markov Models (HMMs). HOG features are considered a suitable descriptor relevant to facial expression variations which provide a good performance in recognition of action units. In a recent study, HOG features and SVM classifiers were used for emotion recognition, which showed better performance than LBP features [14]. After extracting the features, facial expression classifiers are employed to recognize the expressions. In recent studies, SVM classifiers are used as a baseline which can perform well on classifying facial expressions [11, 12]. In summary, there is a more demand for automatic methods to reliably recognize non-posed facial actions in less constrained contexts, which is more common in real life [9, 10]. In the framework presented in this paper, we employed different computer vision algorithms for facial expression recognition and compared them in order to investigate the one with the best performance. This framework is applied on an infants face dataset, a population that is nearly ignored in previous researches. Based on our recent study [16] and a recent work [14], the HOG descriptor performs better than LBPH in representing facial expressions. In this work we applied the combination of PCA+LMNN (Large Margin Nearest Neighbor [16]). We compared this combination (PCA+LMNN) to a nonlinear dimensionality reduction technique (i.e. Laplacian Eigenmap [11]), in order to compare the performance of these algorithms in the area of facial expression recognition. The Principal Component Analysis procedure is employed to remove the redundant information and decrease the dimensionality of feature space for further computations. Next, the LMNN method is applied on the data to rearrange data points in the feature space based on their labels. Finally the K-Nearest Neighbor and the Support Vector Machine classifiers are applied to measure the intensity of action units [16]. The rest of this paper is organized as follows. In Section 2, we present a framework to recognize and measure the nonposed facial expressions for 6-month old infants influenced by their parents/caregivers to express different expressions. The infants show non-posed expressions with head movement. In this framework, different classifiers are tested and compared. Moreover, in Section 3 which is the Experiment Section, the results of these classifiers are also compared with the results of Laplacian Eigenmaps and SVM. Conclusion can be found in Section PROPOSED METHOD 2.1. System Overview The overall system model for detecting and measuring the intensity of Action Units (AUs) consists of four major phases, i.e., image registration, facial feature extraction, dimensionality reduction and feature projection and, intensitybased classification. Fig. 1 outlines the main parts of the proposed framework. According to Fig. 1, first all images are registered to a common view, then the HOG facial features are extracted. The process follows by applying PCA on the extracted features to decrease their dimensionality. Then, the LMNN method is used to project data to a new feature space so that the classification performance will be improved. Different classifiers are tested and compared. KNN, which is regarded as the base classier for LMNN is tested. In addition to that, the results of SVM classifiers are compared. Experimental results uphold that the last classifier (SVM) lead to better classification rate although the LMNN method is not originally designed for it Image Registration In the first phase, which is image registration, the facial landmark points are tracked using Active Shape Model (ASM) [21]. Having the landmark points of all images, the common head position is easily computed. Since changes in pose and head movement are potential confounds, images must be registered to a common frame. This process is done by Procrustes analysis, which is a statistical shape analysis, and performs a linear transformation to minimize the measure of shape differences called Procrustes distance. Fig. 1 better demonstrates this procedure Facial Features Facial feature extraction comprises of facial landmark points tracking using AAM [21, 22], and extracting HOG descriptors. The HOG features code local spatial patterns and shapes from the distribution of intensity gradients and edge orientation. To obtain HOG features, the image is divided into small blocks. A histogram of gradient directions or edge orientations is computed for the pixels in each block. Then, these histograms are combined to represent the HOG features for the image. In order to contrast-normalize the local histograms, a larger block of the image is considered to calculate the measure of intensity. Then, the calculated measure is employed to normalize all smaller blocks within the large one. The mentioned normalization leads to a better robustness against illumination changes. [14]. To implement HOG descriptors, the image is first segmented into 35 small blocks based on the landmark points positions as shown in Fig. 2 (a). Since the coordinate

3 (a) (b) Figure 2. (a) The image is divided into 30 small blocks. (b) The size of the blocks changes when expression occurs. Figure 1. The overview of the system. of the blocks are based on positions of the landmarks, the size of the blocks may vary when an expression occurs, which is shown in Fig. 2 (b). As a result, a specific region of the face is always located in a specific block independent of different expressions and face formations. As a consequence, features are better categorized and covered. Then, for each block, the histogram of gradient directions (edge orientations) is computed. The concatenation of these histograms results in HOG descriptors, which are mapped onto a 64-bin histogram to decrease the size of feature vector. The full image is represented by a feature vector consisting of 1920 features. After extracting HOG features, the Principal Component Analysis (PCA) procedure is applied to decrease the dimensionality of feature space. PCA is a mathematical method that transforms a number of correlated variables into a (possibly) smaller set of uncorrelated variables, which are called the principal components. The number of principal components that are chosen is based on the percentage of the covered cumulative energy. The threshold for the covered accumulative energy is considered as 98% of the total energy. After applying PCA, the high dimensional feature vector is replaced by the lower dimension one from PCA which has enough information to describe the actual data and fasten the classification procedure Feature Projection Large margin nearest neighbor (LMNN) classifier is a newly proposed method by Weinberger et al [15]. It learns Mahalanobis distance metric for K-Nearest Neighbor (KNN) classification. The KNN rule is an effective and plain computer vision method for multi-way classification and its performance extremely depends on the distance metric employed. In the common KNN rule, Euclidian distance is applied to compute the distance between sample points in the feature space. However, without loss of generality the Mahalanobis metric can be considered as a linear transformation applied on all the points of input feature space followed by Euclidean based KNN classifier. The goal of learning a new distance metric is to push away impostors (points from different classes) out of the local margin around the test sample, and to attract target points (neighbors from the same class) by shrinking and penalizing their distance to the test sample. Thereby, the KNN classification accuracy improves. The learning procedure of Mahalanobis metric includes semi-definite programming for optimizing a convex problem. Assume a set of feature vectors of dimension d, x i R d 1,, along with their labels y i in addition to the target neighbors for each point. The parameter L is used to learn a Mahalanobis distance which improves the accuracy of KNN., 2 (1) To minimize the distance to target points, the variable 0, 1 is used to identify whether xj is a target neighbor of xi. Thus, L should be learnt to minimize the following objective function: 1 2 (2) To attract target neighbors comparing to impostor neighbors, say x l, the following function is suggested to minimize the sum of standard hinge loss over x i, x j and x l.

4 ψ 2 (L) = (3) In Equation 2, if, 1,, the hinge loss will be zero which means there is a margin of 1 between target and impostor neighbors. equals to, where is 1 if and only if the argument is true. The function implies that maximum of {f, 0}. The total cost function is written by where 0 and is a constant. This minimization of the cost function can be formulated and solved as a convex SDP since M = L L 0. More details on solving the objective function are provided in [15]. To get a better classification rate, we employed the LMNN method on our feature vectors and learnt the L matrix. This L is used to project all data points to a new feature space. With this projection, the data points are rearranged so that the performance of the classifier in the next phase of the system will be improved. In our framework, we used LMNN for either the KNN classifier or the SVM. According to experiments, applying LMNN leads to better performance for both classifiers Classifiers In order to test and compare the performance of the system with different classifiers, we used K-Nearest Neighbor and SVM classifiers. After the projection of PCAresulted features into action unit sub-spaces, the problem is to measure the intensity of action units described in six levels from these features. This is a multi-label classification problem and we employ KNN and SVM classifiers to solve this problem. In the following, brief explanation for each one is provided K- Nearest Neighbor (KNN) Classifier KNN is a simple pattern recognition rule to classify a new point among a number of points with assigned labels. KNN classifies the label of the new point based on a selected number of its nearest neighbors. The neighbor points are defined according totheir distance to the new point. There are different distance metrics to be considered. The most common one is Euclidian distance that is applied in many KNN classifiers. We used and compared the performance of KNN classifier with other ones in our experiments Support Vector Machine (SVM) To measure the intensity of AUs, we used SVM classifiers that are used in the field of pattern recognition and machine learning, and shown to perform well in facial expression and facial action unit recognition [11, 17]. The technical details for SVM classifier can be found in [18]. Overall, SVM utilizes kernel functions that map not linearly separable data to a new feature space with higher dimensionality, where the linear methods would be applicable. There are different types of kernel functions that provide this mapping approach such as polynomial and Radial Bases Functions (RBF) where the RBF kernel showed better performance than others in our experiments. Since our goal is to classify frames into six classes of AU intensity, we applied the one-against-one approach [19] to turn the binary SVM classifiers into a multi-label classifier. For our experiments, the libsvm library is used [20]. Also, the ground truth for this system is provided by an expert FACS coder who manually labeled the AU intensities frame by frame in a 0-5 scale, for each AU. 3. EXPERIMENTAL RESULTS Our experiments are performed on the parents-infants dataset. In this study, videos of facial expressions of twelve 6-month old infants are considered. The videos are recorded at 30 frames per second during the face to face interaction of the mother and infant. The goal is to recognize and measure the intensity of infants facial expressions. There are 163,380 video frames for 12 subjects that had manual coding for a total of minutes frames are tracked by the AAM and used in this paper. There were 1,163 discriminable events of AUs within manual coding amongst all of the AUs. There were 531 instances of AU6, 398 instances of AU12, and 234 instances of AU20. An instance is defined as a coding of no AU to any intensity 1 or greater. In order to provide the ground truth for training and testing of proposed system, an expert FACS/BabyFACS coder manually labeled all frames based on the intensity of AU6, AU12 and AU20 occurring in all frames. As mentioned, Class 0 shows the absent of an AU and classes 1-5 correspond to five intensity levels of FACS. First, the faces in each frame are tracked by the AAM and registered to a canonical view using the Procrustes analysis. The registered faces are cropped and saved in the dataset. Then, the HOG features are extracted from each image. The images are segmented into 30 small blocks and the HOG computed for each block. All the histograms mapped onto a 64-bin histogram, and concatenated to build a feature vector of size In the next step, PCA which is an unsupervised algorithm for dimensionality reduction is used to replace the high dimension feature vector with a small one of size 218 elements. For each AU (i.e. AU6, AU12 and AU20), a mapping function is learnt by using the supervised method LMNN, to project features to a new feature space which is more correspondent to that AU. In our previous work [24], we tuned and aligned the manifold for each subject and used them for dimensionality reduction and AU classification using support vector machines. Whereas, in this study, we learned one generic manifold for all subjects and used for data dimensionality reduction and then AU classification. Then, the resulted

5 TABLE I. THE ICC VALUES FOR 12 SUBJECTS, USING DIFFERENT DIMENSIONALITY REDUCTION METHODS (I.E. PCA+LMNN, LAPLACIANEIGENMAP), AND DIFFERENT CLASSIFIERS (I.E. KNN AND SVM). THE ICC VALUES ARE REPORTED FOR 3 AUS (I.E. AU6, AU12 AND AU20). AU6 AU12 AU20 PCA+LMNN LE PCA+LMNN LE PCA+LMNN LE KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM feature vectors are used to train the KNN and SVM classifiers. In order to measure the recognition power of the aforementioned methods based on predicted values and manually FACS coding, we used Intra-Class Correlation (ICC) values. ICC value is in the range 0 to 1, and is regarded as a good measure of correlation and conformity when data sets with multiple labels are considered [20]. ICC is used to quantify the degree to which related individuals resemble each other. When n targets of data are rated by k judges (here k=2, and n=6) the ICC can be defined as in Equation 4. 1 (4) BMS is defined as between-targets mean squares, and EMS is regarded as the residual mean squares defined by Analysis Of Variance (ANOVA). In Table I, the ICC values for AU6, AU12 and AU20 using different dimensionality reduction methods and classifiers are shown. Each row corresponds to one subject, and three main columns are defined for three AUs. For each AU, two dimensionality reduction methods (i.e. PCA+LMNN, and Laplacian Eigenmap) are tested. Then, two classifies (i.e. KNN, and SVM) are applied on the data resulted from each dimensionality reduction method. By comparing different methods for measuring the intensity of AUs, not only the performance of different methods can be tested, but also we can evaluate our dataset. As an example, consider the ICC values for AU12 in subject 4. All the methods returned 0 as an output, which shows there are not enough samples for subject 4 in which AU12 occurred (only 12 samples in subject 4 with intensity equal to 1 comparing to the overall samples in this subject). As a consequence, stronger classifiers cannot also get the higher ICC values for this subject. The same situation happened for subject 6 in AU20. To better compare the performance of different methods, we have provided Table II in which the overall ICC values are measured and shown. According to the Table, each method has some pros and cons, however, the combination of PCA+LMNN as dimensionality reduction method and SVM as the classifier lead to higher ICC values. TABLE II. OVERALL ICC VALUES FOR ALL SUBJECTS. PCA+LMNN Laplacian Eigenmap AU 6 AU 12 AU 20 KNN SVM KNN SVM In Table III, the number of occurrences of each AU per subject is provided. As the table shows, for some subjects the number of AU occurrences is very low (e.g. only 12 instances for AU12 for subject 4) and this makes the interpretation of the reliability difficult due to lack of sufficient samples. TABLE III. NUMBER OF OCCURRENCES (INTENSITY GREATER THAN 0) OF EACH AU FOR 12 SUBJECTS. THE LAST COLUMN INDICATES THE TOTAL NUMBER OF FRAMES FOR EACH SUBJECT. AU6 AU12 AU20 # of Frames Total CONCLUSION AND FUTURE WORK In this paper, we studied all combinations of two classifiers and two dimensionality reduction methods to

6 automatically measure the intensity of spontaneous facial expressions of infants with their parents. The videos of faceto-face interaction of twelve infants with their mothers are recorded, and specific Action Units related to positive and negative expressions are manually coded by expert FACS coders. AAM used to track and register facial features to a canonical view. The HOG descriptors are used to code local spatial patterns and shapes from the distribution of intensity gradients and edge orientation. To reduce the dimensionality of HOG features, two approaches are tested and compared which are Principal Components Analysis with Large Margin Nearest Neighbor (PCA+LMNN), and Laplacian Eigenmap. Also, two different classifiers compared, K-Nearest Neighbor and SVM. The KNN classifier is regarded to be more compatible with LMNN method and SVM is known as a powerful classifier for facial expression recognition. Four different combinations of dimensionality reduction and classifiers were compared in order to find the best automated system. Regarding the consistency measure (i.e. ICC value), PCA+LMNN and SVM classifiers gave best results. In order to improve the reliability of our system, in the future, we will study and consider the effect of head pose in measuring the intensity of AUs and this may improve the low reliability of AU12 measurement. ACKNOWLEGMENT This research was supported by the grant BCS from the National Science Foundation. REFERENCES [1] I. Cohen, N. Sebe, L. Chen, A. Garg, and T. S. Huang, Facial expression recognition from video sequences: Temporal and static modelling, Computer Vision and Image Understanding, 2003, 91, pp [2] P. Ekman, W. V. Friesen, and J. C. Hager. Facial Action Coding System, The Manual on CD ROM, [3] H. Oster, Baby FACS: Facial action coding system for infants and young children (Unpublished monograph and coding manual), New York, NY: New York University, [4] K. Kaye, & A. Fogel, The temporal structure of face-to-face communication between mothers and infants. evelopmental Psychology, 1980, vol. 16, pp [5] J. F. Cohn, K. Schmidt, R. Gross, and P. Ekman, "Individual differences in facial expression: Stability over time, relation to selfreported emotion, and ability to inform person identification," Proceedings of the International Conference on Multimodal User Interfaces, October, [6] B. Fasel and J. Luettin, Automatic Facial Expression Analysis: A Survey, Pattern Recognition, 2003, pp [7] M. Bartlett, G. Littlewort, I. Fasel, J. Chenu, and J. Movellan, Fully Automatic Facial Action Recognition in Spontaneous Behavior, Proc. Seventh IEEE Int l Conf. Automatic Face and Gesture Recognition, 2006, pp [8] Z. Yunfeng, F. De la Torre, J.F. Cohn, Z. Yu-Jin, Dynamic Cascades with Bidirectional Bootstrapping for Action Unit Detection in Spontaneous Facial Behavior, IEEE Transactions on Affective Computing, 2011, pp [9] Y. Tian, J.F. Cohn, and T. Kanade, Facial Expression Analysis, Handbook of Face Recognition, Springer, [10] M. Pantic and L. Rothkrantz, Facial Action Recognition for Facial Expression Analysis from Static Face Images, IEEE Trans. Systems, Man, and Cybernetics, 2004, vol. 34, no. 3, pp [11] M. H. Mahoor, S. Cadavid, D. S. Messinger, and J. F. Cohn, "A Framework for Automated Measurement of the Intensity of Non-Posed Facial Action Units", 2nd IEEE Workshop on CVPR for Human communicative Behavior analysis (CVPR4HB), Miami Beach, June 25, [12] D. S. Messinger, M. Mahoor, S. Chow, J. F. Cohn, Automated measurement of facial expression in infant mother interaction: A pilot study. Infancy pp [13] H. Yuxiao, Z. Zhihong, Y. Lijun, W. Xiaozhou, Z. Xi, T.S. Huang, Multi-view facial expression recognition, IEEE International Conference on Automatic Face & Gesture Recognition, 2008, pp. 1-6, [14] M. Dahmane, J. Meunier, Emotion recognition using dynamic gridbased HOG features, Automatic Face & Gesture Recognition and Workshops, 2011, pp [15] N. Zaker, M. H. Mahoor, W. I. Mattson, D. S. Messinger, J. F. Cohn, Intensity Measurement of Spontaneous Facial Actions: Evaluation of Different Image Representations, In the Proceedings of the IEEE International Conference on Development and Learning and Epigenetic Robotics, November [16] K. Q. Weinberger, J. Blitzer, and L. K. Saul, Distance Metric Learning for Large Margin Nearest Neighbor Classification Adv. in Neural Inf. Proc. Sys. (NIPS), [17] S. Lucey, A. B. Ashraf, and J. Cohn. Investigating spontaneous facial action recognition through aam representations of the face, In K. Kurihara, editor, Face Recognition Book. Pro Literatur Verlag, Mammendorf, Germany, April [18] V. Vapnik, S. E. Golowich, and A. J. Smola. Support vector method for function approximation, regression estimation and signal processing. In M. Mozer, M. I. Jordan, and T. Petsche, editors, Proceedings of the 1996 Neural Information Processing Systems Conference, December 2 5, 1996, Dever, CO, USA, pp , [19] C. W. Hsu, C. C. Chang, and C.-J. Lin. A practical guide to support vector classification. Technical report, Department of Computer Science, National Taiwan University, [20] C. C. Chang and C. J. Lin. Libsvm a library for support vector machines. cjlin/libsvm/. [21] T. Cootes, D. Cooper, C. Taylor, and J. Graham. Active shape models - their training and application. Computer Vision and Image Understanding 1995, pp [22] I. Matthews and S. Baker. Active appearance models revisited. International Journal of Computer Vision, 2004, pp [23] T. Kanade, J. Cohn, and Y. L. Tian. Comprehensive database for facial expression analysis. In Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition (FG 00), 2000, pp [24] D.S. Messinger, D.S., W.I. Mattson, M.H. Mahoor, and J.F. Cohn, The eyes have it: Making positive expressions more positive and negative expressions more negative. Emotion, In Press.