Recognizing Facial Expressions

Transcription

1 Recognizing Facial Expressions B.S. de Graaf June 12, 2007 Abstract Truly intelligent Human-computer Interaction is an important challenge, and presently still far beyond our grasp. It remains an active and open area of research, as humans have many ways to express themselves beyond the verbal domain. Whereas computers have improved during the last few decades in comprehending its users, the communication is seriously impaired by the lack of being able to intercept and understand implicit messages. This study focuses on recognizing person-independent facial expressions of high resolution frontal images of faces. The Eigenfaces technique was applied to discover the relevant regions of a face when a person changes from one emotion to another. The eyebrows, mouth and eyes are the regions that deviate most markedly when shifting expressions. Implementing a combination of the well-known mathematical optimization method LLS (Linear least squares) and Hebbian learning proved valuable in matching and fitting the eyebrows and mouths. A template model for each emotion was statistically created, utilizing four different classification techniques to recognize the different emotions. Combining feature selection, Eigenfaces and the several classification methods, it was possible to develop an integrated approach to distinguish six different emotions up to a certain level. The three emotions afraid, neutral and sad had the lowest percentage of correct classification, while the three emotions happy, angry and surprised could be classified quite well. 1 Introduction In the last few decades, various techniques have been developed to detect and/or recognize faces in either images or video. These techniques improved in such a way that it was possible to recognize multiple faces, some even partially occluded, in a complex background [12]. This was considered quite a breakthrough, and the developments of other techniques increased rapidly. On the other hand, there remained a challenging area unexplored. Although computer applications could detect and recognize faces from live video, they lacked the capability of informing us if a certain face was smiling. As there has been a growing interest in improving all aspects of the interaction between computers and humans, recognizing emotions of a user has become increasingly important [6]. Until now, a range of devices have been developed to improve the communication between the user and the computer, from the ancient keyboard to the high-tech webcams and touch screens. To achieve truly intelligent human computer interaction, the computer needs to obtain the ability to communicate naturally with the user, in a similar way that humans communicate with each other. The main difficulty is comprehending the implicit messages humans send to each other. These messages are principally in the form of body language, gestures, voice pitching and facial expressions. As individuals focus more attention on projecting their own facial expressions and perceiving others than they do on the other nonverbal channels [3, 6], the most significant way for humans to express their emotions is through facial expressions. Ekman and Friesen [5] were the first researchers to investigate this field from a psychological view. Their results inspired several other researchers, such as Chen [2], to study this emerging field, creating effective algorithms shown to work. Until now, there have been numerous approaches to this problem, including Hidden Markov Models [3] and a Naïve Bayes classifier [9]. The novelty of this study is the use of the Eigenfaces technique [11] combined with feature selection. Earlier techniques mainly used the whole face to identify emotions, whereas this approach uses only a few features of the face. The Eigenfaces technique calculates the principal areas that change when people shift emotions, and the Linear least squares technique [1] and Hebbian learning [7] are subsequently combined to derive statistics from those regions. As a consequence, this paper investigates the following research question: Can the Eigenfaces technique combined with Linear least squares and Hebbian learning effectively be applied to recognize person-independent facial expressions?

2 B.S. de Graaf Bachelor thesis Various applications exist where it is beneficial for computers to recognize emotions. For example, a computer can develop into a more effective tutor when able to identify the moods of its user. In hospitals, it may prove useful in monitoring the stress level of patients. In addition, computers can help diagnose early psychological disorders by recognizing a person s inability to express certain facial expressions. This article is structured as follows: Section two provides background information of several theories and techniques applied throughout this thesis. A detailed description of the general approach is given in Section three, while Section four reports the experiments and its results. Finally, the conclusions and further recommendations are presented in Section five. 2 Feature extraction This Section offers background information on the various techniques investigated during this thesis. The first Subsection provides information regarding the Eigenfaces technique. The mathematical optimization technique Linear least squares will be examined in the second Subsection, whereas Hebbian learning will be discussed in the third Subsection. 2.1 Eigenfaces The idea of Eigenfaces was introduced by Matthew Turk and Alex Pentland in 1987 [11]. Around that time, many scientists explored the field of face recognition. The Eigenfaces technique those two scientists developed was actually the first facial recognition method that truly worked. Nowadays it is also referred to as the Eigenimages technique, since developers not only apply it in the field of face recognition, but use it as well for handwriting analysis, medical imaging, and other applications. Although this method is sensitive to lighting and angles, it recognizes faces correctly up to 96%, when relying on a database of pictures taken under similar conditions. This technique has several advantages over other techniques, such as efficiency and speed. It is possible to create Eigenfaces of huge databases in a short time. Basically, an Eigenface is some kind of combination of many faces, all situated in one image. Each Eigenface appears different, as each one contains different images and/or another number of images. For example, a picture can exist for 40% out of Eigenface 1, 30% out of Eigenface 2 and 30% out of Eigenface 3. Three Eigenfaces which were created during this thesis are presented in Fig. 1 to give an example. The technique is a form of the well-known Principal Component s Analysis (PCA) [4]. Using a large database of different pictures of frontal faces, Eigenfaces can be extracted by calculating the eigenvectors of the covariance Figure 1: Eigenfaces of the emotions Neutral, Angry and Happy matrix of all faces. Choosing the eigenvectors (principal components) corresponding with the highest eigenvalues, the most relevant Eigenfaces are obtained, containing the largest number of faces. After creating the Eigenfaces, it is possible to compute feature vectors of all faces in the database. The dimension of the feature vector depends on the selected number of Eigenfaces. This process is basically done by calculating the inner product of each image and the Eigenfaces. When aiming to use this technique to recognize a face of a new image, the feature vector of this new image can be calculated in the same manner, subsequently using any classification method to classify the new image. 2.2 Linear least squares Linear least squares is a mathematical optimization method to obtain an affine function that best fits a given set of data. Mathematically, it is applied in this study to find a solution for x in the function: Ax = b. Applying this technique to fit a simple linear line, the matrix A becomes a m-by-n matrix consisting of two columns of the same length. The first column holds all the x-values of the data set, while the second column contains only ones. The vector b contains all the y-values of the data set, and the vector x holds the values for a and b, consequently providing the linear function y = ax + b. In this study, the matrix A always has full column rank, and thus the solution of the system of linear equations is unique and given by x = (A T A) 1 A T b. A 2nd-order polynomial fit can be obtained as well, which mathematically stands for y = ax 2 + bx + c, through a Vandermonde Matrix as matrix A. This matrix basically has a geometric progression in each row. Thus, the difference with the linear case is that the matrix A would contain a third column, containing all x 2 values. Furthermore, the vector x would include the third value c. (v. June 12, 2007, p.2)

3 Bachelor thesis B.S. de Graaf 2.3 Hebbian learning The most famous idea from the scientist Donald O. Hebb, written down in The Organization of Behavior [8], is the following postulate: When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A s efficiency, as one of the cells firing B, is increased. This theory later became known as the eminent Hebbian learning, or the Hebb Rule. In the field of artificial neural networks, this principle can be explained as a technique to determine how to alter the weights between model neurons. The principle is possibly the most basic form of weight selection. In this thesis the Hebb Rule is applied solely to a straightforward neural network, the associative memory, via a linear associator [7]. The main task of this network is to learn Q pairs of prototype input/output vectors: {p 1, t 1 }, {p 2, t 2 },..., {p q, t q }. When the network receives an input p q, it should return its target t q, for q= 1,2,..., Q. Furthermore, when the input is not exactly similar to a previously trained input vector, the output vector will be slightly different than its target. As a consequence of using these target vectors, this is a form of supervised learning. The supervised Hebb Rule is mathematically illustrated (in vector notation) in the following way: W new = W old + t q Pq T. Initializing the weight matrix to zero and applying each of the Q input/output vectors once, the result is: W = T P T. One drawback of the Hebb Rule is that when the input vectors are not orthogonal, the results will contain errors. The pseudoinverse rule is a procedure used to reduce these errors. When applying this procedure, the formula transforms into: W = T P +,where P + is the Moore-Penrose pseudoinverse: P + = (P T P ) 1 P T. 3 Implementation The images used in this thesis were acquired from the Karolinska Hospital in Sweden [10]. They have been doing an extensive study on facial expressions as well, constructing a database of color images along their research, using seventy different models. Of the several emotions they incorporated, six different expressions were selected: (1) afraid, (2) angry, (3) happy, (4) neutral, (5) sad and (6) surprised. The motivation for choosing these particular emotions is the fact that humans are able to recognize these six emotions without any noticeable effort, while perceiving other emotions cause more difficulty [3, 6]. The images of this database are normalized and lined up to the eyebrows and mouths, having a resolution of 762 by 562 pixels. Matlab is the mathematical programming language used for the implementation of the algorithms described in the following Subsections. Subsection one describes the implementation of the Eigenfaces technique. The second Subsection discusses the development of several algorithms for feature selection, while Subsection three briefly explains various classification methods. 3.1 Eigenfaces The Eigenfaces technique was implemented at the beginning of this thesis to generate Eigenfaces of the dataset. At that time, the goal of this process was to eventually use the Eigenfaces to identify emotions. This process consisted of two different steps: preprocessing and feature extraction. P reprocessing The images from the database were colored and contained a large amount of background. When converting color images into matrices, a huge amount of data is acquired, since there are three colors taken into account. Regarding facial expressions, the color of a face is much less significant than other features. Therefore, grayscaling the images appeared practical, as it reduces the amount of data to a third of the original quantity. The large amount of background was useless as well, as this study involves recognizing expressions of a face only. As a consequence, the preprocessing part included grayscaling and cropping all images. After these two processes, it was inevitable that several pictures required stretching as well, as grayscaling influences the contrast of the image. Stretching the image provides a better contrast, which aids the extraction of features. The result of a preprocessed image is illustrated in Fig. 2. Figure 2: Original Image - Grayscaling - Cropping - Stretching F eature extraction After preprocessing the images, the resolution was still relatively high, regarding the Principal Component s Analysis explained in the previous Section. Applying the standard PCA would require extreme CPU and mem- (v. June 12, 2007, p.3)

4 B.S. de Graaf Bachelor thesis ory. Therefore, the prominent Karhunen-Loeve expansion [4] was applied. This technique offers an alternative form of PCA, requiring less memory. The procedure can be described as follows: it computes the outer product of all images, resolving into, when having N pictures converted to vectors, a N-by-N matrix. The relevant eigenvectors can subsequently be determined. To construct the Eigenfaces, it is necessary to carry out the following procedure for each eigenvector: multiply each element with the corresponding image and compute the sum. The result will be an Eigenface. In this thesis, the three eigenvectors corresponding with the highest eigenvalues were selected. As a consequence, three Eigenfaces were generated for each emotion (examples are presented in Section two), followed by the calculation of feature vectors of each face. Plotting these feature vectors in a 3-Dimensional space, it was possible to see if the emotions were clustered, thus being linearly separable. Unfortunately this was not the case, as illustrated in Fig. 3. Figure 4: Standard devations of all Eigenfaces Hebbian learning and Linear least squares. The Eigenface technique again proved very valuable, in view of the fact that Fig 4. could be used to remove some of the irrelevant regions from images while attempting to derive statistics from the important features. Fig 5. illustrates this point clearly. Figure 3: Plot of all feature vectors Although it became clear that the Eigenfaces technique could not be applied solely to recognize emotions in the same manner it is applied to recognize faces, it was very useful for the second approach. This new approach firstly included the decision to determine the regions of the face which change when people shift expressions. Computing the standard deviations of all the constructed Eigenfaces, a new image was produced, showing the deviating regions, as illustrated in Fig. 4. The figure clearly shows that the most important regions within the face are the mouth, the eyes and the eyebrows, which is most likely intuitively clear. Therefore, the second approach would apply feature selection of these main deviating regions of the face, combining Figure 5: Removing irrelevant regions 3.2 Feature selection Eyebrows As the images were lined up to the eyebrows and mouths, the area of the eyebrow was approximately known. As mentioned above, some of the irrelevant regions were removed from the face to reduce possible noise. The main difficulty that appeared concerned the location of the eyebrows. Several eyebrows were positioned very high (v. June 12, 2007, p.4)

5 Bachelor thesis B.S. de Graaf on the face when considering for instance the emotion surprised, while others were positioned very low when considering the emotion angry. Consequently they were partly removed as well during the process of eliminating irrelevant regions. However, implementing the Hebb rule, it was feasible to determine whether an eyebrow was positioned low or high on the face. An algorithm was developed to determine the number of high grey-values of each row and column, mathematically insinuating certain patterns. Hebbian learning ensured a training of a computated weight matrix, therefore providing a method to recognize either a high- or low-positioned eyebrow. The dimensions of the regions were arranged in such a manner that eyebrows positioned in the middle of the face were always captured. Applying the mathematical optimization method Linear least squares, an affine function was produced fitting the pattern, drawing on the aid of an additional algorithm. This algorithm would locate the darkest pixels below a certain variable, which relied on the level of greyness of the image. This measure was taken because there exists a huge difference between the darkness of different eyebrows. The well-known formula to compute the perpendicular distance [1] of each dark pixel to a function proved valuable in calculating the thickness of an eyebrow, subsequently aiding to draw a rectangle around the eyebrow. The formula is structured as follows: The distance d from a point (m, n) to the line Ax + By + C = 0 is given by the following formula: d = (Am + Bn + C)/ A 2 + B 2. The result of the approach described above is illustrated in Fig. 6: Although a basic linear function appears practical for some faces, most eyebrows fail to follow a simple line. Frequently they tend to give the impression to be fitted more appropriately using the polynomial function y = ax 2 +bx+c. The implementation of such a formula could result in following the exact curve of the eyebrow, aiding in discovering more characteristics of the different emotions. The result of implementing this 2nd-order polynomial function is illustrated in Fig. 7. Figure 7: Fitting a parabolic curve Mouth Through observation it became clear that 95% of the faces smiled with their mouths open, showing their teeth. An additional algorithm was developed to recognize teeth, aiding in recognizing the emotion happy. Reusing the same algorithm to insinuate patterns for eyebrows, a pattern for shown teeth was determined. Training a weight matrix with the aid of the Hebb rule, it was possible to recognize either teeth or merely a mouth. To track the line of the mouth, the Linear least squares method was utilized to create a 2nd-order polynomial fit, following the exact middle line of the mouth, as illustrated in Fig. 8. Figure 6: Fitting the right eyebrow Figure 8: Fitting the mouth The perpendicular distance helped estimating how wide/open the mouth was. This feat was quite relevant, as the majority of the surprised and afraid people keep their mouths wide open, while the angry people mainly show a thin line. (v. June 12, 2007, p.5)

6 B.S. de Graaf Bachelor thesis 3.3 Classification Once the features were extracted from the eyebrows and the mouth, a classification method was required to correctly identify the emotions. Four different well-known techniques were implemented, to determine which one performs superior. 1. Nearest Neighbour using Means A classical NN search, simply classifying a new image to the mean of the emotion it is nearest to. 2. k-nearest Neighbour A slightly different NN search, locating firstly the k nearest neighbours, and secondly assigning the new image to the emotion which has the majority of these k neighbours. 3. Fuzzy c-means A more intelligent NN search, clustering the different emotions first. The technique assigns weights to every sample of every emotion. The weight is determined by the distance of every sample to the mean of that emotion. This process causes outliers to have almost no effect on classifying a new image. 4. Edited NN Another form of clustering, which edits the training set before the new image is classified. For every sample of every emotion, it determines its k nearest neighbours. When the majority of those neighbours is not the emotion it is itself, the sample is rejected from the training set. This process eliminates all outliers of each emotion, instead of assigning low weights to those samples. 4 Results This Section reports the experiments performed and the results derived from its outcomes. The experiments were conducted using the Image Processing Toolbox of the mathematical programming language Matlab, version The test set was yet again provided by the Karolinska Hospital in Sweden [10], consisting of seventy different images for every emotion. Subsection 4.1 will present the statistics derived from the eyebrows and mouths, while Subsection 4.2 shows the results of the different classification methods. 4.1 Statistics The statistics derived from fitting a linear line on the eyebrows are presented in Table 1. The numbers in the columns represent the mean and standard deviation of each property. The angle represents the slope of the linear line, transformed into degrees. The thickness represents the thickness of the eyebrow/mouth, while the position represents the position of the eyebrow/mouth on the face. Note that the position values match the y- values running from the top of the image to the bottom. Therefore, the surprised emotion acquired the lowest position value, since it is positioned high on the face. The statistics may seem obvious, as humans are familiar with how his/her eyebrow and mouth are shaped when expressing a certain emotion. Angle Thickness Position Afraid 4.9± ± ±5.9 Angry 4.3± ± ±5.9 Happy 5.9± ± ±4.9 Neutral 5.5± ± ±5.1 Sad 2.7± ± ±5.5 Surprised 7.4± ± ±5.7 Table 1: Fitting a linear line on the eyebrows The statistics derived from fitting a parabolic curve on the eyebrows are provided in Table 2. Angle( 10 2 ) Thickness Position Afraid 0.05± ± ±6.0 Angry -0.09± ± ±6.7 Happy 0.16± ± ±4.9 Neutral 0.14± ± ±5.1 Sad 0.01± ± ±5.6 Surprised 0.14± ± ±5.7 Table 2: Fitting a parabolic curve on the eyebrows Here the angle represents the a-value of the polynomial function y = ax 2 + bx + c. The statistics derived from fitting a parabolic curve on the mouth are presented in Table 3. Angle( 10 2 ) Thickness Position Afraid -0.00± ± ±6.7 Angry 0.06± ± ±6.2 Happy -0.18± ± ±6.7 Neutral 0.03± ± ±4.8 Sad -0.03± ± ±6.6 Surprised 0.12± ± ±6.2 Table 3: Fitting a parabolic curve on the mouth The angle of the emotion happy has the highest negative value. A negative a-value in the formula above would normally more likely represent a sad face. However, since the curve is fitted on an image, the y-values run vertically backwards and a negative value does represent a happy mouth. 4.2 Classification results As explained in the third Section, four different classification methods were used. The first experiment tested all four classification methods on all six emotions. The (v. June 12, 2007, p.6)

7 Bachelor thesis B.S. de Graaf Af raid Angry Happy N eutral Sad Surprised Average M eans 13% 69% 96% 37% 23% 54% 49% k N N 4% 57% 97% 56% 4% 67% 48% F uzzy 9% 66% 99% 46% 11% 27% 43% Edited 10% 74% 93% 34% 16% 40% 45% Average 9% 66% 96% 43% 14% 47% 46% Best 13% 74% 99% 56% 23% 67% 55% Table 4: Classification results - six emotions result of this experiment is displayed in Table 4. The numbers in the columns represent the percentage of correctly classified images, respectively for the means, k-nearest neighbour, fuzzy c-means and the edited nearest neighbour. The results of that experiment inspired to perform two extra experiments, to determine the influence of the size of the database on classification performance. Table 5 displays the classification results of the second experiment, incorporating the three emotions which scored the lowest in the first experiment. Af raid N eutral Sad Average M eans 41% 67% 47% 52% k NN 33% 90% 6% 43% F uzzy 33% 87% 36% 52% Edited 36% 70% 41% 49% Average 36% 79%. 33% 49% Best 41% 90% 47% 59% Table 5: Classification results - three low scoring emotions The classification results of the third experiment, incorporating the three emotions which scored the highest in the first experiment, are presented in Table 6. Angry Happy Surprised Average M eans 77% 97% 63% 79% k NN 76% 97% 71% 81% F uzzy 77% 99% 49% 75% Edited 79% 97% 66% 81% Average 77% 98% 62% 79% Best 79% 99% 71% 83% Table 6: Classification results - three high scoring emotions 5 Conclusions and future research The first Subsection of this Chapter will provide the conclusions drawn based on the results given in the previous Section, consequently answering the central research question. Section 5.2 discusses the shortcomings of this thesis and offers several future recommendations. 5.1 Conclusions The results of the conducted experiments clearly give the impression that the emotions angry, happy and surprised are more easily identifiable than afraid, neutral and sad. The statistics provided in Subsection 4.1. already indicated such a result, as the latter three emotions did not acquire any outstanding characteristics. Whereas the first three emotions often obtained the lowest and/or highest value(s) at several characteristics. The second and third experiment indicate the obvious result that decreasing the size of the database positively effects the classification performance. Every emotion obtained a higher correctly classified percentage. After examining the results per classification method, it became apparent that none of these techniques is truly superior to the others. In the next Subsection several ideas are offered to improve the classification process. Feature selection of the face is a relatively new approach to facial expression recognition, especially in combination with Eigenfaces. Although it can be concluded that the Eigenfaces technique does not suffice in recognizing emotions in the same manner it is applied to recognize faces, it proved quite valuable. The technique helped in calculating the main deviating regions of a face when shifting expressions. These regions are the eyes, the eyebrows and the mouth. In addition, the method could be applied to remove several irrelevant regions within the face, while extracting the important features. The mathematical optimization method Linear least squares combined with Hebbian learning as well proved to be a successful method to (v. June 12, 2007, p.7)

8 B.S. de Graaf Bachelor thesis extract these features of the relevant regions of the face. It can be concluded that the combination of the previous techniques to extract features, subsequently merged with a range of classification methods, is an accurate and effective approach to recognize person-independent facial expressions. 5.2 Future research Analysing the three parts of this thesis; preprocessing, feature extraction and classification, improvement of the last two may be be most effective. For instance, the Hebb networks applied were trained with a low number of samples. Results will almost certainly improve when the network is trained with more samples. A multilayered neural network using hidden layers possibly works superior as well. The Linear least squares technique is quite sensitive to noise. At this moment, long hair on the face and the darkness just above the eyes cause many errors. There are various regularization techniques which can be applied to reduce these errors. When determining the most important regions of the face, it became apparent that the three most relevant areas were the eyebrows, the eyes and the mouth. However, time limited the number of features that could be extracted, thus excluding the eyes. Extracting features from the eyes as well is clearly a good example of future research. As for the last part, the classification methods applied here were each a form of the well-known nearest neighbour search. Many other intelligent classification methods would almost certainly perform superior, for example Hidden Markov Models [3], a Naïve Bayes classifier [9] or multilayered neural networks [7]. An other interesting idea is to apply this program on videos. At this moment it only operates on images of frontal faces having a specified resolution, a property which clearly limits the number of possibilities. Evidently, an implementation of a face detection algorithm is required first, before this technique could be utilized on something else than images of frontal faces. However, once this program obtains a face, it should be able to recognize person-independent facial expressions. [3] Cohen, I., Garg, A., and Huang, T. (2000). Emotion recognition from facial expressions using multilevel hidden markov models. Neural Information Processing Systems. [4] Danner, Tim (1999). Eigenfaces home group. elec301/. [5] Ekman, P. and Friesen, W.V. (1978). Facial action coding system: Investigator s guide. Consulting psychologists Press. [6] Elfenbein, H. A. and Marsh, A. A. (2002). Emotional intelligence and the recognition of emotion from facial expressions. The wisdom in feelings, pp , The Guilford Press, New York. [7] Hagan, M. T., Demuth, H. B., and Beale, M. (1996). Neural Network Design. PWS Publishing Co.,Boston. [8] Hebb, D. O. (1949). The organization of Behavior. Wiley, New York. [9] Sebe, Nicu, Lew, Michael S., Cohen, Ira, Garg, Ashutosh, and Huang, Thomas S. (2002). Emotion recognition using a cauchy naive bayes classifier. 16th International Conference on Pattern Recognition, Vol. 1, p [10] Stockholm, Sweden (1999). Directed Emotional Faces. Karolinska Hospital, Department of Clinical Neuroscience, Psychology section. [11] Turk, M. and Pentland, A. (1991). Eigenfaces for recognition. Journal of Cognitive Neurosicence, Vol. 3, pp [12] Zhao, W., Chellappa, R., Rosenfeld, A., and Phillips, P. (2000). Face recognition: A literature survey. CfAR Technical Report. References [1] Bourne, Murray (2007). Interactive mathematics. [2] Chen, L. S. (2000). Joint processing of audiovisual information for the recognition of emotional expressions in human-computer interaction. Ph.D. thesis, University of Illinois at Urbana-Champaign, Dept. of Electrical Engineering. (v. June 12, 2007, p.8)