FACIAL EXPRESSION DETECTION AND RECOGNITION SYSTEM

Transcription

1 FACIAL EXPRESSION DETECTION AND RECOGNITION SYSTEM W.K. Teo 1, Liyanage C De Silva 2 and Prahlad Vadakkepat 1 ABSTRACT In this paper, the integration of face feature detection and extraction, and facial expression recognition are discussed. In this paper, we propose an algorithm that utilizes multi-stage integral projection to extract facial features. Furthermore, in this project, we propose a statistical approach to process the optical flow data to obtain the overall value for the respective feature region in the face. This approach has eliminated the requirement of accurate identification of the feature boundary. Optical flow computations are utilized to identify the directions and the amount of motions in image sequences that are caused by human facial expressions. The optical flow computation results are processed using Kalman filtering. The filtered results are given to a neural network to realize a mapping into the facial expression space. This technique is used on a set of training and testing face images. Preliminary experiments indicate an accuracy between 60% - 80% on the Kalman filtered data when recognizing four types of expressions: anger, sad, happy and surprise. In an attempt to further improve the recognition results, we proposed a technique to process the optical flow results using a statistical approach instead of using Kalman filtering. The preliminary experiments on this proposal approach produced accuracy between 70% - 100% on the original optical flow results that is better than the Kalman filter technique. INTRODUCTION Detection of face features such as eyes and mouth have been major issues of facial image processing which may be required for various areas such as emotion recognition [1] and face identification [2]. Face feature detection can be used to determine the face features from images to be used later as input for other functions like face and emotion recognition. Facial expression plays an important role in smooth communication among individuals. The extraction and recognition of facial expression has been the topic of various researches subject to enable smooth interaction between computer and their users. In this way, computers in the future will be able to offer advice in response to the mood of the users. 1 Department of Electrical and Computer Engineering, National University of Singapore, 4, Engineering Drive 3, Singapore Institute of Information Sciences and Technology, Massey University, Palmerston North, Pvt. Bag , New Zealand 14

2 Computer-based recognition of facial expressions goes a long way, and various methods have been proposed. All the method can be classified into two broad-based category: probabilistic approach and feature based approach. The feature-based method utilizes the Facial Action Coding System (FACS) designed by Ekman and Friser [3]. In FACS, the motions of the face are divided into 44 action units (AU), and their combinations may describe any facial expression. More than 7,000 combinations of AU have been observed [4]. However, FACS itself is purely descriptive, uses no emotion or other inferential labels, and provides the necessary background to describe facial expression. The probabilistic-based method does not give preference to facial features such as eyes and mouth. Instead, the feature vector can be the random distribution of image intensities and these vectors may differ from each emotion. The vectors are calculated per emotion and classification algorithms like HMM, Neural Network (NN) or a hybrid approach (HMM an NN) [5] are applied. In this work, we propose a method to combine feature detection and extraction and facial expression recognition into an integrated system so that the recognition results will not be influenced by subjective factors and the bound of areas are invariant during the whole sequence. We propose a method for facial expression recognition that uses integral projection, statistical computation, a neural network and kalman filtering. The face feature detection method uses multi-stage integral projection. Optical flow computation [6] will be used on the detected feature namely, the eyebrows and the lip to extract movement. One advantage of optical flow is that optical flow information can be extracted easily even at low contrast edges. The extracted feature vectors are also preprocessed using Kalman filtering [7], [8] to smooth the data. The original as well as the Kalman filtered feature vectors are then fed in separately into the neural network to realize a mapping into the expression space. The face image database created by Carnegie Mellon University (CMU) is used in this analysis. In this paper, firstly, we propose a robust face feature detection method that we called multi-stage integral projection. At the moment, there are several approaches such as the one using color space [9] and using deformable template matching [10]. The performance of these approaches is affected by several external conditions such as illumination and skin color. Our proposed method makes use of edges extracted from the facial image that are considered as general features during our feature detection process. Then we make use of multi-stage integral projection to obtain the feature positions. After each stage of integral projection, we will get closer to the position of the features. Our detection algorithm makes use of a few parameters that we designed using a facial image database containing faces from 30 different people. Our detection algorithm is robust and it works well for people of different skin color. We make two assumptions: 1) the subjects are in front of a background that contains only a few elements; 2) the subject does not wear glasses and hats. Secondly, we use a statistical approach and Kalman filter approach to process the values obtain from the optical flow computation on the detected features. Kalman filtering is applied on the optical flow values of the images sequences in an attempt to improve the overall recognition rate of the system. Kalman filter provides an efficient computational 15

3 (recursive) solution of the least-squares method. The filter is very powerful in several aspects it supports estimations of past, present, and even future states, and it can do so even when the precise nature of the modeled system is unknown. With a statistical approach, there is no need to track the feature movement in the subsequent image frames, as the one with the highest possibilities will indicate the average movement of the feature. Figure 1 gives a complete overview of the facial feature detection, extraction and emotion recognition proposed in this paper. In Section 2, a feature detection method called multistage integral projection to detect the position of feature is explained. In Section 3, facial feature extraction is discussed. After this process, the average output is also process using Kalman filter. Here we also utilize a statistical approach to process the optical flow values in an attempt to improve the results. In Section 4, the structure of the neural network is discussed. In Section 5, the experimental results after feeding the feature vectors into the neural network are discussed. Finally, conclusion to the paper is presented. Image Sequences for Training Feature Detection Feature Extraction Feature Vectors Kalman Filter/ Statistical Approach Training Neural Network Neural Network Image Sequences to be Recognitzed Feature Detection Feature Extraction Feature Vectors Kalman Filter/ Statistical Approach Mapping to expression space Recognition Results Figure 1: Block diagram of the proposed facial expression recognition system FACIAL FEATURE EXTRACTION In this section, the edge detection will be discussed here. Next, head location and size estimation, and location of feature using integral projection is discussed. Edge detection The forming of the image of the face on the retina does not include fine edge details of the faces when the person is far away. Instead, we see a blurred replica of the face. We can get the same effect when an image is blurred using a gaussian filter with a large σ and window size. In the first step, the image is blurred using a Gaussian operator h that can be given by g( w = w w w h( α, β ) f ( x α, y β ) dαdβ (1) 16

4 where, 2 2 x + y h( = exp( ), 2 2σ w = window size of 7 x 7, and σ = 7, standard deviation of the Gaussian Filter. To calculate the edges of the image, we convolve the blurred image with vertical and horizontal Sobel edge operator, such that: g ( = g( s x x ( = g( x 1, y + 1) + g( x + 1, y + 1) 2g( x 1, + 2g( x + 1, g( x 1, y 1) + g( x + 1, y 1) (2) g ( = g( s y y ( = g( x 1, y + 1) g( x 1, y 1) + 2g( y + 1) 2g( y 1) + g( x + 1, y + 1) g( x + 1, y 1) (3) The edge image G( can be given, G( = g( = 1 if t > T = 0 else (4) 2 2 where t = ( g x ( ) + ( g y ( ) and 2 2 T is the root mean value of {( g x ( ) + ( g x ( ) } over the total image area. This edge detection method uses a standard algorithm, the Sobel operator. This step is necessary as a pre-process before using integral projection to obtain the feature location. One advantage of this step is that the threshold is based on the statistics of the image data instead of manual value and is acceptable to small amount of noise. Head size and location estimation, and feature location using multi-stage integral projection In this section, estimation of head size and location, and feature location using the proposed multi-stage integral projection is discussed. The integral projection technique was originally proposed by Kanade [11] and modified by the authors to suit the problem of face position, size estimation and feature location. 17

5 In the 1 st stage, we implement a method proposed in a paper by De Silva [12] that estimates the head size and location. However, we have changed the parameters to fit the statistics of the image database. The head size and location is estimated to separate the head area from the background and the other body parts. The edge-detected image is subjected to vertical integral projection. The algorithm will detect the eye position, eyepos of the face, starting from the top of the head. One important aspect to note is that the value eyepos is not precise but only an estimate. The value eyepos will be used in the next stage that will be discussed later. The object width of the head is taken as the width of the head. By observing the faces of 30 different people, we got 2.0 times eyepos as the head height. Instead of an elliptical area, we have implemented a rectangular area with the breadth equal to the head width (HW) and the length equal to the head length (HL). See Figure 2 for the results of the head area estimation. HW eyepos HL Figure 2: Results of head area estimation In the 2 nd stage, the head area is subjected to horizontal projection. The horizontal integral projection of an edge image is given by H ( = HW x= 1 G head area ( (5) We find the exact position of the eyeplane by observing the maximum projection in a pre-defined search area. The search area is limited using the eyepos; the lower limit is equal 0.5 times eyepos and the upper limit equal 1.2 times eyepos. Next, the eyebrow position is located by the next highest horizontal projection before the eyeplane. In addition, a threshold condition is built into the algorithm. A threshold value is use to determine if the eyebrow slit passed detected earlier is actually the true eyebrow slit and not the forehead. By doing horizontal projection in the detected eyebrow slit and comparing the maximum value with the threshold of 15 pixels, it determines if the detected eyebrow slit is the actual one. We also find the lip slit by finding the maximum projection in a pre-defined search area. The search area is also limited using the eyepos, the lower limit is times eyepos and the upper limit is HL. Both the search area and 18

6 the threshold value is obtained from the statistical data of 30 different faces. See Figure 3 for the horizontal projection indicating the position of the eyebrow, lips and eyes. In the 3rd stage, the exact location of the eyebrow is obtained by using vertical projection of their edge image slit. A hierarchical searching process is adopted to find the location of the left and right eyebrows. Here when searching from the left to the right side of the head, we can observe that the projection is divided into ranges. The two largest ranges are observed to be bounded by the left and right extremes of eyebrows. However if the eyebrow is covered by hair this condition maybe violated. Figure 3: Horizontal projection of head area Next, the exact location of the lip is located by using vertical projection of their edge image slit. A hierarchical searching process is adopted to find the location of the lip. Here when searching from the left to the right side of the head, we can observe that the projection is divided into several ranges. The broadest range is observed to be bounded by left and right extremes of the lip position. A horizontal projection is done in the window, bounded by the left and right extremes of the lip position. The maximum horizontal projection is the center of the upper and lower lip. See Figure 4 for the vertical projection of the eyebrows and lip slit. Figure 4: Vertical projection of eyebrows and lips 19

7 In Figure 5, results of the multi-stage projection are shown. The main advantage is that this is a simple algorithm that is very fast. The detected feature location is to be used for the facial feature extraction that is discussed next. Figure 5: Final Result of Projection FACE FEATURE EXTRACTION In this section, facial feature extraction is discussed. We will explain about utilizing the proposed statistical approach to process the optical flow values in an attempt to improve the results. Next, we will discuss the usage using Kalman filter. Optical flow computation The optical flow algorithm proposed by P. Anandan [6] is implemented here. The optical flow computation is applied on the detected eyebrows. In the case of the lips, it is divided into 4 areas as shown in Figure 6 and the optical flow computation is applied on each of the regions. A statistical approach is used to compute the average optical flow of the facial feature. A statistics is created from the optical flow field for each window. The most common motion is selected as the overall movement of the feature. The motions components x and y are divided into a few possible ranges. The range with the highest percentage is then selected and the average motion inside this range is taken as the average flow. This is illustrated in Figure 7. % Min pixel Max Figure 6: Lips partition Figure 7: Illustration of motion statistics 20

8 Kalman filter The Kalman filter is a set of mathematical equations that provides an efficient computational (recursive) solution of the least-squares method. The filter is very powerful in several aspects: it supports estimations of past, present, and even future states, and it can do so even when the precise nature of the modeled system is unknown. It has been the subject of extensive research and application, particularly in the area of autonomous or assisted navigation. This is applied to perform smoothing of the optical flow values. In this case, we attempt to apply Kalman filter-smoother to recognition of emotion in an attempt to improve the recognition rate. A Linear Dynamical System is a partially observed stochastic process with linear dynamics and linear observations, both subject to Gaussian noise. It can be defined as follows, where X(t) is the hidden state at time t, and Y(t) is the observation. The conditional independence assumptions that are being made can be concisely modeled using a Bayesian network as shown in Figure 8. (Circles denote random variables with linear-gaussian distributions, clear means hidden, shaded means observed.) The Kalman filter is an algorithm for performing filtering on this model, i.e., computing P(X(t) Y(1),..., Y(t)). Figure 8: Bayesian network We filtered the optical flow values of the velocity (pixels/frame) for a sequence of images leading to the surprise emotion and show the smooth value and the observed value using statistical computation in Figure 9. The starting and ending point of the filtered estimate is now above X2 = 0 line compared to the observed value that is below it. We know that it is better as the upper lip of the surprise expression should be moving up instead of down. However, it has smoothed the sudden changes in facial feature movement that forms an important part in facial expression. The summation of the observed value and the smooth estimates for the lip and eyebrows is fed into the neural network that will be discussed in the next section. (a) Left eyebrow (b) Right eyebrow 21

9 (c) Left section of Lip (d) Right section of Lip (e) Upper section of Lip Figure 9: Plot of observed and filter values (f) Lower section of Lip STRUCTURE OF THE NEURAL NETWORK AND ITS TRAINING ALGORITHM Neural network learning methods provide a robust approach to approximating real-valued, discrete-valued, and vector-valued target functions. For certain types of problems, such as learning to interpret complex real-world sensor data, artificial neural networks are among the most effective learning method currently known. The backpropagation algorithm that has proven successful in many practical problems is implemented here. In this paper, the one-hidden-layer neural network is proposed. There are 6 inputs units into the neural network: left and right eyebrows (2 units), and left, right, upper and lower section of lip (4 units). The output layer has four units that correspond to the four facial expression categories: anger, happiness, sadness and surprise. See Figure 10 for an illustration of the structure. Inputs hidden Outputs Figure 10: Neural network structure 22

10 The Levenberg-Marquardt algorithm is used to update the network weights and biases. One iteration of this algorithm can be written as T 1 T [ J J I] J e x + 1 = x + µ k k (7) where x k is a vector of current weights and biases, J is the Jacobian matrix that contains first derivatives of the network error with respect to the weights and biases, and e is a vector of network errors. This algorithm appears to be the fastest method for training moderate- sized feed-forward neural network. The tan-sigmoid transfer function is used for the activation of the hidden network units and the linear transfer function is used for the activation of the output network units. The network identify the facial expression by finding the output unit that shows the largest value among all of the output units, and taking the order of that unit as the facial expression category (winner takes all). EXPERIMENTS AND RESULTS The experiments are done under the following conditions: 1. Only the frontal view of the facial image sequences is analyzed through the whole image of sequence. 2. The head motion between two consecutive frames is considered small. 3. The subjects do not have facial hair and they are not wearing glasses. Our image database has 122 image sequences taken from 30 subjects. They are 30 of anger, 32 of happiness, 30 of sad and 30 of surprise. We use 20 of each emotion to train the neural network. Our experiments include two parts: (i) Using kalman filtered resultant optical flow as neural input. (ii) Using statistically computed optical flow value as neural network input The recognition results of using the Kalman filter and statistically computed optical flow are shown in Tables 1 and 2 respectively. From the result, we can see that using the original resultant optical flow yield high accuracy than using the Kalman filter output. While using the original resultant optical flow, a recognition accuracy as high as 100% is achieved for the emotion of surprise. Table 1: Recognition results using kalman filter Anger Happiness Sad Surprise Anger Happiness Sad Surprise Accuracy 70% 75% 60% 70% 23

11 Table 2: Recognition results using statistical computed optical flow Anger Happiness Sad Surprise Anger Happiness Sad Surprise Accuracy 70% 75% 70% 100% CONCLUSION In this paper we have proposed to combine feature detection and extraction and recognition of facial expression into one system. We proposed a new method of feature detection and statistical approach is introduced in an attempt to improve the recognition rate. The proposed feature detection using multi-stage integral projection is simple, robust and efficient. Using integral projection, we were able to locate the eyebrows and lips. Then using a statistical approach on the optical flow field, we found the overall movement of the features in the window detected earlier without the need to pinpoint the exact location of the feature. The main advantage of this approach is that it does not require any initial manual settings such as location of head. The initial settings are predetermined using normalized coefficients obtained using a facial image database. Second, Kalman filtering is applied on the resultant optical flow value to calculate the recognition rate. From the recognition rate we can see that directly feeding in the Kalman filter to the neural network lead to a recognition rate of 70%. However, by applying the proposed statistical approach on the optical flow results and feeding it to the neural network lead to an improvement to 80%. This is because of the facial movement due to expressions consists of sudden changes and it is difficult to model a facial expression system. In Kalman filtering the system model has to be defined beforehand -- in the case of expression, the amount of movement need to be known. However, the amount of movement of the facial features varies from person to person and to some people for example, happiness is only a slight movement of the lips while to others, the movement can be very large. Therefore, the use of Kalman filtering lead to poor performance because of the difficulty in designing a suitable system model. FUTURE DIRECTIONS Future areas of development can include using other types of Kalman filters like the extended Kalman filter and the unscented Kalman filter that are designed for system with non-linear Gaussian noise. Global motion of the head also affects the results obtained from the optical flow values. By taking into consideration global motion, the average optical flow movement of the facial features can be more accurately extracted. We can also extent the proposed method to a larger database that consists of people from different races and ages. We can also further analyze the selection of the network structure and to modify the network-training algorithm for improved recognition rates. 24

12 REFERENCES Yingli Tian, T. Kanade and J. F. Cohn, Recognizing Action Units for Facial Expression Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 2, February, R. Brunelli and T. Poggio, Face recognition: features versus templates, IEEE PAMI, vol.15. no 10, pp , Oct P. Ekman and W.V. Frisen, Facial action coding system, Consulting Psychologists Press, 1977 P. Ekman, Methods for measuring facial actions, In K.R. Scherer & P. Ekman (Eds.), Handbook of Methods in Nonverbal Behavior Research, Cambridge: Cambridge University 1982, pp Tianming Hu, Liyanage C. De Silva, Kuntal Sengupta, A Hybrid Approach of NN and HMM for Facial Emotion Classification, in ELSEVIER Pattern Recognition Letters Journal, vol. 23, no. 11, pp , November P. Anandan, A Computational Framwork and an Algorithm for the Measurement of Visual Motion, Interational Journal of Computer Vision, 2, pp , Kalman, Rudolph and Emil, A New Approach to Linear Filtering and Prediction Problems, Transactions of the ASME--Journal of Basic Engineering, vol 82, series D, pp.35-45, 1960 Greg Welch and Gary Bishop., An Introduction to the Kalman Filter, UNC-Chapel Hill, TR , Feb J. Yang, W. Lu, and A. Waibel, Skin-color modeling and adaption, Technical Report CMU-CS , School of Computer Science, Carnegie Mello University, A.L. Yuille, P.W. Hallinan, and D.S. Cohen, Feature extraction from faces using deformable templates. Int. Journal of Computer Vision, vol. 8, no.2, pp , T. Kanade, Picture Processing System by Computer Complex and Recognition of Human Faces, Ph.D. thesis, Kyoto University, Japan, Nov Liyanage C. De Silva, K. Aizawa and M. Hatori, Detection and Tracking of Facial Features by using a Facial Feature Model and Deformable Circular Templates, in IEICE (Inst. of Electronics, Information and Communication Engineers, Japan) Transactions on Information and Systems, vol.e78-d, no.9, pp , Sep Burt, Fast filter transforms for image processing, Computer Graphics Image Processing 16:20-51,

13 R.Y. Wong and E.L. Hall, Sequential hierarchical scene matching, IEEE Transaction on Computer, vol. 27, no. 4, pp , A. Rosenfeld and A.C. Kak, Digital Picture Processing. Academic Press: New York, P.J. Burt, T.H. Hong and A. Rosenfeld, Image segmentation and region property computation by cooperative hierarchical computation, IEEE Transactions On Systems, Man, And Cybernetics, vol. 11, pp ,