A Facial Expression Classification using Histogram Based Method

Transcription

1 2012 4th International Conference on Signal Processing Systems (ICSPS 2012) IPCSIT vol. 58 (2012) (2012) IACSIT Press, Singapore DOI: /IPCSIT.2012.V58.1 A Facial Expression Classification using Histogram Based Method Darli Myint Aung 1+ and Nyein Aye 2 1 University of Computer Studies, Mandalay 2 Computer University (Meikhtila) Abstract. With a number of emerging new applications, automatic recognition of facial expressions is a research area of current interest. In this paper, a simple and effective approach for accurate facial expression recognition using morphological based method along with histogram method had been proposed. This paper focuses on facial expression to identify five universal human expressions: neutral, happy, anger, sad and surprise. Proposed system consists of four stages. In the first stage, using median filter which reduces noise that cause over segmentation and pre-processes the face image. In the second stage, mouth region is segmented using the morphological method. After segmentation part, the geometric feature (especially mouth) is extracted in the third stage. Lastly, histogram calculations are used for facial expression classification. The proposed system tested on JAFFE facial expression database. The effectiveness of the proposed system can be confirmed through the experimental results. Keywords: expression recognition, facial feature, morphological method, histogram frequency. 1. Introduction There is a long history of interest in the problem of recognizing human emotion from facial expressions, as well as extensive studies on face perception over the last three decades [1]. Facial expression recognition system is a considerably challenging field to generate an intelligent computer that is able to identify and understand human emotions for various vital purposes, e.g. security, society, entertainment. In order to facilitate a more intelligent and natural human machine interface of new multimedia products, automatic facial expression recognition had been studied worldwide in the last ten years [2]. The domain of application is human interaction with computers in smart environments. Analyzing the emotional expression of a human face requires a number of preprocessing steps which attempt to detect and locate characteristic facial regions, extract facial expression features, and model facial gestures using anatomic information about the face. We consider the geometric feature points in this system. The distance between features point represent the shape of the mouths which express emotion of the face. The major contribution of this paper is to present a new facial expression classification method based on the histogram sequence of the feature vector. Summarizing, the proposed system consists of four main tasks: pre-processing, mouth segmentation, feature extraction and classification. The following will cover these tasks in detail and provide evaluation results. Details of this system are described in the remainder of this paper. The paper is organized as follows: Section 1 gives introduction, Section 2 highlights related works, and Section 3 present Data Collection and Section 4 covers proposed method. In Section 5, experimental results and analysis are presented. Finally, conclusions are summarized in Section Related Works Mondal et.al. [3] proposed a method for face detection by using a geometric definition of human face. They computed the mean and variance of the input image to obtain a median filter that was used to reduce + Corresponding author. address: darlimyintaung@gmail.com. 1

2 the noise effect within the image. Gamathi et.al. [4] used uniform local binary pattern (LBP ) histogram technique for feature extraction and MANFIS ( Multiple Adaptive Neuro Fuzzy Inference system) for expression recognition. Lekshmi and Sasikumar [5] presented a method to analyze four expressions using Gabor features derived from facial images. Their method showed an average recognition rate of 88%. The system had a low computational complexity as compared with existing methods. Murthy and Jadon [6] proposed a method to recognize facial expressions as defined in Ekman and Friesen work using Eiganface method. Their method was effective for facial expression recognition but it requires a high storage of data because six subspaces are calculated according to each expression. Their best recognition rate was 83% for the surprise category. Le Hoang Thai et.al. [7] proposed a novel approach using Canny, Principal Component Analysis and Artificial Neural Network. The facial expression classification of their proposed methods was 85.7%. Lyons et.al. [8] presented a method for extracting information about facial expressions from images. Their results showed that it is possible to construct a facial expression classifier with Gabor coding of the facial images as the input stage. 3. Data Collection Data required for experimentation is collected from JAFFE database for neural network training and testing. JAFEE stands for The Japanese Female Facial Expression (JAFFE) Database [9]. The database contains 213 images of 7 facial expressions posed by ten different Japanese female models. Sixty Japanese subjects have rated each image on 6 emotion adjectives. The photos were taken at the Psychology Department in Kyushu University. Few samples are shown in Figure Proposed System Fig. 1: Samples of facial expressions. In this paper a new simple facial expression classification system is proposed to automatically recognize facial expressions to identify five universal human expressions: neutral, happy, anger, sad and surprise using relatively face images. Figure 2 illustrates the block diagram of the facial expression recognition system. The proposed system involves four phases: preprocessing, segmentation of the mouth, feature extraction from the mouth region and finally recognition of the mouth. Firstly, this system uses preprocessing step before extracting the facial feature in the image. We take the input image and then use a median filter to eliminate the noise in image. This system emphasizes mainly geometric feature such as the mouth region. Secondly, segmentation is an essential preliminary step in automatic facial expression recognition system. The aim of segmentation is making image more analyzable. We use edge detection method and morphological method. Thirdly, after getting the segmented mouth image, this feature points use for a calculation of Euclidean distances between these features. Then we calculate frequency of the feature vector. Lastly, this system is classified by using histogram based method. The input image s expression is displayed what type of expression Mouth segmentation Fig. 2: Block diagram of the facial expression recognition system. 2

3 After preprocessing, the first process is the segmentation. The mouth area is chosen by taking the portion of the face image from 66.67% of the face image height of the face image height after the top boarder to 5% of the face image height before the bottom boarder and 25% of the face image width after the left boarder and before the right border. The mouth region to be segmented differs greatly in contrast from the background image. First, we apply Sobel edge operator to calculate the threshold value. We then adjust the threshold value and use edge again to obtain a binary mask that contains the segmented mouth image. (See in Figure 3-6) Feature extraction The most important step in the field of facial expression recognition is the facial feature extraction which is based on finding a set of features that are conveying the facial expression information. The segmented mouth region is processed with the proposed method of finding more exact feature. This system performs morphological method. Morphological operation has been used as means to identify and extract meaningful image descriptors based on properties of form or shape within the image. In this operation, the value of each pixel in the output image is based on a comparison of the corresponding pixel in the input image with its neighbors [10]. The purpose of morphological processing is primarily to remove imperfections added during segmentation. This system applied erosion and dilation operation, so we can remove small holes. Finally, mouth region is segmented from the image as shown in Figure Calculation of mouth feature distance The proposed system evaluates the distances between certain points of the mouth as described below. The features (F1 F6) are obtained from the geometric features (see Figure 8). This system calculates Euclidean distance. This is probably the most commonly chosen type of distance. In 2-D, it is computed as: distance(x, y) = i ( ) y i x i 2 Fig. 3: The 6 distance based geometrical features of a face. F1, F2: distance between the upper lip and the left/right mouth corner F3, F4: distance between the lower lip and left/right mouth corner F5: vertical distance between the left and right mouth corner F6: horizontal distance between the upper and the lower lip In addition, normalization is considered the F1, F2, F3, F4, F5 are divided by F6. The facial feature vector is then: V = [F1; F2; F3; F4; F5] This vector represents each person by a unique way. These parameters will be stored in database Facial expression classification In the classification step, the current facial expression is assigned to one of the trained categories. At present, up to five classes, that is, neutral, happy, surprise, anger and sad are considered. This system is classified by using Histogram based method. The horizontal axis depicts the range and scale of observations involved and vertical axis shows the number of data points in various intervals i.e. the frequency of observations in the intervals. A histogram consists of multiple bins where each bin corresponds to a range of values. When any data is provided to system, it decides the number of intervals amongst which the data should be distributed. The sample feature histogram for neutral expression as shown in Figure 9. The values exist on the horizontal axis are the upper limits of bins (intervals) of feature points. As an example, the bar shows against 160 has a frequency of 40. That means 40 feature points lie in the range above 120. As is evident, the histogram gives a fairly good idea about the shape and spread of data at a glance. We employ the histogram base method for classification. 3

4 In training phase, histograms of feature vectors are built for each facial expression using training set of images. For classification process, feature vector is extracted and probabilities for each facial expression are calculated from histograms of facial expression. Facial expression can be determined using calculated probabilities for facial expression. 5. Experimental Results and Analysis Fig. 4: Sample feature histogram for neutral expression. The experiment is performed on the JAFFE Database, we consider the last five images from each subject facial expression were taken and used the odds (60%) in training and the evens (40%) in testing. This system focuses on facial expression to identify five universal human expressions: neutral (NE), happy (HA), anger (AN), sad (SA) and surprise (SU). Based on the Morphological method and Histogram method, we conduct the training and obtained the result as shown in Table 1. The average facial expression classification rate of our proposed method is 81.6%. Table 1: Confusion matrix for Facial expression recognition. HA NE SU AN SA % HA NE SU AN SA The above analysis results prove that the proposed system could classify surprise, neutral and happy in maximum rates but the Anger in minimum rate in JAFFE database. According to observation, anger was the minimum rate because the proposed system fails to segment the mouth region of the some subjects in segmentation part, although the feature vector is extracted well. Typically, the eye never classified correctly an expression that the mouth caught correctly. Surprise was the well-segmented easily, being found the highest classification rate of 90%. In future, we plan to use the eyebrows region in extraordinary case such as anger. If we are angry, the eyebrows are lowered and drawn together. We will classify the anger expression correctly. 6. Conclusions In this paper, we present a new simple method using Morphological and Histogram method applies for facial expression classification. In this work the proposed system deals with the static images. Morphological method applies for local facial feature segmentation. A facial image is separated to mouth region. This region feature is calculated by Euclidean distance and frequency. So that image representation space is reduced. To experience the effectiveness of our method, we built recognition of five basic facial expressions system on JAFFE database. The experimental result shows the effectiveness of our proposed system. With our proposed work the efficiency is 81.6%.Histogram Based method leads to large memory requirement due to processing of each feature histogram as a result of which speed may decrease but on the other hand accuracy level is increased which is an advantage to the system. Fig.5: Original image. Fig.6: Mouth region. Fig.7: Binary gradient mask. Fig.8: Binary Image with filled holes 4

5 7. Acknowledgements Fig.9: Segmented mouth region. I wish to thank Dr. Nyein Aye for his extreme support and guidance. I also thank my family and colleagues for their valuable feedback and thoughtful suggestions. 8. References [1] M. Pantic and L.Rothkrantz. Automatic Analysis Of Facial Expression: The State Of The Art. IEEE.Trans. On PAMI, vol. 22, pp [2] S. Yoshimori, H. Takimoto, and Y. Mitsukura. A Proposal Of Feature Extraction For Impression Analysis. The 11 th IEEE International Workshop on Advanced Motion Control, Japan. March 21-24, [3] T. Mondal, A. Nath, A. Das, and M. Banerjee. An Approach of Face Detection Using Geometrical Definition of Human Face. National Conference on Computational Instrumentation/Central Scientific Instruments Organization (CSIO), Chandigarh, India, March. pp [4] V. Gomathi, Dr. K. Ramar, and A. S. Jeeyakumar. A Neuro Fuzzy Approach for Facial Expression Recognition using LBP Histograms. International Journal of Computer Theory and Engineering. April 2010, 2(2): [5] V. Lekshmi, S. Kumar, and D. S. Vidyadharan, Face Detection and Localization of Facial Features in Still and Video Images, First International Conference on Emerging Trends in Engineering and Technology, pp , [6] G. R. S. Murthy and R. S. Jadon. Effectiveness of Eigenspaces for Facial Expressions Recognition. International Journal of Computer Theory and Engineering, 1(5), pp [7] L. H. Thai, N. D. T. Nguyen, and T. S. Hai, Member, IACSIT, A Facial Expression Classification System Integrating Canny, Principal Component Analysis and Artificial Neutral Network, International Journal of Machine Learning and Computing, 1( 4), October [8] M. Lyons, M. S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with Gabor wavelets. 3rd International Conference on Automatic Face and Gesture Recognition, [9] C. Anitha, M. K. Venkatesha. A Survey on Facial Expression Databases. International Journal of Engineering Science and Technology, 2(10): [10] R. C., Gonzalez, R. E Woods, and S. L. Eddins Digital Image Processing Using Matlab. Published by Dorling Kindersley (India) Pvt. Ltd. ISBN Darli Myint Aung received the Master degree in computer science from the University of Computer Studies, Mandalay in February She is a PhD candidate at the University of Computer Studies, Mandalay. Her research interests are pattern recognition and image processing. Nyein Aye received the PhD degree from Engineering Physics Institute, Moscow. He currently serves as the headmaster of Computer University, Meikhtila. His current research interests include pattern recognition, image processing and information security. 5