Emotion Recognition With Facial Expressions Classification From Geometric Facial Features

Transcription

1 Reviewed Paper Volume 2 Issue 12 August 2015 International Journal of Informative & Futuristic Research ISSN (Online): Emotion Recognition With Facial Expressions Classification From Geometric Facial Features Paper ID IJIFR/ V2/ E12/ 044 Page No Electronics & Subject Area Telecommunication Key Words Emotion Recognition, Self-Organizing Map, Geometric Features, Confusion Matrix Received On Accepted On Published On Patil Pooja Adgonda Dr. Shubhangi B. Patil M.Tech. Student, Department of Electronics & Telecommunication Dr. J. J.Magdum College of Engineering, Jaysingpur Shivaji University, Kolhapur - Maharashtra Professor, Department of Electronics Dr. J. J.Magdum College of Engineering, Jaysingpur Shivaji University, Kolhapur - Maharashtra Abstract This paper presents a novel emotion recognition model using the system identification approach. A comprehensive data driven model using an extended self-organizing map (SOM) has been developed whose input is a 26 dimensional facial geometric feature vector comprising eye, lip and eyebrow feature points. The analytical face model using this 26 dimensional geometric feature vector has been effectively used to describe the facial changes due to different expressions. This paper thus includes an automated generation scheme of this geometric facial feature vector. This also include a method like single input and single output, by giving single image as an input we can obtained a result of its specific expression which is observed, we also able to observe the confusion matrix of all the images which were used in training in which the overlapping of their expressions are given. This paper gives the provision of using real time images as an input test image. Copyright IJIFR

2 1. Introduction Face analysis such as facial features extraction and face recognition is one of the most flourishing areas in computer vision like identification, authentication, security, surveillance system, human-computer interaction, psychology and so on [1]. Facial expressions are a kind of nonverbal communication. They are considered to be one of the most powerful and immediate means of recognizing one's emotion, intentions and opinion about each other. Mehrabian found that when people are communicating feelings and attitudes, 55% of the message is conveyed through facial expression alone, vocal cues provide 38% and the remaining 7% is via verbal cues. Ekman and Friesen did a rigorous study on facial expression and came to conclusion that facial expressions are universal and innate. They also stated that there are six basic expressions, these include happiness, sadness, disgust, anger,surprise and fear. Much efforts have gone towards the study of facial expression and emotion recognition, initially by cognitive scientists and later by computer vision researchers. Facial features extraction is the initial stage for face recognition in the field of vision technology. Automatic analysis of facial features requires extraction of relevant facial features from either static images or video sequences, which can either be further classified into different action units (AUs) or can be applied directly to the classifiers to give the respective emotion. Efficient extraction of facial features from faces of different persons is a crucial step towards accurate facial expression recognition. Generally, two common types of features are used for facial expression recognition: Geometric features data and appearance features data. Geometric features give clues about shape and position of the feature; whereas appearance based features contain information about the wrinkles, bulges, furrows, etc. Appearance features contain micropatterns which provide important information about the facial expressions. We humans have a very extraordinary ability to recognize expressions. Even if we are given a cartoon image having only some contours, we can easily recognize the expression. In many cases, it is observed that features obtained from facial contours alone can convey adequate information to recognize various expressions on the face. The goal of this work is to introduce a completely automatic method of facial expression recognition using geometric facial features alone. The features extracted from the region of the eyes, eyebrows, lips, etc. play a significant role in providing sufficient information to recognize the presence of any of six basic expressions happiness, sadness, disgust, anger, surprise and fear. We propose a classification method using Self-Organizing Map (SOM) [2, 3] to classify the features data into six basic facial expressions. SOM has an extra-ordinary ability to arrange the data in an order that maintains the topology of the input data. Applying SOM technology on input images, confusion matrix is generated which gives the information about all the basic expressions. We also proposed a method like single input and single output, by giving single image as an input we obtained a result of its specific expression which is observed. We also given the provision of using real time images as an input test image. 2 Related Works Facial expression analysis approaches can be broadly classified into three basic stages: face detection, facial features extraction, facial expression classification. For decades, researchers are working on human facial expression analysis and features extraction. Substantial efforts were made during this period [4].Major challenge was the automatic detection of facial features. Representation of visual information in order to reveal the subtle movement of facial muscles due to changes in expression is one of the vital issues. Several attempts were made to represent the 4537

3 visual informations accurately. Some of them are: optical flow analysis, local binary patterns (LBPs), level set, active appearance model (AAM), geometric analysis of facial features. The major drawback with model based methods like AAMs and ASM is that they need prior information about the shape features. Generally, during the training phase of AAM and ASM, the shape features are marked manually. Moore et al. found appearance based features by dividing the face image into subblocks. They used LBPs and variations of LBPs as texture descriptors. Gu et al. [5] used contours of the face and its components with a radial encoding strategy to recognize facial expansions. They applied self-organizing map to check the homogeneity of the encoded contours. Kobayashi and Hara modeled local facial features using geometric facial points. Zang et al. used geometric components of facial points along with multi-scale and multi-orientation Gabor wavelet coefficients computed from every pixel of facial images. Many techniques have been proposed for classification of facial expressions, such as multilayer perceptron (MLP) [6], radial basis function network (RBFN) [7], support vector machine (SVM) [1] and rule based classifiers [4]. 3 Region Of Interest Detection A rectangular portion of an image to perform some other operation and also to reduce the computational cost for further processing is known as region of interest (ROI).By applying the Viola-Jones face detector algorithm, the detected face region is cropped first then we divide the face area vertically into upper, middle and lower parts [8]. From the human frontal face structure concept, eyebrows & eyes, nose, and mouth areas are situated in upper, middle, and lower portions of the face image respectively. Again, the upper portion is partitioned horizontally into left and right segments for isolating right-eyebrow and right-eye and also left-eyebrow and left-eye, respectively. Finally, the smallest ROI regions are segmented for right-eyebrow, right-eye, lefteyebrow, left-eye, nose, and mouth in order to increase the detection rate. Fig.3.1 and Fig. 3.2(d) are shown the location and size of six ROIs and cropped images, respectively. Figure 3.1: Location and size of six ROIs of a face image such as (a) Right Eyebrow (Size:0.375W 0.12H), (b) Left Eyebrow (Size:0.375W 0.12H),(c) Right Eye (Size:0.375W 0.25H, (d)left Eye (Size:0.375W 0.25H, (e)nose(size:0.50w 0.19H), and (f) Mouth (Size:0.50W 0.16H) where, W=Image Width and H=Image Height. 4538

4 Figure 3.2 : Procedure of our proposed algorithm: (a) Input image, (b) Detected and cropped the face, (c) Face is divided into three vertical parts, which are indicated eyebrows, eyes, nose and mouth areas, (d) Six ROIs show the exact right-eyebrow, right-eye, left-eyebrow, left-eye, nose and mouth regions, (e) Applying CH method, all of the six ROIs are converted into new filtering images. 4 Automatic Facial Features Extraction Techniques The first and most crucial aspect of automatic facial expression recognition is the accurate detection of the face and prominent facial features, such as eyes, nose, eyebrows and lips. We used CascadeObjectDetector, a matlab function for the detection. CascadeObjectDetector Detect objects using the Viola-Jones algorithm DETECTOR = vision.cascadeobjectdetector creates a System object that detects objects using the Viola-Jones algorithm. The DETECTOR is capable of detecting a variety of objects, including faces and a person's upper body. The type of object to detect is controlled by the ClassificationModel property. By default, the DETECTOR is configured to detect faces. DETECTOR = vision.cascadeobjectdetector(model) creates a System object, DETECTOR, configured to detect objects defined by MODEL.MODEL is a string describing the type of object to detect. There are several valid MODEL strings. Examples include 'FrontalFaceCART', 'UpperBody', and 'ProfileFace'. DETECTOR = vision.cascadeobjectdetector(name,value) creates an object, DETECTOR, with the specified property Name set to the specified Value. You can specify additional name-value pair arguments in any order as (Name1,Value1,...,NameN,ValueN). Step method syntax: BBOX = step(detector,i) performs multi-scale object detection on the input image, I, and returns, BBOX, an M-by-4 matrix defining M bounding boxes containing the detected objects. Each row in BBOX is a four-element vector, [x y width height], that specifies the upper left corner and size of a bounding box in pixels. I must be a grayscale or truecolor (RGB) image. CascadeObjectDetector properties: ClassificationModel - Name of the classification model according to which part we have to detect MinSize - Size of the smallest object to detect 4539

5 MaxSize ScaleFactor MergeThreshold - Size of the biggest object to detect - Scaling for multi-resolution object detection - Threshold for merging collocated detections 5 Self Organizing Map(SOM) Self-organizing maps learn to cluster data based on similarity, topology, with a preference (but no guarantee) of assigning the same number of instances to each class. self-organizing map (SOM) [9] has an extra ordinary capability of clustering the data in an order that maintains the topology of input data. Because of this property of SOM, the features data of similar facial expressions (small changes in features) get clustered into closer zones. This in turn makes the classification much better. This property of SOM motivates us to use it for classifying the features data into basic expressions. From the ontological prospective, the emotion space may not be topologically related. 6 Experimental Results This section presents the results of features detection and classification of facial expressions into five basic emotions (happiness (H), sadness (Sa), disgust (D), anger (A), surprise (Sur), fear(f)) demonstrating the accuracy of the proposed methodologies. In our experiments, we used 81 different images. The directional displacement along x- and y-coordinate of each facial point is used as input feature for training the SOM. First the SOM is clustered to ordered zones. The clustered weights are further used to model the five basic emotions as a function of 26 directional displacement features data. The performance of the detection results is evaluated by comparing them against the ground truth (marked feature points).the fig.6.1 shows detection of different parts of frontal face image. Figure 6.1: Final result of detection The experimental results focuses on the determination of classification accuracy obtained on using proposed SOM based classification method. Table 6.1 demonstrates the classification accuracy when it is tested with new displacements data. The SOM classification accuracy is found to be 4540

6 highest (98.33%) for anger and least (88.7%) for sadness with average recognition rate as 93.53%.For the input image shown in figure 6.1,experiment gives the output as an anger. Table 6.1: Confusion matrix of emotion recognition for the 26 dimensional geometric features data using SOM. H Sa D A Sur F H Sa D A Sur F Conclusion A completely automated system for facial geometric features detection and facial expression classification is proposed. The extensive experiment illustrates that the effectiveness and accuracy of SOM based facial expression recognition system using only geometric features. References [1] W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, Face Recognition: A Literature Survey, ACM Computing Surveys, Vol. 35, No. 4, December [2] S. Wu, T.W. Chow, Clustering of the self-organizing map using a clustering validity index based on inter cluster and intra-cluster density, Pattern Recognit. 37 (2) (2004) [3] M.H. Khosravi, R. Safabakhsh, Human eye sclera detection and tracking using a modified timeadaptive self-organizing map, Pattern Recognit. 41 (August (8)) (2008) [4] F. Tsalakanidou, S. Malassiotis, Real-time 2dþ3d facial action and expression recognition, Pattern Recognit. 43 (5) (2010) [5] W. Gu, Y. Venkatesh, C. Xiang, A novel application of self-organizing network for facial expression recognition from radial encoded contours, Soft Comput. Fusion Found. Methodol. Appl. 14 (2) (2010) [6] Z. Zhang, Feature-based facial expression recognition: sensitivity analysis and experiments with a multilayer perceptron, Int. J. Pattern Recognit. Artif. Intell. 13 (1999) [7] M. Rosenblum, Y. Yacoob, L. Davis, Human expression recognition from motion using a radial basis function network architecture, IEEE Trans. Neural Netw. 7 (5) (1996) [8] P. Viola and M. J. Jones, Robust Real-time Object Detection, International Journal of Computer Vision, Vol. 57, No.2, p , [9] T. Kohonen, The self-organizing map, Proc. IEEE 78 (9) (1990)