AUTOMATIC FACIAL EXPRESSION RELATED EMOTION RECOGNITION USING MACHINE LEARNING TECHNIQUES

Transcription

1 International Journal of Computer Engineering & Technology (IJCET) Volume 8, Issue 5, Sep-Oct 2017, pp , Article ID: IJCET_08_05_014 Available online at Journal Impact Factor (2016): (Calculated by GISI) ISSN Print: and ISSN Online: IAEME Publication AUTOMATIC FACIAL EXPRESSION RELATED EMOTION RECOGNITION USING MACHINE LEARNING TECHNIQUES V. Sathya Research Scholar, A.V.V.M.Sri Pushpam College, Poondi, Tamilnadu, India T.Chakravarthy Associate Professor, Department of Computer Science, A.V.V.M Sri Pushpam College, Poondi, Tamilnadu, India ABSTRACT Facial expression are commonly used in everyday human communication for express the emotions. Emotions are reflected on the face, hand, body gesture and voice to express our feelings. In human communication, the facial expression is understanding of emotions help to achieve mutual sympathy. It is a nonverbal communications. Computer vision based technology is placed an important role in various applications especially in human emotion recognition process because emotions are related to the peoples mental ability and thinking process[1]. More ever, one single emotions leads to create the difficult health problems. Peoples affected by single emotions due to their stress, over thinking, personal problems and so on. So, their mental ability need to be maintained continoulsy for avoiding their health issues which is done by linking the emotion recognition system with computer vision area that effectively utilize the intelligent techniques [2]. The intelligent techniques analyze the human emotions from different parameters such as facial expression had electroencephalogram (EEG) brain activities with successful way. Among the parameters, facial expression based emotion recognition process is one of the easiest method because it does not require high cost, easy to capture the face expression [3] with the help of the digital camera, minimize the computation complexity also the impact of the facial expression is related with the brain activities and social impacts. There are there are 100 types of facial expressions such as blinking, cheerless, coy, blithe, deadpan, brooding, glowering, faint, grave, dejected, derisive, leering, moody, hopeless, slack-jawed and so on. These facial expressions are derived from the basic expressions such as Happy, Sad, Anger, Disgust, Surprise, Fear and Neutral. Keywords: facial expression, emotion recognition, the non-local median filtering, neural networks, hidden markov model editor@iaeme.com

2 Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques Cite this Article: V. Sathya and T.Chakravarthy, Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques, International Journal of Computer Engineering & Technology, 8(5), 2017, pp INTRODUCTION Facial expression is one of the important computer vision based process that helps to detect the human emotions, feelings and mental ability in different situations. To detecting the emotions, the well-defined automatic facial emotion recognition system has been developed in the recent years because [7] it has several attraction in the mental ability detection process. from the literature of the different authors opinions, facial epression related images [8] are used to detect their emotions with accurate manner. Even though the facial expression images are consume low time, cost, minimize the complexity, some times of the facial points are difficult to detect with accurate manner. In addition to this, the detected facial points are fails to classify the exact emotions using the traditional classification techniques. Hence earlier classification techniques only uses the particular features and develop a classifier for emotion recognition process. Therefore a new novel combination of techniques has been proposed to remove the noise, affected region segmentation, feature extraction and the good classification techniques which improve the classification performance. Here the major objective is to recognize the facial expression using the effective classification techniques. 2. PROPOSED WORK In this Proposed work, Jaffe and Cohn Kanade database is used to recognize the facial expression related emotion using different classifiers. Both the dataset consists of collections of images which was captured in various emtoions that used to detect the emotions in automatic way. The database images are used to for both training and testing process which is done by utilizing the different image processing techniques. Then the sample captured database image is shown in the figure 2. Figure 1 Sample Database Facial Expression image By using the above images the facial expressions has been classified by applying the proposed methods which is shown in the figure 3. Initially the face expression images has been captured and the facial point is detected with the help of the geometric method and the unwanted noise is removed by using the Non local median filter. Then the feature points are extracted by using the local binary pattern and progression invariant sub space learning method editor@iaeme.com

3 V. Sathya and T.Chakravarthy Optimal features are selected using the the Particle Swarm Optimization Process (PSO) Extracted features are trained by back propagation neural networks (BPNN). Finally the emotion recognition is done with the help of the different classifiers like Hidden Markov Model (HMM), Support Vector Machine (SVM) and Back Propagation Neural Networks (BPNN) 2.1 Noise Removal Initially the facial expression images are captured by digital camera, the geometric facial points [9] are detected as follows, = (ln) (1) In the above eqn (1), h. The facial points are detected with minimum delay,. After detecting the facial points, the color of the images are changed, if the captured image is color image. The color transformations is done by as follows, = !!"() !!" () !!" (() (2) The color transmitted images may contains several noise that reduces the emotion recognition system. So, the noise present in the image is eliminated using the non-local median filter [10]. First intensity of the images are estimated as follows, )() = *() +!() (3) Where)() is defined as the observed value from the given image, *() is defined as the true value and!() is defined as the noise agitation at a pixel. Then the noise influenced by the images are examined, then the images are mostly affected by Gaussian noise that is eliminated with the help of the following assumptions,!() areindependent, identicallydistributed Gaussian values with variance + 2 and zero mean. Based on the assumptions, the neighborhood pixel value is estimated with the help of the weighted values,(,.1)!,(,.2).according to the above process, each pixel,is investigated and the non-local median filter is estimated as follows, /(0)() =,(,.)0(.) 234 (4) Where 0 is defined as the noisy image, and weights,(,.) meet the subsequent conditions 0,(,.) 1 and 2,(,.) = 1. After estimating the non-local value, the similarity value between the neighborhood values is calculated as follows, (,.) = 0( 7 ) 0( 2 ),9[1,2] (5) Where < is defined as the neighborhood filter employed to the neighborhood s squared difference. The weights is defined as follows,(,.) = (BC>CDC(E,F) >?@A G (6) =(7) Where+ is as defined as the standard deviation of the noise and 2+ are set to 1. Where H() is defined as the normalizing constant is defined as follows H() = IJ(7,2) 2 [1,2] (7) K Where h is defined as the weight-decay control parameter. As earlier mentioned, < is known as the neighborhood filter with L MNO. The weights of < are computed is as follows < = P QRS 1/(2 1) P QRS NWO (8) Where is defined as the distance the weight is from the neighborhood filter s center. This process is repeated until to eliminate the noise from the image with effective manner editor@iaeme.com

4 Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques After detecting the face using the geometric approach, different features are derived this is done with the help of local binary pattern and progression invariant sub space learning method. Figure 3 Proposed System Architecture 2.2 Feature Extraction The next step is feature extraction which is done by using the local binary pattern and progression invariant sub space learning method [11]. First the local binary pattern process is applied to the image, that analyze each and every pixel present in the image and the particular operator is assigned to each pixel. After that the threshold value is examined by using the 3*3 neiughboring pixel image. According to the process the general local binary pattern representation is shown in the following figure 3. Figure 3 Local Binary Pattern Representation From the detected pixels and threshold value, the image has been represented using the circle of center. Based on the above neighboring pixel representation, the facial texture features are derived by combining the local descriptors with global descriptors because it manages the variations and illuminations also it effectively examines the ordinary features with effective manner. The method examines the features in different directions, rotations and relationship between the pixels or key points by dividing the captured face image as follows, editor@iaeme.com

5 V. Sathya and T.Chakravarthy Figure 4 Different Divisions of Face Images After segmenting the different regions, the features has been estimated using the maximum and minimum ccorner information which is done as follows, X(,",+) = /(,",Y N +) /(,",Y Z +) (9) Where X(,",+)the difference of the Gaussian image is, /(,",Y+) is the convolution value of the image, (,") is the Gaussian blur value, /(,",Y+) = (,",+) (,") (10) According to the above process, the facial key point features have been detected using the Taylor series as follows, X() = X + [\] + _ [C \ (11) [^ [^C Then the orientation has been assigned as follows, which is used to identify the direction of the particular key point is measured by the magnitude and orientation estimation. (,") = `a/( + 1,") /( 1,")b + a/(," + 1) /(," 1)b (12) c(,") =!2a/(," + 1) /(," 1)b,a/( + 1,") /( 1,")b (13) Where, (,") =!* d h ", c(,") =!! h "! Based on the above process, the key point features are derived in different orientation using the 4*4 histogram orientation process that consists of 16*16 region of the key point which has 8 bins and 28 elements. The extracted elements are normalized with the help of the threshold value 0.2. According to the histogram and threshold value, different features such as nose, mouse, eyes and eye brows are derived from face images. The extracted features consist of lot of information which is difficult to process, so, the optimized features are selected for making the system so effective. 2.3 Feature Selection The next step is feature selection which is done by using the particle swarm optimization method. The PSO [12] method analyze extracted features and the best solution has been detected which is relevant to the human emotions. In the search space, each feature treated as the particle and the position, velocity of the feature is estimated because the features are moved in the searching space while examining the optimal features. Based on the above process, global features are selected from the feature space that helps to detect the facial emotions with effective manner. 2.4 Feature Training and classification The extracted features are trained by using the back propagation neural networks (BPNN) because it effectively train the features that helps to detect the new facial expression related editor@iaeme.com

6 Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques facial points. The BPNN network is one of the supervised learning method but in this work, it treated as the unsupervised learning concept which computes the activation value of each selected features. The networks has three layers namely, input, hidden and output layer each layer has particular weights and bias value. During the training process the network use 70 input nodes,1 hidden nodes and 70 output node with mean square error function as the training function. Initially the activation value of the layer has been computed as follows, ef)! )g* = h N, NZ (14) In eqn (14), h N h!* )g* d h!*!, NZ h,h )g* d h! By using the activation value, the minimum activation is saved as the index pair and the output value 1 is assigned to the maximum activation value else the output is assigned as 0. Thus the output value of each neuron weighted value is estimated as follows,, NZ (!,) =, NZ (g) + f i N +, NZ (g)jk Z (15) In eqn (15),, NZ (!,) *,h )g*, NZ (g) g,h )g* These weight updating process helps to minimize the error value while train the features. The trained features are stored as template in the database for further emotion recognition process in the testing stage. These trained features are classified by using different classifiers like Hidden Markov Model (HMM), Support Vector Machine (SVM) and Back Propagation Neural Networks (BPNN) Hidden Markov Model (HMM) The first classifier is Hidden Markov Model (HMM), the method uses the tested features which is driven from the image preprocessing, feature extraction and feature selection process that is discussed in the section 2.1 to 2.4. The tested features are compared with the trained features which is discussed in the section 2.5. This model works according to the statistical Bayesian network approach which utilizes the probability value. Then the probability value of each feature is computed as follows, l(k) = ^ l(k h)l(h) (16) In the above eqn (16), the P(Y) is the probability value of the testing feature sequence which is compared with the trained features. Based on the comparison process the human emotions are effectively recognized. Support Vector Machine (SVM) The tested features are classified using the support vector machine (SVM). This classifier is statistical method which classifies the features using the hyper plane that reduces the misclassification data. Let x and y are the input and related output class, then the hyper plane has been chosen to divide the output class labels y{1,-1} and the hyper plane is,,. + ( = 0 (17) Then, " N (,. + () 1,h = 1,2,3,. (18) The hyper plane should be separable the data and minimize the difference between data. Then the difference between the hyper plane is calculated as follows, p + I = q (19) editor@iaeme.com

7 V. Sathya and T.Chakravarthy Based on the train data, the new entered features are matched with the template present in the hyper plane as follows, r X!f = h r NW N k N (20) Where h N is the given iris template and k N is the stored template in the database. Based on the distance the templates are classified into the emotions. Back Propagation Neural Networks (BPNN) The last classifier is back propagation neural networks (BPNN) which is one of supervised neural network. The network has three layers such as input, hidden and output layer. Each layer uses the tested features and it has been passed to the hidden layers and the output is estimated as follows, r ** = NW N, N + ( (21) During the output estimation process, the network uses the radial basis activation function that reduces the error rate while computing the facial emotions. In addition to this the weights and bias values are continoulsy updated for minimizing the mis-classification data. Thus the mentioned classification methods such as Hidden Markov Model (HMM), Support Vector Machine (SVM) and Back Propagation Neural Networks (BPNN) successfully recognize the facial emotions. Then the efficiency of the system is analyzed using the experimental results 3. EXPERIMENTAL RESULTS AND DISCUSSION This section describes the performance evaluation of the methods described in the proposed system. During the efficiency estimation process, the automatic expression system uses two databases such as Jaffe and Cohn Kanade, the noise present in the images are eliminated and different local binary features are extracted which helps to detect the feature points such as eyes, eye brows, mouth and nose. Then the detected features are shown in the figure 5. Figure 5 Face Expression Steps and Relevant outputs editor@iaeme.com

8 Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques The assessment of these methods is done in terms of accuracy, specificity and the sensitivity. These three assessment terms are specified in the following forms. eff*f" = (tl + t) (tl + t + <l + <) 100% fdf" = t / (t + <) 100%!)" = tl / (tl + <) 100% Where, TP (True Positives) = correctly classified positive cases, FP (False Positives) = incorrectly classified negative cases, TN (True Negative) = correctly classified negative cases, FN (False Negative) = incorrectly classified positive cases. The face expression has been recognized by HMM, SVM and BPNN which reduces the reduces the error rate while classify the face exprressions. The minimize error rate is increase the classification accuracy. Then the performance of the proposed system error rate is shown in the figure 6. Figure 6 Performance of Means Sqaure Error Rate The reduced error rate leads to increases the overall efficiency of the system which is examined by using different emotions such as happy, sad and anger from differnet regions and the obtained results are shown in the following figure 7. Figure 7 Accuracy of Different Classifiers The above figure 7 shows that the three classifers successfully classifies the extracted features with highest accuracy. Overall the obtained accuaracy value is shown in the figure editor@iaeme.com

9 V. Sathya and T.Chakravarthy Figure 8 Overall Accuracy of the classifier From the above discussions the face expression has been classified using Hidden Markov Model (HMM) with 98.3%, the Support vector Machine (SVM) classifies with 99.64% and Back propogation Neural Networks ensures 99.87% when compared to other existing methods. This makes the additional advantage to the proposed system and acts as a medical image analysis device for the medical experts to classify the emotions witheffective manner. 4. CONCLUSION This paper examining the effectiveness of the proposed back propagation neural networks with hidden markov model based face emotion recognition process using the different face database such as Jaffe and Cohn Kanade. thesis, facial expression related emotions recognition system is done by using the Hidden Markov Model (HMM), Support Vector Machine (SVM) and Back Propagation Neural Networks (BPNN). The captured face digital images, geometric facial points are detected and the noise present in the images areeliminatd by using the non-local median fitler. From the noise free image, facial features are extracted by segmenting the images into the local binary patterns and the key points are detected in differnet directions and locations using progression invariant sub space learning method. From the extracted features, optimized global features are selected using the particle swarm optimization method. The extracted features are trained by back propagation neural networks and the classification is done by proposed classifiers. The performance of the proposed system is analyzed by using the Jaffe and Cohn KanadeDataset which consumes the minimum error rate. These reduced error rates increase the classification accuracy when compared to previous work and shows that the proposed classification method brings the result with more sensitivity and accuracy. REFERENCE [1] ShrutiBansal, Pravin Nagar, Emotion Recognition From FacialExpression Based On Bezier Curve, International Journal of Advanced Information Technology, volume5, number 3. [2] SpirosV.Ioannou,.Amaryllis T.Raouzaiou, VasilisA.Tzouvaras, TheofilosP.Maili,.Kostas C.Karpouzis, StefanosD.Kollias, Emotion recognition through facial expression analysis based on a neurofuzzy network, Neural Networks, Volume 18, Issue 4, May 2005, Pages [3] Ekman, P., Friesen, W. V: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. Prentice-Hall, New Jersey (1975) [4] ArunaChakrabortyAmitKonar, Fuzzy Models for Facial Expression-Based Emotion Recognition and Control, Emotional Intelligence in elesviewerpp editor@iaeme.com

10 Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques [5] Ludmila I Kuncheva and William J Faithfull, PCA Feature Extraction for Change Detection in Multidimensional Unlabelled Streaming Data, International Conference on Pattern Recognition (ICPR 2012), November 11-15, [6] Jia-FengYu,Yue-Dong Yang, Xiao Sun, and Ji-Hua Wang, Sequence and Structure Analysis of Biological Molecules Based on Computational Methods, BioMed Research International, Volume 2015, [7] Tallapragada, Rajan, Improved kernel-based IRIS recognition system in the framework of support vector machine and hidden markov model, IET Image Processsing in IEEE, volume 6, [8] Tallapragada, Rajan, Improved kernel-based IRIS recognition system in the framework of support vector machine and hidden markov model, IET Image Processsing in IEEE, volume 6, [9] Hai Nguyen, KatrinFranke, and Slobodan Petrovic, Optimizing a class of feature selection measures, Proceedings of the NIPS 2009 Workshop on Discrete Optimization in Machine Learning: Submodularity, Sparsity &Polyhedra (DISCML), Vancouver, Canada, December [10] Hongjun Li, Ching Y. Suen, A novel Non-local means image denoising method based on grey theory, JournalPattern Recognition in ACM, [11] Ying-li Tian, Takeo Kanade, and Jeffrey F. Cohn, RecognizingAction Units for Facial Expression Analysis, IEEETransactions on Pattern Analysis and Machine Intelligence,23(2), 2001, [12] Tallapragada, Rajan, Improved kernel-based IRIS recognition system in the framework of support vector machine and hidden markov model, IET Image Processsing in IEEE, volume 6, editor@iaeme.com