Discovery Engineering

Transcription

1 Discovery Engineering An International Journal ISSN EISSN Discovery Publication. All Rights Reserved ANALYSIS Cognitive emotion recognition for elder care in homes Publication History Received: 03 November 2015 Accepted: 18 December 2015 Published: 1 January 2016 Citation Rajashree T, Ramya R. Cognitive emotion recognition for elder care in homes. Discovery Engineering, 2016, 4(11), Page70

2 Cognitive emotion recognition for elder care in homes Rajashree T 1 and Ramya R 2 1. ME CSE second year student ( batch), Kamaraj college of Engineering and Technology, Virudhunagar, India, iamrajashree.raji@gmail.com 2. Assistant Professor (Selection Grade) /CSE, Kamaraj college of Engineering and Technology, Virudhunagar, India ABSTRACT The elderly cognitive function decreased significantly due to the change in social roles and physical ability, which seriously affects the quality of their life and mental state. So we are proposing a system to understand emotions. The system is designed to determine the facial landmarks and to apply feature extraction methods for emotion recognition. For an input image, preprocessing is done to remove noise in the facial images, followed by face detection using Viola Jones technique. The different regions of interest are eye and nose. To reduce the computational complexity as well as the false detection rate, the coarse region of interest is extracted based on the position of facial landmarks. The active patches are positioned below the eyes, in between the eyebrows, around the nose and mouth corners. Based on the localization of left and right eye, the 19 active facial patches are detected. Feature extraction is done by Local Binary Pattern (LBP) algorithm. In Local Binary Pattern algorithm, the image is divided into equal blocks, pattern is extracted and the histogram is drawn for each block. Finally all the histograms are concatenated for processing. Principal Component Analysis (PCA) is a dimensionality reduction technique which is used to reduce the features and then they are classified by Multi-class SVM classification algorithm. Japanese Female Facial Expression (JAFFE) dataset is used to show the performance of the system. Keywords: cognitive, landmark points, viola Jones technique, active patches, multi class SVM 1. INTRODUCTION Automatic facial expression recognition systems have wide range of applications. Expression recognition systems will helps to create an interface between the human and the machine. Humans communicate effectively and are responsive to each other's emotional states. Systems are designed to gain this ability. Affective Computing is an area that focuses on Human-Computer Interaction to recognize the emotions of the elderly people. Facial Action Coding System (FACS) helps to recognize the human emotions. Facial Action Coding is a muscle-based approach that causes changes in the features of facial images. These changes in the face and the one or more muscles that caused these changes are called Action Units (AU). The movement of the feature points is used to understand and recognize facial expressions. The human face is an elastic muscle area that consists of organs, skins, and bones. When a muscle contracts, the transformation of the corresponding skin area attached to the muscle result in a certain type of facial expression. The task of facial landmark localization is aimed to find the accurate positions of the facial feature points. An expression is defined as any change in the face from a neutral display (i.e., no expression) to a non-neutral display (i.e., anger, fear, happy, etc) and back to a neutral display. The automation of the entire process of facial expression recognition is highly beneficial. Reliable expression recognition by machine is still a challenge. A key challenge is achieving facial expression recognition is optimal preprocessing, feature extraction, feature selection and classification, particularly under conditions of input data variability. To attain successful recognition performance, most current expression recognition approaches require some control over imaging conditions. The Facial Expression Recognition (FER) is the base for affective computing that helps to recognize the human expression effectively. Human facial expressions contain abundant information about the human behaviors. It is a non-verbal communication and thus systems are designed for the recognition of Page71

3 emotions of the people. The goal of recognizing emotion is the analysis of people s facial expressions from a facial image using standardized algorithms. Algorithms for face detection, feature extraction and facial expression recognition were integrated and extended to develop a facial expression analysis system that has the potential to be used in intelligent tutoring systems, smart homes, driver safety, etc. The primary reason for developing a new system is based on the perceived need for a facial emotion. 2. LITERATURE REVIEW The multi class SVM algorithm for classification in which the system was performed worst for anger expression and classification error was maximum between anger and sadness since they involve similar and subtle changes. With facial data being the information source for the facial expression recognition task, the processing complexity and expression recognition performance are strongly dependent on the data capture technique used. As more the number of patches used, more is the size of the feature vector. This increases the computational burden. The machine learning challenges are related to facial feature extraction and classification to achieve a high performance of the facial expression recognition. The classification method must be capable to define appropriate rules in order to derive a specific type of facial expression from the facial features provided, even when the output from the preceding processing stages, such as facial data acquisition and facial feature extraction, is noisy or incomplete (S L Happy, 2015). The Cognitive emotional model uses three feature extraction methods such as Gabor wavelet, Local Binary Pattern (LBP) algorithm and Euclidean distance. By applying Gabor wavelet in eight different orientations which causes more overhead in memory and lead to higher dimension (HAN Jing, 2015). The Hidden conditional Random Field (HCRF) models are limited by their independence suppositions which may reduce classification accuracy. One problem that can be associated with the use of global feature methods is the fact that they are very sensitive to variations in pose, illumination, occlusion, aging, and rotation changes of the face. These techniques are poor at handling data where classes do not follow the Gaussian distribution. Also, these techniques do not work well in case of a small sample size. Furthermore, complexitywise, most of these techniques are much expensive because of considering the entire face, as this requires more memory. Local Fisher Discriminant Analysis (LFDA) fails to determine the essential assorted structure when face image space is highly nonlinear. The performance of this approach also degrades with variation in illumination. HCRF utilizes diagonal covariance Gaussian distributions in the feature function and does not guarantee the convergence of its parameters to some specific values at which the conditional probability is modeled as a mixture of normal density functions. Because of this property, the existing HCRF losses a lot of information. This is one of the main disadvantages of the HCRF model. Lack of accuracy can be attributed to various causes, such as the failure to extract prominent features, and the high similarity among different facial expressions that results due to the presence of low between-class variance in the feature space (Muhammad Hameed, 2015). Spontaneous facial behavior analysis can be very challenging due to several factors, such as out-of-plane head motion and different poses, subtle facial expressions and intra-subject variability in dynamics and timing of different facial actions. In addition it has been shown that the dynamics and patterns of spontaneous facial expressions can be very different from the posed ones. Analyzing spontaneous facial expressions, especially for intensity measurement, is not as robust and accurate as the posed one, because of the challenging factors. Results demonstrate that recognizing spontaneous expressions is more challenging than posed expressions. To model the dependencies among AUs and achieved improvement over single image-driven methods especially for recognizing Aus that is difficult to detect but have strong relationships with other action units. Dynamic nature of facial actions, individually recognizing each action unit intensity is not accurate and reliable for spontaneous facial expressions. These works are all limited to detection of the presence and absence of AUs mainly in posed expressions (Yongqiang Li, 2015). A statistical method for automatic facial landmark localization is used. Statistical models can fail if the variance captured during training is not rich enough to generalize to new test settings. Training such a system is more difficult and costly than training a land marker. However, both the appearance and the structure are Page72

4 changed under expression variations and in different ways. The joint estimation of landmarks (e.g., in AAM approaches) will be problematic. Closely placed landmark points are difficult to separate with statistical landmark classification methods, as the models need to deal with idiosyncratic variations and noise and to thus be sufficiently general in admitting a landmark (Hamdi Dibeklio glu, 2012). Most of the facial representations use a set of features from the original images. Different features have been applied on the entire whole face or specific regions of the face to extract the facial features (Lin Zhong, 2012). 3. METHODOLOGY It describes the architecture for the proposed system INPUT IMAGE PREPROCESSING IDENTIFICATION OF EYE, NOSE REGIONS SALIENT PATCHES IDENTIFICATION FEATURE EXTRACTION FEATURE SELECTION MULTI CLASS SVM CLASSIFICATION RECOGNIZE EMOTION Figure 1 Proposed Architecture female images and it is a gray scale image. The size of the JAFFE dataset is 256x256 pixels. Figure 2 JAFFE Dataset 3.2. Preprocessing The first step in optimizing the data is preprocessing. Preprocessing involve operations that are normally required prior to the main data analysis and extraction of information. The main purpose of applying preprocessing is to reduce the amount of errors or inconsistencies in image brightness values that may affect the ability to analyze images. Image preprocessing will increase the reliability of an image. The proposed method uses Gaussian filter of 3x3 mask followed by face detection using viola-jones technique as the preprocessing technique. It is a low pass filter that removes high-frequency components from the image. Convolution of Gaussian with self is another Gaussian. The sum of all the Gaussian mask values lies between 0 to 1 which represents only the positive values. By applying Gaussian mask to an image reduces the contents of the edge to be processed. It minimizes the ringing effect problem which means that the transition between one color to another cannot be predefined precisely. It is also called as Gaussian blur. The 2D Gaussian can be expressed as the product of two functions, one a function of x and other a function of y. The two functions are calculated as follows 3.1. Input Image The input image is taken from widely used facial expression database, Japanese Female Facial Expression (JAFFE). The JAFFE dataset consists of 211 images in total which includes anger, disgust, fear, happy, sad and surprise and neutral expressions which are in Tagged Image File Format (TIFF). This dataset consists of only G σ (x, y) = 1/2πσ 2 exp -x2 + y 2 /2σ 2 (1) The advantage of applying Gaussian filter to an image includes many advantages such that it is rotationally symmetric for large size filters. The filter weight decrease from central peak and give most weight to central pixels. It is simple and it shows the relationship between size of sigma and the smoothing. It is separable that reduces to 2k operations per pixel. Page73

5 It smoothens the image with a small Gaussian and the result is smoother the original image with a large Gaussian. This filter is a discrete filter and that can be used for approximation. Convolution with an image averages the pixels in the image that decrease the difference in value between neighboring pixels. The viola Jones technique turns input image into integral image by making each pixel equal to the entire sum of all pixels. It analyzes a given sub window using features consisting of two or more rectangles. The feature results in single value, calculated by subtracting the sum of the white rectangles from the sum of the black rectangles. A detector of 24x24 pixels is used to detect the faces. The localized face was extracted and scaled to bring it to a common resolution. This made the algorithm shift invariant, i.e., insensitive to the location of the face on image. The localized face is brought to a common resolution of 96x96. Histogram equalization was carried out for lighting corrections. The main purpose of histogram equalization is to enhance the contrast of images by transforming the values in an intensity image or color map. Histogram of output image matches a specified histogram and it is approximately flat. time with a new size of the detector. It should discard non-faces faster, instead of finding faces. The detector returns the M-by-4 matrix in which M represents the number of bounding boxes and these boxes include the detected objects. The detector detect objects within rectangular search region specified by the Region of Interest (ROI).The output matrix of the detector is a four element vector that includes x, y, width and height of the ROI in pixels. Figure 4 Detection of eye and nose Then the landmark points are identified by finding the mean of the coordinates. This is performed for both the left and right eyes separately. Similar to eyes, nose land mark points are detected. Plotting a plus mark point shows the localization point. Figure 5 Eye and Nose localization Figure 3 Preprocessing 3.3. Identification of Eye and Nose regions Active patches are positioned below the eyes, in between the eyebrows, around the nose and mouth corners. To extract these patches from face image, we need to locate the facial components first Eye and Nose Detection To reduce the computational complexity as well as the false detection rate, the coarse region of interests (ROI) for eyes and nose were selected. Both the eyes were detected separately using Haar classifiers trained for each eye. It considers rectangular regions at a specific location in a detection window. The cascade classifier scans the detector many times through the same image and each 3.4. Salient Facial Patches Identification The active facial patches usually lie around eyes, lips, eyebrows, and nose. The use of whole face for expression analysis is a drawback since most of the facial regions do not take part in producing facial expressions. Therefore, we selected the face sub regions that undergo a change during different expressions and call it as active facial patches. To locate the active patches, we need to localize some of the facial landmarks such as eyes, nose, and lips corners accurately. Based on the position of left and right eye localization, the 19 active facial patches are extracted. These 19 active facial patches are responsible for facial expression recognition. The locations of these patches are highly related to expression types. The first five patches were found from the left side of the facial image. The first two patches were below the eye region. Based on the Page74

6 two patches found, next three patches were found from the facial image. It constitutes the first five patches. Similarly next five patches were found from the right side of the facial image. Thus total of ten patches were found from the eye regions. Next between the left and right eye region and just above the nose then next patch is found and then subsequent patches were found from the eyebrow region. It constitutes of four patches. Then in the mouth region five patches were found. This is done by patch which is found from below the eye region. These patches were found across the corner of the mouth regions. On the left side two patches were found and similarly on the right side another two patches were found. Then the last patch is found below the mouth region. Finally 19 active facial patches were found from the facial images. These 19 active facial patches were responsible for recognition of emotion of the people. It will helpful for feature extraction process and it will lead to dimensionality reduction process followed by feature classification. Figure 6 19 active facial patches 3.5. Feature Extraction Feature extraction is the process of defining a set of features which will meaningfully represent the information that is important for analysis and classification. The extracted features are expected to contain the relevant information from the input data. The goal of feature extraction is to increase the effectiveness and efficiency of analysis and classification. It helps to know about variability in the image data Feature Extraction by Local Binary Pattern (LBP) Algorithm The basic idea is to summarize the local structure in an image by comparing each pixel with its neighborhood. Take a pixel as center and threshold its neighbors against. If the intensity of the center pixel is greater or equal to its neighbor, then denote it with 1 and 0 if not. With 8 surrounding pixels and it will end up with 2^8 possible combinations, which are called Local Binary Patterns or sometimes abbreviated as LBP codes. The Local Binary Patterns algorithm has its roots in 2D texture analysis. The length of the feature vector for a 3x3 window is calculated as follows LBP(x, y) = Σ 7 n=0 s(i n - i c )2 n (2) Where i n represents the nearest pixels and i c represents the center pixel and s(x) = {1, x>=0; 0,0therwise} (3) Local Binary Pattern method provides very good results, both in terms of speed and discrimination performance. This method is robust against face images with different facial expressions, different lightening conditions, image rotation and aging of persons. It provides various benefits such as that it saves memory. The second benefit is that LBP of uniform pattern detects only the important region of interest. The histograms of these patterns are also called labels that form a feature vector, and are thus a representation for the texture of the image. It also seems that much of the pixels around the mouth, the nose and the eyes have uniform patterns. The local binary pattern operator is an image operator which transforms an image into an array or image of integer labels describing small-scale appearance of the image Feature Selection Feature subset selection is the process of removing the irrelevant and redundant information in the extracted feature. This reduces the dimensionality of the data and may allow classification algorithms to operate faster and perform more effectively. In pattern recognition, feature selection can have an impact on accuracy and complexity of the classifier Feature Selection by Principal Component Analysis Principal component analysis is a dimensionality reduction technique. It is useful when it have a large number of variables and that there exist some redundancy in those variables. The redundancy refers that some of the variables in the extracted feature are correlated with one another, and they are measuring the same construct. Because of this redundancy, it will reduce the observed variables into a smaller Page75

7 number of principal components that will account for most of the variance in the observed variables. A principal component is defined as a linear combination of optimally weighted observed variables. The linear combination of variables refers the scores of a component and is created by adding together scores on the observed variables being analyzed. Optimally weighted refers that the observed variables are weighted in such that the resulting components account for a maximal amount of variance in the extracted feature. The first component extracted in a principal component analysis accounts for a maximal amount of total variance in the observed variables. This means that the first component will be correlated with at least some of the observed variables. The second component will be uncorrelated with the first component. In principal component analysis, the total eigen values of the correlation matrix is equal to the total number of variables being analyzed. The purpose of principal component analysis is to reduce a number of observed variables into a relatively smaller number of components. PCA is a statistical procedure concerned with elucidating the covariance structure of a set of variables. It identifies the principal directions in which the data varies. PCA gives us a way of reducing the dimensionality of the extracted feature. The principal components are found by calculating the eigen vectors and eigen values of the data covariance matrix. This process is equivalent to finding the axis system in which the covariance matrix is diagonal. The eigen vector with the largest eigen value is the direction of greatest variation Feature Classification Classification is the process of identifying a set of categories in which a new observation belongs on the basis of a training set of data containing observations whose category label is known. This refers the supervised learning where a training set of correctly identified observations is available in the dataset. The set of known labels is called the training set because it is used by the classification programs to learn how to classify input images. In the training phase, the training set is used to decide how the parameters ought to be weighted and combined in order to separate the various classes of output labels Multi Class SVM Classification Algorithm Support Vector Machine (SVM) is based on determining the location of decision boundaries that produce the optimal separation of classes. The selected decision boundary will be one that finds the greatest margin separation between the two classes. Maximum margin plane is defined as the sum of the distances to the hyper plane from the closest points of the two classes. The data points that are closest to the hyper plane are used to measure the margin and hence these data points are termed support vectors. SVMs was developed to perform binary classification that class labels can only take two values and they are + or -1. For multi-class problems determine n hyper planes. Each classifier being trained on only two out of n classes, giving a total of n (n- 1)/2 classifiers. Applying each classifier to the test data vectors gives one vote to the winning class. Then the data is assigned the label of the class with most votes. In one against one method, SVM classifiers for all possible pairs of classes are created. Therefore, for M classes, there will be binary classifiers. The output from each classifier is in the form of a class label is obtained. The class label that occurs the most is assigned to that point in the data vector. The Sequential Minimal Optimization (SMO) algorithm in weka shows the support vector machine performance, in which it provides an efficient way of solving the duality problem that arises from the calculation of support vector machine. It is a quadratic programming used for training of support vector machines. SMO is an iterative algorithm for solving the optimization problem. It solves these problems into a number of smaller sub problems that uses linear equality constraint involving Lagrangian multipliers. This algorithm finds the Lagrangian multiplier that violates the conditions for the optimization problem. Then select second Lagrangian multiplier that optimizes the pair of first and second Lagrangian multipliers. Pair wise coupling is used to improve the performance of the system. It requires a binary classifier for each possible pair of classes. One of the classifier helps to take the decision function of the SVM classifier for positive and negative points. In one against all strategy, it constructs large number of binary classifiers each of which Page76

8 separates one class from the rest. The SVM is trained with all training samples of i th class with positive labels and the others with negative labels. The separating hyper plane in a feature space is a polynomial of degree 3 polynomial in input space. The kernel function used for support vector machine is normalized polynomial kernel. It is mainly used in support vector machine model that represent the similarity of vectors in a feature space over polynomials of the original variables that learns the non linear models Emotion Recognition The system can correctly recognize the emotions of the people. The output is produced in the form of confusion matrix. It will show the accuracy of the system. The output labels are of basic emotions like happy, sad, fear, anger, disgust, neutral, surprise, etc. A confusion matrix also known as a contingency table or an error matrix is a specific table layout that allows visualization of the performance of an algorithm. Each column of the matrix represents the instances in a predicted class while each row represents the instances in an actual class. 4. RESULTS The proposed system is tested on Japanese Female Facial Expression (JAFFE) dataset. It shows the accuracy of %. The dataset contains all basic expressions. The input image is classified by the multi class support vector machine algorithm. Among all the images in the JAFFE dataset, 70% of the image is taken for training and the rest of 30% of image is taken for testing. The training and testing of the dataset is calculated by support vector machine algorithm. The performance of the system is tested on weka tool. It uses one against all strategy in which it compares its output label with all other output labels and the final output label is assigned. The training data for each such classifier is the subset of the available training data and it will only contain data for the two involved classes. The data will be labeled as positive and negative points. This classifier can be combined with a voting scheme to give final classification results. The voting scheme needs a pair wise probability. The polynomial kernel is used to convert a linear learner into a non linear learner. At training the SVM classifier it finds the hyper plane in a feature space generated by the kernel. The hyper plane has a maximum margin in feature space with minimal training error. Kernel in support vector machine is used as a similarity measure. The features in the same class have high kernel value and the feature in the different classes have low kernel value. Normalization can be applied on the kernel function for selecting the relative weight of implicit features. The support vector machine shows good performance and has various applications in bioinformatics, image recognition, etc. The accuracy of the system relies on the classification algorithm used. The maximum discrimination has been shown by the support vector machine algorithm. The various output labels are of basic emotions like anger, disgust, fear, happy, neutral, sad, surprise. Table 1 Confusion Matrix Anger Disgust Fear Happy Neutral Sad Surprise Anger Disgust Fear Happy Neutral Sad Surprise The proposed system shows true positive rate of 96.8% and false positive rate of 5%. Both determine the corrected and wrongly classified images. The precision rate of the proposed system is 97.4% and a recall of 96.8%. The proposed system provides misclassification between happy and anger expressions. The 20% of the image is produced as the result of misclassification. If the normalized polynomial kernel is set to 2, the accuracy of the proposed system is %. It shows true positive rate of 82.5%, false positive rate of 2.9%, recall of 82.5% and a precision of 92.1%. It shows misclassification between anger and with all other expressions. The proposed work is of significant importance and has a large influence on the development of modern day automatic facial expression recognizers with the advances in the fields of robotics, computer graphics and computer vision, animators and computer scientists. This led to the development of robust face detection and face tracking algorithms. Page77

9 All of these factors led to a renewed interest in the development of automatic facial expression recognition systems. A facial expression is a visible manifestation of the affective state, cognitive activity, intention, personality, and psychopathology of a person. It plays a communicative role in interpersonal relations. The work involves analyzing the facial features and/or the changes in the appearance of facial features and classifying this information into some facial expression and interpretative categories such as facial muscle activations like smile or frown, emotion categories like happiness or anger, attitude categories like (dis)liking or ambivalence, etc. Automated analysis of nonverbal behavior, and especially of facial behavior, has attracted increasing attention in computer vision, pattern recognition, and human-computer interaction. This system is useful for medical applications and it is important for emotion recognition purposes. The result is expressed in the form of confusion matrix. It shows the accuracy of the proposed system. Automatic analysis of facial expressions forms the essence of numerous next generation computing tools including affective computing technologies, learner-adaptive tutoring systems, patient profiled personal wellness technologies, etc. REFERENCES 6. L. Zhong, Q. Liu, P. Yang, B. Liu, J. Huang, and D. N. Metaxas.Learning active facial patches for expression analysis, in Proc. IEEE Conf. Comput.Vis. Pattern Recog., 2012, pp Facial Expression Recognition - Wikipedia, the free encyclopedia.htm. 8.Action units Wikipedia, the free encyclopedia.htm. 9. Sequential minimal optimization Wikipedia, the free encyclopedia.htm. 1. S L Happy and Aurobinda Routray. Automatic Facial Expression Recognition Using Features of Salient Facial Patches. IEEE Transactions On Affective Computing, Vol. 6, No. 1, January-March HAN Jing, XIE Lun, LI Dan, HE Zhijie, WANG Zhiliang. China Communications April Muhammad Hameed Siddiqi, Rahman Ali, Adil Mehmood Khan, Young-Tack Park and Sung young Lee. IEEE Transactions On Image Processing, Yong qiang Li, S.Mohammad Mavadati, Mohammad H, Mahoor Yongping Zhao, Qiang Ji. Elsevier, Hamdi Dibeklio glu, Albert Ali Salah and Theo Gevers. A Statistical Method for 2-D Facial Landmarking. IEEE Transactions on Image Processing, Vol. 21, No. 2, February 2012 Page78