Facial Expression Analysis

Size: px

Start display at page:

Download "Facial Expression Analysis"

Darleen Booth
5 years ago
Views:

1 Facial Expression Analysis

2 Faces are special Face perception may be the most developed visual perceptual skill in humans. Infants prefer to look at faces from shortly after birth (Morton and Johnson 1991). Most people spend more time looking at faces than at any other type of object. We seem to have the capacity to perceive the unique identity of a virtually unlimited number of different faces

3 Understanding Emotion: from facial expressions Facial expressions as a communicative tool We laugh more if in a group/ show distress more if in a group Babies (10 months) almost only smile in presence of caregiver Babies look to caregiver and behave according to caregiver response when encountering novel object. E.g. a barking dog or a snake This is known as social referencing and is also seen in chimpanzee societies A similar process, observational fear, is seen in other monkeys. Infant monkeys show fearful unconditioned response to mother s expression of fear when the mother could see a snake, but the infants could not. That is, infants showed a fear response to the mother s fear response.

4 facial expressions as communication Facial expressions allow for rapid communication They are produced when there is an emotional stimulus and an audience present Our interpretation of another s emotion modulates our behaviour and vice versa The ability to recognise emotion expressions appears very early first few days (neonates) Four- to six-month seven months can distinguish between expressions of happiness, sadness, and surprise show preferences for facial expressions of happiness over neutral and angry expressions can distinguish among expressions of fear, anger, surprise, happiness, and sadness

5 Movements in the facial area: macro-expressions micro-expressions eye blinks changes in gaze direction talking tongue yawn... Etc.

6 Determine the emotional state of the face Regardless of the identity of the face Sensors: remote cameras Input: Images or Videos Macro expressions Micro expressions Analyze: Typical emotional states Action Units Intensity

7 Illumination Pose Scale Face Preprocessing Alignment Facial Feature Presentation Face Acquisition Facial Feature Extraction Facial Expression Classification Facial Features Whole Face Geometrical Features Appearance Features Recognition Interpretation Generic Facial Expression Analysis Framework

8 Face detection Preprocessing Feature extraction Recognition

9 Face detection To locate the faces in images/videos YzVk

10 Viola-Jones face detector Integral Image AdaBoost Cascade Testing phase Training phase Training Set (subwindows) Cascade trainer Integral Representation Feature computation AdaBoost Feature Selection Classifier cascade framework Strong Classifier 1 (cascade stage 1) Strong Classifier 2 (cascade stage 2) Strong Classifier N FACE IDENTIFIED (cascade stage N)

11 pros Extremely fast feature computation Efficient feature selection Scale and location invariant detector Instead of scaling the image itself (e.g. pyramid-filters), we scale the features. Such a generic detection scheme can be trained for detection of other types of objects (e.g. cars, hands) and cons Detector is most effective only on frontal images of faces can hardly cope with pose changes Sensitive to lighting conditions Might get multiple detections of the same face, due to overlapping sub-windows.

12 Results

13 Face preprocessing Face Acquisition Landmark Detection Face Registration Face crop

14 Landmark detection Locate key facial landmarks in face image The landmarks can be defined by MPEG-4 International Standard (ISO14496) MPEG-4 International Standard (ISO14496) 68 points model 49 points model

15 Landmark detection Method Commonly used methods: Active Shape Model (ASM) Active Appearance Model (AAM) Constrained Local Model (CLM) Incremental Parallel Cascade of Linear Regression Method (Chehra) Discriminative Response Map Fitting (DRMF)

Matching a statistical model of object shape and appearance to a new image.

16 Landmark detection Active Shape Model (ASM, 1995) Active Appearance Model (AAM, 1998) Statistical models of the shape of objects which iteratively deform to fit to an example of the object in a new image. Matching a statistical model of object shape and appearance to a new image. They are built during a training phase. A set of images, together with coordinates of landmarks that appear in all of the images, is provided to the training supervisor.

17 Landmark detection Incremental Parallel Cascade of Linear Regression Method (Chehra) * A. Asthana, S. Zafeiriou, S. Cheng and M. Pantic. Incremental face alignment in the wild. CVPR 2014.

18 To eliminate Face Registration rigid head motions in the input sequence Interperson variations

19 Face Registration Procedure Register each frame of the input image sequence with the candiate frame using an affine spatial transform T Step 1: Calculate transformation matrix Transformation Matrix T Candiate Model Step 2: Affine transformation Apply T

20 Face Crop d Distance of eyes d *0.4 d *0.6 d d *0.4 d*2.2 d *1.8

21 Preprocessing X. Tan, B. Triggs, Enhanced local texture feature sets for face recognition under difficult lighting conditions, International Workshop on Analysis and Modeling of Faces and Gestures, 2007, pp

22 Preprocessing chain: Gamma correction: a nonlinear gray-level transformation. It has the effect of enhancing the local dynamic range of the image in dark or shadowed regions, while compressing it in bright regions and at highlights. Difference of Gaussian (DoG) filtering: to remove the influence of overall intensity gradients such as shading effects Masking: to suppress facial regions that are felt to be irrelevant or too variable Contrast equalization: globally rescales the image intensities to standardize a robust measure of overall contrast or intensity variation

23 Features Appearance features Geometric features Deep learning features

24 Facial expression recognition Determine the emotional state of the face Regardless of the identity of the face

25 Local Binary Pattern and Contrast operators Ojala T, Pietikäinen M & Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recognition 29: An example of computing LBP and C in a 3x3 neighborhood: example thresholded weights Pattern = LBP = = 241 C = ( )/5 - (5+2+1)/3 = 4.7 Important properties: LBP is invariant to any monotonic gray level change computational simplicity

26 Uniform patterns U=0 Uniform patterns (P=8) U=2 Examples of nonuniform patterns (P=8) U=4 U=6 U=8

27 Texture primitives ( micro-textons ) detected by the uniform patterns of LBP 1 = black 0 = white Spot Spot / flat Line end Edge Corner

28 LBP from Three Orthogonal Planes (LBP-TOP) Length of Feature V ec tor 10 x Concatenated LBP VLBP P: Number of Neighboring Points

29 3 2 1 Y T X Y 0 T 0 T X X Y

30 LBP-TOP

features (c) Concatenated features for one block volume

31 (a) Non-overlapping blocks(9 x 8) (b) Overlapping blocks (4 x 3, overlap size = 10) (a) Block volumes (b) LBP features (c) Concatenated features for one block volume from three orthogonal planes with the appearance and motion

32 AU detection in 3D N. Bayramoglu, G. Zhao and M. Pietikainen. CS-3DLBP and Geometry based Person Independent 3D Facial Action Unit Detection. Proc. IAPR International Conference on Biometrics (ICB 2013), Madrid, Spain.

33 The majority of the previous works have focused primarily on the 2D data (images and videos) Broad range of use, Being readily available Computational limitations Accessible 3D Face

34 Main work Modified LBP on the orientation of 3D face points New set of geometric features (RbG) for detecting 3D facial action units (AUs)

35 Benefit of 3D data Robust to illumination changes

36 Benefit of 3D data Robust to background clutter

37 Why AU? A small set of emotional expressions Prototypic expressions are infrequent in everyday life. Expressions are often composed of few facial feature movements.

38 Facial ACTION CODING SYSTEM (FACS) FACS : An anatomically based system for measuring facial movements (Ekman and Friesen, 1978). Describe all visually distinguishable facial activity on the basis of 44 unique action units (AUs). The system codes the intensity of AUs on a five point scale.

39 Facial expressions may contain Single AU Combinations of several AUs. AUs Basis spanning the facial expressions space by proper combinations

40 Method Local Binary Patterns (LBP) Center Symmetric Local Binary Patterns (CS-LBP)

41 CS-3DLBP

42 Person Independent Geometric Properties AU12R (Lip Right Corner Puller) Samples : Shapes and sizes are different There is a similar harmony in each person s face We tried to formulate the rules of this harmony What we use: ratios of these distances and areas and also angles defined on a single 3D face data

43 Formulation Ratios

44 Lip Stretcher Lip Funneler Lip Corner Puller Lip Corner Depressor Chin Raiser Lip Tightener

45 Person Independent Geometric Properties

46 Detector Random Forests Classifier (2 class problem : Detector) Is a collection of tree predictors Provides probabilistic output Easily combined in the decision level For each AU: CS-3DLBP RbG Combination of CS-3DLBP and RbG

conditions 29 professional actors/actresses Most of

47 Database 105 subjects from structured-light based 3D system various poses, expressions and occlusion conditions 29 professional actors/actresses Most of the subjects are Caucasian Up to 24 AUs (35 expressions)

48 Results Decision level combination of CS-3DLBP and RbG features achieved the highest detection performance. No reference neutral face is needed. Limitations: Marker based 3D experiments : Person is not in a natural environment Laboratory environment They have been told to pose

49 Classifiers KNN (K-Nearest Neighbors): In k-nn classification, the output is a class membership. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.

50 SVM (Support Vector Machine): constructs a hyperplane or set of hyperplanes in a high- or infinite-dimensional space, can be used for classification, regression, or other tasks. a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class the larger the margin the lower the generalization error of the classifier. H1 does not separate the classes. H2 does, but only with a small margin. H3 separates them with the maximum margin.

51 Dynamic facial expression analysis with deep learning Mengyi Liu, Shaoxin Li, Shiguang Shan, Ruiping Wang, Xilin Chen, Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis, ACCV 2014, Singapore, Dec., After mean pooling Part-wise discriminative training to obtain the part-based estimation scores. Finally a full connection layer is used to predict the class label 64 filters total 13*c*k filters total, 13 facial parts, c classes, and k filters for a certain class An adapted 3D CNN: jointly takes into account localizing the facial action parts and learning part-based Representations.

52 Deformable Facial Action Parts Model 13 anchor positions of facial action parts Learned action parts filters for different expressions S2 C3/S4 D5

53 Experimental Results DB Anger Contempt Disgust Fear Happy Sadness Surprise CK MMI An example clip from CK+ database. (Surprise) An example clip from MMI database. (Anger) *15-fold cross validation on CK+ and 20-fold cross validation in MMI

54 Results Comparison on CK+ database Method Anger Contempt Disgust Fear Happy Sadness Surprise Mean CLM [1] AAM [2] ITBN [3] HOG3D [4] LBP-TOP [5] DCNN [6] DCNN-DAP S. Chew, et al., Person-independent facial expression detection using constrained local models. FG P. Lucey, et al., The extended cohn-kanadedataset (ck+): A complete dataset for action unit and emotionspecied expression. CVPRW. (2010) 3. Z. Wang, et al., Capturing complex spatio-temporal relations among facial muscles for facial expression recognition. CVPR A. Klaser, et al., A spatio-temporal descriptor based on 3d-gradients. BMVC G. Zhao, et al., Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE T PAMI 29 (2007) S. Ji, et al., 3d convolutional neural networks for human action recognition. IEEE T PAMI 35 (2013)

55 Results Comparison on MMI database Method Anger Disgust Fear Happy Sadness Surprise Mean ITBN HOG3D LBP-TOP DCNN DCNN-DAP

International Journal of Computer Techniques Volume 4 Issue 1, Jan Feb 2017

International Journal of Computer Techniques Volume 4 Issue 1, Jan Feb 2017 RESEARCH ARTICLE OPEN ACCESS Facial expression recognition based on completed LBP Zicheng Lin 1, Yuanliang Huang 2 1 (College of Science and Engineering, Jinan University, Guangzhou, PR China) 2 (Institute