Facial Expression Analysis

Transcription

1 Facial Expression Analysis Jeff Cohn Fernando De la Torre Human Sensing Laboratory Tutorial People June 2012 Facial Expression Analysis F. De la Torre/J. Cohn People (CVPR-12) 1

2 Outline Introduction Facial Action Coding System (FACS) Discrete vs. dimensional approaches Applications of FEA Databases Algorithms Supervised Unsupervised Conclusions and open problems 2

3 Outline Introduction Facial Action Coding System (FACS) Discrete vs. dimensional approaches Applications of FEA Databases Algorithms Supervised Unsupervised Conclusions and open problems 3

4 Supervised FacialExpressionAnalysis(FEA)

5 SupervisedFEA (II) Most work on FEA has been supervised using different registration, features and classifiers. 2D/3D Face tracking (AAM) Registration (remove 3D rigid motion) AU present? Classifiers (identity, discriminate Aus) Features (illumination, identity)

6 Facial feature detection Generative (Parameterized Appearance Models) Active Appearance models (e.g., Cooteset al. 98, Romdhaniet al. 99, De la Torre 00, Matthews & Baker 05, De la Torre & Nguyen 08, Gong et al. 00) Eigentracking (e.g., Black & Jepson 98) Morphablemodels (e.g., Jones & Poggio98, Blanz& Better 99) Discriminative Regression: Classifier fitting (e.g., Liu 09) Continuous regression (e.g., Sauer et al. 11, Saragih 11) Cascaded regression (e.g., Dollar et al. 10, Cao et al. 12) Local models: Constrained Local model (e.g., Cristanace& Cootes08, Luceyet al. 09, Saragihet al. 10) Part-based model (Zhu & Ramanan 12)

7 Parameterized Appearance Models Shape normalised images Procrustes Hand-Labeled Training Data Shape modes Appearance modes B 0 B 1 B 2

8 Detection as an Optimization Problem Translation rotation, scale c s 0 c s n s 0 (f(x, a)) s n (f(x, a )) Non-rigid parameters d( ) Bc Appearance parameters c 2 2 Learned off-line B 0 B 1 B 2 8

9 Prone to local minima Problems Not generalize well (e.g., different people) (Nguyen & De la Torre 10)

10 Discriminative models In general improve generalization (e.g., Liu 09, Sauer et al. 11, Saragih11, Dollar et al. 2010, Cao et al. 2012) 0.7 Non-rigid parameters Rigid parameters [ ] s a 1 c 1 features 0 S ( f( S(c + c 1 ), a 0 + a 1 )) [ ] S a 10 2 c 2 features 0 s ( f( S(c + c ), a 0 + a 2 2 ) )

11 Discriminative models (II) Local discriminative models Constrained Local model (e.g., Cristanace& Cootes 08, Lucey et al. 09, Saragih et al. 10) Part-based model (Zhu & Ramanan 2012) Thanks Saragih/Lucey

12 Face registration What are the three most important aspects of face recognition? registration, registration, registration (Takeo Kanade 90) Similarity registration (e.g., Barlettet al. 05, Whitehillet al. 11) Rotate, scale Piece-wise warping (e.g., Cooteset al. 98, Gong et al. 00, Tong et al. 07, De la Torre & Nyugen08, Jones & Poggio, 1998, Luceyet al. 09, Saragihet al. 10) Piece-wise warping Benefits: - Subtle AUs - Out-of plane rotation (3D models)

13 3D registration (Thanks Laszlo Jeni) Face Registration

14 SupervisedFEA (II) Most work on FEA has been supervised using different registration, features and classifiers. Face tracking (AAM) Registration AU present? Classifiers Features

15 Features Three types: (1) Shape, (2) Appearance, (3) Temporal features. Shape features (e.g., Sebeet al. 07, Asthanaet al. 09, Luceyet al., 2007; Chew et al., 2011, Zhou et al. 10; Valstaret al. 12) (e.g., Zhou et al. 2010)

16 Raw pixels Appearance features SIFT/HOG Box filters (e.g., Kanade et al., 2000) Gabor bank (e.g., Zhu et al. 2011, Simon et al., 2010, Dhall et al. 11) Local binary patterns (e.g., Whitehill& Omlin, 2006) NMF (e.g., Donato et al., 99; Barlett 04, Littlewort et al., 2006, Whitehill et al. 11) (e.g., Shan et al 09, Zhao et al., 10 Jiang et al. 11) (e.g., Zhiet al. 11, Zafeiriou and Petrou 10) Warning!!: Appearance features typically need dimensionality reduction and/or feature selection

17 Temporal features Motion units/trajectories Optical flow (e.g., Cohen et al., 02, Li et al. 01) Bag of temporal words (e.g., Essa and Pentland 97, Gunes and Piccardi 05) Motion history (e.g., Simon et al., 10) (e.g., Valstaret al., 04, Koelstra et al. 10)

18 SupervisedFEA (II) Most work on FEA has been supervised using different registration, features and classifiers. Face tracking (AAM) Registration AU present? Classifiers Features

19 Classifiers Static Exemplar + GMM (Wen and Huang, 2003) Neural Network (Kapoor and Picard, 2005) SVM/Adaboost (Bartlett et al., 2005) Linear Discriminant Classifiers (Wang et al., 2006) Gaussian Process (Chen et al., 2009) Boosting (Shan et al. 2006, Zhu et al. 2010) Dynamic Hidden Markov models (Lien et al, 2000) Dynamic Bayesian Network (Tong et al., 2007) Conditional random field (Chang and Liu, 2009) Temporal Bag of Words (Simon et al. 2010)

20 The million $ question Which is the best feature and classifier? Data Have access to reliable and well annotated data The more data the better Features It is AU dependent In general feature fusion is the best (e.g., multiple kernel learning) Classifier Depends on the amount of training data How familiar you are with the classifier

21 SupervisedFEA (II) Most work on FEA has been supervised using different registration, features and classifiers. Face tracking (AAM) Registration AU present? Classifiers Features

22 Sample selection Most work on FEA has been supervised using different registration, features and classifiers. Onset Peak Offset Intensity - AU Time + - Make good use of the data!!! (Zhu et al 11, Simon 10)

23 Results for AU4 and AU12 The first number between lines denotes the area under the ROC, the second number is the size of positive samples in the testing dataset and separated by / is the size of negative samples in the testing dataset. The third number denotes the size of positive samples in training working sets and separated by / the total frames of target AU in training data sets.

24 Bayesian networks Bayesian networks to model spatial and temporal relationships among different Aus (Tong et al. 05, Shang et al 07).

25 Outline Introduction Facial Action Coding System (FACS) Databases Applications of FEA Algorithms Supervised Unsupervised Conclusions and future work 25

26 Motivation Mining facial expression for one subject

27 Motivation Mining facial expression for one subject Summarization Visualization Indexing

28 Mining facial expression for one subject Looking up Sleeping Motivation Waking up Looking forward Smiling Summarization Visualization Indexing

29 Motivation Mining facial expression of one subject Summarization Embedding Indexing

30 Mining facial expression across subjects RU-FACS database (Bartlett et al. 06) Summarization Embedding Indexing

31 Aligned Cluster Analysis (Zhou et a. 10) h1 h2 Labels (G) h 3 Start and end of the segments (h) hm h m+ 1

32 Kernel k-means and spectral clustering (Ding et al. 02, Dhillonet al. 04, Zassand Shashua 05, De la Torre 06) J 2 ( M, G) = ϕ( X) MG F x G= X= y M = J x y ( G ) G= = tr ( K ( I n G T MG= ( GG T x y ) 1 G )) y K=ϕ( X) T ϕ( X) x

33 Problem formulation for ACA X h 1.. h ) X h 2.. h ) [ 2 [ 3 X[ h m.. h m+ 1) h1 h2 h 3 h4 Labels (G) Start and end of the segments (h) hm h m+ 1 J H 2 aca kkm( ( M, G, ) = ϕ ( X[.. ), X[.. ),..., X[.. ) ) MG h 1 h 2 h 2 h 3 h m h m+ 1 F Dynamic Time Alignment Kernel (Shimodaira et al. 01)

34 Matrix formulation for ACA K T = ϕ( X) ϕ( X) J k = tr ( KL ) km with L = T T I n G ( GG ) 1 G J aca = tr ( K ( L o W )) with 23 frames, 3 clusters L = T T T I n H G ( GG ) 1 GH segments W R samples H 23 { 0,1} 7 clusters Dynamic Time Alignment segments Kernel (ShimodairaG et al. 01) 7 { 0,1} 3

35 Facial image features Active Appearance Models (Baker and Matthews 04) Image features Shape Appearance Upper face Lower face

36 Facial event discovery across subjects Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger)

37 Facial event discovery across subjects Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger)

38 Facial event discovery across subjects Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger) 10 sets of 30 people ACA Spectral Clustering (SC) 0.87(.05) 0.56(.04)

39 Unsupervised facial event discovery

40 FACS coding Outer Brow Raiser (AU2) Upper lid raiser (AU5) Lip Tightener(AU23) Nose Wrinkler(AU9) ACA SpectralClustering (SC) Lower face 0.53(.09) 0.39(0.14) Upper face 0.69(.12) 0.47(0.12)

41 Conclusions and open problems Supervised and unsupervised algorithms for FEA Tracking/registration Registration, registration, registration changes in pose (3D models) Robustness to occlusion Features Subtle facial expressions Dynamics (e.g., temporal envelope) Classifiers Expression intensity Individual differences Predicting onset/offset Truly multi-class AU detection

42 Conclusions and open problems Data Attention to reliability of ground truth Shared, well-annotated video Innovative ways to use video that cannot be shared Segmentation and timing Intra-personal Interpersonal User-in-loop approaches User-assisted coding, e.g., Fast FACS (e.g., Simon et al., 2011) Combining manual and automated measurement (e.g. Ambadar et al. 2009) Person-dependent classifiers Other issues Multimodal Context

43 Questions? Jeff Cohn Fernando De la Torre Human Sensing Laboratory Tutorial People June 2012 Facial Expression Analysis F. De la Torre/J. Cohn People (CVPR-12) 43