Multi-view Facial Expression Recognition Analysis with Generic Sparse Coding Feature

Transcription

1 0/19.. Multi-view Facial Expression Recognition Analysis with Generic Sparse Coding Feature Usman Tariq, Jianchao Yang, Thomas S. Huang Department of Electrical and Computer Engineering Beckman Institute for Advanced Science and Technology University of Illinois at Urbana-Champaign {utariq2, jyang29, October 13, 2012

2 1/19. Outline. 1

3 1/19. Outline. 1

4 2/19. Motivation It is not trivial in real application to always have a frontal face A detailed analysis of the effect of large variations in pose (both pan and title angles) on expression recognition performance is needed, e.g. for positioning the cameras Bulk of the existing literature assumes frontal or near-frontal face view manually/automatically detected key points and/or presence of a neutral face

5 3/19. Highlights This work deals with single image multi-view facial expression recognition It works with translation invariant sparse coding to obtain features and then does linear classification This work, despite achieving state of the art results, does an extensive analysis of effect of variation in pan and tilt angles

6 4/19. Database The database used in this work is the publicly available BU3D-FE database. It has 3D face scan of a 100 subjects, each performing 6 expressions at 4 intensity levels Facial expressions in the database: anger (AN), disgust (DI), fear (FE), happy (HA), sad (SA), surprise (SU) (and neutral) Out of 100 subject, 56 are females. The dataset is quite diverse and contains subjects with various racial ancestries. Views with seven pan angles (0, ±15, ±30, ±45 ) and five tilt angles (0, ±15, ±30 ) are generated for each subject with each expression-intensity combination, resulting into a dataset with images.

7 . Database - expressions and intensity levels Face images with different expressions and intensities Level 1 Level 2 Level 3 Level 4 AN DI FE HA SA SU Facial Expressions Rendered facial images of a subject with various expressions and intensity levels. The intensity levels are labeled from 1 to 4 with 4 being the most intense. 5/19

8 . Database - pan and tilt angles Face images with different pan and tilt angles 30 Tilt angles Pan angles Rendered facial images of a subject with various pan and tilt angles. 6/19

9 6/19. Outline. 1

10 7/19. Translation Invariant Sparse Coding (ScSPM) Bag of Features (BoF) model Spatial Pyramid Matching (SPM) framework the given image is partitioned then the BoF histograms are extracted for each of those partitions these histograms are then concatenated together When we relax the SPM cardinality constraint by solving a lasso problem and instead of histogram representation, use max-pooling, we approach at the ScSPM setting. it is robust to translation misalignments, since it is computed in an SPM framework. it gives significant improvement over SPM

11 7/19. Translation Invariant Sparse Coding (ScSPM) Bag of Features (BoF) model Spatial Pyramid Matching (SPM) framework the given image is partitioned then the BoF histograms are extracted for each of those partitions these histograms are then concatenated together When we relax the SPM cardinality constraint by solving a lasso problem and instead of histogram representation, use max-pooling, we approach at the ScSPM setting. it is robust to translation misalignments, since it is computed in an SPM framework. it gives significant improvement over SPM

12 7/19. Translation Invariant Sparse Coding (ScSPM) Bag of Features (BoF) model Spatial Pyramid Matching (SPM) framework the given image is partitioned then the BoF histograms are extracted for each of those partitions these histograms are then concatenated together When we relax the SPM cardinality constraint by solving a lasso problem and instead of histogram representation, use max-pooling, we approach at the ScSPM setting. it is robust to translation misalignments, since it is computed in an SPM framework. it gives significant improvement over SPM

13 7/19. Translation Invariant Sparse Coding (ScSPM) Bag of Features (BoF) model Spatial Pyramid Matching (SPM) framework the given image is partitioned then the BoF histograms are extracted for each of those partitions these histograms are then concatenated together When we relax the SPM cardinality constraint by solving a lasso problem and instead of histogram representation, use max-pooling, we approach at the ScSPM setting. it is robust to translation misalignments, since it is computed in an SPM framework. it gives significant improvement over SPM

14 7/19. Translation Invariant Sparse Coding (ScSPM) Bag of Features (BoF) model Spatial Pyramid Matching (SPM) framework the given image is partitioned then the BoF histograms are extracted for each of those partitions these histograms are then concatenated together When we relax the SPM cardinality constraint by solving a lasso problem and instead of histogram representation, use max-pooling, we approach at the ScSPM setting. it is robust to translation misalignments, since it is computed in an SPM framework. it gives significant improvement over SPM

15 8/19. ScSPM - Learning the code-book Suppose X is a matrix whose columns are image features. Now let X = [x 1,...,x N ] R p N. We solve the following in an alternative iterative fashion to come up with V, N min W,V n=1 x n Vw n λ w n 1 subject to v k 2 1, k 1,2,...,K (1) Here, V = [v 1,...,v K ] R p K is the code-book or dictionary, W = [w 1,...,w N ] R K N, and λ is a regularization parameter, whose value controls the sparseness in the solution w n

16 9/19. ScSPM - Coding In the ScSPM framework, to extract image level features, firstly the patches are densely sampled from a given image Then some low-level features like SIFT, are extracted on these patches These features are then sparsely coded using the code-book V, by solving min w n x n Vw n λ w n 1

17 10/19. ScSPM - Pooling These sparse vectors are then pooled in an SPM framework z = Φ(W) for instance, z i = max{ w i1,..., w in } The resulting image level feature vectors are extracted by concatenation of the vectors obtained for various partitions/levels

18 11/19. Experimental Setup Dense SIFT features with 3 pixel shift are extracted V R is used for sparse coding in an SPM framework, followed by pooling Experiments are done in 5 subject independent fold cross-validation setting on the images extracted from the BU-3DFE database. A universal is adopted approach for classification Linear SVMs are used for single image expression recognition L2-regularized logistic regression is used to get probability estimates for fusion

19 11/19. Outline. 1

20 12/19. Confusion matrix Class confusion matrix for over-all recognition performance (69.1%) averaged over all poses and expression intensity levels Overall classification Predicted AN DI FE HA SA SU Ground Truth AN DI FE HA SA SU

21 13/19. Recognition performance for different intensities Average recognition rates for various expressions of different intensities Intensity 1 (min) Intensity 2 Intensity 3 Intensity 4 (max) Average Percentage Recognition Rate AN DI FE HA SA SU Expression Labels Recognition performance for various expressions with different intensities

22 14/19. Performance vs pan and tilt angle variations (exp.) Average recognition rates vs pan angles for various expressions Average recognition rates vs tilt angles for various expressions Percentage Recognition Rate Percentage Recognition Rate AN DI FE HA SA SU Average AN DI FE HA SA SU Average Pan Angles Tilt Angles Effect of change in pan angles on the recognition performance of various expressions Effect of change in tilt angles on the recognition performance of various expressions

23 15/19. Performance vs pan and tilt angle variations (int.) Average recognition rates vs pan angles for various expression intensties Average recognition rates vs tilt angles for various expression intensties Percentage Recognition Rate Percentage Recognition Rate Intensity 1 (min) Intensity 2 Intensity 3 Intensity 4 (max) Average Intensity 1 (min) Intensity 2 Intensity 3 Intensity 4 (max) Average Pan Angles Tilt Angles Effect of change in pan angles on the recognition performance of various expression intensity levels Effect of change in tilt angles on the recognition performance of various expression intensity levels

24 16/19. Performance vs simultaneous pan and tilt variations Average recognition rates for various combinations of pan and tilt angles Tilt angles Pan angles Effect of change in both pan and tilt angles on the overall single image expression recognition performance.

25 17/19. Comparison with State-of-the-Art Zheng et al. and Tang et al. follow the same experimental setting by restricting themselves to only the strongest expression intensity level. Hence, their image dataset consists of = images. To compare, we repeat the experiments in the same experimental setting Performance comparison with earlier works on strongest expression intensity in terms of the percentage recognition rates Zheng et al Tang et al. Ours 68.2% 75.3% 76.1%

26 18/19. Concluding Remarks Our work sets a new state-of-the-art for multi-view facial expression recognition on the BU3D-FE database Unlike many other works, our method neither requires any key point detection nor does it need a neutral face A significant analysis of variations in expression recognition with changes in a range of pan angles, tilt angles or both is done The most subtle expression are the most difficult to recognize We do not find any conclusive evidence that non-frontal views give significantly better performance than frontal view

27 18/19. Concluding Remarks Our work sets a new state-of-the-art for multi-view facial expression recognition on the BU3D-FE database Unlike many other works, our method neither requires any key point detection nor does it need a neutral face A significant analysis of variations in expression recognition with changes in a range of pan angles, tilt angles or both is done The most subtle expression are the most difficult to recognize We do not find any conclusive evidence that non-frontal views give significantly better performance than frontal view

28 18/19. Concluding Remarks Our work sets a new state-of-the-art for multi-view facial expression recognition on the BU3D-FE database Unlike many other works, our method neither requires any key point detection nor does it need a neutral face A significant analysis of variations in expression recognition with changes in a range of pan angles, tilt angles or both is done The most subtle expression are the most difficult to recognize We do not find any conclusive evidence that non-frontal views give significantly better performance than frontal view

29 18/19. Concluding Remarks Our work sets a new state-of-the-art for multi-view facial expression recognition on the BU3D-FE database Unlike many other works, our method neither requires any key point detection nor does it need a neutral face A significant analysis of variations in expression recognition with changes in a range of pan angles, tilt angles or both is done The most subtle expression are the most difficult to recognize We do not find any conclusive evidence that non-frontal views give significantly better performance than frontal view

30 18/19. Concluding Remarks Our work sets a new state-of-the-art for multi-view facial expression recognition on the BU3D-FE database Unlike many other works, our method neither requires any key point detection nor does it need a neutral face A significant analysis of variations in expression recognition with changes in a range of pan angles, tilt angles or both is done The most subtle expression are the most difficult to recognize We do not find any conclusive evidence that non-frontal views give significantly better performance than frontal view

31 19/19 The End