A Coarse-to-Fine Classification Scheme for Facial Expression Recognition

Transcription

1 A Coarse-to-Fine Classifiation Sheme for Faial Expression Reognition Xiaoyi Feng 1,, Abdenour Hadid 1 and Matti Pietikäinen 1 1 Mahine Vision Group Infoteh Oulu and Dept. of Eletrial and Information Engineering P. O. Box 4500 Fin University of Oulu, Finland {xiaoyi,hadid,mkp}@ee.oulu.fi College of Eletronis and Information, Norwestern Polytehni University Xi an, China fengxiao@nwpu.edu.n Abstrat. In is paper, a oarse-to-fine lassifiation sheme is used to reognize faial expressions (angry, disgust, fear, happiness, neutral, sadness and surprise) of novel expressers from stati images. In e oarse stage, e sevenlass problem is redued to a two-lass one as follows: First, seven model vetors are produed, orresponding to e seven basi faial expressions. Then, distanes from eah model vetor to e feature vetor of a testing sample are alulated. Finally, two of e seven basi expression lasses are seleted as e testing sample s expression andidates (andidate pair). In e fine lassifiation stage, a K-nearest neighbor lassifier fulfils final lassifiation. Experimental results on e JAFFE database demonstrate an average reognition rate of 77% for novel expressers, whih outperforms e reported results on e same database. 1 Introdution Numerous algorims for faial expression analysis from stati images have been proposed [1,,3] and e Japanese Female Faial Expression (JAFFE) Database is one of e ommon databases for testing ese meods [4-10]. Lyons et al. provided a template-based meod for expression reognition [4]. Input images were onvolved wi e Gabor filters of five spatial frequenies. Then e amplitude of e omplex-valued filter responses were sampled on 34 manually seleted fiduial points and ombined into a single vetor, ontaining 100 elements. The prinipal omponents analysis (PCA) was used to redue e dimensional of data and finally a simple LDA-based lassifiation sheme was used. Zhang et al. [5,6] used a similar representation for fae while ey applied wavelet of 3 sales and 6 orientations. They also onsidered geometri position of e 34 fiduial points as features and used a multi-layer pereptron for reognition. Guo and Dyer [7] also adopted a similar fae representation and ey used linear programming tehnique to arry out simultaneous feature seletion and lassifier training. Buiu et al. [8]

2 adopted ICA and Gabor representation for faial expression reognition. Neural Networks have been onsidered in [9, 10]. Reognizing e expressions of novel individual is still a hallenging task and only few works have addressed is issue [1, 4, 10]. In is paper, a modified template-based lassifiation meod is proposed for novel expressers expression reognition. The template-based tehniques are simple fae representation and lassifiation meods. They have only limited reognition apabilities, whih may be aused by smooing of some important individual faial details, by small misalignment of e faes, and also by large inter-personal expression differenes, but ey an disriminate typial and ommon features. In our work, a oarse-to-fine lassifiation meod is adopted, aiming to make use of e advantages of template-based meods and at e same time to weaken eir shortomings mentioned above. In e oarse lassifiation stage, seven model vetors (templates) are formed for e seven basi faial expressions. Then distanes between eah template and a testing sample are alulated wi e Chi square statisti. The two nearest expression lasses (andidate pair) are seleted as andidate expressions. As a result, seven-lass lassifiation is redued to a two-lass lassifiation. Sine e traditional template-based meods have e ability to disriminate main faial expression features, e real expression lass of e testing sample has a high probability of belonging to one of e two andidate expressions. To minimize e disadvantage of traditional template-based meods, seven templates are substituted for multi-template pairs, and weighted Chi square statisti replaes former Chi square statisti as dissimilarity measure in e fine lassifiation stage. A simple K-nearest neighbor lassifier follows to finally lassify e testing sample. The rest of e paper is organized as follows: fae representation is introdued in setion. In setion 3, e oarse-to-fine expression lassifiation meod is presented. Experimental results are desribed in setion 4. Finally, we onlude e paper. Fae Representation Fig. 1 illustrates e basi LBP operator [11]. The 3 3 neighborhood is reshold by e value of e enter pixel, and a binary pattern ode is produed. The LBP ode of e enter pixel is obtained by onverting e binary ode into a deimal one. Based on is operator, eah pixel of an image is labeled wi an LBP ode. The 56-bin histogram of e labels ontains e density of eah label over a loal region, and an be used as a texture desriptor of e region. Reently, an LBP based faial representation has shown an outstanding result in fae reognition [1]. In our work, we use a similar faial representation as at proposed in [1]: Divide e fae image into small regions. The size of eah pre-proessed image is After experimenting wi different blok sizes, we hoose to divide e image into 80 (10 8) non-overlapping bloks of pixels (see Fig. ). Calulate e LBP histogram from eah region. The LBP histogram of eah region is obtained by sanning it wi e LBP operator.

3 Original neighborhood Threshold Loal binary pattern Binary ode: LBP ode: 9 Fig. 1. The basi LBP operator Conatenate e LBP feature histograms into a single feature vetor. LBP histogram of eah region is ombined togeer to form a single feature vetor representing e whole image. Fig.. An example of a faial image divided into 10 8 bloks The idea behind using our approah for feature extration is motivated by e fat at emotion is more often ommuniated by faial movement, whih will hange visible appearane. Our feature extration meod is apable of presenting faial appearanes and so it an be used for representing faial expressions. 3 Coarse-to-Fine Classifiation Though traditional template-based approahes have only limited reognition apabilities, ey are quite simple and an reflet main and ommon features. Experiments have shown at ey are effetive in reognizing intense and typial expressions. Based on at, ey are used in our oarse lassifiation proedure to redue a sevenlass to a two-lass lassifiation problem. To overome eir shortomings, multitemplate pairs and a K nearest neighbor lassifier are used in e fine lassifiation.

4 3.1 Coarse Classifiation In is stage, e lassifiation is performed using a two-nearest neighbor lassifier wi Chi square as dissimilarity measure. Feature vetors of same expression lass of training samples are averaged to form model vetors and so seven model vetors are onstruted. The testing vetor is extrated from a testing sample. Distanes from eah model vetor to e testing vetor are alulated. Consider a training set X ontaining n d -dimensional feature vetors. The training set is divided into seven subsets and eah subset orresponds to one expression. subset wi ( = 1,,...7 ve- Let X { x i < n } tors and = 0 denotes e i x i is e i feature vetor. So n ) X = 7 X = 1 = 7 n = 1 The model vetor (denoted as m ) of e of e subset. m n (1) = n n 1 1 i= 0 expression lass is e luster enter A hi square ( χ ) statisti is used as dissimilarity measure between a testing sample and models. Suppose s is e test vetor and x i s j is its j element, we have () χ d 1 ( s j m j ) = = s + m ) ( s, m j 0 j j (3) The weighted hi square ( χ ) statisti [1] is defined as follows and will be used in our fine lassifiation later. χ d 1 ( s j m j ) = w j = s + m ) w ( s, m j 0 j j Instead of lassifying e test sample into one expression lass, we hoose two ex- and subjet to pression lasses as andidates { 1, } =, min χ ( s, m ) = 7 χ ( s, m ) (4)

5 min χ ( s, m ) = 1 7, 1 χ ( s, m ) (5) 3. Fine Classifiation To overome e shortomings of traditional template-based tehniques, multitemplate pairs are used in e fine lassifiation stage, replaing simple seven templates. A simple K-nearest neighbor lassifier is also used in is stage. Our experimental results favor our fine lassifiation ideas. When we analyzed e results of e oarse lassifiation, we notied at more an 50% of testing samples at were wrongly reognized have e seond nearest expression lass as eir real expression lass. This shows at e template-based meod has e ability to disriminate expressions in a oarse level and we need some oer meods to disriminate expressions in a fine level. The following steps are used in fine lassifiation. First, multi-template pairs are formed for eah pair of andidate expressions. Eah template pair orresponds to one expresser in training set. The multi-template pairs are formed as follows: In e ase of one expression andidate is neutral: Suppose e oer expression is. For eah expresser in e training set, distanes between eah feature vetor in expression and at in neutral are alulated by formula (3). A template pair wi e nearest distanes is seleted as one template pair for e neutral- lassifiation. The above proedure is repeated for all expressers in e training set. Regions ontaining more useful information for expression lassifiation are given a relatively high weight value. The aim for forming template pairs in e above way is to minimize e distane between eah pair to ensure at expressions wi weak intensity are lassified orretly. In e ase neier of e expression andidates is neutral: Denote e two expression andidates as 1 and. For eah expresser in e training set, suppose feature vetor v 1 orresponds to e enter of feature vetors of expression 1, and v orresponds to e enter of feature vetors of expression. So vetor pair v1 - v forms one template pair for e 1 - lassifiation. The above proedure is repeated for all expressers in e training set and so e number of template pair is at of expressers in e training set. One multi-template pairs are formed for one andidate pair, e weighted hi square ( χ ) statisti is used as dissimilarity measure. Sine more an one template pairs are employed for one andidate pair, we use a simple K-nearest neighbor lassifier for e two-lass lassifiation in is stage.

6 4 Experiments and Results Our meod is tested on e Japanese Female Faial Expression (JAFFE) Database [13]. The database ontains 13 images of ten expressers posed 3 or 4 examples of eah of e seven basi expressions (happiness, sadness, surprise, anger, disgust, fear, neutral). Sample images from e database are shown in Fig. 3. Fig. 3. Samples from e Japanese Female Faial Expression Database There are mainly two ways to divide e JAFFE database. The first way is to divide e database randomly into 10 roughly equal-sized segments, of whih nine segments are used for training and e last one for testing. The seond way is to divide e database into several segments, but eah segment orresponds to one expresser. In our experiments, image pre-proessing is onduted by e pre-proessing subsystem of e CSU Fae Identifiation Evaluation System [14]. As a result, e size of eah pre-proessed image is (see Fig. 4). Fig. 4. Samples from e preproessed images To ompare our results to ose of oer meods, a set of 193 expression images posed by nine expressers is used. These images are partitioned into nine segments, eah orresponding to one expresser. Eight of e nine segments are used for training and e nin for testing. The above proess is repeated so at eah of e nine partitions is used one as e test set. The average of reognizing e expression of novel expressers is 77% (Reognition results of eah trail are in Table 1).

7 Table 1. Reognition auray of eah trail in our meod Trial Auray/Total % Corret Trial Auray/Total % Corret 1 18/3 78.3% 6 13/1 61.9% 15/ 68.% 7 0/1 95.5% 3 0/ 90.9% 8 15/1 71.4% 4 11/0 55.0% 9 0/ 90.9% 5 17/1 81.0% Now we ompare e reognition performane to oer published meods using e same database. In [4], a result of 75% using Linear Disriminant Analysis (LDA) was reported wi 193 images. In [10], an average reognition result of 30% was reported wi 13 images. Oer reports [5-9] on e same database did not give e reognition rate for novel expressers expression. It should be pointed out at in [4], 34 fiduial points have to be seleted manually. In our meod, we need only e position of two pupils for fae normalization and oer proedures are ompletely automati. It should also be noted at in e JAFFE database, some expressions had been labeled inorretly or expressed inaurately. Wheer ese expressional images are used for training or testing, e reognition result is influened. Fig. 5 shows a few examples wi e labeled expression and our reognition results. Fig.5. Examples of disagreement. From left, e labeled expressions are sadness, sadness, sadness, surprise, fear, disgust, happiness, and e reognition results are happiness, neutral, neutral, happiness, sadness, angry and neutral, respetively 5 Conlusion How to reognize faial expressions of a novel expresser from stati images is one of e hallenging tasks in faial expression reognition. The template-based tehniques an reflet main and typial features but ey will smoo some important individual features. A oarse-to-fine lassifiation sheme is used so at e lassifiation an utilize e advantages of e template-based tehniques and minimize eir disadvantages. The ombination of multi-template pairs, e weighted Chi-square and K- nearest neighbor lassifier provides a good solution. Experimental results demonstrated at our meod performs better an oer meods on e JAFFE database.

8 Aknowledgement The auors ank Dr. M. Lyons for providing e Japanese Female Faial Expression (JAFFE) Database. The auors also ank CIMO of Finland and e China Sholarship Counil for eir finanial support for is researh work. Referenes 1. M. Panti, Leon J.M. Rokrantz: Automati analysis of faial expressions: e state of e art, IEEE Transations on Pattern Analysis and Mahine Intelligene, Vol. (000) B. Fasel and J. Luettin: Automati faial expression analysis: A survey, Pattern Reognition, Vol. 36 (003) W. Fellenz, J. Taylor, N. Tsapatsoulis, S. Kollias: Comparing template-based, feature-based and supervised lassifiation of faial expression from stati images, Computational Intelligene and Appliations, (1999) 4. M. Lyons, J. Budynek, S. Akamastu: Automati lassifiation of single faial images, IEEE Trans. Pattern Analysis and Mahine Intelligene, Vol. 1(1999) Z. Zhang: Feature-based faial expression reognition: Sensitivity analysis and experiment wi a multi-layer pereptron, Pattern Reognition and Artifiial Intelligene, Vol. 13(1999) Z. Zhang, M. Lyons, M. Shuster, S. Akamatsu: Comparison between geometry-based and Garbor-wavelet-based faial expression reognition using multi-layer pereptron, In: Third International Conferene on Automati Fae and Gesture Reognition. (1998) G. D. Guo, C. R. Dyer: Simultaneous feature seletion and lassifier training via linear programming: A ase study for fae expression reognition. In: IEEE Conferene on Computer Vision and Pattern Reognition. (003) I. Buiu, C. Kotropoulos, I. Pitas: ICA and gabor representation for faial expression reognition. In: International Conferene on Image Proessing. (003) B. Fasel: Head-pose invariant faial expression reognition using onvolutional neural networks. In: Four IEEE Conferene on Multimodal Interfaes. (00) M. Gargesha, P. Kuhi: Faial expression reognition using artifiial neural networks, EEE 511: Artifiial Neural Computation Systems, (00) 11. T. Ojala, M. Pietikäinen, T. Mäenpää: Multiresolution gray-sale and rotation invariant texture lassifiation wi Loal Binary Patterns, IEEE Transations on Pattern Analysis and Mahine Intelligene, Vol. 4(00) T. Ahonen, A. Hadid, M. Pietikäinen: Fae reognition wi loal binary patterns. In: 8 European Conferene on Computer Vision. (004) M. Lyons, S. Akamastu, M. Kamahi, J. Gyoba: Coding faial expressions wi Gabor wavelets. In: Third IEEE Conferene on Fae and Gesture Reognition. (1998) D. Bolme, M. Teixeira, J. Beveridge, B. Draper: The CSU fae identifiation evaluation system user s guide: its purpose, feature and struture. In: Third International Conferene on Computer Vision Systems. (003)