An AAM-based Face Shape Classification Method Used for Facial Expression Recognition

Transcription

1 Internatonal Journal of Research n Engneerng and Technology (IJRET) Vol. 2, No. 4, 23 ISSN An AAM-based Face Shape Classfcaton Method Used for Facal Expresson Recognton Lunng. L, Jaehyun So, Hyun-Chool Shn, and Youngjoon Han Abstract Facal expresson recognton s a key element n human-computer communcaton. However, some noses such as dentty, gender and face shape may serously have an effect on mult-person's expresson recognton. In ths paper, an AAM-based method s proposed to remove face shape nose for the purpose of mprovng facal expresson recognton performance. Frstly, Actve Appearance Model (AAM) s used to extract the facal feature, whch ncludes abundant face geometry nformaton n the form of shape parameters. Subsequently, based on the shape parameters of AAM, an SVM-based classfcaton method s proposed to classfy the man face shape--melon seed, round and square. Fnally, a proposal for facal expresson recognton usng face shape classfcaton s gven, whch can mprove the recognton rate. Keywords Actve Appearance Model, facal expresson recognton, face shape, SVM. I. INTRODUCTION ACIAL expresson s one of the most powerful, natural and Fmmedate means for humans to share ther emotons and ntentons. Psychologcal studes focus on the nterpretaton on ths mean to nteract and descrbe that there are sx basc emotons unversally recognzed [], namely: joy, sadness, surprse, fear, anger and dsgust. An automatc, effcent and accurate facal expresson extracton system would thus be a powerful tool assstng on these studes, allowng also other knds of applcatons such as Human Computer Interface (HCI), smart nteractve systems, vdeo compresson, etc. In order to work toward these capabltes, efforts have recently been devoted to ntegratng affect recognton nto human-computer applcatons [2]. Applcatons exst n both emoton recognton and agent-based emoton generaton [3]. Irene kotsa used the well-known Candde wreframe to locate the facal feature ponts. Combnng wth Facal Acton Codng Systems (FACS) and mult-class SVM, facal expresson recognton acheves a very hgh rate. However, ths system can be only used for one person [4]. The proposed face shape classfcaton method used for facal expresson recognton s based on the dea that all the face Lunng L, Jaehun Soo, Hyun-Chool Shn, and Youngjoon Han are wth Department of Electronc Engneerng, Soongsl Unversty, Seoul, Korea. (phone: ; fax: ; e-mal: young@ssu.ac.kr, runyonzeelee@63.com ).. nformaton such as dentty, age, and gender are ncluded n one face, whch can be the man noses for facal expresson recognton. In order to obtan a more satsfyng recognton result, useless nformaton should be fltered. In ths work, Actve Appearance Models (AAM) [5] s used as the facal feature extracton method, modelng both shape and texture from an observed tranng set, beng able to extract relevant face nformaton wthout background nterference. Both face shape and expresson nformaton are represented by means of shape parameters after buldng AAM model. The face shape classfcaton process s based on a one-versus-one multclass Support Vector Machne (SVM), whch can classfy 3 classes of face shapes melon seed, round and square. Ths paper s organzed as follows: Secton 2 gves an ntroducton to AAM, ncludng the man process of buldng AAM and model fttng algorthm. Secton 3 manly descrbes how to classfy face shapes by means of one-versus-one multclass SVM method. Secton 4 and 5 dscuss expermental results and concluson. II. ACTIVE APPEARANCE MODELS Actve Appearance Models (AAM) s a statstcal based template matchng method, where the varablty of shape and texture s captured from a representatve tranng set. Prncpal Components Analyss (PCA) on shape and texture data allow buldng a parameterzed face model that fully descrbes wth photorealstc qualty the traned faces as well as unseen. For further detals refer to [5]. A. Shape Model The shape s defned as the qualty of a confguraton of ponts whch s nvarant under Eucldan Smlarty transformatons [6]. The representaton used for a sngle n-pont shape s a 2n vector gven by wth n shape annotatons, follows a statstcal analyss where the shapes are prevously algned to a common mean shape usng a Generalzed Procrustes Analyss (GPA) removng locaton, scale and rotaton effects. Applyng a Prncpal Components Analyss (PCA), we can model the statstcal varaton wth s = s +Φ sbs () Where s s the mean shape, Φs s a weghted lnear 64

2 Internatonal Journal of Research n Engneerng and Technology (IJRET) Vol. 2, No. 4, 23 ISSN combnaton of egenvectors of the covarance matrx. b s s a vector of shape parameters whch represents the weghts. We can change the form of () as n s s ps = + (2) = In ths expresson the coeffcents p are the shape parameters. Snce we can easly perform a lnear reparameterzaton, wherever necessary we assume that the vectors s are orthogonal. B. Texture Model For m pxels sampled, the texture s represented by the vector g = [g, g 2,..., g m-, g m ]. Buldng a statstcal texture model requres warpng each tranng mage so that the control ponts match those of the mean shape. Ths texture mappng process uses a pece-wse affne warp by a set of trangles usng the Delaunay trangulaton. A texture model s obtaned by applyng a low-memory PCA on the normalzed textures. Defnng pxel coordnates as x = ( xy, ) T, the appearance of the AAM s an mage, A(x), defned over the pxels x s such m as A( x) A ( x) λ A( x) x s. The appearance = +, = A ( x) can be expressed as a base appearance A ( x) plus a lnear combnaton of m appearance mages A ( x). The coeffcents λ are the appearance parameters. C. Inverse Compostonal Image Algnment Fttng an AAM to the face actually s to mnmze the texture error between the model nstance A ( x) and the nput back warped mage on to the base mesh IW ( ( xp ; )). [ A( x) + λa( x) IW ( ( xp ; ))] (3) x s m = In (3) the warp W s the pecewse affne warp from the base mesh s to the current AAM shape s. Hence, W(x;p) s a functon of the shape parameters p. The ICIA algorthm s a modfcaton of the forwards compostonal algorthm where the roles of the template and example mage are reversed [7]. Rather than computng the ncremental warp wth respect to I(W(x;p)), t s computed wth respect to the template A (x). By means of some mathematcal trcks, computaton of fttng process has been greatly decreased. So far, ICIA algorthm s the fastest fttng method for AAM. Readers may refer to [7] to obtan the specfcaton about ICIA. Fg. shows the nput mages and fttng results ncludng teratons, whch fnally the appearance of AAM nstance over the face s almost the same wth the nput mage. (a) Input (b) st (c)5 nd (d) th Fg. Input mage and teraton results. III. FACE SHAPE CLASSIFICATION A. Bref Introducton of Support Vector Machne Support Vector Machnes are maxmal margn hyperplane classfcaton methods that rely on results from statstcal learnng theory to guarantee hgh generalzaton performance. It s frstly used n bnary stuatons. SVM s capable to solve lnear and nonlnear classfcaton problems. In the nonlnear case a kernel functon s used to map the nput data nto the feature space, normally wth hgher dmensonalty. The propose of SVM s to map ths nonlnear data nto a hgher dmensonal space, maybe nfnte, and make them lnearly separable n that space. Gven a tranng set of nstance-label n R and {, } l pars (x, y ), =,,l where requres the soluton of the followng optmzaton problem: T mn / 2( ww) + C wb,, ξ x l = ξ y, SVM T Subject to y ( w φ( x ) + b) ξ, ξ (4) The most common used kernels nclude: lnear, polynomnal, radal bass functon (RBF) and sgmod. Currently, there are several strateges for solvng multclass SVM classfcaton problems, whch almost extended from bnary SVM. The frst way s to combne several bnary SVMs together and generalze a multclass SVM, such as one-versus-all, one-versus-one, DAGSVM. The other method s to consder all optmzaton problems of sub-classfers parameter n one formulaton, such as SVM lght, whch s mplemented by Vanpnk[8]. In ths paper, a one-versus-one multclass SVM s used as an effcent classfer both for face shape and expresson classfcaton. B. One-versus-One Multclass SVM One-versus-one multclass SVM strategy s one of the most effcent classfyng method, whch can be consdered as an votng process by means of several bnary SVMs. Suppose A, B and C represent the 3 classes of face shapes-- melon seed, round and square. We compose every two the them and do bnary SVM tranng process,.e. (A,B), (A,C) and (B,C). For n classes stuaton, the number of bnary SVMs s n(n-)/2. Subsequently, a test sample wll be classfed by the 3 bnary SVMs. Fnal result s obtaned by means of votng,.e. the class that has the most votes s the fnal result. The votng process s as follows: 65

3 Internatonal Journal of Research n Engneerng and Technology (IJRET) Vol. 2, No. 4, 23 ISSN ) Intalzaton : Vote(A)= Vote(B)= Vote(C)=; 2) If the test sample s classfed nto class A, usng SVM(A,B), Vote(A)=Vote(A)+. Otherwse, Vote(B)=Vote(B)+; Respectvely, test sample s also classfed by SVM(A,C) and SVM(B,C). 3) Fnal result: Test sample s class = Max {Vote(A), Vote(B), Vote(C)}. In ths paper, the famous BSVM lbrary s used as multclass SVM classfer, whch s developed by ChhJen Ln[9] C. Face Shape Classfcaton As s known to all, the most obvous dfference of human face appearance s the dfference of face shape. Some people have a face lke melon seed, someone s face seems lke round and some people s face unfortunately has a square shape. See Fg.2. Snce the mult-person s expresson recognton performance s mostly affected by dentty, gender and especally face shape, t s pretty necessary to remove the man nose resulted from dfferent people s face shape. The proposed soluton s as follows: ) AAM modelng and feature extracton: Select mages ncludng 3 face shapes and 6 expressons as many as possble and choose the most representatve samples to buld AAM model, where the shape parameters p s used as the extracted facal feature. 2) SVM tranng: Select sub-database ncludng 6 expressons for every face shape and tran a 3-class SVM model for face shape classfcaton. 3) Face shape classfcaton: Test mage can be classfed after tranng the face shape SVM model. Note that: face mage wth any expresson can be classfed nto dfferent face shape. IV. EXPERIMENT RESULTS A. Database Selecton For the purpose of buldng an AAM model whch can gve a satsfyng fttng performance, two facal expresson databases are used. One s JAFFE data base, ncludng 6 Japanese females and each female has 6 expressons. The other one s bult by Soongsl Unversty students n Korea, ncludng 4 persons and each person has 6 expressons. Expresson moves also exst from whch t s convenent to obtan enough mage sequences for SVM tranng. The total number of mages n the two databases s 485, whch s enough for the two-stage SVM s tranng. Some samples are as follows accordng to dfferent face shape: Fg.3 Sample mages for 3 face shapes ncludng 6 expressons B. Fttng Results In order to acheve a satsfyng fttng performance and smultaneously avod the SVM over-fttng problem resulted from hgh dmenson of shape parameters p, we just use the most representatve mages to buld the AAM model. The dmenson of p s 35, whch s far smaller compared wth the tranng mage number 485. The fttng results are as follows: Fg.4 Fttng results of some sample mages. (a) melon seed (b) round (c) square Fg. 2 Dfferent face shape Summarzng, the system has a feature extractng mechansm based on AAM algorthm and a multclass SVM model based on one-versus-one votng strategy. For an nput mage, the AAM fttng framework extracts the normalzed shape parameter p whch represents the mage. Furthermore, a multclass SVM model s traned usng all face mages wth 6 expressons by means of shape parameters. Fnally, face shape classfcaton can be performed. C. Face Shape Classfcaton: For face shape classfcaton, we select 53 mages as melon seed face shape tranng data, ncludng 6 expressons. Respectvely, 538 mages for round faces and 434 mages for square faces. Consequently, we randomly select 8 mages as test mages. The face shape classfcaton results are as follows: TABLE I FACE SHAPE CLASSIFICATION RESULTS Face shape Melon seed Round square Recognton rate(%)

4 Internatonal Journal of Research n Engneerng and Technology (IJRET) Vol. 2, No. 4, 23 ISSN D. Analyss of classfcaton results TABLE I shows that not all face shapes can be classfed correctly. The reason s that tranng faces for face shape SVM classfer nclude 6 expressons, whch to some degree may change the orgnal face shape. As a result, a test mage may be classfed nto a false shape. Obvously, ths stuaton mostly happens n surprse expresson. For neutral, sadness and anger expresson, face shape can be almost classfed correctly. Correct classfcaton samples are gven n Fg.5. False classfcaton samples are gven n Fg.6. Images a-c are the false classfcaton results that round face shape are classfed n to melon seed ; Images d and e show that square face shape are falsely classfed nto melon seed. (b) square Fg. 5 Test mages that are correctly classfed nto melon seed (c) round Fg.4 Experment result Fg.6 False shape classfcaton results. surpse expresson s wrongly recognzed as melon seed. Fg. 4 shows the man process of operatng the system and gves the recognton result. Frstly, AAM model and a face detector usng haar-lke feature are loaded n order to complete the ntalzaton process. Secondly, a 3-class SVM for face shape classfcaton s loaded. Fnally, ICIA fttng algorthm s used to do AAM model matchng and extract facal feature by means of shape parameters, whch are used as the test data for SVM. V. CONCLUSION AAM-based face shape classfcaton method used for facal expresson recognton was effcently acheved by means of a multclass SVM usng one-versus-one votng strategy, whch s very mportant for the subsequent facal expresson recognton. However, one thng that cannot be gnored s that face shape may change wth the dfferent expresson, especally for surprse expresson. For example, a square face can be changed nto melon seed f the expresson surprse s performed. But for ths case, even though the face shape s wrongly classfed, the fnal expresson recognton result wll also be rght, snce surprse s a very obvously recognzed expresson for SVM. Furthermore, our method s very effcent especally for the expresson of neutral, sad, and dsgust, snce these expressons have a slght effect on face shape. (a) Melon seed ACKNOWLEDGMENT Ths research was supported by Next-Generaton Informaton Computng Development Program through the Natonal Research Foundaton of Korea (NRF) funded by the Mnstry of Educaton, Scence and Technology (No. 22M3C4A732 82)). And ths research was also supported by the MSIP (Mnstry of Scence, ICT & Future Plannng), Korea, under the ITRC(Informaton Technology Research Center) support program (NIPA-23-H3-3-26) supervsed by the NIPA(Natonal IT Industry Promoton Agency). REFERENCES [] W. John and L. Sons, Handbook of Cognton and Emoton, Dalglesh, 999, pp

5 Internatonal Journal of Research n Engneerng and Technology (IJRET) Vol. 2, No. 4, 23 ISSN [2] D. J. Ltman and K. Forbes-Rley, Predctng student emotons n computer-human tutorng dalogues. n Proc. of the 42nd Annual Meetng on Assocaton for Computatonal Lngustcs ACL, Barcelona, 24, pp. 35. [3] L. Gralewsk, N. Campbell, B. thomas, C. Dalton, and D. Gbson, Statstcal synthess of facal expressons for the portrayal of emoton, In Proc of the 2nd Internatonal conference on Computer graphcs and nteractve technques, Australasa and South East Asa, 24, pp [4] I. Kotsa, and I. Ptas, Facal Expresson Recognton n Image Sequences Usng Geometrc Deformaton Features and Support Vector Machnes, IEEE J. mage processng, Vol. 6, pp [5] G. J. Edwards T. F. Cootes and C. J. Taylor, Actve appearance models, IEEE Trans Pattern Analyss and Machne Intellgence, 2. [6] S. Sclaroff and J. Isdoro, Actve blobs, Proc. 6th IEEE Internatonal Conference on Computer Vson, 998, pp [7] I. Matthews and S. Baker, Actve appearance models revsted, Internatonal Journal of Computer Vson, 24, vol. 6, pp [8] V. N. Vapnk, The Nature of Statstcal Learnng Theory. 995, Sprnger, pp [9] C. J. Ln, BSVM,