Generating a Mapping Function from one Expression to another using a Statistical Model of Facial Shape

Transcription

1 Generatng a Mappng Functon from one Expresson to another usng a Statstcal Model of Facal Shape John Ghent Computer Scence Department Natonal Unversty of Ireland, Maynooth Maynooth, Co. Kldare, Ireland jghent@cs.may.e John McDonald Computer Scence Department Natonal Unversty of Ireland, Maynooth Maynooth, Co. Kldare, Ireland johnmcd@cs.may.e Abstract We demonstrate a novel method of generatng a mappng functon from the shape of a neutral face to the shape of one showng an expresson. It s proposed that ths mappng functon can be used to automatcally generate facal expressons from stll mages of never seen before faces or classfy whch expresson a person s portrayng. Ths new technque draws on the work of Ekmans [8] Facal Acton Codng System (FACS), Cootes [4,5] Actve Shape Model (ASM) and Artfcal Neural Networks (ANN). To buld an ASM t s requred to have a tranng phase, where each mage n the tranng phase s scored by the FACS system and a neural network s used to generate a mappng functon as a face moves from a neutral expresson to an alternatve expresson. We descrbe ths method n detal and gve results ndcate the effectveness of the technque. Keywords: Facal Expresson Generaton, Actve Shape Model (ASM), Facal Acton Codng System (FACS), Prncple Component Analyss (PCA), Feedforward Heteroassocatve Memory Network. Introducton The purposes of ths paper are, frstly to demonstrate how facal expressons can be modelled n a consstent manner, secondly, to buld a statstcal model of facal expressons, and thrdly, to show that a mappng may be derved that maps an mage correspondng to a neutral expresson to an mage correspondng to a non-neutral expresson. We address the problem by generatng a mappng from a neutral expresson to a non-neutral expresson ndependent of the subject. Ths s acheved by buldng a statstcal model of the expresson n queston from a number of subjects showng that expresson n a tranng set. The change n shape of each face n the tranng phase s then analysed and used to derve a mappng functon, whch takes ther neutral face to one depctng the new expresson. Ths mappng functon from one expresson to another s a dffcult problem. To decrease the dmensonalty of ths mappng the varance n shape of each face n the tranng set s analysed usng prncple components analyss [0]. Ths approach can model a large amount of the varance n the tranng set by usng only a few prncple axes or prncple components. In other words, the goal s to obtan a smaller set of features or vectors that reduces the complexty of the mappng but can accurately represent the orgnal tranng set. Before a model of facal expresson can be developed, a system measurng facal expressons would have to be n place. Ths s necessary for measurng the accuracy of results and for allowng consstency n expresson descrpton regardless of dentty. Ths s acheved usng the Facal Acton Codng System (FACS) developed by Ekman [8]. The FACS system separates the muscles n the face nto Acton Unts (AU) and uses these AU's as a bass for facal movement. As wth many vson and graphcs applcatons, the generaton of facal expresson s often a dffcult computatonal problem. Solutons to ths problem are often are too complcated to run n real tme and therefore cannot be used n adaptve applcatons. Those that can run n real tme are often naccurate and could never be used for applcatons that requre a hgh level of robustness. Pcard [20] detals a procedure that has a 98 percent accuracy rate at classfyng facal expressons but takes fve mnutes to process a facal expresson. A method has been developed that augmented Ekmans FACS for facal expresson and recognton usng templates of moton energy, but ths cannot be run n real tme. Yaser Yacoob and Larry Davs of the Unversty of MaryLand, use templates of moton energy, but use a combnaton of templates and smaller sub-templates and combne them wth rules to formulate expressons [20]. Ths method s also not real-tme, because computng the moton flow s very tme consumng.

2 The queston now becomes: what s the most effectve, effcent and accurate method or combnaton of methods sutable for automatcally generatng and recognzng arbtrary facal expressons n real tme? A method s proposed n ths paper that detals a procedure that provdes accurate results that could be run n real tme. Ths technque could be used n many applcatons such as an arbtrary anmated agent (a photo realstc anmaton of facal expresson), automatng an nteractve web host (nstead of streamng vdeo across a network, a sngle mage could be transmtted and the mage could be manpulated on the machne the user s usng), face and expresson verfcaton (explaned below), and addng personalty and emotonal expressons to an arbtrary mage of a face. Other useful applcatons would be n the feld of verfcaton and authentcaton. Ths paper shows that there s a dstnct change of facal parameters regardless of dentty (see secton fve). Ths suggests that any expresson can be automatcally classfed or generated usng ths novel technque. Ths has useful mplcatons, f a user of a system had a web camera mounted on top of ther montor, ths technque can be used to correctly dentfy that persons expresson. It s mportant to note that ths s not facal recognton but expresson recognton. However, f ths mappng could be nverted, the technque could be could be used n facal recognton systems. 2 Measurng facal expresson An understandng of how the face creates expresson s necessary before a model of facal expresson can be generated. As everyone's facal features are unque, the only feasble way to measure an expresson s by the movement of the muscles on the face. For ths reason the anatomy of the face s a very mportant aspect n understandng expresson 2. Facal Acton Codng System (FACS) The FACS s based on an anatomcal analyss of facal actons [8]. A movement of one or more muscles of the face s known as an acton unt (AU). Sometmes t s hard to dstngush f a set of muscles are accountable for an expresson or f a sngle muscle s, for ths reason the term 'Acton Unt's used. All resultng expressons can be descrbed usng the AU's or a combnaton of the acton unts. The FACS System can account for over 0,000 expressons. 3 Buldng a statstcal model In order to buld a relable model of facal expresson, the model must provde adequate flexblty to cover the vast dfferences n human expressons. However, t was also mportant that the models only deform n ways consstent wth the FACS. In other words the model should only be allowed generate expressons that are consstent wth the tranng set. The followng sectons detals how such a model s bult. 3. Labellng the tranng set In order to represent a shape we frst have to label t wth a set of ponts. These are located around the face and around key areas such as the eye, nose, mouth and eyebrow. These ponts are weghted on ther level of mportance, dependng on whch AU the statstcal model s beng bult for. For example, f a model were beng bult for AU 23 (Orbculars Ors) then the ponts around the mouth would have a greater weght than the ponts around the eye. There are 22 ponts n total, 27 around the outlne of the face, 24 around the mouth, 5 around the nose, 8 around each eye, 8 around each pupl and 2 around each eyebrow. Fg shows a labelled face. Fg. A Screen shot of the software used to label the face

3 3.2 Algnng the tranng set To analyse the varance of the ponts that descrbes the shape of the face t s necessary that the faces n the tranng set are as closely algned as possble. One way of dong ths s to use a technque called the generalzed procrustes algnment (GPA)[4]. Ths technque algns two shapes wth respect to poston, rotaton and scale by mnmzng the sum of the squared dstances between the correspondng landmark ponts dscussed n the prevous secton. The algnment wll depend on the weghts gven to each of the ponts, whch n turn depends on whch synthetc AU s beng generated. To algn two shapes x and x 2, we choose a rotaton, scale and translaton that mnmses the sum of the squared dstances between x and x 2. Let x be T x = ( x 0, y0, x, y,, xn, yn ) () where n s the number of ponts used to descrbe the shape of the face. In ths case n s 22. Equaton (2), s the rotaton, scale and translaton equaton that s to be mnmsed [3]. T ( x M ( s )[ x ] t) W( x M ( s, θ )[ t) E = θ ] (2), 2 x 2 M ( s, θ )[ x] s the rotaton θ, and scale s of x, an t s a translaton of x. The complexty of the mappng to be learned depends on the way t s represented. In ths case the mappng of an expresson change needs to be ndependent of dentty. For each dentty, the neutral expresson s algned to the non-neutral expresson usng the labelled ponts that do not move as an expresson s formed. For example, f AU 4 (brow lowerer) were beng mapped, the neutral face and the face showng AU4 would be algned usng every pont except the ponts along the eyebrow. Ths wll emphasze the varance along the eyebrow as that expresson s formed and allow for a less complex mappng functon to be generated. Ths realgnment s the key aspect n fndng a mappng functon for ths expresson that s ndependent of dentty. 3.3 Prncple Component Analyss (PCA) Before any sgnfcant analyss can be performed on the shape of the faces, the mean must be acqured. Ths s found usng where N s the number of mages n the tranng set. N x = x (3) N = The way n whch the landmark ponts tend to move wth respect to each other can be analysed usng a method known as Prncpal Component Analyss (PCA, also known as the Karhunen-Loéve transform). Ths method takes a set of data ponts and constructs a lower dmensonal lnear subspace that bests descrbes the varaton of these data ponts from ther mean. Ths s computed by x = x x (4) where (4) represents the dfference the frst face has wth s correspondng ponts n the mean. A 2n x 2n covarance matrx can be calculated usng: N S = x x N = T (5) The egenvalues and egenvectors of the covarance matrx are then calculated. The egenvector correspondng to the largest egenvalue descrbes the most sgnfcant mode of varaton. Only a few prncple axes are needed to descrbe the majorty of varance of the tranng set. If P s the set of egenvectors and b s a vector of weghts, then new shapes can be generated usng equaton (6): x = x + Pb (6) where P s a 2n x m matrx, b s a m x vector and m << 2n.

4 4 Generatng a mappng To compute the mappng we use a Feedforward Heteroassocatve Memory Network (FHMN) [9]. Ths knd of network s smply a one-layer network that stores patterns. FHMN nclude the class of networks known as content addressable memores or memory devces that permt the retreval of data from pattern keys that are based on attrbutes of the stored data. A smple FHMN s shown n Fg 2. Fg 2. A Feedforward Heteroassocatve Network As the dagram suggests, f the nput to a group of neurons s x and the output exctatons are y = y(x), then the synaptc weght values W are gven by j where α s a normalzng constant. To retreve a pattern k k y, the assocated pattern x s nput to the network, thus: W = αx y (7) j P T k ( x ) x P k P Wx = y (8) P= In other words, f we had fve nput-to-output assocatons to be learned, we would frst compute each W by usng T 2 5 W = y (x ), then create the fnal matrx W usng W = W + W + + W. It s then possble to retreve patterns usng equaton (8). 5 Experments and results Both of the experments descrbed n ths paper demonstrate two dstnct propertes of ths technque. The frst experment demonstrates that usng a small tranng set and a subtle change n facal expresson t s stll possble to generate a mappng functon. The second experment demonstrates how a mappng can be generated for a more substantal expresson change such as a smle and shows how ths can be used to predct the most probable change n facal shape n never seen before facal mages. 5. Experment one Ten people were used to create a mappng from a neutral face to one showng AU 4. Each mage was taken usng a Sony EVI- D3 CCD camera. The lghtng, pose and orentaton of the faces were kept consstent so that the varance measured would be due to change n expresson and dentty only. Each person was asked to learn AU 4 as descrbed by the FACS system and perform the acton unt wth ntensty C. AU4 s a brow lowerer. There are 5 ntenstes A,B,C,D,E, where A stands for the mnmum, up to E, the maxmum ntensty. AU C s the most common so ths s what was modelled.

5 Fg 3. Percentage of egenvalues Fg 4. Mappng taken from AU0 to AU4 n egenspace Each face was labelled usng 22 ponts, as n Fg. They were algned wth each other usng procrustes algnment and then for each dentty the neutral expresson was algned to the non-neutral expresson n a way that best showed the varance of shape across the eyebrows. PCA was performed on the data and the top three egenvalues were used to generate the approprate mappng. The top three egenvalues explan 85.05% of the total varance, as shown n Fg3. A mappng was then generated usng the top three egenvalues, whch represented the neutral faces as nput, and the largest three egenvalues that represented the faces showng AU4C (acton unt 4 wth ntensty C) as output. A FHMN was used to generate the mappng. From equaton (6), the parameters b, used for the nputs and outputs can be acqured usng ( y x) b = V T (9) where V are the egenvectors whch explan the varance shown n the tranng set and y s the shape of a face. Fg 4 shows the mappng for a number of subjects from a neutral expresson to AU4. The '+'sgn represents the nput vector (the largest three egenvalues used to portray AU0 or a neutral face for a specfc dentty) and the 'o'symbol represents the output vector (the largest three egenvalues used to portray AU4 for a specfc dentty). Once the mappng was created the model was tested by puttng the nputs through the mappng functon and comparng them to the orgnal outputs. The output b of each nput was converted nto a facal shape usng equaton (6). Fg 5 shows the results of puttng the orgnal nput vectors through the mappng functon. Fg 5.AU0 mage supermposed over AU4 mage Fg 6. The error of the mappng The '+'sgns n Fg 5 represents the shape of a face showng AU4 whle the 'box'symbol represents the shape of the face when showng a neutral expresson. It can be seen that the '+'s of the eyebrows are lowered n the mage. Ths shows that t s possble to map a subtle dfference n shape wthout affectng the rest of the shape. There s a small error here as only the largest three egenvalues were used n the mappng functon, ths means that only 85.05% of the total varance was taken nto consderaton. Ths can be seen more clearly usng Fg 6. The greatest varance s wth the ponts that mark the shape of the eyebrows; these ponts are around the 200 mark on the x-axs. Ths provdes a

6 measure of the error; t shows the devaton from the orgnal mage. Fg 6 s result of subtractng an orgnal neutral mage from a synthetc mage portrayng AU4 of the same ndvdual. The purpose of ths mage s to show how ths model captures the holstc nature of facal expressons. Although only the eyebrows move under ths transformaton, ths model uses the entre face to calculate how they move. 5.2 Experment two Thrty mages of dfferent subjects were used to create a mappng from a neutral expresson to a smle. Images were acqured usng the same method as experment one and usng the 'AR face database'developed by Martnez [7]. The lghtng n these mages was constant and the pose and orentaton of the mages were kept the same, as n experment one. The procedure for computng the mappng functon was the same as experment one. As some of the faces were taken from a database that was not consstent wth the FACS system, the mappng from a neutral expresson to one showng a smle was expected to be less accurate than the mappng generated n the frst experment. That s the ntensty of the smle n the mages n the tranng set was not unform. Fg 7 shows ths mappng and ts dscrepancy. Fg 7. The Mappng taken from a neutral expresson to a smle The '+'sgn represents the nput vector (the largest three egenvalues used to portray AU0 or a neutral face for a specfc dentty) and the 'o'symbol represents the output vector (the largest three egenvalues used to portray a smle for a specfc dentty). The non unform mappng of a face movng from a neutral expresson to a smle can be accounted for n two ways, frstly, the mages used n the tranng set were not consstent wth the FACS system and secondly, only the frst three egenvalues are represented n the dagram. The frst three egenvalues account for only 65.69% of the total varance found durng the tranng phase. Fg 8 shows how a larger tranng set ncreases the amount of components needed to explan the total varance of the tranng set. Fg 8. Top 30 egenvalues The purpose of experment two was to demonstrate that a mappng functon can be created that can predct accurately how a facal shape wll change as the formaton of a smle emerges. Fg 9 shows how three never seen before facal shapes would change as predcted by the mappng functon as they move from AU0 to a smle. The mappng functon was bult up usng the same FHMN used n experment one.

7 Fg 9. Faces on the bottom row are the results of applyng the mappng functon to the faces on the top row The results shown n Fg 9 suggest that any expresson can be mapped usng the novel technque descrbed n ths paper. The mages n the top row are representatons of the shapes of the never seen before faces and the mages n the bottom row are how the mages look after been passed through the mappng functon whch takes an mage from a neutral expresson to a smle. 6 Concludng remarks and future work Ths paper detals a method of generatng a mappng functon as a face moves from a neutral expresson to a non-neutral expresson. Ths new method combnes the Facal Acton Codng System (FACS), Actve Shape Model (ASM) and Artfcal Neural Networks (ANN) namely Feedforward Heteroassocatve Memory Networks (FHMN). The procedure nvolves the creaton of a database of mages of faces, the mages would portray neutral expressons and non-neutral expressons. The nonneutral expressons selected for a specfc mappng would all score the same AU s as stpulated by the FACS. All the mages would be used n a tranng phase to buld an ASM. The largest prncple axes used to portray a neutral expresson would be used as nput to a FHMN and the largest prncple axes used to portray a specfc AU would be used as output to a FHMN. In ths manner a mappng functon s created. The problem of generatng a mappng functon as an expresson moves from a neutral expresson to a non-neutral expresson can be smplfed through the use of the Actve Shape Model (ASM). There were eght modes of varaton n the example n experment one n ths paper and the three largest prncple components accounted for 85.05% of the total varance. These three prncple components were used n a neural network to frstly, see f a mappng exsts and secondly, to create a mappng functon, f possble. Fg 4 suggests that there s a clear mappng of expresson as a face went from a neutral expresson to one showng AU4 (brow lowerer), The mappng functon was generated and was appled to the nputs. The results show a clear lowerng of the eyebrow n each face shape (see Fg 5). Ths strongly suggest that usng the prncple components of the shape model as nputs to a traned neural net, any AU can be modelled and therefore a functon for expresson formaton can be created. In experment two a larger tranng set was used and twenty of the mages were not consstent wth a specfc AU as descrbed by the FACS system. The results of ths experment demonstrate two thngs, frstly that wthout a proper measure of expresson such as the FACS system, ths mappng functon would be less accurate as suggested by Fg 7, and secondly, that the mappng can be successfully used on never seen before faces (Fg 9). Wthout a large enough egenspace to represent a new face t would be mpossble to generate the parameters for that facal shape (see equaton 9) and hence produce ts most probably change n shape. These functons would allow for classfcaton and generaton of expresson once a neutral mage of a face shape was known. It s predcted that a greater accuracy would be acheved f more examples where used n the tranng set.

8 Buldng the ASM and the ANN to model expresson s a computatonally complex problem, but because most of the work takes place n the tranng phase, ths new hybrd method consstng of the FACS, the ASM and neural networks can be mplemented n real-tme. Ths means that ths technque can be used n adaptve applcatons to create and run vrtual avatars and also to dentfy the users expresson. It s mportant to note that ths technque would not classfy emotons as a further tme dependent system would need to be n place before tryng to classfy emotons. It s planned to develop ths technque to take nto account the grey scale level of the faces as well as the shape, ths would be acheved by performng PCA on the shape, the grey scale level and the by performng statstcal analyss on how the shape changes wth respect to the grey scale levels (ths technque s better known as the Actve Appearance Model (AAM), Cootes [5]) It s also planned to analyse the trajectory of the expresson vector n feature space as an expresson changes. Ths wll be done usng a tranng set of people showng dfferent ntenstes of an AU as descrbed by the FACS system. Ths wll show f the prncple components take a specfc path through egenspace as an expresson s formng. References [] Anton, and Rorres, Elementary Lnear Algebra, Seventh Edton, Wley and Sons, 994. [2] Balke and Isard Actve Contours, The Applcaton of technques from graphcs, vson, control theory and statstcs to vsual trackng of shapes n moton, Sprnger, 998. [3] Cootes, Taylor, Cooper, and Graham, Actve Shape Models - Ther Tranng and Applcaton, Computer Vson and Image Understandng, Vol. 6, No., January, pp , 995. [4] Cootes and Taylor Statstcal Models of Appearance for Medcal Image Analyss and Computer Vson, Imagng Scence and Bomedcal Engneerng, Unversty of Manchester, UK, Proceedngs, SPIE Medcal Imagng, 200. [5] Cootes, Edwards, and Taylor Actve Appearance Model, 5 th European Conference on Computer Vson, Burkhardt and Neumann, eds., Vol. 2, pp , Sprnger, Berln.998. [6] Cootes and Taylor Statstcal Models of Appearance for Computer Vson, Wolfson Image Analyss Unt, Imagng Scence and Bomedcal Engneerng, Unversty of Manchester, Manchester M3 9PT, U.K. October 26th, 200. [7] Crstann, and Shawe-Taylor Support Vector Machnes, Cambrdge Unversty Press, [8] Ekman and Fresen Facal Acton Codng System, Human Interacton Laboratory, Dept. of Psychatry, Unversty of Calforna Medcal Centre, San Francsco, Consultng Psychologsts Press, Inc. 577 College Avenue, Palo Alto, Calforna 94306, 978. [9] Fagan The Artst's gude to Facal Expressons, Watson-Guphll Publcatons, 990. [0] Forsyth and Ponce Computer Vson: A Modern Approach, Prentce Hall, st Edton, August 4, [] Ghent, McDonald, and Harper, A Statstcal Model for Expresson Generaton usng the Facal Acton Codng System, NUIM, TR No:NUIM-CS-TR , 2003 [2] Gladln, Zachow, Deuflhard, Hege, Towards a Realstc Smulaton of Indvdual Facal Mmc, Zuse-Insttute- Berln (ZIB) Takustr, 7, D-495 Berln, Germany, November 2-23, 200. [3] Gosseln and Schyns Bubbles: a technque to reveal the use of nformaton n recognton task, Vson Research, Department of Psychology, Unversty of Glasgow, 56, Hllhead Street, Glasgow G2 8QB, Scotland, UK, December 3rd, [4] Gower Generalsed Procrustes Analyss, Psychometrka, Vol. 40, pp , 975. [5] Johnson and Wchern, Multvarate Statstcs, A Practcal Approach, Chapman and Hall, 988. [6] Lants, Taylor and Cootes, A Generc system for classfyng varable objects usng flexble template matchng, Proceedngs, Brtsh Machne Vson Conference, 993 (J. Illngworth, Ed.), Vol, pp , BMVA Press, 993. [7] Martnez, and Benavente, The AR face database CVC Tech. Report No.24, 998. [8] McDonald, Markham, and McLoughln, Selected Problems n Automated Vehcle Gudance, Sgnals and Systems Group, Department of Computer Scence, NUI Maynooth, 27th Aprl 200. [9] Patterson Artfcal Neural Networks, Theory and Applcaton Prentce Hall, 995. [20] Pcard, Affectve Computng MIT Press, 998. [2] Prncpe, Eulano, and Lefebvre, Neural and Adaptve Systems, Wley,2000. [22] Stephens Vsual Basc Graphcs Programmng, Second Edton, Wley, 999. [23] Vernon Machne Vson, Automated Vsual Inspecton and Robot Vson, Prentce Hall, 99. [24] Xue, L, and Teoh, Facal Feature Extracton and Image Warpng usng PCA Based Statstcal Mode, Mcrosoft Research Chna, Zh Chun Road, Hadan Dstrct Bejng, 00080, P.R. Chna, 2000