A Statistical Discriminant Model for Face Interpretation and Reconstruction

Transcription

1 A Statstcal Dscrmnant Model for Face Interpretaton and Reconstructon Edson C. Ktan, Carlos E. homaz, and Duncan F. Glles 2 Department of Electrcal Enneern, Centro Unverstáro da FEI, São Paulo, Brazl 2 Department of Computn, Imperal Collee, London, UK {ektan,cet}@fe.edu.br, 2 d.lles@mperal.ac.uk Abstract Multvarate statstcal approaches have played an mportant role of reconsn face maes and characterzn ther dfferences. In ths paper, we ntroduce the dea of usn a two-stae separatn hyper-plane, here called Statstcal Dscrmnant Model (SDM), to nterpret and reconstruct face maes. Analoously to the well-known Actve Appearance Model proposed by Cootes et. al, SDM requres a prevous alnment of all the maes to a common template to mnmse varatons that are not necessarly related to dfferences between the faces. However, nstead of usn landmarks or annotatons on the maes, SDM s based on the dea of usn PCA to reduce the dmensonalty of the ornal maes and a maxmum uncertanty lnear classfer (MLDA) to characterse the most dscrmnant chanes between the roups of maes. he expermental results based on frontal face maes ndcate that the SDM approach provdes an ntutve nterpretaton of the dfferences between roups, reconstructn characterstcs that are very subjectve n human bens, such as beauty and happness.. Introducton he most successful statstcal models for vsual nterpretaton of face maes have been based on Prncpal Component Analyss (PCA) [, 3, 0]. hese approaches have used as features ether shapes [3] or textures [0] alone, or a combnaton of both []. Unfortunately, however, even n the PCA approach based on a combnaton of features, the sources of the shapes and textures varatons have to be solated n order to extract and nterpret the most expressve dfferences n the trann samples. For nstance, n the well-known Actve Appearance Model proposed by Cootes et. al. [] the shape model s dssocate from the texture model and a manual annotaton of landmarks s necessary to perform the statstcal analyss. In ths paper, we ntroduce the dea of usn a twostae separatn hyper-plane, here called Statstcal Dscrmnant Model (SDM), to nterpret and reconstruct face maes. Analoously to the Cootes et al. approaches [ 4], SDM requres a prevous alnment of all the maes to a common template to mnmse varatons that are not necessarly related to dfferences between the faces. However, nstead of usn landmarks or annotatons on the maes, SDM s based on the dea of usn PCA to reduce the dmensonalty of the ornal maes and a maxmum uncertanty lnear classfer (MLDA) [8] to characterse the most dscrmnant dfferences between the samples of maes. he remander of ths paper s dvded as follows. In secton 2, we brefly revew PCA and hhlht ts mportance on reducn the hh dmensonalty of face maes. Secton 3 descrbes the standard lnear dscrmnant analyss (LDA) and states the reasons for usn a maxmum uncertanty verson of ths approach to perform the face experments requred. he estmaton of the separatn hyper-plane and the mplementaton of the Statstcal Dscrmnant Model are descrbed n Secton 4. In secton 5, we present expermental results of the PCA and SDM approaches on a face database mantaned by the Department of Electrcal Enneern at FEI. hs secton ncludes reconstructon experments of face maes usn the SDM approach proposed. In the last secton, secton 6, the paper concludes wth a short summary of the fndns of ths study and future drectons. 2. Prncpal Component Analyss (PCA) PCA s a feature extracton procedure concerned wth explann the covarance structure of a set of varables throuh a small number of lnear combnatons of these varables. It s a well-known statstcal technque that has been used n several mae reconton problems, especally for dmensonalty reducton. A comprehensve descrpton of ths multvarate statstcal analyss method can be found n [6]. Let us consder the face reconton problem as an example to llustrate the man dea of the PCA. In any

2 mae reconton, and partcularly n face reconton, an nput mae wth n pxels can be treated as a pont n an n-dmensonal space called the mae space. he coordnates of ths pont represent the values of each pxel of the mae and form a vector [ x, x2, x ] x =, n obtaned by concatenatn the rows (or columns) of the mae matrx. It s well-known that well-framed face maes are hhly redundant not only own to the fact that the mae ntenstes of adjacent pxels are often correlated but also because every ndvdual has one mouth, one nose, two eyes, etc. As a consequence, an nput mae wth n pxels can be projected onto a lower dmensonal space wthout snfcant loss of nformaton. Let an N x n trann set matrx X be composed of N nput face maes wth n pxels. hs means that each column of matrx X represents the values of a partcular pxel observed all over the N maes. Let ths data matrx X have covarance matrx S wth respectvely Φ and Λ eenvector and eenvalue matrces, that s, K P SP = Λ. () It s a proven result that the set of m ( m n ) eenvectors of S, whch corresponds to the m larest eenvalues, mnmses the mean square reconstructon error over all choces of m orthonormal bass vectors [6]. Such a set of eenvectors that defnes a new uncorrelated coordnate system for the trann set matrx X s known as the prncpal components. In the context of face reconton, those Ppca components are frequently called eenfaces [0]. herefore, althouh n varables are requred to reproduce the total varablty (or nformaton) of the sample X, much of ths varablty can be accounted for by a smaller number m of prncpal components. hat s, the m prncpal components can then replace the ntal n varables and the ornal data set, consstn of N measurements on n varables, s reduced to a data set consstn of N measurements on m prncpal components. 3. Maxmum Uncertanty LDA (MLDA) he prmary purpose of the Lnear Dscrmnant Analyss, or smply LDA, s to separate samples of dstnct roups by maxmsn ther between-class separablty whle mnmsn ther wthn-class varablty. Althouh LDA does not assume that the populatons of the dstnct roups are normally dstrbuted, t assumes mplctly that the true covarance matrces of each class are equal because the same wthn-class scatter matrx s used for all the classes consdered. Let the between-class scatter matrx S b be defned as S b = = and the wthn-class scatter matrx S w = = N ( x x)( x x) (2) N = j= S w be defned as ( N ) S = ( x x )( x x ) (3), j, j where x, j s the n-dmensonal pattern j from class π, N s the number of trann patterns from class π, and s the total number of classes or roups. he vector x and matrx S are respectvely the unbased sample mean and sample covarance matrx of class π [6]. he rand mean vector x s ven by x = N = N x = N N = j= x, j, (4) where N s the total number of samples, that s, N = N + N2 + L + N. It s mportant to note that the wthn-class scatter matrx S w defned n equaton (3) s essentally the standard pooled covarance matrx S p multpled by the scalar ( N ), where S p can be wrtten as 2 ) S 2 ( N ) S + ( N S p = ( N ) S = N = N + L + ( N ) S. (5) he man objectve of LDA s to fnd a projecton matrx P lda that maxmzes the rato of the determnant of the between-class scatter matrx to the determnant of the wthn-class scatter matrx (Fsher s crteron), that s, P lda P SbP = ar max. (6) P P S P he Fsher s crteron descrbed n equaton (6) s maxmsed when the projecton matrx P lda s composed of the eenvectors of S w S b wth at most ( ) nonzero correspondn eenvalues [5]. hs s the standard LDA procedure. However, the performance of the standard LDA can w

3 be serously deraded f there s only a lmted number of total trann observatons N compared to the dmenson of the feature space n. Snce the wthn-class scatter matrx S w s a functon of ( N ) or less lnearly ndependent vectors, ts rank s ( N ) or less. herefore, S w s a snular matrx f N s less than ( n + ), or, analoously, mht be unstable f N s not at least fve to ten tmes ( n + ) [7]. o avod the aforementoned crtcal ssues of the standard LDA n lmted sample and hh dmensonal problems, we have calculated P lda by usn a maxmum uncertanty LDA-based approach (MLDA) that consders the ssue of stablsn the S w estmate wth a multple of the dentty matrx [8, 9]. In a prevous study [8] wth applcaton to the face reconton problem, homaz and Glles showed that the MLDA approach mproved the LDA classfcaton performance wth or wthout a PCA ntermedate step and usn less lnear dscrmnant features [8]. he MLDA alorthm can be descrbed as follows:.fnd the Φ eenvectors and Λ eenvalues of where S = S [ N ] ; p.calculate the w S p averae eenvalue λ, that s, n S p, trace( S p ) λ = j = n λ ; (7a) n = j.form a new matrx of eenvalues based on the follown larest dsperson values Λ = da[max( λ, λ ),...,max( λ, λ )]; (7b) v.form the modfed wthn-class scatter matrx S w p = S ( N ) = ( ΦΛ Φ )( N ). (7c) he maxmum uncertanty LDA (MLDA) s constructed by replacn S w wth S w n the Fsher s crte- ron formula descrbed n equaton (6). It s based on the dea [8] that n lmted sample sze and hh dmensonal problems where the wthn-class scatter matrx s snular or poorly estmated, the Fsher s lnear bass found by mnmsn a more dffcult but approprate nflated wthn-class scatter matrx would also mnmse a less relable shrvelled wthn-class estmate. 4. Statstcal Dscrmnant Model (SDM) he Statstcal Dscrmnant Model proposed n ths work s essentally a two-stae PCA+MLDA lnear n classfer that reduces the dmensonalty of the ornal maes and extracts dscrmnant nformaton from maes. In order to estmate the SDM separatn hyperplane, we use trann examples and ther correspondn labels to construct the classfer. Frst a trann set s selected and the averae mae vector of all the trann maes s calculated and subtracted from each n- dmensonal vector. hen the trann matrx composed of zero mean mae vectors s used as nput to compute the PCA transformaton matrx. he columns of ths n x m transformaton matrx are eenvectors, not necessarly n eenvalues descendn order. We have retaned all the PCA eenvectors wth non-zero eenvalues, that s, m = N, to reproduce the total varablty of the samples wth no loss of nformaton. he zero mean mae vectors are projected on the prncpal components and reduced to m-dmensonal vectors representn the most expressve features of each one of the n-dmensonal mae vector. Afterwards, ths N x m data matrx s used as nput to calculate the MLDA dscrmnant eenvector, as descrbed n the prevous secton. Snce n ths work we have lmted ourselves to two-roup classfcaton problems, there s only one MLDA dscrmnant eenvector. he most dscrmnant feature of each one of the m- dmensonal vectors s obtaned by multplyn the N x m most expressve features matrx by the m x MLDA lnear dscrmnant eenvector. hus, the ntal trann set of face maes consstn of N measurements on n varables, s reduced to a data set consstn of N measurements on only most dscrmnant feature. Once the two-stae SDM classfer has been constructed, we can move alon ts correspondn projecton vector and extract the dscrmnant dfferences captured by the classfer. Any pont on the dscrmnant feature space can be converted to ts correspondn n-dmensonal mae vector by smply: () multplyn that partcular pont by the transpose of the correspondn lnear dscrmnant vector prevously computed; (2) multplyn ts m most expressve features by the transpose of the prncpal components matrx; and (3) addn the averae mae calculated n the trann stae to the n-dmensonal mae vector. herefore, assumn that the spreads of the classes follow a Gaussan dstrbuton and applyn lmts to the varance of each roup, such as ± 2 sd, where sd s the standard devaton of each roup, we can move alon the SDM most dscrmnant features and map the results back nto the mae doman.

4 (a) (b) Fure. Samples of the female versus male (a) and non-smln versus smln trann sets (b). 5. Expermental Results We have used frontal maes of a face database mantaned by the Department of Electrcal Enneern of FEI to carry out the experments. hs database contans a set of face maes taken between June 2005 and March 2006 at the Artfcal Intellence Laboratory n São Bernardo do Campo, wth 4 maes for each of 8 ndvduals a total of 652 maes. All maes are colourful and taken aanst a whte homoenous backround n an uprht frontal poston wth profle rotaton of up to about 80 derees. Scale mht vary about 0% and the ornal sze of each mae s 640x480 pxels. o mnmse mae varatons that are not necessarly related to dfferences between the faces, we alned frst all the frontal face maes to a common template so that the pxel-wse features extracted from the maes correspond rouhly to the same locaton across all subjects. In ths manual alnment, we have randomly chosen the frontal mae of a subject as template and the drectons of the eyes and nose as a locaton reference. For mplementaton convenence, all the frontal maes were then cropped to the sze of 64x64 pxels and converted to 8-bt rey scale. We have carred the follown two-roup statstcal analyses: female versus male experments, and nonsmln versus smln experments. he dea of the frst dscrmnant experment s to evaluate the statstcal approaches on a dscrmnant task where the dfferences between the roups are evdent. In contrast, the second experment,.e. non-smln versus smln samples, poses an alternatve analyss where there are subtle dfferences between the roups. Snce the number of female maes s lmted and equal to 49, we All these maes are avalable upon request (cet@fe.edu.br). have composed the female/male trann set of 49 frontal female maes and 49 frontal male maes. For the smln/non-smln experments, we have used the 49 frontal male maes prevously selected and ther correspondn frontal smln maes. All faces are manly represented by subjects between 9 and 30 years old wth dstnct appearance, harstyle, and adorns. Fure shows some examples of these two trann sets selected. 5.. PCA Results In ths secton, we descrbe the most expressve features captured by PCA. As the averae face mae s an n-dmensonal pont ( n = 4096 ) that retans all common features from the trann sets, we could use ths pont to understand what happens statstcally when we move alon the prncpal components and reconstruct the respectve coordnates on the mae space. Analoously to the works by Cootes et al. [ 4], we have reconstructed the new averae face maes by chann each prncpal component separately usn the lmts of ± λ, where λ are the correspondn larest eenvalues. Fure 2 llustrates these transformatons on the frst three most expressve prncpal components usn the female/male trann set. As can be seen, the frst prncpal component (on the top) captures essentally the varatons n the llumnaton and ender of the trann samples. he second prncpal component (mddle), n turn, models varatons related to the rey-level of the faces and har, but t s not clear whch specfc varaton ths component s actually capturn. he last prncpal component consdered, the thrd component (bottom), models manly the sze of the head of the trann samples. It s mportant to note that as the female/male trann set has a very clear separaton be-

5 Fure 2. PCA results usn the female/male trann set. tween the roups, the prncpal components have kept ths separaton and when we move alon each prncpal component axs we can see ths major dfference between the samples, even thouh subtly, such as n the thrd prncpal component llustrated. Fure 3 presents the three most expressve varatons captured by PCA usn the non-smln/smln trann set, whch s composed of male maes only. Analoously to the female/male experments, the frst prncpal component (on the top) captures essentally the chanes n llumnaton, the second prncpal component (mddle) models varatons partcularly n the head shape, and the thrd component (bottom) captures varatons n the facal expresson amon others. As we should expect, these expermental results show that PCA captures features that have a consderable varaton between all trann samples, lke chanes n llumnaton, ender, and head shape. However, f we need to dentfy specfc chanes such as the varaton n facal expresson solely, PCA has not proved to be a useful soluton for ths problem. As can be seen n Fure 3, althouh the thrd prncpal component (bottom) models some facal expresson varaton, ths specfc varaton has been captured by other prncpal components as well ncludn other mae artefacts. Lkewse, as Fure 2 llustrates, althouh the frst prncpal component (top) models ender varaton, other chanes have been modelled concurrently, Fure 3. PCA results usn the non-smln/smln trann set (male maes only).

6 -2sd -sd Mean +sd +2sd Fure 4. SDM results usn the female/male trann set. such as the varaton n llumnaton. In fact, when we consder a whole rey-level model wthout landmarks to perform the PCA analyss, there s no uarantee that a snle prncpal component wll capture a specfc varaton alone, no matter how dscrmnant that varaton mht be SDM Results As descrbed earler, n order to estmate the SDM separatn hyperplane, we have used the female/male and non-smln/smln trann sets prevously selected and ther correspondn labels to construct the classfer. Snce n these experments we have lmted ourselves to two-roup classfcaton problems, there s only one SDM dscrmnant eenvector. herefore, assumn that the spreads of the classes follow a Gaussan dstrbuton and applyn lmts to the varance of each roup, such as ± 2 sd, where sd s the standard devaton of each roup, we can move alon the SDM most dscrmnant features and map the results back nto the mae doman for vsual analyss. Fure 6 presents the SDM most dscrmnant features for the ender experments. It dsplays the mae reons captured by the SDM approach that chane when we move from one sde (left, male) of the dvdn hyper-plane to the other (rht, female), follown lmts to the standard devaton ( ± 2 sd) of each sample roup. As can be seen, the SDM hyper-plane effectvely extracts the roup dfferences, shown clearly the features that manly dstnct the female samples from the male ones, such as the sze of the eyebrows, nose and mouth, wthout enhancn other mae artefacts. Fure 5 shows the SDM most dscrmnant features for the facal expresson experments. Analoously to the ender experments, Fure 5 dsplays the mae reons captured by the SDM classfer that chane when we move from one sde (left, smln) of the dvdn hyper-plane to the other (rht, non-smln), follown lmts to the standard devaton ( ± 2 sd) of each sample roup. As can be seen, the SDM hyperplane effectvely extracts the roup dfferences, shown exactly what we should expect ntutvely from a face mae when someone chanes ther expresson from smln to non-smln. In fact, t s possble to note that the SDM most dscrmnant drecton has predcted a facal expresson not necessarly present n our correspondn smln/non-smln trann set, that s, the defntely non-smln or may be aner status represented by the mae +2sd n Fure 5. Analoously to the PCA experments, all SDM reconstructons have been made usn the averae face mae of the correspondn trann sets. However, t s possble to project any face mae on the SDM feature space, move alon ts correspondn most dscrmnant features, and map the chanes back to the ornal mae space. Fure 6 shows these expermental results when we move an example mae alon the male/female (Fure 6a) and smln/non-smln (Fure 6b) hyper-planes prevously calculated. As can be seen n Fure 6a, the most dscrmnant features be- -2sd -sd Mean +sd +2sd Fure 5. SDM results usn the smln/non-smln trann set.

7 (a) (b) Fure 6. SDM results when we move an example mae alon the male/female (a) and smln/non-smln (b) hyper-planes. tween a male and female face maes have been ncorporated on the example mae when we move t to the male sde of the dvdn hyper-plane, such as the thckenn of the lps, nose, and eyebrows. In contrast, snce the example chosen s from a woman, almost no facal chanes occurs when we move the same example to the other sde of the hyper-plane, that s, to the female sde. Also, accordn to Fure 6b, t s possble to see that the SDM lnear classfer has ncorporated all the most dscrmnant facal chanes that we ntutvely expect when we chane our facal expresson from smln to non-smln status. It s mportant to note n ths case where most of the facal chanes are localsed around the mouth that only the dfferences related to the facal expresson dfferences have chaned on the mae wth no mpact on other face features, such as har-style, forehead, eyebrows, and chn. 6. Concluson In ths work, we ntroduced the dea of usn the PCA+MLDA two-stae lnear classfer to nterpret and reconstruct frontal face maes rather than reconsn subjects. Dfferently from other statstcal approaches, our method s based on a supervsed separaton between the whole maes and not on the use of landmarks and solated models for the shapes and textures varatons. he experments carred out n ths work showed that subjectve nformaton such as beauty and happness can be effcently captured by a lnear classfer when we pre-process the face maes usn a smple affne transformaton. he results presented n ths paper suested that the statstcal dscrmnant model proposed could be useful to reconstruct not only frontal face maes but also face maes wth dfferent profles. Further work s ben undertaken to nvestate ths possblty. Acknowledments he authors would lke to thank Leo Leonel de Olvera Junor for acqurn and normalzn the FEI database under the rant FEI-PBIC References []. F. Cootes, G. J. Edwards, C. J. aylor, Actve Appearance Models, H.Burkhardt and B. Newmann edtors, n Proceedns of ECCV 98, vol. 2, pp , 998. [2].F Cootes, A. Lants, Statstcal Models of Appearance for Computer Vson, echncal report, Unversty of Manchester, 25 paes, [3]. F. Cootes, C. J. aylor, D. H. Cooper, J. Graham, Actve Shape Models- her rann and Applcaton, Computer Vson and Imae Understandn, vol. 6, no., pp , 995. [4].F. Cootes, K.N. Walker, C.J. aylor, Vew-Based Actve Appearance Models, In 4 th Internatonal Conference on Automatc Face and Gesture Reconton, Grenoble, France, pp , [5] P.A. Devjver and J. Kttler, Pattern Classfcaton: A Statstcal Approach. Prentce-Hall, 982. [6] K. Fukunaa, Introducton to Statstcal Pattern Reconton, second edton. Boston: Academc Press, 990. [7] A. K. Jan and B. Chandrasekaran, Dmensonalty and Sample Sze Consderatons n Pattern Reconton Practce, Handbook of Statstcs, 2, pp , 982. [8] C. E. homaz and D. F. Glles, A Maxmum Uncertanty LDA-based approach for Lmted Sample Sze problems - wth applcaton to Face Reconton, n Proceedns of SIBGRAPI 05, IEEE CS Press, pp , 2005.

8 [9] C. E. homaz, D. F. Glles and R. Q. Fetosa, A New Covarance Estmate for Bayesan Classfers n Bometrc Reconton, IEEE ransactons on Crcuts and Systems for Vdeo echnoloy, vol. 4, no. 2, pp , [0] M. urk, A. Pentland, Eenfaces for Reconton, Journal of Contve Neuroscence, MI, vol. 73, pp. 7-86, 99.