FACIAL ACTION TRACKING USING PARTICLE FILTERS AND ACTIVE APPEARANCE MODELS. Soumya Hamlaoui & Franck Davoine

Transcription

1 FACIAL ACTION TRACKING USING PARTICLE FILTERS AND ACTIVE APPEARANCE MODELS Soumya Hamlaoui & Franck Davoine HEUDIASYC Mixed Research Uni, CNRS / Compiègne Universiy of Technology BP 20529, Compiègne Cedex, France Soumya.Hamlaoui@hds.uc.fr, Franck.Davoine@hds.uc.fr ABSTRACT Tracking a face and is facial feaures in a video sequence is a challenging problem in compuer vision. In his view, we propose a sochasic racking sysem based on a paricle-filering scheme. In his paradigm, he unobserved sae includes global face pose and appearance parameers coding boh shape and exure informaion of he face. The adoped observaions disribuion is derived from an Acive Appearance Model (AAM). The ransiion disribuion and he paricles number are adapive in he sense ha hey are guided by an AAM deerminisic search. This opimizaion sage adjuss he explored area of he sae space o he qualiy of he predicion and enables a subsanial gain in compuing ime. The observaion model uses a robus disance measure in order o accoun for occlusions. Experimens on real video show encouraging resuls. 1. INTRODUCTION This work addresses he problem of racking in a single video he global moion of a face as well as he local moion of is inner feaures, due for insance o expressions or eye blinking. This ask is required in many emerging applicaions, like surveillance, eleconferencing, emoional compuer inerfaces, human-compuer communicaion, moion capure for video synhesis, auomaed lipreading, driver drowsiness monioring, ec. Face racking poses challenging problems because of he variabiliy of facial appearance wihin a video sequence, mos noably due o changes in head pose, expressions, lighing or occlusions. Much research has hus been devoed o he problem of face racking, as a specially difficul case of non-rigid objec racking. Noe ha in he applicaions argeed by his work, he person looks approximaely in he direcion of he camera. The face remains hus in a near fronal orienaion, so ha a 2D model of he face is assumed o be able o capure he expeced variaions. In he objec racking lieraure, he following formulaion of he racking problem is convenienly used: a each ime sep, he goal is o infer he unobserved sae of he objec, denoed x X, given all he observed daa unil ime, denoed z 1: (z 1,...,z ). When racking a face in 2D, he unobserved sae x includes moion or pose parameers like he posiion, scale and orienaion of he face; when facial feaures are also racked, he unobserved sae should conain parameers describing he face inner moion. The observed daa z consiss of measuremens derived from he curren video frame, such as grey level paches, edges, or color hisograms. In order o evaluae a hypohesized sae, he measuremens are acually only considered in he image area corresponding o he hypohesized locaion. For insance, he mos naural measuremen consiss of he pixel grey level values hemselves. Basically, a given sae x (moion parameers) is hen evaluaed by comparing he moion-compensaed grey level image pach g image (z,x ) wih a grey level emplae face pach g model. The racking ask hen essenially consiss in searching he curren sae ˆx X ha maches a bes he measuremens z in he curren image. The racking hisory ˆx 1:( 1) is mainly used as a prior knowledge in order o search only a small subse of he sae space X. In a non-probabilisic formulaion of he racking problem, he sae ˆx is usually seeked so as o minimize an error funcional d [g image (z,x );g model ], e.g. an euclidean or robus disance. Acually, in a racking seing, he sae is supposed o evolve lile beween consecuive ime seps. The soluion is hus searched as a small displacemen from he previous frame esimaion: ˆx = ˆx 1 + x. The opimal displacemen is hen ypicallly obained by a gradienlike descen mehod. The well-known Lucas-Kanade algorihm is a paricular case of such a formulaion, and has been recenly generalized in [1]. Insead of being specified by a single face greylevel emplae g model, he face model can span a subspace of greylevel paches, learn by principal componen analysis (PCA) from a face raining se. The error funcional is hen a disance from he image pach g image (z,x ) o he face subspace, usually aken o be he disance o he projecion in face subspace. The subspace modeling allows o accoun for some variabiliy of he global face appearance. The eigen-racking mehod is based on such a principle [3]. Using also principal componen analysis, he Acive Appearance Models (AAMs) encode he variaions of face appearance by learning he shape and exure variaions [4].

2 They enable hus he racking of boh global moion and inner feaures. In he case of AAMs, he gradien jacobian marix is pre-compued in order o save processing ime. In pracice, racking using he deerminisic AAM search appears o work well while he lighing condiions remain sable and only small occlusions are presen. However, large occlusions ofen make he AAM search converge o incorrec posiions and loose rack of he face. In probabilisic formulaions, he hidden sae and he observaions are linked by a join disribuion; his saisical framework offers rich modeling possibiliies. A Markovian dynamic model describes how he sae evolves hrough ime. An observaion model specifies he likelihood of each hypohesised sae, i.e. he probabiliy ha he considered sae may generae he observed daa. Such generaive models represen he variabiliy in he moion and appearance of he objec o rack. Noe ha even he non-probabilisic minimizaion of an error funcional can be recas as he maximizaion of a likelihood: p(z x ) exp d [g image (z,x );g model ] Based on such a generaive model, Bayesian filering mehods recursively evaluae he poserior densiy of he arge sae a each ime sep condiionally o he hisory of observaions unil he curren ime. Sochasic implemenaions of Bayesian filering are generally based on sequenial Mone Carlo esimaion, also known as paricle filering [5]. Paricle filering approximaes he poserior sae densiy by a se of random weighed samples (paricles) a each ime sep. The CONDENSATION algorihm consiss in propagaing his sample se hrough ime using a dynamic model and in weighing each sample proporionally o is likelihood funcion value [7]. When compared wih he analyical soluion provided by he wellknown Kalman filer, paricle filering has wo advanages: i is no resriced o he case of linear and Gaussian models, and i can mainain muliple hypoheses of he curren sae, a desirable propery when he background is complex and conains disracing cluer. For video racking, he CONDENSATION algorihm was firs proposed in conjuncion wih edge measuremens produced by an edge deecor [7]. Since hen, his algorihm has araced much ineres, and oher kinds of measuremens have given valuable varians. For insance, he color hisogram yields fas, deformaion- and orienaion-robus racking [8]. However, since color hisograms are global, hey do no allow o rack he moion of inernal facial feaures as is he goal here. A greylevel pach is used as measuremen vecor by Zhou e al. [10]. In order o cope wih he changing appearance of he face, he likelihood is aken o be a mixure of hree mean appearance emplaes, and he parameers of he mixure are re-esimaed during he racking. Their model considers however only global face appearance emplaes in he likelihood, and global moion parameers in he sae vecor. Ross e al. [9] propose o consanly updae he appearance model of he racked objec o accoun for he lighing, expression and pose variaions bu heir racker doesn handle occlusion well. The AAM paradigm provides an elegan and simple way o rack boh he global pose and he inernal facial feaures. The idea proposed in his paper consiss in combining he AAM wih he CONDENSATION sochasic search in order o augmen is robusness o occlusions. Regarding exising works we are aware of combining AAM wih emporal dynamics, hey model facial behaviours in order o generae video-realisic animaed faces (see e.g. [2]). In hose papers, he racking iself uses he AAM frame-by-frame search wih no emporal dynamics. In secion 2, we recall he main principles of AAMs and paricle filering, and inroduce a few relaed noaions. In secion 3, we presen he proposed racking algorihm. In secion 4, experimenal resuls are shown on real video. In secion 5, we draw concluding remarks and discuss he perspecives opened by his work. 2. BACKGROUND 2.1. Face Acive Appearance Models A face AAM is a saisical model which describes shape and exure variaions of he human face class [4]. The appearance variabiliy is linearly modelled by a Principal Componen Analysis (PCA) of shape and exure. The se of model parameers which conrol he differen modes of shape and exure variaion are learned from a raining se of annoaed images. Each raining image is manually annoaed using a se of landmark poins o ouline he facial srucures. The face shape consiss hen of he spaial coordinaes of he landmark poins. The raining se shapes are aligned o he Procruses mean shape of he raining se. The corresponding exures are described by he inensiy of he pixels inside he area delimied by he shapes. These exures are all warped o he mean shape. A PCA is hen applied o shape and exure daa, denoed respecively s and g. Each raining face is hen represened by shape and exure model parameers b s, b g : s = s m + φ s b s g = g m + φ g b g where s m, g m are respecively he mean shape and exure, φ s, φ g are he eigenvecors of shape and exure covariance marices. A hird PCA is hen performed on a concaenaed shape and exure parameers b, o obain a combined model vecor c: b = φ c c where: ( ) Ws b b = s W s is an esimaed weighing marix beween shape and exure and φ c is a se of eigenvecors. b g

3 From he combined appearance model vecor c, a new insance of shape and exure can be generaed: s model (c) = s m + Q s c g model (c) = g m + Q g c In his paper, our approach is esed using person-specific appearance models. The model is rained using 20 annoaed sill images of he person o be racked. Figure 1 shows he second mode variaion of he combined model vecor c hrough ± 1 sandard deviaion from he mean for one of he raining persons. Fig. 1. Second combined mode of appearance variaion for one raining person, i.e. exure insance g model (c) warped back ino shape insance s model (c), wih c 2 = mean - sd, mean, mean + sd and c = (0,c 2,0,...,0) T. In order o mach a arge face in a given image, he shape and exure insances have o be ranslaed, scaled and roaed. This affine ransformaion can be represened by a vecor of four 2D pose parameers p = ( x, y,α,θ). Those parameers denoe respecively he x and y ceners of graviy of he shape, and he scaling facor and inplane roaion relaively o he learn mean shape. The AAM paradigm provides an ieraive gradien-like search echnique in order o compue auomaically he pose and appearance parameers (ˆp,ĉ) ha bes approximae he arge face in he image [4]. The minimized crierion is he norm of he error vecor r(p,c) = g model (c) g im (p,c) (1) whereg model (c) denoes he model face exure andg im (p,c) he image exure sampled a he hypohesized pose p and shape s model (c). Saring from an iniial guess (ˇp,č), he opimal correcions o apply, ( p, c), are linear funcions of he error vecor: p = R p.r(ˇp,č) c = R c.r(ˇp,č) (2) The marices R p and R c can be precompued from raining daa, as in [4] CONDENSATION The CONDENSATION algorihm employs he Mone Carlo echnique of facored sampling in order o recursively approximae he poserior sae densiy. Approximaion is done by means of he empirical disribuion of a sysem of paricles. The paricles explore he sae space following independen realizaions from a sae evoluion model, and are redisribued according o heir consisency wih he observaions, he consisency being measured by a likelihood funcion. The skech of he CONDENSATION algorihm is recalled in Figure 2. For a good inroducion, he reader is refered o he seminal paper of Isard and Blake [7]. A Sep 2d, he sae ˆx could also be esimaed using he mean ˆx = E[x z 1: ] N n=1 π(n) a (n) ; since boh esimaes appeared very similar in he experimens, he MAP will be used in he following. 1. Iniializaion = 0: Generae N sae samples a (1) 0,...,a(N) 0 according o he prior densiy p(x 0 ) and assign hem idenical weighs, π (1) 0 =... = π (N) 0 = 1/N. 2. A ime sep, we have N weighed paricles (a (n) 1,π(n) 1 ) ha approximae he poserior disribuion of he sae p(x 1 z 1:( 1) ) a previous ime sep. (a) Resample he paricles proporionally o heir weighs, i.e. keep only paricles wih high weighs and remove paricles wih small ones. Resampled paricles have he same weighs. (b) Draw N paricles according o he dynamic model p(x x 1 = a (n) 1 ). These paricles approximae he prediced disribuion p(x z 1:( 1) ). (c) Weigh each new paricle proporionally o is likelihood: π (n) p(z x = a (n) 1 = ) N m=1 p(z x = a (m) 1 ) (3) The se of weighed paricles approximaes he poserior p(x z 1: ). (d) Give an esimae of he sae ˆx as he MAP: ˆx = arg max x p(x z 1: ) arg max π (n) a (n) Fig. 2. CONDENSATION algorihm. 3. PROPOSED SCHEME: AAM-BASED CONDENSATION We propose o adap he CONDENSATION algorihm o our racking ask in hree aspecs, each being deailed below: he sae space spans he global and inner moion of he face; he observaion model is based on sampled pixel grey level paches and a previously rained AAM subspace; he dynamics are adapive as in [10].

4 3.1. Sae space spans global and local moion The sae vecor x conains he parameers o infer abou he objec: he face global 2D pose p = ( x, y,α,θ) T he facial acions, conained in he AAM shape and exure, which are hemselves capured in a compac way by he combined appearance parameer vecor c. Our experimens sugges ha for a person-specific AAM, reaining only he firs K = 4 modes of he appearance parameer c allows o span he facial changes of ineres and provides saisfying racking resuls. The sae vecor is hus of dimension 4 + K = 8: x = (p,c ) T 3.2. AAM-based observaion model The observaion model consiss of he likelihood p(z x ), according o which he paricles are weighed in he formula (3) of he CONDENSATION algorihm. This likelihood indicaes he probabiliy ha a hypohesized saex = (p,c ) T gives rise o he observed daa. This probabiliy should be high whenever here is a good mach beween: he image pach sampled a he hypohesized pose and shape, g im (p,c ) he hypohesized appearance of he face, given by he model exure g model (c ). The adoped likelihood funcion has hus he following form: p(z x ) = C exp d [g model (c );g im (p,c )] (4) where C is he normalizing consan of his disribuion, and he exure disance d [;] is an error measure, summed over all L pixels of boh exures: d [g;g ] = L l=1 gl g l ρ( ) σ l This error is weighed by he sandard deviaion σ l of each pixel, in order o accoun for face pars wih higher variabiliy. The error funcion ρ() can be chosen in differen ways: (5) a simple square error funcion ρ(x) = 1 2 x2 yields a weighed euclidean disance d [;] and a gaussian densiy p(z x ); a robus error funcion can be used insead (see e.g. [6]); in our experimens, we esed he following funcion: where h is a fixed hreshold above which he difference x is considered o be an oulier. Such a robus measure reduces he influence of occluded pixels, which would oherwise dominae he oal error measure (5) and rule ou a poenially good sae candidae Adapive dynamics The sae ransiion model p(x x 1 ) is used in he CON- DENSATION algorihm in order o draw he paricles approximaing he prediced disribuion (Sep 2b of Figure 2). Following he ideas developed in Zhou e al. [10], he dynamics used here are adapive by having he following model for sae evoluion: where x = ˆx 1 + v + S u (7) ˆx 1 is he esimae of he sae vecor a he previous ime sep, he velociy v indicaes he prediced shif in pose / appearance he random componen u is a vecor of 4+K = 8 independen normal random variaes having zero mean and uni variance he diagonal marixs = diag(σ (x),...,σ (c4) ) specifies he sandard deviaion of he random draw for each pose/appearance parameer. The prediced shif v = ( p, c) T is obained by an AAM search in he curren frame 1, using he updae equaions (2). The search is iniialized wih he previous sae esimae (ˇp,č) = (ˆp 1,ĉ 1 ) = ˆx 1. The prediced sae yielded by his search will be denoed by x = ˆx 1 + v. This deerminisic search aims o focus he paricle drawing in a region ha is mos likely o conain good candidaes, and hus reduce he volume of he sae space o explore. According o he sae ransiion model (7), pose / appearance parameers are drawn around he prediced sae x wih dispersions (sandard deviaions) given by S. Those dispersions should be adapive: he generaed paricles need o explore a wide area around x only when he prediced sae x gives a poor soluion. To measure he qualiy of he prediced sae x = ( p, c ), we average he exure error over he L pixels of he exures: ǫ = 2 L ( g model l ( c ) g im ) l ( p, c ) ρ L l=1 σ l (8) { 1 2 ρ(x) = x2 if x h h x 1 2 h2 if x > h (6) 1 In [10], he shif in global posev = p is also compued by using he curren frame, a principle of consan brighness consrain being applied o calculae he prediced moion.

5 Since his error is a measure of variance, is square roo ǫ is used o scale he sandard deviaions of he pose/appearance drawing: (σ (x),...,σ (c4) ) T = R (σ (x) 0,...,σ (c4) 0 ) T where (σ (x) 0,...,σ (c4) 0 ) are fixed reference sandard deviaions, and R is a diagonal marix: R = diag(r (x),...,r (c4) ) where R (i) are scaling facors associaed o he 8 componens of he sae vecor (subscriped by he index i) and proporional o ǫ, wih according bounding values [R (i) min ;R(i) max]: R (i) = max(min( ǫ,r (i) max),r (i) min ) When R (i) are large, he prediced disribuion has a high variance and requires herefore a large number of paricles o approximae i. In oher words, he larger is he area of he sae space subregion covered by he prediced disribuion, he more paricles are needed o explore i. This suggess having an adapive number N of paricles, using he formula: N = N i=1 R (i) 4. EXPERIMENTAL RESULTS The proposed mehod was implemened in non-opimized C++ and esed on a PC running WinXP a 2.4 GHz wih 512 Mb of RAM. Fig. 4. Tracking on a video sequence wih occlusions, frames 921, 928 and 940. Top row: deerminisic AAM racking. Boom row: CONDENSATION based racking. The performance of our approach was also esed in presence of occlusions. We compared i wih a purely deerminisic AAM racking, where he opimal sae configuraion is obained by ˆx = x = ˆx 1 + v (using he noaions defined in paragraph 3.3). As is highlighed in he op row of Figure 4, when he occlusion occurs, he deerminisic search appears o be rapped in an incorrec local opimum, and he racking diverges hus from ha momen. This problem is overcome by he sochasic racking: he occlusion induces a high exure error ǫ for he prediced sae x, and consequenly he variance of drawn paricles and heir number N are increased (see he peaks in Figure 5). Fig. 3. AAM-based adapive CONDENSATION racking, for frames 69, 109 and 188. On each image, he drawn shape shows he esimaed sae ˆx ; he model and image exure g model (c ) and g im (p,c ) are displayed in he lower righ corner Resuls are firs shown for a video sequence where a face in near-fronal view undergoes large variaions in pose, expressions and lighing (see Figure 3). The racking of boh global pose and facial feaures appears saisfying. Seing N 0 = 500, he number of paricles N evolves beween abou 20 and 80, and increases each ime he change in pose and/or appearance is rapid; using such adapive dynamics allows o process on average 2 frames per second. This represens a drasic improvemen over a mehod using a zero-velociy sae evoluion model, which required 1000 paricles o successfully rack his sequence (according o experimens no shown here). Fig. 5. Evolving number of paricles N on he video sequence wih occlusions. The nearly full occlusion of frame 921 induces a high peak, while a parial occlusion occurring around frame 1400 induces a lower peak. The paricles cover hus a greaer area of he sae space, hopefully including he correc soluion x ; as hose paricles are evaluaed by he robus disance likelihood (4) and (5), he reained soluion ˆx sands beer chances o be a good candidae, which allows o correc he deerminisic search (boom row of Figure 4). The effeciveness of he appearance model, largely proved in lieraure, remains condiioned by he fac ha he racked appearance mus be beforehand learned and modelled. This modelling is sensiive o he recording condiions of he raining images. In order o cure his problem, our curren works consis o replace he AAM by an adapive appearance esimaed on he fly (on-line).

6 In his case, he general racking algorihm principle remains he same in he sense ha we use he same adapive dynamics described on paragraph (3.3). However, he likelihood funcion, given by he equaion (4), is now based on he disance beween he image exure sampled a he hypohesized sae and he model exure esimaed and updaed on he fly. The new exure model g fly () is iniialized manually using he face exure in he firs image of he video sequence and updaed a each ime sep using he following equaion: g fly () = (1 α)g fly ( 1) + αg im (,ˆx 1 ) where α is a forgeing facor which deermines he imporance of he exure model updae. g im (,ˆx 1 ) is he curren image exure esimaed according o he sae hypohesis seleced a 1, ˆx 1. In his case, he hidden sae space encodes he pose p and he firs four modes of he shape parameers b s obained from he face model: x = (p,b s ) T This new model is robus vis-a-vis he lighing variaions and occlusion schemes. The racking problem is adaped o each arge face wihou being condiioned by a preliminary raining of is appearance. Some resuls are presened in (Figure 6). We also proposed o replace he exure model given by he AAM by an adapive exure esimaed on he fly o accoun for a necessary beforehand learning of he racked appearance. Now ha a robus racking sysem is available, we can sudy he recogniion of facial acions: he inpu being given by he combined appearance parameers a each ime sep, differen recogniion approaches can be esed, from a simple linear discriminan analysis on sill frames, o dynamic graphical models. In his regard, he paricle filer paradigm provides a naural inference framework for richer models for insance, he facial acion o be recognized could be included as a discree componen of he sae vecor. 6. REFERENCES [1] S. Baker and I. Mahews. Lucas-kanade 20 years on: A unifying framework. In. Journal of Compuer Vision, 56(3): , February [2] F. Beinger, T.F. Cooes, and C.J. Taylor. Modelling facial behaviours. In Proc. BMVC 2002, volume 2, pages , [3] M. Black and A. Jepson. Eigen-racking: Robus maching and racking of ariculaed objecs using a view-based represenaion. In. Journal of Compuer Vision, 36(2): , [4] T. F. Cooes, G. J. Edwards, and C. J. Taylor. Acive appearance models. IEEE Trans. on Paern Analysis and Machine Inelligence, 23(6): , June Fig. 6. Tracking pose and facial acions when replacing he exure model previously obained from he AAM by an adapive exure updaed on he fly. Frames 75, 470 and α = CONCLUSION For he purpose of racking he 2D global pose of a face and is inner facial acions, his paper proposes o combine an adapive paricle filering scheme wih an acive appearance model. The sae vecor is composed of four pose parameers and four combined appearance parameers. The likelihood measures he fi beween he hypohesized model exure and he image exure sampled a he hypohesized locaion and shape; a robus disance accouns for occluded pixels. Following he ideas of [10], he dynamics in sae space are guided by a deerminisic AAM search; his allows o reduce significanly he number of paricles, which is only increased when he AAM search fails o converge o a saisfying soluion. The experimens show ha he proposed algorihm can successfully rack a face and is facial acions undergoing quick moion and nearly full occlusions. [5] A. Douce, J. F. G. De Freias, and N. Gordon. Sequenial Mone Carlo Mehods in Pracice. Springer- Verlag, [6] P. J. Huber. Robus saisics. Wiley, [7] M. Isard and A. Blake. Condensaion - condiional densiy propagaion for visual racking. In. Journal of Compuer Vision, 29(1):5 28, [8] P. Pérez, C. Hue, J. Vermaak, and M. Gangne. Colorbased probabilisic racking. In Proc. Europ. Conf. Compuer Vision, pages , [9] D. Ross, J. Lim, and M.-H. Yang. Adapive probabilisic visual racking wih incremenal subspace updae. In Proceedings of he European Conference on Compuer Vision, [10] S. Zhou, R. Chellappa, and B. Moghaddam. Visual racking and recogniion using appearance-adapive models in paricle filers. IEEE Trans. on Image Processing, To appear, 2004.