Sequential mean field variational analysis of structured deformable shapes

Transcription

1 Computer Vson and Image Understandng 101 (2006) Sequental mean feld varatonal analyss of structured deformable shapes Gang Hua *, ng Wu Department of Electrcal and Computer Engneerng, 2145 Sherdan Road, Northwestern Unversty, Evanston, IL 60208, USA Receved 15 March 2004; accepted 25 July 2005 Avalable onlne 3 October 2005 Abstract A novel approach s proposed to analyzng and trackng the moton of structured deformable shapes, whch consst of multple correlated deformable subparts. Snce ths problem s hgh dmensonal n nature, exstng methods are plagued ether by the nablty to capture the detaled local deformaton or by the enormous complexty nduced by the curse of dmensonalty. Takng advantage of the structured constrants of the dfferent deformable subparts, we propose a new statstcal representaton,.e., the Markov network, to structured deformable shapes. Then, the deformaton of the structured deformable shapes s modelled by a dynamc Markov network whch s proven to be very effcent n overcomng the challenges nduced by the hgh dmensonalty. Probablstc varatonal analyss of ths dynamc Markov model reveals a set of fxed pont equatons,.e., the sequental mean feld equatons, whch manfest the nteractons among the moton posterors of dfferent deformable subparts. Therefore, we acheve an effcent soluton to such a hgh-dmensonal moton analyss problem. Combned wth a Monte Carlo strategy, the new algorthm, namely sequental mean feld Monte Carlo, acheves very effcent Bayesan nference of the structured deformaton wth close-to-lnear complexty. Extensve experments on trackng human lps and human faces demonstrate the effectveness and effcency of the proposed method. Ó 2005 Elsever Inc. All rghts reserved. Keywords: Structured deformable shapes; Dynamc Markov network; Mean feld Monte Carlo 1. Introducton Structured deformable shapes consst of multple correlated deformable subparts. For example, a human face s composed of outer face contour, eyebrows, eyes, nose, and mouth. Analyzng the moton of structured deformable shapes has many real applcatons such as trackng human lps for speech recognton [1], locatng human faces for face recognton [2], and medcal applcatons such as trackng the endocardal wall [3]. The structured deformaton s dfferent from artculated moton. In structured deformaton, each subpart s a deformable shape whle artculaton conssts of a lnked structure of rgd subparts. For structured deformaton analyss, the frst problem s how to represent the deformable shapes. Exstng methods * Correspondng author. Fax: E-mal addresses: ganghua@ece.northwestern.edu (G. Hua), yngwu@ ece.northwestern.edu (. Wu). ether represent the deformable shapes as splne curves [4 7], or as a set of control ponts [2,8]. The second problem s how to recover the structured deformaton from the vdeo sequences. There are manly two approaches: the top-down approach and the bottom-up approach. The top-down approach takes a two-step strategy,.e., the hypothess generaton and mage observaton verfcaton as n the partcle flterng algorthm [5 7]. The bottom-up approach estmates the moton parameters by mnmzng determnstc cost functons. The SNAKES [4] s the representatve usng such an approach. The hgh-dmensonal nature of the structured deformable shapes causes a lot of challenges. Both approaches mentoned above are confronted by the hgh dmensonalty: for the frst approach, e.g., the partcle flterng algorthm [5 7], the number of partcles needed to acheve a good result may ncrease exponentally wth the dmensonalty, so does the computaton cost; for the second approach, e.g., the SNAKES [4], the cost functon /$ - see front matter Ó 2005 Elsever Inc. All rghts reserved. do: /j.cvu

2 88 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) needs to be optmzed n a very hgh-dmensonal space. It s confronted by the enormous local mnma nduced by the hgh dmensonalty. In addton, snce these methods treat the deformable shapes as a whole part, detaled local deformaton of a structured deformable shape s hardly to be analyzed. Our method employs a Markov network to represent the structured deformable shapes. The structural constrants are modelled n the Markov network as potental functons, whch are n fact the measures of the probablty of two constraned subparts beng n certan states of deformaton. The structured deformaton s then modelled by a dynamc Markov network, whch s a temporal extenson of the Markov network that models the structured deformaton at each tme nstant. The probablstc mean feld varatonal analyss of the dynamc Markov network results n a set of mean feld fxed pont equatons. Snce t s dffcult to obtan the closed form soluton to such a set of fxed pont equatons, a sequental mean feld Monte Carlo (SMFMC) [9] algorthm s proposed as a non-parametrc approxmaton. Under ths formulaton, dfferent subparts are tracked by dfferent trackers whle these trackers exchange nformaton wth one another to renforce the structural constrants at the mean tme. By way of ths, we acheve very effcent Bayesan nference of the posteror deformaton wth near-to-lnear complexty and the local deformaton of each subpart can be recovered very well. The remander of ths paper s organzed as follows: n Secton 2, related work n the lterature s dscussed; then n Secton 3, we propose the Markov network representaton of the structured deformable shapes and the MFMC [9] algorthm whch can perform effcent approxmate Bayesan nference n such a Markov netowrk; n Secton 4, by extendng the Markov network and the MFMC algorthm temporally, we propose a dynamc Markov network to model the structured deformaton and a SMFMC algorthm to effcently mplement the Bayesan nference n the dynamc Markov network; expermental results are presented and dscussed n Secton 5; and we present the concluson and the future work n Secton Related work Because of the hgh-dmensonal nature of the moton of the structured deformable shapes, the analyss of them faces many problems. We use a dynamc Markov network representaton to approach to the hgh-dmensonal problem and the trackng algorthm s formulated as the Bayesan nference n ths graphcal model. Therefore, related work n the lterature can be categorzed nto three: the frst category addresses the formulaton of vsual trackng algorthm; the second category refers to the endeavors to conquer the curse-of-dmensonalty; and the thrd category s the Bayesan nference algorthms n complex graphcal models. For vsual trackng algorthms, there are two man approaches just as brefly mentoned n Secton 1. For the top-down approach, there s a sgnfcant lterature n the partcle flterng tracker [5 7] and ts numerous varants [10 14], whch bascally formulate the vsual trackng algorthm as the propagaton of the condtonal probablty of the target state at the current tme nstant gven the mage observatons up to the current tme nstant. For the bottom-up approach, the mean-shft blob tracker [15,16], the SNAKES [4], and the effcent regon tracker wth the parametrc model of geometry and llumnaton [17] are the representatves, to lst a few. To attack the problems caused by hgh dmensonalty, there are manly two methodologes. The frst methodology s to learn the ntrnsc lower-dmensonal descrpton of the manfold spanned by the vald states n the very hgh-dmensonal state space. To menton some, the actve shape model [2] uses PCA to fnd the best possble low-dmensonal lnear approxmaton to the vald shape model state space. The same approach has also been adopted to learn a unon of lnear manfolds spanned by a set of bass confguratons for artculated natural hand moton [18]. Whle the Bayesan PCA [19,20] provdes a more prncpled way to learn both the ntrnsc dmensonalty and the statstcal mxture model for the vald states. More recent work ncludes usng ISOMAP algorthm for dmenson reducton [21]. However, learnng a lower-dmensonal descrpton of the object needs a set of tranng data and sometmes t would be very dffcult to obtan. On the other hand, the second methodology to attack the curse-of-dmensonalty focuses on usng more effcent means of computaton. Parttoned samplng [22] samples dfferent parts n a herarchcal way and thus acheves lnear complexty w.r.t. the number of parts, but the nformaton flow s undrectonal and t volates the symmetrc constrans of the dfferent subparts n a structured deformable shape. The MFMC algorthm actually has been used for trackng artculated human body moton [9], whch keeps the symmetrc constrans between two dfferent artculated lnks. In ths paper, we adopted the same MFMC algorthm to analyze structured deformable shapes, whch can keep the symmetrc constrans between two dfferent deformable subparts as well as acheve very effcent Bayesan nference wth nearto-lnear complexty w.r.t. the number of subparts. For Bayesan nference n graphcal model, belef propagaton can acheve exact results n the case of drected acyclc graphcal model (DAG). When the graphcal model has loops, loopy belef propagaton [23] or generalzed belef propagaton (GBP) [24] can obtan good approxmate results n many applcatons [25,26]. Probablstc varatonal analyss s a prncpled approxmate nference technque. It uses a more tractable approxmate form of the true jont posteror probablty [13,20,27 29] and an approxmate nference s acheved by mnmzng the Kullback Lebler (KL) dvergence between the approxmated posteror and the real posteror dstrbuton. Partcle flterng or sequental Monte Carlo (SMC) [5 7,30] employs a set of weghted

3 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) samples to represent the real dstrbutons and statstcal nference s mplemented based on these samples. More recent work usng partcle flterng to perform the nference of general graphcal model ncludes the non-parametrc belef propagaton (NBP) algorthm [31] and the PAMPAS algorthm [32], where Monte Carlo method s used n the message passng process n belef propagaton. In both algorthms, the messages are modelled as Gaussan mxtures and Markov chan Monte Carlo (MCMC) samplers are desgned to draw the samples of the components of the Gaussan mxtures for the newly updated messages from the old message dstrbutons. Applcatons of these probablstc modellng and nference algorthms for computer vson problems nclude the part-based object recognton by mxture of trees [33], fndng deformable shape by loopy BP [34], trackng 3-D human body moton through a loose-lmbed model [35], and the multple frame artculated moton nference usng BP [36], to lst a few. In the proposed method, each subpart of the structured deformable shape s modelled by a parametrc descrpton of a splne curve n the moton space. The constrant between each two correlated subparts s modelled explctly n the dynamc Markov network representaton. As descrbed n the followng sectons, our method of performng nference n such a dynamc Markov network actually combnes the dea of varatonal nference methods [13,20,27 29] wth the dea of partcle flterng [5 7]. Dfferent from NBP [31] and PAMPAS [32], whch combnes the BP algorthm wth MCMC sampler and all the dstrbutons are modelled as Gaussan mxtures, the proposed SMFMC s a fully nonparametrc nference algorthm wth close to lnear complexty. 3. Markov network representaton of structured deformable shapes and mean feld Monte Carlo Suppose the structured deformable object conssts of K subparts, then we denote the state of each subpart of the object as a random varable x, whch can be any parametrc descrpton of the moton such as the affne moton n our experment. Then, we can construct a sutable potental functon w(x,x j ) between two dfferent subparts, whch n fact models the probablty of two random varables beng subject to a certan constrant. In addton, suppose the mage observaton for each x s z, and the observaton functon s /(z x ), a Markov network can thus be constructed to model the structured deformable object. Fg. 1 shows a Markov network of human face where the subscrpts ÔOf,Õ ÔLeb,Õ ÔReb,Õ ÔLe,Õ ÔRe,Õ ÔN,Õ and ÔMÕ correspond to outer face contour, left eyebrow, rght eyebrow, left eye, rght eye, nose, and mouth, respectvely. Each undrected lnk n the graphcal model represents a potental functon w(x,x j ) and each drected lnk n the graphcal model represents an observaton functon /(z x ). Denote X ¼fx ; ¼ 1...Kg and Z ¼fz ; ¼ 1...Kg, the jont probablty correspondng to the graphcal model n Fg. 1 s X Le Z Le X N Z N pðx; ZÞ ¼ 1 wðx ; x j Þ /ðz jx Þ; ð1þ Z C ð;jþ2e 2V where Z C s a normalzaton constant, E denotes the set of undrected lnk and V denotes the set of drected lnks n the graphcal model. The nference problem of such a loopy graphcal model s to calculate the posteror probablty p(x Z). We employ a varatonal mean feld nference algorthm to obtan an approxmate soluton to t,.e., the jont posteror probablty s approxmated by PðXjZÞ Q ðx Þ; ð2þ where Q (x ) s an ndependent approxmate dstrbuton of P(x Z). Assumng such an fully factorzed mean feld approxmaton whch has ndependent posteror margnal dstrbuton does not mean that the correlaton of the group of random varables n the graphcal model s n anyway lost or gnored, all t means s that any dependences among ths set of random varables X remaned gven the observaton Z cannot be captured n the mean feld approxmaton [20]. Then, we can construct a cost functon! JðQÞ¼logðPðZÞÞ KL Q ðx ÞkPðXjZÞ ð3þ I Q ¼logPðZÞ Q ðx Þlog Q ðx Þ dx ð4þ PðXjZÞ X ¼ X Z H ðq Þþ Q ðx ÞE Q flogpðx;zþjx gdx ; ð5þ x where the KL(Æ) s the KL dvergence and H ðq Þ¼ X x Q ðx Þ log½q ðx ÞŠ ð6þ s the entropy of the dstrbuton Q (x ) and E Q flog PðX; ZÞjx g¼ X Leb Z Leb I X Of Z Of fjgn fx jgnx X M XReb Z Reb Fg. 1. Markov network for human face. Z M X Re Z Re Q j ðx j Þlog ½PðX; ZÞŠdX. ð7þ

4 90 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) ggdx Maxmzng J(Q) wth the constrant that R x Q ðx Þdx ¼ 1 wll lead to Q (x ) f P(x Z). Takng the dervatve of J(Q) w.r.t. Q (x ), and settng t to zero, we obtan that Q ðx Þ¼e E Qflog PðX;ZÞjx g. ð8þ In conjuncton wth the constrant that R x Q ðx Þdx ¼ 1, we can solve the equaton set to obtan a set of fxed pont equatons for Q (x ) [27,9]: Q ðx Þ 1 e E Qflog PðX;ZÞjx g ; Z E ð9þ where Z Z E ¼ expfe Q flog PðX; ZÞjx ð10þ x s a normalzaton constant to constran that Q (x ) s a vald probablstc dstrbuton. Embeddng the factorzed jont dstrbuton of the graphcal model n Eq. (1) nto Eqs. (7) and (9), we can obtan a set of factorzed mean feld fxed pont equatons [9]: Q ðx Þ 1 /ðz jx Þ Z 0 E ( X Z ) exp Q j ðx j Þ log wðx ; x j Þdx j ; ð11þ x j j2n ðþ where ( Z 0 E Zx ¼ X Z ) exp Q j ðx j Þ log wðx ; x j Þ dx x j j2n ðþ ð12þ s agan a normalzaton constant and N ðþ denotes the set of subscrpts of the neghborng nodes of x. Snce ths set of fxed pont equatons nvolves ntegraton of complex probablstc dstrbutons, closed form soluton would be dffcult. We smulate t by Monte Carlo technques where several sets of weghted samples are used to represent the probablstc dstrbutons,.e., Q ðx Þfs ðnþ ; p ðnþ g. The teraton of the fxed pont equaton set s then mplemented based on these samples. Ths leads to the MFMC algorthm, whch s shown n Fg. 2. Compared wth the NBP algorthm [31] and the PAMPAS algorthm [32], the MFMC algorthm employs weghted samples for all the dstrbutons whle both NBP and PAM- PAS model all the dstrbutons as Gaussan mxtures. In ths sense, our algorthm s fully non-parametrc whle thers are sem-parametrc. Also, our algorthm employs mportance samplng thus t avods usng slow MCMC samplers whch have been used n both NBP and PAMPAS. Although how to determne the best mportance functon s stll an art more than scence, the use of mportance samplng has two advantages: frst, t avods samplng drectly from the complex Gbbs dstrbuton of Eq. (9) or Eq. (11); second, we can easly ntegrate both our doman knowledge and heurstcs nto the mportance functon to generate samples from more confdent areas. A natural choce of the mportance functon would be the potental functon w(x,x j ) snce t s lke a condtonal probablty f one of the x and x j s fxed. In our experments n Secton 5, we ndeed adopted the potental functon as the mportance functon, Fg. 2. Mean fled Monte Carlo algorthm.

5 .e., f one subpart s connected wth several other subparts, we mx the samples from each potental functon together. 4. Modellng structured deformaton by dynamc Markov network and sequental MFMC Assumng each subpart of the structured deformable shape has an ndependent dynamc moton model p(x,t x,t 1 ), we can descrbe the moton of a structured deformable object through a dynamc Markov network as n Fg. 3. It s the temporal extenson of the Markov network n Fg. 1. Each horzontal arrow represents the dynamc moton model of the correspondng subpart. Denote X t ¼fx ;t ; ¼ 1...Kg, Z t ¼fz ;t ; ¼ 1...Kg, and Z 1:t ={Z 1,...,Z t }, we can easly fgure out that: PðX t ; Z 1:t Þ/ wðx ;t ; x j;t Þ /ðz ;t jx ;t Þ ð;jþ2e t 2V t Z PðX t jx t 1 ÞPðX t 1 jz 1:t 1 ÞdX t 1. ð13þ X t 1 Here, the nference problem s to recover Pðx ;t jz 1:t Þ; ¼ 1...K at tme nstant t based on the nference result Pðx ;t 1 jz 1:t Þ; ¼ 1...K at tme nstant t 1. Under the context of mean feld varatonal analyss, suppose we already have PðX t 1 jz 1:t 1 Þ Q ;t 1 ðx ;t 1 Þ G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) ð14þ and then we need to fnd the best ÕQ,t (x,t ) to approxmate the real P (X t Z 1:t ). Assumng ndependent dynamc model for each subpart,.e., PðX t ;Z 1:t Þ/ wðx ;t ;x j;t Þ /ðz ;t jx ;t Þ ð;jþ2e t 2V t Z Pðx ;t jx ;t 1 Þ Q ;t 1 ðx ;t 1 ÞdX t 1 ð16þ X t 1 / wðx ;t ;x j;t Þ ð;jþ2e t " Z # /ðz ;t jx ;t Þ Pðx ;t jx ;t 1 ÞQ ;t 1 ðx ;t 1 Þdx ;t 1. 2V t x ;t 1 ð17þ The mean feld approxmaton of P(X t Z 1:t )s PðX t jz 1:t Þ Q ;t ðx ;t Þ. ð18þ Then, followng almost the same steps n Secton 3, wefrst construct the cost functon J t (Q) as n Eq. (21): J t ðqþ ¼logðPðZ 1:t ÞÞ KL! Q ;t ðx ;t ÞkPðX t jz 1:t Þ ð19þ I Q ¼ log PðZ 1:t Þ Q ;t ðx ;t Þlog Q ;tðx ;t Þ dx t ð20þ PðX t jz 1:t Þ X t ¼ X Z H ;t ðq ;t Þþ Q ;t ðx ;t ÞE Q flog PðX t ; Z 1:t Þjx ;t gdx ;t ; x ;t ð21þ where H ;t ðq ;t Þ¼ X Q ;t ðx ;t Þ log Q ;t ðx ;t Þ ð22þ x ;t s the entropy of the dstrbuton Q,t (x ) and I E Q flog PðX t ; Z 1:t Þjx ;t g ¼ Q j;t ðx j;t Þlog PðX t ; Z 1:t ÞdX t. ð23þ fjgn fx j;tgnx ;t PðX t jx t 1 Þ¼ Pðx ;t jx ;t 1 Þ. ð15þ Embeddng Eqs. (14) and (15) nto Eq. (13), we have Takng the dervatve of J t (Q) w.r.t. Q,t (x,t ) and settng t to zero, wth the constrant that R x ;t Q ðx ;t Þdx ;t ¼ 1, we can solve the equaton set to get the followng fxed pont equaton: Fg. 3. DMN for human face deformaton.

6 92 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) Q ;t ðx ;t Þ 1 Z E;t e E Qflog PðX t;z 1:t Þjx ;tg ; ð24þ where Z E;t s a normalzaton constant. By embeddng Eq. (13) nto Eq. (24), we obtan the followng fxed pont equaton: Q ;t ðx ;t Þ 1 /ðz ;t jx ;t Þ Z C Z Pðx ;t jx ;t 1 ÞQ ;t 1 ðx ;t 1 Þdx ;t 1 x ;t 1! X exp Q j;t ðx j;t Þ logðwðx ;t ; x j;t ÞÞ ; ð25þ j2n ðþ where Z C s a normalzaton constant whch can make sure that Q,t (x,t ) s a vald probablstc dstrbuton. Therefore, by usng the mean feld varatonal analyss, we actually approxmate the propagaton of the condtonal posteror probablty of P(X t Z 1:t ) [5 7] to the propagaton of the varatonal mean feld dstrbuton Õ Q,t (x,t ). Ths s the sequental verson of the mean feld varatonal method, whch we call sequental mean feld varatonal analyss. Comparng Eq. (25) wth Eq. (11), we can see that there s one more tem n Eq. (25), whch exactly models the dynamc predcton pror. If we do know the pror probablty of each node n Fg. 1, we can also ncorporate a local pror tem n Eqs. (9) and (11). Ths set of fxed pont equatons s more complex because they nvolve more multple ntegratons of complex probablstc dstrbutons. Agan, we adopt a Monte Carlo strategy to mplement t. We call t the SMFMC algorthm, as shown n Fg. 4. The SMFMC algorthm has two steps, the frst step s the SMC process; then, the result s used as the ntalzaton of the MFMC process. The MFMC process runs untl convergence. One queston here would be the choce of the mportance functon, here we can construct a sutable mportance functon through our doman knowledge and vald heurstcs, e.g., a more accurate dynamc model of the structured deformaton, or a rough detecton result of the dfferent subparts of the structured deformable shape, or a learned condtonal probablty of one subpart gven one of the connected subparts, whch n fact could be from the learned potental functon w(x,x j ), etc. 5. Experments and dscusson 5.1. Trackng human lps Potental functon and observaton lkelhood for lp trackng A human lp can be decomposed nto upper-lp and lowerlp, each of them s represented by an affne deformaton of a splne curve. Thus, a two-node Markov network can be constructed and each node represents a sx-dmensonal random varable of the affne deformaton. The constrant between the upper-lp and lower-lp s that the two pars of the end ponts should be as close as possble, thus a potental functon can be constructed based on ths. Denote x u and x l as the affne moton of the upper-lp and lower-lp, Pt u 1 and Pt u 2 as the two end ponts of the upper-lp curve, and Ptl 1 and Pt l 2 as the two end ponts of the lower-lp curve, we adopt a Gaussan potental functon to model the constrant,.e., wðx u ; x l Þ¼pffffffffff 1 2p r exp! kptu 1 Ptl 1 k2 þkpt u 2 Ptl 2 k2. 2r 2 ð26þ Snce t s a zero mean Gaussan dstrbuton, the only parameter whch needs to be estmated s r 2. It can be easly traned from a set of manually labelled lp mages by the maxmum lkelhood (ML) estmaton. Nevertheless, our experments show that ths parameter s not that senstve, t can be set n a large range around the traned one. Therefore, another choce to save the tedous manually labellng work s to manually set the parameter to see whether t can acheve good emprcal results or not. The observaton functon /(z s x s ), s 2 {u,l} s smlar as that n [14]. We follow the dea of usng a set of n normal measurement lnes of length L to collect mage features [7,5]. For the th measurement lne, =1,...,n, we denote the contour pont poston by x. A 1-D edge detecton s performed along the measurement lne to fnd all the m locatons of the edge ponts fz 1 ;...; zm g. It s easy to observe that only one of the fz k g could be produced by the object contour and all the other edge ponts should be produced by the cluttered background. Therefore, we assume a Gaussan dstrbuton of the locaton of the edge pont produced by the object boundary, whch could only be one of the m (W) edge ponts nsde a wndow W < L around x, and a posson dstrbuton wth spatal arrval rate k on the number of edge ponts produced by the cluttered background. Then, the lkelhood p(z s, x ) s calculated as To demonstrate both the effectveness and the effcency of the SMFMC algorthm, t has been mplemented to track both human lps and human faces. All the experments run on a PC wth 2.4 GHz CPU. The code s programmed wth C++ and no code optmzaton s performed. pðz s; jx Þ¼aN W kl m 2 0; r2 0 e kl þð1 aþ m! P z k 2W N zk jx ; r 2 0 kl m 1 mðw Þ ðm 1Þ! e kl ; ð27þ

7 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) Fg. 4. Sequental mean fled Monte Carlo algorthm. where a s the probablty of mssng detecton of the edge pont produced by the object boundary. Introducng such a probablty may ncrease the robustness of the observaton lkelhood functon [14,7,5]. Then, the full observaton lkelhood s smply the product of the lkelhood of all the measurement lnes,.e., /ðz s jx s Þ¼ n ¼1 pðz s; jx Þ; s 2fu; lg. ð28þ We refer the reader to [14] for a more detaled dscusson of the observaton lkelhood functon used here. The same observaton lkelhood functon s also adopted n the face trackng experment n Secton Convergence of the MFMC teratons To demonstrate the convergence of the proposed MFMC algorthm, we manually ntalze the MFMC algorthm from dfferent ntal ponts on the frst frame of the lp vdeo. Our expermental results show that the MFMC algorthm usually converges nto the correct results after fve teratons, even wth ntalzatons whch are sgnfcantly devated from the real result, as shown n Fg. 5.

8 94 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) Fg. 5. Convergence of the MFMC teratons: A.(a) A.(g), the convergence of the teraton under ntalzaton A; B.(a) B.(g), the convergence of the teraton under ntalzaton B. Fg. 5 presents the convergence of the MFMC teratons under two ntalzatons. The frst two rows and the last two rows show the convergence of the MFMC teratons under ntalzaton A and ntalzaton B, respectvely. As we can observe n the fgure, both ntalzatons are largely devated from the real locatons of the lp but the MFMC teraton does converge to the correct result n fve teratons. We extensvely run the experments many tmes and obtaned smlar results as shown n Fg. 5. However, the convergence of the MFMC algorthm does depend on the ntalzaton,.e., wth a better ntalzaton n the sense that t s nearer to the real result, the convergence of the MFMC teraton s faster, and vce versa. Too wld ntalzaton may result n the falure of the MFMC teraton to converge n a reasonable tmes of teraton, e.g., for the ntalzaton shown n Fg. 6. Snce durng trackng, we usually try to manually ntalze the tracker as accurate as possble, the MFMC teraton usually needs less teratons to converge. We generally set the number of teratons to three durng the trackng process usng the SMFMC algorthm. Fg. 6. The MFMC algorthm faled to converge under ths knd of wld ntalzaton.

9 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) Lp trackng by SMFMC The lp vdeo sequence has more than 150 frames. Sample frames of the results of the SMFMC algorthm are shown n Fg. 7. For comparson, a sngle SMC tracker and a multple ndependent SMC (MSMC) tracker are also mplemented. The results are shown n Fgs. 8 and 9, respectvely. From the results, we observe that the sngle SMC tracker cannot catch the detaled local deformaton well. The reason s that the upper-lp and lower-lp are actually subject to dfferent correlated affne moton nstead of the same moton, but the sngle SMC algorthm treats them as the same. We also observe that the MSMC cannot keep the structure of the upper- and lower-lp, and t easly loses track because of the lack of constrants between the two ndependent trackers. On the contrary, the proposed SMFMC algorthm acheves much better results. It tracks both the upper-lp and lower-lp deformaton very well, and the structure of the combned upper-lp and lower-lp has also been kept because t has been renforced by the dynamc Markov network A unfed vew of SMFMC, MSMC, and sngle SMC In fact, f a very tght potental functon s used n the Markov network,.e., the state of one node can fully decde the state of the other, then the SMFMC tracker degenerates to a sngle SMC tracker. On the contrary, f the potental functon s very loose,.e., each node s almost ndependent wth the others, then the SMFMC tracker degenerates to a MSMC tracker. Thus, we can treat SMC and MSMC as two extreme cases of the SMFMC tracker, as showng n Fg Face algnment and trackng Potental functon and observaton lkelhood for face Just as we have shown n Fg. 1, the face contour are modelled by a seven-node Markov network. It s obvous that two subparts on the face are constraned by the rgdty of the human face, e.g., the relatve locaton of the eye and the nose can only be changed n certan range. Denote the centrod pont of the subpart curve as Pt and the centrod pont of the subpart curve j as Pt j, then the potental Fg. 7. Trackng human lp va SMFMC. Fg. 8. Trackng human lp va a sngle SMC tracker.

10 96 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) Fg. 9. Trackng human lp va a MSMC tracker. loose constrant Sngle SMC functon between these two s defned by a Gaussan dstrbuton as wðx ;x j Þ¼ 1 2pjR j j exp SMFMC tght constrant MSMC Fg. 10. The relatonshp of SMFMC, SMC, and MSMC. 1 2 ðpt Pt j ~l j Þ T R 1 j ðpt Pt j ~l j Þ ; ð29þ where ~l j and R j are the two-dmensonal mean vector and 2 by 2 covarance matrx of the relatve translaton between two sub-parts. We easly obtaned the ML estmaton of the ~l j and R j from a set of manually annotated face mages. The traned potental functons are then adopted to perform the face trackng. Just as we have stated, the observaton lkelhood functon s the same as that n Secton Face trackng by SMFMC The proposed SMFMC algorthm acheves good results for automatc face algnment and face trackng over the vdeo sequence of more than 200 frames. After manual ntalzaton n the frst frame, the algorthm automatcally algns and tracks the face over the sequence. Sample frames of the expermental result are shown n Fg. 11. Compared wth the ASM model [2], our face pror model s actually a Gbbs model whle the ASM uses a Gaussan model. Thus, our dstrbuted model s more general than the ASM model, and more capable of capturng detaled shape deformaton. Just as we have mentoned n Secton 4, the frst step of the SMFMC algorthm s the SMC step, then the result s used as the ntalzaton of the MFMC step. In most cases, the result of the SMC step s unsatsfactory, then the MFMC step wll use the structure nformaton of the face to gude an teratve algnment to a better result Quanttatve analyss By manual labellng the tracked face vdeo sequence, quanttatve evaluaton of the proposed SMFMC algorthm was performed. Snce durng the SMFMC trackng process, the affne parameters are estmated wth respect to the manually ntalzed B-splne curve n the frst frame, we are facng the problem of recoverng the ground-truth affne moton from the manually labelled vdeo sequence. Denote the labelled curve at the frst frame as x 1 and the labelled curve at tme nstant t as x t, ths could be acheved to solve a least square fttng problem,.e., A t ¼ mn ka t ðx 1 Þ x t k 2 ; ð30þ A t where A t denotes the affne moton, A t denotes the estmated ground-truth affne moton, and Æ denotes the Eucldean dstance. However, solvng the above least square fttng problem nvolves the evaluaton of the squared resdue error on certan selected ponts on the two curves. And usng the ftted affne moton as the ground-truth may ntroduce more numercal errors. Therefore, nstead we take a drect evaluaton approach by usng the recovered affne moton from the moton trackng algorthm to transform the reference curve n the frst frame to the current tme nstant. We can then evaluate the root mean square error (RMSE) between the transformed curve and ground-truth labelled curve on certan selected ponts. The selected ponts are sampled evenly from the splne curve. For example, denote the evenly sampled n ponts on the reference curve as {s 1,...,s n }, the n ponts on the ground-truth curve at the current tme nstant as fs t 1 ;...; st n g, and the recovered affne moton as A t, then the RMSE s evaluated as sffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 1 X n RMSE ¼ ka t ðs Þ s t n k 2 ; ð31þ where Æ s agan the Eucldean dstance. Our face model contans sever subparts. Each of them has ts own curve

11 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) Fg. 11. Trackng human face va sequental MFMC. representaton and ndvdual but correlated affne moton. We evenly sampled n ponts from the curve of dfferent subparts. Then, the overall RMSE s calculated as vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff u1 X 7 1 X n RMSE ¼ t ka j t 7 n ðsj Þ s j;t k 2. ð32þ j¼1 The number of ponts evenly sampled from each subpart curve s proportonal to the length of the curve, e.g., 30 ponts are sampled for face outer contour, 20 ponts are sampled for mouth contour, and 12 ponts are sampled for the other subpart. We show the overall RMSE of the SMFMC tracker, the MSMC tracker, and the sngle SMC tracker on the tested face vdeo sequence on Fg. 12. We can observe that the proposed SMFMC trackng algorthm generally acheved better accuracy than the MSMC trackng algorthm and the sngle SMC trackng algorthm on the tested vdeo sequence Computaton effcency In the SMC algorthm, the major computaton comes from the evaluaton of the observaton lkelhood. Snce n each tme step, the SMC algorthm needs to evaluate the observaton lkelhood once for each sample, the computaton complexty s almost proportonal to the number of samples. For conventonal partcle flterng methods, the needed number of samples s exponental ncreasng w.r.t. the dmensonalty. Suppose for a sngle object, the number of samples needed to make the tracker work s N. Then, for a structured deformable shape wth K subparts, f mean feld teraton converges n M steps, the computaton complexty s OðMKN Þ [9], whch s clearly lnear w.r.t. the number of subparts, thus to the dmensonalty. Moreover, our algorthm can let us allocate the computaton resources to dfferent subparts accordng to ther needs,.e., some of the more complex subparts may need more samples whle some of the others may need less samples. In ths way, the computaton resources can be used more effcently as we have done n the experments. Table 1 shows the rough processng frame rates of dfferent methods n our experments. Notce that the lp has two subparts, the face has seven subparts and the number of mean feld teraton s three, t verfes that the SMFMC tracker does have lnear complexty w.r.t. the dmensonalty. 6. Concluson and dscusson In ths paper, a novel SMFMC algorthm s proposed to analyze and track structured deformable shapes based on a Fg. 12. The overall RMSE of the SMFMC, the SMC, and the MSMC on the test face vdeo sequence. The green lne, the red lne, and the blue lne represent the overall RMSEs of the SMFMC algorthm, the MTSMC algorthm, and the sngle SMC algorthm, respectvely. (For nterpretaton of the references to color n ths fgure legend, the reader s referred to the web verson of ths paper.) Table 1 Processng frame rate of dfferent trackers Algorthms: SMCLp MSMCLp SMFMCLp SMFMCFace Frame rate (F/s):

12 98 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) dynamc Markov network representaton. It approxmates the propagaton of the posteror moton of the structured deformable shapes to the propagaton of the varatonal mean feld dstrbutons. The SMFMC algorthm combnes both analytcal and non-parametrc nference methods,.e., the probablstc mean feld varatonal analyss and Monte Carlo methods. More nterestngly, t has lnear complexty w.r.t. the dmensonalty. In addton, t shows that the sngle SMC tracker and the MSMC tracker can be regarded as two extreme cases of our algorthm. Experments have demonstrated the effectveness and effcency of the proposed SMFMC algorthm. Although the SMFMC algorthm s employed n ths paper for analyzng structured deformable shapes, t s actually generalzable for many other vson problems such as artculated moton analyss [9]. The dea of dealng wth the local subparts whle renforcng the global constrant proved to be an effectve way to overcome the curse-of-dmensonalty. Our future work ncludes applyng SMFMC to other vson problems and more thorough theoretcal study of the SMFMC such as the theoretc analyss of the convergence rate. Acknowledgments Ths work was supported n part by Natonal Scence Foundaton Grant IIS , IIS , Northwestern faculty startup funds for ng Wu and Walter P. Murphy Fellowshp for Gang Hua. References [1] R. Kaucc, B. Dalton, A. Blake, Real-tme lp trackng for audovsual speech recognton applcatons, n: Proc. Eur. Conf. on Computer Vson, vol. 2, 1996, pp [2] T. Cootes, D. Cooper, C. Taylor, J. Graham, Actve shape models ther tranng and applcaton, Comput. Vs. Image Understand. 61 (1) (1995) [3] D. Comancu, Nonparametrc nformaton fuson for moton estmaton, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, [4] M. Kass, A. Wtkn, D. Terzopoulos, SNAKES: actve contour models, Int. J. Comput. Vs. (1988) [5] M. Isard, A. Blake, Contour trackng by stochastc propagaton of condtonal densty, n: Proc. Eur. Conf. on Computer Vson, vol. 1, 1996, pp [6] M. Isard, A. Blake, CONDENSATION condtonal densty propagaton for vsual trackng, Int. J. Comput. Vs. 29 (1) (1998) [7] A. Blake, M. Isard, Actve Contours, Sprnger-Verlag, Berln, [8] C. Lu, H.-. Shum, C. Zhang, Herarchcal shape modelng for automatc face localzaton, n: Proc. Eur. Conf. on Computer Vson, 2002, pp [9]. Wu, G. Hua, T. u, Trackng artculated body by dynamc Markov network, n: Proc. IEEE Int. Conf. on Computer Vson, 2003, pp [10] M. Isard, A. Blake, Icondensaton: unfyng low-level and hgh-level trackng n a stochastc framework, n: Proc. Eur. Conf. on Computer Vson, [11] M. Isard, A. Blake, A smoothng flter for condensaton, n: Proc. Eur. Conf. on Computer Vson, [12] M. Isard, A. Blake, A mxed-state condensaton tracker wth automatc model-swtchng, n: Proc. 6th Internat. Conf. on Computer Vson, 1998, pp [13]. Wu, T.S. Huang, Robust vsual trackng by co-nference learnng, n: Proc. IEEEInternat. Conf. oncomputervson, vol. II, 2001, pp [14]. Wu, G. Hua, T. u, Swtchng observaton models for contour trackng n clutter, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, vol. 1, 2003, pp [15] D. Comancu, V. Ramesh, P. Meer, Real-tme trackng of non-rgd objects usng mean shft, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, vol. 2, Hlton Head Island, South Carolna, 2000, pp [16] R.T. Collns, Mean shft blob trackng through scale space, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, vol. 2, Madson, WI, 2003, pp [17] G. Hager, P. Belhumeur, Effcent regon trackng wth parametrc models of geometry and llumnaton, IEEE Trans. Pattern Anal. Mach. Intell. 20 (10) (1998) [18]. Wu, J. Ln, T.S. Huang, Capturng natural hand artculaton, n: Proc. IEEE Internat. Conf. on Computer Vson, vol. II, 2001, pp [19] C.M. Bshop, J.M. Wnn, Non-lnear bayesan mage modelngproc. Sxth Eur. Conf. on Computer Vson, vol. 1, Sprnger, Berln, 2000, pp [20] J.M. Wnn, Varatonal message passng and ts applcaton, Ph.d. thess, Department of Physcs, Unversty of Cambrdge, [21] Q. Wang, G. Xu, H. A, Learnng object ntrnsc structure for robust vsual trackng, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, vol. 2, 2003, pp [22] J. MacCormk, M. Isard, Parttoned samplng, artculated objects, and nterface-qualty hand trackng, n: Proc. Eur. Conf. on Computer Vson, [23] K. Murphy,. Wess, M. Jordan, Loopy-belef propagaton for approxmate nference: an emprcal study, n: Proc. Ffteenth Conf. on Uncertanty n Artfcal Intellgence, [24] J. edda, W. Freeman,. Wess, Understandng belef propagaton and ts generalzaton, n: Explorng Artfcal Intellgence n the New Mllenum, Elsever Scence and Technology Books, Amsterdam, 2003, pp (Chapter 8). [25] W.T. Freeman, E.C. Pasztor, Learnng low-level vson, n: Proc. IEEE Internat. Conf. on Computer Vson, 1999, pp [26] W.T. Freeman, H. Zhang, Shape-tme photography, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, [27] T.S. Jaakkola, Tutoral on varatonal approxmaton method, Advanced Mean Feld Methods: Theory and Practce, MIT Press, Cambrdge, MA, [28] J. Vermaak, N.D. Lawrence, P. Pérez, Varatonal nference for vsual trackng, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, vol. 1, 2003, pp [29] M.J. Beal, Varatonal algorthms for approxmate bayesan nference, Ph.d. thess, Gatsby Computatonal Neuroscence Unt, Unversty College London, [30] J.S. Lu, R. Chen, Sequental Monte Carlo methods for dynamc systems, J. Am. Stat. Assoc. 93 (443) (1998) [31] E.B. Sudderth, A. Ihler, W. Freeman, A. Wllsky, Nonparametrc belef propagaton, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 2003, pp [32] M. Isard, PAMPAS: real-valued graphcal models for computer vson, n: Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 2003, pp [33] S. Ioffe, D.A. Forsyth, Mxtures of trees for object recognton, n: Proc. of IEEE Conf. on Computer Vson and Pattern Recognton, vol. 2, 2001, pp [34] J.M. Coughlan, S.J. Ferrera, Fndng deformable shapes usng loopy belef propagaton, n: Proc. Eur. Conf. on Computer Vson, Lecture Notes n Computer Scence, vol. 2352, 2002, pp

13 G. Hua,. Wu / Computer Vson and Image Understandng 101 (2006) [35] L. Sgal, M. Isard, B. Sgelman, M. Black, Attractve people: assemblng loose-lmbed models usng non-parametrc belef propagatonadvances n Neural Informaton Processng System, vol. 16, MIT Press, Cambrdge, MA, [36] J. Gao, J. Sh, Multple frame moton nference usng belef propagaton, n: Proc. IEEE Inter. Conf. on Automatc Face and Gesture Recognton, 2004.