A Particle Filter without Dynamics for Robust 3D Face Tracking

Transcription

1 A Partcle Flter wthout Dynamcs for Robust 3D Face Trackng Le Lu ang-tan Da Gregory Hager Computatonal Interacton and Robotcs Lab, Computer Scence Department the Johns Hopkns Unversty, Baltmore, MD 21218, USA Abstract Partcle flterng s a very popular technque for sequental state estmaton problem. However ts convergence greatly depends on the balance between the number of partcles/hypotheses and the ftness of the dynamc model. In partcular, n cases where the dynamcs are complex or poorly modeled, thousands of partcles are usually requred for real applcatons. Ths paper presents a hybrd samplng soluton that combnes the samplng n the mage feature space and n the state space va RANSAC and partcle flterng, respectvely. We show that the number of partcles can be reduced to dozens for a full 3D trackng problem whch contans consderable nose of dfferent types. For unexpected motons, a specfc set of dynamcs may not exst, but t s avoded n our algorthm. The theoretcal convergence proof [1, 3] for partcle flterng when ntegratng RANSAC s dffcult, but we address ths problem by analyzng the lkelhood dstrbuton of partcles from a real trackng example. The samplng effcency (on the more lkely areas) s much hgher by the use of RANSAC. We also dscuss the trackng qualty measurement n the sense of entropy or statstcal testng. The algorthm has been appled to the problem of 3D face pose trackng wth changng moderate or ntense expressons. We demonstrate the valdty of our approach wth several vdeo sequences acqured n an unstructured envronment. Key words: Random Projecton, RANSAC, Partcle Flterng, Robust 3D Face Trackng. 1 Introducton In recent years there has been a great deal of nterest n applyng Partcle Flterng (PF), also known as Condensaton or Sequental Importance Samplng (SIS), to computer vson problems. Applcatons on parameterzed or non-parameterzed contour trackng [11, 12, 2], and human trackng [2, 14] have demonstrated ts usefulness. However, the performance of SIS depends on both the number of partcles and the accuracy of the dynamc model. Gven a specfc error margn, the number of the partcles requred are generally determned by the dmenson and structure of the state space [5]. A typcal 6-DOF trackng problem usually requres thousands of partcles [5]; reducng the number of partcles by tranng a fnely tuned dynamc model s not trval [16], sometmes even mpossble. On the other hand, the RANSAC method [6] s often appled as a robust estmaton technque. The fnal stage n RANSAC s to apply a robust estmator that results n a good soluton whch ncludes as many non-outlers as possble. However, RANSAC by tself does not preserve multple solutons from frame to frame n a probablstc nference framework. In our proposed algorthm RANSAC-PF (or RANSAC- SIS), randomly selected feature correspondences are used to generate state hypotheses between pars of frames n vdeo sequences. However, nstead of lookng for a sngle best soluton, the projectons are used to gude the propagaton of the resampled partcles. These partcles are then reweghted accordng to a lkelhood functon and resampled. Consequently, the combned process not only serves as a robust estmator for a sngle frame, but provdes stablty over long sequences of frames. The convergence property of RANSAC-PF s emprcally analyzed. The evaluaton of qualty s an ssue of crtcal mportance for all trackng problems, stochastc or determnstc. Wth the samplng concept, t s straghtforward to nfer the trackng qualty from the state parameters posteror probablstc dstrbuton. We defne an entropy-based crteron to the statstcal qualty characterstcs of the tracked densty functon and evaluate t numercally. More mportantly, the entropy curve computed durng trackng can help us extract some well tracked frames as exemplars [2]. When necessary, these exemplars are then archved to stablze the trackng 1. The remander of ths paper s organzed as follows. Related work s presented n secton 2, followed by a descrpton of a 3D face trackng applcaton that uses our RANSAC-PF algorthm. We also address our entropy and statstcal testng based crteron for trackng qualty evaluaton n ths secton. Secton 4 shows some expermental results. Fnally, we offer conclusons and dscuss future work. Notaton: x (t) s the current state of the -th partcle at 1 Much of computer vson research can be regarded as lyng on a contnuum between explct models and exemplars [2]. To extract exemplars whch are tracked statstcally well, we dscuss the trackng qualty evaluaton ssue from the tracked posteror probablstc dstrbutons. 1

2 tme t, whle (t) represents the current and prevous states of the -th partcle at tme t. (t) = fx (1) ;:::;x (t) g. Assumng a second order Markov property, (t) represented as fx (t 1) ;x (t) g, orfx (t) ;v (t) x (t 1) x (t)! (t). can also be g, where v (t) = s the normalzed weght for the -th partcle at tme t. These weghts collectvely represent the partcles posteror probablty dstrbuton. z (t) delegates the current observaton (mage features, for example). z (t) = fz (t) 1 ;:::;z(t) g, where M M (t) s the num- (t) ber of detected features at tme t. 2 Related Work Determnstc parameter estmaton algorthms normally produce more drect and effcent results when compared wth Monte Carlo-style samplng methods. On the other hand, determnstc algorthms are unfortunately easly based and cannot recover from accumulated estmaton errors. Robust estmators, such as LMedS [23] and MLESAC [18], follow the strategy of Wnner Takes All and get the maxmum lkelhood (ML) result. However, those estmators are not sutable for a sequental estmaton problem for a dynamc system because the ML estmaton error n each stage can accumulate and result n falure. It s shown n [1] that sequental mportance samplng or partcle flterng may have non-zero probablty to condense nto an ncorrect absorbng state when the number of samples are fnte, even though PF-lke technques are accurate n the asymptotc sense. In our work, we propose a hybrd samplng approach to acheve a good balance of samplng effcency and dynamc stablty. From a partcle flter vewpont, random geometrc projectons wth the RANSAC samplng of mage features and mportance resamplng of t 1 gude the tme seres evoluton of state partcles to acheve the trade-off between varance and bas. There are some papers ntegratng partcle flters wth varatonal optmzaton or observaton-based mportance functon. For the sake of computatonal effcency, Sullvan et al. showed n [17] that random partcles can be guded by a varatonal search, wth good convergence when the mage dfferences between frames are low. They used a predefned threshold to swtch the probablstc or determnstc trackng engnes, whch could be problematc. Isard et al. [12] presented an approach (ICondesaton) to combne low-level and hgh-level nformaton by mportance resamplng wth a partcle flter. Our most related work s Torr and Davdson s research [19] on structure from moton by hybrd samplng (IMPSAC). They bult a herarchcal samplng archtecture wth a RANSAC-MCMC estmator at the coarse level and a SIR-MCMC estmator at the fner levels. In ths paper, we use sequental samplng-mportance-resamplng (SIR) technque to regularze and smooth the object pose estmaton from spatal RANSAC samplng wth sgnfcant ob- (a) (b) (C) Fgure 1: An example of RANSAC samplng of feature ponts (red dots are nlers, and green squares are outlers.) to track planar motons. servaton nose 2. Our technque consders the robust estmaton problem from the vewpont of tme seres analyss, whle Torr et al. constraned the output of RANSAC wth a MCMC formulated buldng model n 3D scene reconstructon. There are varous solutons for face pose trackng. Here we manly dscuss two technques: SSD (sum of squared dstances) and partcle flterng based trackng. SSD [7, 13] has attracted much nterest and has become a standard technque for varous trackng problems. However, a sngle template can not cover a large range of motons. Dynamc template updatng s usually a non-trval problem and causes error accumulaton. Moreover, the relatve moton between frames s requred to be smooth enough to satsfy the local lnearty prerequste of Jacoban approxmatons n SSD trackng. Jumps can cause loss of trackng. Moon et al. [14] proposed a new approach for the mage lkelhood measurement based on the shape dstance between the reprojected eye and mouth curve wth the detected curve n the mage. Ths observaton model s smple and effcent to be mplemented, but may have some problems wth large outof-plane rotatons and expressonal changes. We propose a new method to acheve comparable trackng results [13] under many strong dstractons, wth much fewer (5 ο 2) partcles, auto-recoverable for a wderange full 3D trackng problem 3. Small jumps (up to 1 o ) between consecutve frames s also not an ssue for our approach. 3 The RANSAC-PF Algorthm 3.1 Motvaton In order to explan our approach to trackng, consder the stuaton shown n Fgure 1. Here we consder n-plane rotatons and translatons of a planar object through three frames 2 We consder three sources of nose n the probablstc robust trackng work; 1) mage feature matchng outlers manly due to expresson deformatons; 2) the naccuracy of our 3D face model; 3) manual algnment errors for ntalzaton. The hybrd samplng strategy s used to handle the based or unbased noses, playng the smlar role as a robust estmator. 3 Face pose trackng s a well-studed topc, but stll remans dffcult for a deformable subject wth non-semantc, wde-range motons. Here we attempt to get 3D pose estmates from a movng talkng face. Expressonal deformatons are treated as clutter nose. Please refer to lelu/ransacpf.htm for more results. 2

3 Rotaton Angle (n rad) Fgure 2: Feature based trackng for a planar patch s cyclc nplane moton sequence. Blue represents ground truth of rotaton angles; black and pnk represent the results of RANSAC wth and wthout outlers; red (MAP) and green (MWP) (closely supermposed wth blue) are results for RANSAC-PF. of a vdeo. In each frame, a set of feature ponts fz (t) g are detected on the object. Some of these features are common among frames, and some are nconsstent or spurous. We develop a RANSAC flter for computng ncremental moton between successve frames (to smplfy features matchng), and then ntegrate these solutons over tme to compute a state estmate. Whle ths s computatonally convenent, the lack of dstrbutonal nformaton means that a sngle ncorrect estmaton step can be dsastrous. More precsely, note that RANSAC tres a number of subsets of features; whle the probablty that at least one of the subsets yelds an adequately correct result can be set to a hgh probablty 4 q by adjustng the tolerance threshold, the overall probablty of correct solutons over t frames, q t ; quckly decreases to zero as t grows. One way to avod ths problem s to nclude a tme seres model that mantans and regularzes multple solutons over tme. Ths s exactly the goal of our RANSAC-PF algorthm. In Fgure 2, we show the smulated results of RANSAC whle trackng a planar patch. The blue lne s the ground truth of n-plane rotaton angles, the magenta lne s the RANSAC trackng result when sub-pxel feature matchng accuracy s unavalable, and the black lne s the result when matchng outlers are ntroduced. We also test the RANSAC-PF algorthm n the syntheszed sequence: the red lne s the trajectory of partcles wth maxmal weght (MAP) and the green lne s the mean trajectory (MWP). In ths smple case, drft n the parameter estmaton from RANSAC s clearly vsble, whle there s no apparent drft wth RANSAC-PF of 1 partcles. For a second example, we use a partcle flter to track a sequence of smulated data. A 2nd order Markov dynamcs s adopted and the observaton lkelhood functon s based on the dfference wth the ground truth 5. We see that the performance of the partcle flter degrades dramatcally when 4 q =1only happens wthout exstence of any knd of nose. 5 Ths observaton measurement s near perfect, because the partcle weghts are penalzed respect to ther bas wth the ground truth. the number of dmensons of the state space ncreases. Fgure 3 shows the smulaton results for a partcle flter tested on varous dmensons. Qualtatvely, we see that even when estmatng only 2 parameters, a 2-partcle flter tends to compute poor solutons after 2 to 3 frames. It s evdent the results for 6 DOF trackng are meanngless wth 8 partcles. Ths s not nconsstent wth actual practce, where a few thousand partcles are used for 2D (four parameter) person trackng applcatons [5]. Kng et al. [1] also addressed the convergence problem of partcle flter wth fnte samples. 3.2 The General Algorthm a) From the ntal results ^x (1) ; ^x (2), construct partcles for the frst two frames. ffl x (1) =^x (1) ;=1;:::;N ffl x (2) = N (^x (2) ;ff); =1;:::;N, where N s a Gaussan Normal dffuson functon. b) From the prevous partcle set f(x (t 1) ;! (t 1) )g N at =1 tme t 1, construct a new partcle set f(x (t) ;! (t) )g N for =1 tme t by 1. For =1;:::;N, generate x (t) (a) Randomly select x (t 1) by wth probablty! (t 1) (b) Randomly select a subset Z p of M (t) features from z (t) by RANSAC. (c) Let x (t) = R(Z p ; x (t 1) ): 2. For =1;:::;N, compute! (t)! (t) P N =! (t) = =1!(t). = L(z (t) ; x (t) ) and 3. Compute P the emprcal entropy crteron H (t) = N =!(t) log 2! (t), or perform other statstcal testngs. Fgure 4: The RANSAC-PF algorthm We consder the object beng tracked as descrbed wth known models but unknown parameters (or states) x. Gven an observaton z (t) of the object for each mage frame t, the objectve s to estmate the object state x (t) at every tmestep (frame) from Z (t) = fz (1) ;:::;z (t) g. Assume that the underlyng observaton and dynamc models F and G are known: Z (t) = F ( (t) ; ) (1) x (t) = G( (t 1) ; ) (2) where the nose terms and have known or assumed dstrbuton. We note that the mage lkelhood functon L(z (t) ; (t) ) = p(z (t) j (t) ) and the state propagaton functon p(x (t) j (t 1) ) can be derved from ths stated nformaton. 3

4 Parameter Value Parameter Value 1 5 Parameter Value 2 2 Parameter Value (a) (b) (c) (d) Fgure 3: We smulate the trackng accuracy wth varyng numbers of partcles n 2, 4 and 6 state space dmensons. Because data s syntheszed, the ground truth s known and used to observe the lkelhood va an ndependent Gaussan process assumpton. To llustrate, only the trackng results of parameter 1 (no physcal meanng) s shown; smlar results are obtaned for other parameters. (a) 2 Partcles for 2 parameters (b) 2 Partcles for 4 parameters (c) 2 Partcles for 6 parameters (d) 8 Partcles for 6 parameters. Colors code same nformaton as n fgure 2. Let z (t) be a set of M (t) elements fz (t) g: z (t) = f ( (t) ; ); =1;:::;M (t) (3) We assume that x (t) can be computed from Z p, a subset of all M (t) elements of observaton. Let R(Z p ; (t 1) ) represent ths nverse of the above equaton. We ntroduce a set of partcles fx (t) g N =1 and ther relatve weghts f! (t) g N =1 that are mantaned for all tmesteps t as a representaton of the posteror dstrbuton p( (t) jz (t) ). Naturally, any functon e of (t) can be estmated by N e( (t) )=! e(x (t) ) (4) =1 Wth these defntons, we can see from Fgure 4 that RANSAC-PF operates roughly as follows. For each frame t, partcles are generated usng R by randomly selectng Z p and (t 1) and computng (t) : A graphcal model representaton of the algorthm s llustrated n Fgure 6 (b). Optonally, some other partcles are sampled from the dynamc model p( (t) j (t 1) ) usng randomly resampled partcles of (t 1) : These two sets of partcles can be mxed together. Weghts! (t) are then computed usng the mage lkelhood functon L as s normally done n mportance samplng. An entropy crteron or other testng to evaluate the trackng result from the partcles s also computed. Note that our RANSAC-PF algorthm does not necessarly need to be combned wth the standard partcle flterng, but ths combnaton makes t convenent to compare these two algorthms n the followng experment secton. 3.3 Convergence Analyss There are two mportant assumptons for the convergence analyss of sequental mportance samplng, or partcle flterng-style algorthms [3, 1]. Frst, the mportance functon s chosen so that the weghts of partcles are bounded above. No super-nodes wth unbounded hgh weghts exst that may domnate the dstrbuton and resamplng. Second, selecton scheme does not ntroduce too strong a dscrepancy. The frst prerequste s easly satsfed by the ndependent Gaussan process assumpton for lkelhood measurement (.e. computng the partcle weghts n factor samplng [11]). Each process returns a real value between and 1, and the number of processes s lmted. On the other hand, the computatonal effcency and convergence of partcle flter algorthms heavly depend on the selecton (resamplng) scheme. The deal case wll be that the lmted number of partcles are sampled from the hgh peak areas n the posteror densty resultng n an estmated expectaton 6 wth a tght varance. The lmted computatonal resource s not wasted n the unlkely zones. We also requre havng low bas to obtan good trackng results, so randomness s ntroduced around hgh densty peaks. The trade-off between varance and bas must be subtly handled. From Fgure 5, we show the partcles lkelhood values (weghts n factor samplng [11]) extracted from a face vdeo sequence. For comparson, red crcles represent partcles drven by RANSAC-PF and blue stars are partcles propagated through a second order Markov dynamcs. In Fgure 5 (a), there are a few hgh weght blue stars appearng n the cloud of red dots. As tme passes, both red and blue partcles ntally decrease ther weghts n (b), then stablze ther weghts at a reasonable level. (c) depcts a hard-totrack frame wth very poor object appearance 7 resultng n even lower weghts. However, the clear recovery s found n (d) where the partcles lkelhood values return to the same level as (b). The lkelhood dstrbuton of blue partcles s normally a very few hgh stars wth mostly low ones, whle the red partcles mantaned by RANSAC-PF have an opposte dstrbuton. In our experments, mult-modal estmates of partcles are converged to have a domnant result wth mnor (low-weghted) nosy estmates most of the tme. When dstractons are very strong, we obtan many estmates floatng around the true value, and no domnant estmate exsts. Our ntenton s not to gve a theoretcal convergence proof, but attempt to llustrate the basc concept of how RANSAC-PF works by an example. The possblty of poor samplng s lower for a dstrbuton wth more hgh lke- 6 The expectaton s also a random varable. 7 The subject s face s turnng down deeply, so the face regon s small and hghly tlted. It causes dffcultes for any face tracker. 4

5 lhood partcles [3, 1]. Most of the tme, the resamplng process on very low weght partcles generates a poor result. Smlarly, Tu et al. showed that usng data-drven technques (lke RANSAC n our case), such as clusterng and edge detecton, to compute mportance proposal probabltes (for partcle flter n our case), effectvely drves the Markov chan dynamcs and acheves tremendous speedup n comparson to the tradtonal jump-dffuson method [21]. 3.4 Trackng Evaluaton Once the trackng results are obtaned, our next step s evaluatng the soundness of the results. By lookng nto the lkelhood functon L(z (t) ; x (t) ), we can calculate a ftness value for each partcle x (t). An ntutve argument can be whether there exsts at least one partcle x (t) wth the ftness L(z (t) ; x (t) ) above a certan threshold. However, we prefer to fnd a more sound answer f possble, usng the property of the obtaned posteror densty fx (t) ;! (t) g of the state varables An Entropy-Based Crteron for Trackng Whle the mean of weghted partcles can serve as a representatve or estmate of the set of partcles, t s the entre set that does the trackng. To evaluate how well t s dong, we can ntroduce an entropy based crteron. Snce we are trackng only one object confguraton, a sngle and sharp peak of the posteror dstrbuton s deal, whle a broad peak probably means poor or lost trackng. Entropy can be used as a scale to dscrmnate these two condtons. Low entropy means less trackng uncertanty, thus better performance. Nevertheless, the weghted partcles fx ;! g N =1 are only a set of samples from a probablty dstrbuton p(x), not the dstrbuton tself. There are a number of ways to estmate the entropy of underlyng dstrbuton. The smplest method s to compute the entropy drectly from weghts of dscrete samples: H 1 = N =1! log 2 p(x )= N =1! log 2! (5) H 1 converges to H when N approaches nfnty, but they may have a sgnfcant dfference when N s small. An alternatve n ths case s to nclude a wndow functon to spread the support of a partcle lke a kernel. We have performed numercal evaluatons that suggest there s no sgnfcant dfference between these two methods wth 5 ο 2 partcles. Whle the entropy tself s a good ndcator, we sometmes need to better dscrmnate between unmodal and multmodal dstrbutons. To do so, we artfcally merge any par of partcles nto one super-partcle provded they are near enough n state space. In ths way, we further lower the outputs of entropy-estmate functons for sngle mode dstrbutons; thus promotng them. In our experments, the entropy curve s very stable most of the tme as expected, ndcatng the stable trackng performance. For the extreme cases (Fgure 5 (c)), the entropy value does ncrease Statstcal Testng for Mode Detecton Though partcle flterng s well known to tackle the non-gaussan trackng problem, we may stll want to test whether a sngle Gaussan s a vald posteror assumpton for a certan frame. For a sngle object trackng problem, the current frame computng result can be consdered favorable f the trackng densty can be well approxmated by a sngle Gaussan wth tght varance. Furthermore, well tracked frames can be used as exemplars to buld an adaptve multvew object model [15] for many purposes. We address the problem as verfyng whether a Gaussan mxture model (GMM) can acheve a statstcally superor approxmaton for trackng posteror densty, compared wth a sngle Gaussan model. Snce we are only concerned wth whether the dstrbuton s unmodal or mult-modal, the testng on 1 or 2-component GMM s adequate. Ths model selecton problem can be solved by evaluatng the condtonal mxture densty (log-)lkelhoods of 1 or 2-component GMM va Expectaton Maxmzaton [9] 8. Alternatvely, the modalty of the trackng densty may be observed drectly by detectng whether the partcle of maxmum a posteror convergng to the estmated posteror expectaton, as an emprcal measure. It s clearly shown from our vdeos that ths convergence (unmodal) s mostly kept n our real experments, except for the poorly tracked frames. 4 Experments on 3-D face trackng The dagram of our face trackng system s shown n Fgure 6 (a). We use a generc trangle based face model, whch s hghly parameterzed and can be easly manpulated wth geometrc modelng software. Dfferent from [22], the approxmate 3D face models are suffcent to acheve the reasonable good trackng results n our experments, thanks to the stochastc property of RANSAC-PF. When ntatng trackng, we regster the generc 3D model to the frst vdeo frame by manually pckng 6 fducal (mouth and eye) corners n the face mage. A two-vew geometrc estmate [4, 23] s then computed for the face pose on the next frame, followed by a Gaussan dffuson. Consequently, bx (1) and bx (2) are obtaned as the state vectors that encode the face pose (three components for rotaton and three components for translaton). 4.1 Feature Detecton and Random Projecton In our face trackng applcaton, we frst detect Harrslke [8] mage corner features n two frames. Then, a cross correlaton process for feature matchng and a rough feature clusterng algorthm based on eppolar geometry are performed to form an ntal set of correspondng feature pars. To compute a relatve pose change, we spatally and unformly sample 9 matched mage features between two 8 The dfference s that partcles are data samples wth dfferent weghts, makng non-equal contrbutons for the mxture densty modellng. 5

6 Lkelhood Value Lkelhood Value 4 3 Lkelhood Value 15 Lkelhood Value Partcle Number (a) Frame 3 Partcle Number (b) Frame 15 Partcle Number (c) Frame 36 Partcle Number (d) Frame Fgure 5: The lkelhood dstrbuton of partcles n a vdeo sequence. There are 1 (blue star) dynamcs drven partcles and 1 (red crcle) RANSAC-PF guded partcles).) Face Pose PDF at Image Frame: t-1 P( t -1 ) Image Frame: t-1 Select 9 pars of mage feature by RANSAC Partcle Importance Resamplng 3D Face Model 3D Facal Features Image Frame: t Select 9 pars of mage feature by RANSAC Partcle Geometrcal Projecton Face Pose PDF at Image Frame: t t P( ) t t-1 x x t t-1 t-1 (a) Partcle Flterng Z t -1 Zt t Z t+1 t+1 t = f ( Zt, Zt, t ) -1-1 p( Z t t) (b) Fgure 6: (a) Dagram of RANSAC-PF as appled to 3D face pose trackng. (b) The graphcal representaton of RANSAC-PF where the new state t s a functon of new observaton Z t, former observaton Z t 1 and state t 1. (a) (b) (c) (d) (e) Fgure 7: The ntalzaton process of face trackng. (a) The ntal frame s manually algned wth a 3D face model usng 6 fducal corners. (b) The next frame s tracked through the two-vew moton estmaton. The RANSAC-PF trackng begns from the thrd frame. We show the Maxmum A Posteror (MAP) result wth a red color reprojected face mesh overlad on the mages, whle the mean of weghted partcles (MWP) wth a black color. (c)map and MWP are dfferent at the begnnng frames of the RANSAC-PF trackng. (d) MAP and MWP converge together quckly. (e) Image features are selected spatally and unformly va RNASAC. frames by RANSAC, llustrated n Fgure 7 (c). We now obtan a set of rgd feature matches f(m (t 1) j ; m (t) j )g, where descrbed n [4]. A par of ( ^R ; ^t ) corresponds to a certan on ^T. We compute ^T wth a lnear least-squares technque 9 m (t 1) j and m (t) j are a par of (probably non-perfectly) partcle as (t). Therefore, ths lnear geometrc projecton matched ponts n two successve mages. For each pont behaves as a brdge between the propagaton of state partcles from (t 1) to (t) ; = 1;:::;N p on frame t. m t 1 j n the reference mage, we cast a 3D ray from the We camera center through that pont, and compute the ntersecton Z j of that ray wth the face mesh model, usng a resampled (factor samplng) pose state x (t 1) at frame (t 1). The relatve pose ^T ^R ^t = can then be computed T 1 accordng to the followng equaton AP ^T ~ Z j = ~m t j (6) where Z ~ j =(Z T j ; 1)T and ~m j =(m T j ; 1)T. The ntrnsc matrx A, the standard projecton matrx P, Z j and m t j are known. Each of the above equatons gves two constrants call ths process random projecton (RP). 4.2 Dynamcs and Image lkelhood Image observatons are modelled as a Gaussan process. Wth each x (t) and ts former state hstory x (t 1), we can project the poston of mage pont features at mage (t 1) 9 We use 9 as the number of mage features for the random projecton n our algorthm. In theory, 3 s the mnmal possble number to compute the 3D object pose. By consderng the sub-pxel matchng errors, too few (e, 3) features can not provde stable geometrc estmates normally. On the contrary, too many features lose the advantage of robustness by random samplng. We emprcally fnd 9 s a good number for the trade-off. More theoretcal and expermental analyss wll be explored for future work. 6

7 to mage (t). The reprojecton errors are the 2D Eucldean dstances d 2 m between mage features (u (t) m ;v m (t) ) at frame t and reprojected mage features (~u (t) m ; ~v m (t) ). d 2 m =(u(t) m ~u (t) m )2 +(v m (t) ~v m (t) )2 (7) Then the condtonal probablty for lkelhood s p(zjx) / 1 p 2ßff m e d2 m 2ff 2 (8) where the standard dervaton ff can be estmated from the set of all feature reprojecton dstances fd m g for each par of x (t 1) and x (t). In the experments, we set ff to 2:5 pxels for smplcty. No apparent mprovement was found when estmatng ff from data. 4.3 Results of Our Algorthm We use a smple constant velocty model to gude the temporal evoluton of partcle flterng. In Fgure 7, we show a short face trackng vdeo (Comparson.av) wth large out-of-plane rotatons. In ths case, a subject s face s consdered as a rgd object wthout facal expressons. From ths fgure, reasonable trackng accuracy 1 s acheved, though the generc face model s not very accurate for the gven subject and feature msmatchng does exst. After few frames, the estmates of MAP and MWP estmates converge together. For the convenence of comparson, we generalze our RANSAC-PF algorthm wth partcles gudng by a second order Markov (constant velocty) dynamcs n parallel (see the dashed lne n Fgure 6). The testng results on the above short sequence (Comparson.av) s shown n Fgure 8. We name the partcles drven by RANSAC-PF RP (random projecton) partcles, and the others as DP (dynamc propagaton) partcles. Note that the constant velocty dynamcs can be consdered as a reasonable assumpton for ths smple yawng vdeo sequence. Nevertheless, the trackng n Fgure 8 (c) s quckly lost due to the relatvely small number of partcles accordng to the 6-DOFs requred by 3D tasks. On the other hand, our algorthm performs better wth the same or smaller number of total partcles. From Fgure 8 (a) and (Comparson.av), good trackng results are obtaned wth 1 RP partcles and 1 DP ones. When reducng the RP partcles to 1 n Fgure 8 (d), slght trackng accuracy s lost for MWP and the MAP results become to flcker around MWP estmates. It means that the computed MWP s stable and MWP s not. Here 1 can be thought of as a lower bound for the number of RP partcles. The decrease of DP partcles (comparng (b) to (a) n fgure 8) does not apparently nfluence trackng qualty. We also test our algorthms on trackng people faces from dfferent races. Reasonably good trackng performance s acheved. Two vdeo sequences (cher.av, donald.av) are 1 Snce we do not have the ground truth for trackng, no explct numercal comparson s provded. The valdty s shown by overlayng the 3D face mesh model to mages. lnked n author s webste. (Cher.av) has moderate expresson changes and results n better trackng, compared to (donald.av), where ntensve expressonal deformatons occur. Both of the vdeos (tracked wth 8 partcles) have long rotaton ranges over 2 ο 3 seconds, and subjects move ther face arbtrarly. Automatc recoveres from poorly tracked frames can also be found. To test the robustness to msalgnments, we manually algn the frst frame n the trackng sequence wth some moderate errors. Our algorthm shows the remarkable stablty from Fgure 9. The ntal regstraton errors do not ncrease wth tme, and a sgnfcant accumulaton of trackng errors s not observed. A general 3D face model s used for trackng though partcular adjustments of face model to a subject may mprove the trackng. In our experments, the relatve moton between successve frames s not requred to be very smooth. We have concluded that random projecton s most successful when handlng rotatons of o to 5 o degrees. One way to test ths robustness s to smply leave frames out. In experments, mages used for trackng can be sub-sampled every 3 to 1 frames. 5 Summary and Future Work In ths paper, we have presented a stochastc method for full 3D face trackng wth a small number of partcles and no learned dynamcs. RANSAC-based mage feature selecton s ntegrated wthn the Monte Carlo samplng framework. The convergence ssue and trackng qualty evaluaton problem are also dscussed. Our RANSAC-PF algorthm does not depend on the exstence of a specfc, fne-tuned dynamcs for the dverse object movng sequences. Furthermore, our algorthm can help the labellng problem for the new trackng data, whch can be valuable for dynamcs learnng and moton recognton. We also ntend to extend our work to mult-face trackng. Our local feature matchng algorthm s expected to dstngush features from dfferent faces by appearance and spatal neghborhood constrants. Ths step can help RANSAC generate proposals from each person s matched feature set respectvely. Data assocaton problem wll be much easer. Fnally, fndng sutable methods to compute mportance proposal probabltes for Monte Carlo-style algorthms s our emphass on future work. References [1] D. Crsan and A. Doucet, A Survey of Convergence Results on Partcle Flterng for Practtoners, IEEE Trans. Sgnal Processng, vol. 5, no. 3, pp , 22 [2] J. Deutscher, A. Blake and I. Red, Artculated Body Moton Capture by Annealed Partcle Flterng. CVPR. [3] D. Crsan and A. Doucet, Convergence of Sequental Monte Carlo Methods. CUED/F-INFENG/TR381, 2. [4] O. Faugeras, Three-Dmensonal Computer Vson: a Geometrc Vewpont, MIT Press,

8 (a) (b) (c) (d) Fgure 8: The trackng result comparson of RANSAC-PF under dfferent confguratons. (a) 1 RP partcles and 1 DP partcles (b) 1 RP partcles and 1 DP partcles (c) 2 DP partcles (d) 1 RP partcles and 1 DP partcles (a) (b) (c) (d) Fgure 9: Robustness testng for a msalgned vdeo sequence. (a) The ntal frame wth the vsble algnment error (b) Frame 195 (c) Frame 628 (d) Frame 948 [5] D. Fox, KLD-Samplng: Adaptve Partcle Flters. NIPS 1. [6] M.A. Fschler and R.C. Bolles, Random Sample Consensus: A Paradgm for Model Fttng wth Applcaton to Image Analyss and Automated Cartography. Commun. Assoc. Comp. Mach., vol. 24:381-95, [7] G. Hager and P. Belhumeur, Effcent Regon Trackng Wth Parametrc Models of Geometry and Illumnaton. IEEE Trans. PAMI, 2:1, [8] C. Harrs and M. Stephens, A combned corner and edge detector. 4th Alvey Vson Conf., pp , [9] T. Haste and R. Tbshran, Dscrmnant Analyss by Gaussan Mxtures. Journal of Royal Statstcal Socety Seres B, 58(1): [1] O. Kng and D. Forsyth, How does CONDENSATION behave wth a fnte number of samples? ECCV, pp [11] M. Isard and A. Blake, CONDENSATION condtonal densty propagaton for vsual trackng. IJCV 29:1, pp. 5-28, [12] M. Isard and A. Blake, ICONDENSATION: Unfyng lowlevel and hgh-level trackng n a stochastc framework. ECCV 98, pp [13] M. La Casca, S. Sclaroff, and V. Athtsos, Fast, Relable Head Trackng under Varyng Illumnaton: An Approach Based on Robust Regstraton of Texture-Mapped 3D Models. IEEE Trans. PAMI, 22:4, Aprl, 2. [14] H. Moon, R. Chellappa, and A. Rosenfeld, 3D Object Trackng usng Shape-Encoded Partcle Propagaton. ICCV 1. [15] L. Morency, A. Rahm and T. Darrell, Adaptve Vew- Based Appearance Models. CVPR 3. [16] B. North, A. Blake, M. Isard, and J. Rttscher, Learnng and classfcaton of complex dynamcs. IEEE Trans. PAMI, 22:9, pp , Sep., 2. [17] J. Sullvan and J. Rttscher, Gudng Random Partcles by Determnstc Search. ICCV 1, Vol.I, pp [18] P.H. Torr and A. Zsserman, MLESAC: A New Robust Estmator wth Applcaton to Estmatng Image Geometry. CVIU, 78, pp , 2. [19] P.H. Torr and C. Davdson, IMPSAC: Synthess of Importance Samplng and Random Sample Consensus. IEEE Trans. PAMI, 25:3, March, 23. [2] K. Toyama and A. Blake, Probablstc Trackng n a Metrx Space. ICCV 1. [21] Z. Tu and S.C. Zhu, Image Segmentaton by Data-Drven Markov Chan Monte Carlo. IEEE Trans. PAMI, 24:5, May 22. [22] L. Vacchett, V. Lepett, P. Fua, Fusng Onlne and Offlne Informaton for Stable 3D Trackng n Real-Tme. CVPR 3. [23] Z. Zhang, et al., A Robust Technque for Matchng Two Uncalbrated Images Through the Recovery of the Unknown Eppolar Geometry, Artfcal Intellgence J., 78 pp: , Oct