Elsevier Editorial(tm) for Image and Vision Computing Manuscript Draft

Transcription

1 Elsever Edtoral(tm) for Image and Vson Computng Manuscrpt Draft Manuscrpt Number: Ttle: FPIV'4:Effcent Partcle Flterng Usng RANSAC wth Applcaton to 3D Face Trackng Artcle Type: Specal Issue Paper Keywords: Random Projecton; RANSAC; Partcle Flterng; Robust 3D Face Trackng Correspondng Author: Le Lu Other Authors: Xangtan Da; Gregory D. Hager,

2 * Manuscrpt Effcent Partcle Flterng Usng RANSAC wth Applcaton to 3D Face Trackng Le Lu Xangtan Da Gregory Hager Computatonal Interacton and Robotcs Lab Computer Scence Department the Johns Hopkns Unversty Baltmore, MD 21218, USA Abstract Partcle flterng s a very popular technque for sequental state estmaton. However, n hgh-dmensonal cases where the state dynamcs are complex or poorly modeled, thousands of partcles are usually requred for real applcatons. Ths paper presents a hybrd samplng soluton that combnes RANSAC and partcle flterng. In ths approach, RANSAC provdes proposal partcles that, wth hgh probablty, represent the observaton lkelhood. Both condtonally ndependent RANSAC samplng and boostng-lke condtonally dependent RANSAC samplng are explored. We show that the use of RANSAC-guded samplng reduces the necessary number of partcles to dozens for a full 3D trackng problem. Ths s method s partcularly advantageous when state dynamcs are poorly modeled. We show emprcally that the samplng effcency (n terms of lkelhood) s much hgher wth the use of RANSAC. The algorthm has been appled to the problem of 3D face pose trackng wth changng expresson. We demonstrate the valdty of our approach wth several vdeo sequences acqured n an unstructured envronment. Key words: Random Projecton, RANSAC, Partcle Flterng, Robust 3D Face Trackng. 1 Introducton In recent years there has been a great deal of nterest n applyng Partcle Flterng (PF), also known as Condensaton or Sequental Importance Samplng (SIS [16, 17]), to computer vson problems. Applcatons on parameterzed or non-parameterzed contour trackng [13, 14, 28], and human trackng [3, 18] have demonstrated ts usefulness. However, the performance of SIS depends on both the number of partcles and the accuracy of the dynamc model. Gven a specfc error margn, the number of the partcles requred s generally determned by the dmenson and structure of the state space [7]. A typcal 6-DOF trackng problem usually requres thousands of partcles [7]; reducng the number of partcles by tranng a fnely tuned dynamc model s not trval [2] and sometmes not even possble. On the other hand, the RANSAC algorthm [8] s often appled as a robust estmaton technque. The basc dea behnd RANSAC s to evaluate many small batches of observatons, and from those to choose a larger set of good observaton from whch a robust state estmate can be produced. Although well-suted for producng sngle state estmates, RANSAC by tself does not preserve multple solutons from frame to frame n a probablstc nference framework. In our proposed algorthm RANSAC-PF (or RANSAC- SIS), randomly selected feature correspondences are used to generate state hypotheses between pars of frames n vdeo sequences. However, nstead of lookng for a sngle best soluton, the projectons are used to gude the propagaton of resampled partcles. These partcles are then reweghted accordng to a lkelhood functon and resampled. Consequently, the combned process not only serves as a robust estmator for a sngle frame, but provdes stablty over long sequences of frames. The evaluaton of soluton qualty s an ssue of crtcal mportance for all trackng problems, stochastc or determnstc. Wth the samplng concept, t s straghtforward to nfer the trackng qualty from the state parameters posteror probablstc dstrbuton. We defne an entropy-based crteron to evaluate the statstcal qualty of the tracked densty functon. In addton to provdng a qualty measure, the entropy values computed durng trackng can help us extract some well tracked frames as exemplars [28]. When necessary, these exemplars can be archved as key frames whch are used to further stablze the trackng. The remander of ths paper s organzed as follows. Related work s presented n secton 2, followed by a descrpton of the RANSAC-PF algorthm n secton 3. In secton 4, we dscuss the samplng effcency of RANSAC-PF. Secton 5 descrbes a 3D face trackng applcaton that uses our RANSAC-PF algorthm and presents expermental results. Fnally, we offer conclusons and dscuss future work. 1

3 2 Related Work Determnstc parameter estmaton algorthms, e.g. lnear mean-square methods, normally produce more drect and effcent results when compared wth Monte Carlo-style samplng methods. On the other hand, determnstc algorthms are unfortunately easly based and cannot recover from accumulated estmaton errors. Robust estmators, such as LMedS [33] and MLESAC [25], follow the strategy of Wnner Takes All to get, wth hgh probablty, the maxmum lkelhood (ML) estmate from contamnated data. However, as dscussed n the next secton, those estmators are are often not sutable for a sequental estmaton problem for a dynamc system because the estmaton error n each stage can accumulate and result n falure on sequental data. Partcle flters (or, more generally, sequental mportant samplng [?]), are a famly of technques for recursve estmate that use samplng methods to approxmate the optmal Bayesan flter. However, t s shown n [12] that sequental mportance samplng or partcle flterng may have non-zero probablty to condense nto an ncorrect absorbng state when the number of samples are fnte, even though partcle flterng technques can, under sutable assumptons, be accurate n an asymptotc sense. In our work, we propose a hybrd samplng approach to acheve a good balance of samplng effcency and dynamc stablty. From a partcle flter vewpont, both random geometrc projectons from RANSAC samplng of mage features and mportance resamplng gude the tme seres evoluton of state partcles. Generally, smple resamplng ncludes many low weght partcles and thus generates poor results. Tu et al. showed that usng data-drven technques (lke RANSAC n our case), such as clusterng and edge detecton, to compute mportance proposal probabltes (for partcle flter n our case), effectvely drves the Markov chan dynamcs and acheves tremendous speedup n comparson to the tradtonal jump-dffuson method [29]. There are several other papers ntegratng partcle flters wth varatonal optmzaton or observaton-based mportance functons. Sullvan et al. showed n [24] that random partcles can be guded by a varatonal search, wth good convergence when the mage dfferences between frames are low. They used a predefned threshold to swtch between the probablstc and determnstc trackng engnes. Isard et al. [14] presented an approach (ICondesaton) to combne low-level and hgh-level nformaton by mportance resamplng wth a partcle flter. In more closely related work, Torr and Davdson [26] compute structure from moton by hybrd samplng (IMP- SAC). They bult a herarchcal samplng archtecture wth a RANSAC-MCMC estmator at the coarse level and a SIR- MCMC estmator at the fner levels. In ths paper, we use sequental samplng-mportance-resamplng (SIR) technque to regularze and smooth the object pose estmaton from spatal RANSAC samplng. As such, our technque consders the robust estmaton problem from the vewpont of tme seres analyss, whle Torr et al. constraned the output of RANSAC wth a MCMC formulated buldng model n 3D scene reconstructon. To demonstrate the effcacy of our technque, we develop a system for 3-D face pose trackng. Although there are many solutons for 3-D face pose trackng, most algorthms are some varaton on drect methods (e.g. SSD trackng) or feature-based matchng. SSD-lke methods [9, 15, 32, 1] have attracted much nterest and have become a standard technque for many trackng problems. However, a classcal determnstc SSD tracker requres a good predcton of target locaton n order to converge relably. As a result, unexpected moton jumps can cause loss of trackng, and thus requre addtonal apparatus to guarantee robustness [27]. On the other hand, feature-based methods have the advantage of potentally provdng good solutons, provded stable features and good correspondences are avalable. The advantage of partcle-flterng-based methods s that such a correspondence need not be produced explctly, but s mplct n the lkelhood functon. Thus, good samplng combned wth a good lkelhood functon can yeld good trackng results, even n cluttered stuatons [18]. However, n cases where motons are abrupt and poorly modeled, and features are unstable, partcle-based methods can stll fal unless extremely large numbers of samples are generated. For ths reason, the majorty of partcle-flter-based vsual trackng consder restrcted cases of two-dmensonal moton. By ncludng an explct correspondence search as part of the samplng process, we are able to acheve comparable trackng results [15] under many strong dstractons, wth much fewer (5 2) partcles, auto-recoverable for a full 3-D pose trackng problem. We also develop a boostng-lke RANSAC method to even more effcently sample partcles from the mage observatons. Okuma et al. [21] utlze Adaboost [31] to generate new detecton hypotheses of mxture partcle flterng for mult-target trackng. In comparson, we employ the boostng prncple to gude the nteracton of samplng between two partcles. Our method s smlar n nature to the teratvely boosted mxture modelng of Pavlovc [23]. He descrbes an optmzaton based approach to maxmze the data s lkelhood gven an estmated Gaussan mxture model. Data are then weghted nversely to ther current lkelhood values and these weghts are then used to to calculate the new kernel of the mxture, and so on. As a result, data poorly represented by the prevous mxture model have more weghts n the next tme step. 2

4 4 2 2 Rotaton Angle (n rad) (a) (b) (C) Fgure 1: An example of RANSAC samplng of feature ponts (red dots are nlers, and green squares are outlers.) to track planar motons. 3 The RANSAC-PF Algorthm 3.1 Motvaton In order to motvate RANSAC-PF, consder the stuaton shown n Fgure 1. Here we consder n-plane rotatons and translatons of a planar object through three frames of a vdeo sequence. In each frame, a set of feature ponts {z (t) } are detected on the object. Some of these features are common among frames, and some are nconsstent or spurous. To track the object, we make use of RANSAC [8] to compute the ncremental moton between successve frames, and then ntegrate these solutons over tme to compute a state estmate. Whle ths s computatonally convenent, the lack of dstrbutonal nformaton means that a sngle ncorrect estmaton step can be dsastrous. More precsely, RANSAC samples some a number, l of subsets of the observed features. Let q denote the probablty that one or more of these l subsets yelds a correct correspondence whch s detected by RANSAC. Although q can, n prncple, be made arbtrarly large by ncreasng l, t wll stll generally be less than 1. As a result, the overall probablty of consstently correct solutons over t frames, q t, quckly decreases to zero as t grows. One way to avod ths problem s to nclude a tme seres model that mantans and regularzes multple solutons over tme. Ths s exactly the goal of our RANSAC-PF algorthm. In Fgure 2, we show the smulated results of RANSAC whle trackng a planar patch. The patch s movng forward whle rotatng n the plane. The blue lne s the ground truth of n-plane rotaton angles, the magenta lne s the RANSAC trackng result when sub-pxel feature matchng accuracy s unavalable, and the black lne s the result when matchng outlers are ntroduced. We also test the RANSAC-PF algorthm (ntroduced formally n the next secton) on the syntheszed sequence. In the fgure, the red lne s the trajectory of the partcle wth maxmal weght at each tme pont (an approxmaton to the MAP estmate), and the green lne s the mean state value at each tme pont Frame Number Fgure 2: Feature based trackng for a planar patch s cyclc nplane moton sequence. Blue represents ground truth of rotaton angles; black and magenta represent the results of RANSAC wth and wthout outlers, respectvely. The results for RANSAC-PF are: red, hghest weghted partcle, and green, mean state value. In most cases, the latter are closely supermposed wth blue. In ths smple case, drft n the parameter estmaton from RANSAC s clearly vsble, whle there s no apparent drft wth RANSAC-PF usng only 1 partcles. For a second example, we use a partcle flter to track a sequence of smulated data. A 2nd order Markov dynamcs s adopted and the state s drectly observed under Gaussan nose. The observaton lkelhood functon s computed based on the dfference wth the ground truth state and s also Gaussan. Fgure 3 shows the smulaton results for a classcal partcle flter tested on varous state dmensonaltes. Qualtatvely, we see that even when estmatng only 2 parameters, a 2-partcle flter tends to compute poor solutons after 2 to 3 frames. It s evdent the results for 6 DOF trackng are meanngless wth 8 partcles. In short, the performance of the partcle flter degrades dramatcally when the number of dmensons of the state space ncreases. Ths s consstent wth actual practce, where a few thousand partcles are used for 2D (four parameter) person trackng applcatons [7] and also reflect the results of Kng et al. [12]. 3.2 The General Algorthm We consder the object beng tracked as descrbed wth known models but unknown parameters (or state) X. Gven an observaton Z (t) of the object for each mage frame t, the objectve s to estmate the object state X (t) at every tmestep (frame) t. We assume that the underlyng observaton and dynamc models F and G are known: Z (t) = F (X (t),η) (1) X (t) = G(X (t 1),ζ) (2) where the nose terms η and ζ have known or assumed dstrbuton. We note that the mage lkelhood functon 3

5 Parameter Value Parameter Value 1 5 Parameter Value 2 2 Parameter Value Frame Number (a) Frame Number (b) Frame Number (c) Frame Number (d) Fgure 3: We smulate the trackng accuracy wth varyng numbers of partcles n 2, 4 and 6 state space dmensons. Because data s syntheszed, the ground truth s known and used to observe the lkelhood va an ndependent Gaussan process assumpton. To llustrate, only the trackng results of parameter 1 (no physcal meanng) s shown; smlar results are obtaned for other parameters. (a) 2 Partcles for 2 parameters (b) 2 Partcles for 4 parameters (c) 2 Partcles for 6 parameters (d) 8 Partcles for 6 parameters. Colors code same nformaton as n fgure 2. L(Z (t) ; X (t) ) = p(z (t) X (t) ) and the state propagaton functon p(x (t) X (t 1) ) can be derved from ths stated nformaton. Let Z (t) be a set of M (t) elements {z (t) }: z (t) = f (X (t),η), =1,...,M (t) (3) We assume that X (t) can be computed from Z p, a subset of all M (t) elements of observaton and, optonally, a pror state value X (t 1). Let R(Z p ; X (t 1) ) denote ths functon whch s essentally an nverse of (1). A specfc nstantaton for face trackng s gven n secton 5. We ntroduce a set of partcles {x (t) } N+Q atve weghts {ω (t) =1 and ther rel- } N+Q =1 that are mantaned for all tmesteps t as a representaton of the posteror dstrbuton p(x (t) Z (t) ). Naturally, any functon e of X (t) can be estmated by e(x (t) )= N+Q =1 ω e(x (t) ) (4) are then computed usng the mage lkelhood functon L as s normally done n mportance samplng. Note that our RANSAC-PF algorthm does not necessarly need to be combned wth the standard partcle flterng, but ths combnaton makes t convenent to compare these two algorthms n the subsequent experments. A graphcal model representaton of the algorthm s llustrated n Fgure 5. Wth these defntons, we can see from Fgure 4 that RANSAC-PF operates roughly as follows. For each frame t, partcles are generated usng R(Z p ; X (t 1) ) by randomly selectng Z p and X (t 1) and computng X (t). Optonally, some other partcles are sampled from the dynamc model p(x (t) X (t 1) ) usng randomly resampled partcles of X (t 1). These two sets of partcles can be mxed together. Weghts ω (t) Condtonal ndependent/ dependent RANSAC samplng We have also ncluded a boosted verson of RANSAC- PF n Fgure 4. The smple RANSAC samplng procedure produces current partcles n the state space wthout further evaluatng the partcles produced n prevous samples. In order to ncrease the samplng effcency, especally for mult-modal densty functons, we have found that a boostng-lke strategy mproves algorthm performance. To llustrate the dea, consder agan the feature based planar trackng problem n Fgure 2. Any selecton of two pars of feature correspondences can be used to calculate a soluton for the translaton and rotaton parameters. Assume now that there are three ndependent motons, wth one half of the feature ponts followng one moton and a quarter of feature ponts followng each of the other two motons. Therefore, we can obtan a correctly sampled hypothess of the moton only when the par of selected feature correspondences belong to the same moton mode. Otherwse, a faulty partcle hypothess s generated from features of dfferent motons. Thus, the probablty that the frst moton s sampled s 1/4 =(1/2) (1/2), the probablty of the other two motons beng sampled s 1/16 = (1/4) (1/4), and random faulty hypotheses are generated wth the remanng probablty of 5/8 = 1 1/4 1/8. As a result, a correct state hypothess has the probablty of only 3/8 of beng generated. Ths s referred to as the samplng effcency, P s. In practce, P s s should be as large as possble. The boostng-lke heurstc n Fgure 4 means that the features that are consstent wth a current hypothess have a hgher chance to be selected to generate the next hypothess. In the case when correct and ncorrect matches are perfectly dstngushed, the correspondng probabltes are 1/2, 5/9 or at least 3/8 1. As a result, the samplng effcency of the 1 Assume that the frst mode s frst sampled by the partcle and all the features assocated wth ths moton are not elgble for the samplng of next partcle. Because there are half and half features left for mode 2 and 4

6 RANSAC Partcle Flterng Algorthm nputs: Z (t) ; ˆx (1) and ˆx (2) ; N; Q; BoostFlag; outputs: X (t) Z t -1 Zt Z t+1 a) From the ntal results ˆx (1), ˆx (2), construct partcles for the frst two frames. x (1) =ˆx (1),=1,...,N x (2) = N (ˆx (2),σ), =1,...,N, where N s a Normal dffuson functon. b) From the prevous partcle set {(x (t 1) tme t 1, construct a new partcle set {(x (t) tme t by 1. If (BoostFlag = False) For =1,...,N, generate x (t) by (a) Randomly select x (t 1),ω (t 1) )} N =1 at )} N =1 for,ω (t) wth probablty ω (t 1). (b) Unformly Randomly select a subset Z p of Z (t) from M (t) features (RANSAC). (c) Let x (t) = R(Z p ; x (t 1) ). 2. If (BoostFlag = True) For =1,...,N/2, generate x (t) and x (t) +N/2 by (a) Randomly select x (t 1) wth probablty {ω (t 1) }. (b) Unformly randomly select a subset Z p of Z (t) from M (t) features (RANSAC). (c) Let x (t) = R(Z p ; x (t 1) ). (d) For j = 1,...,M (t), weght each feature j as ν (t) j = 1/p(z (t) j x (t) ) and ν (t) j = ν (t) j / N =1 ν(t) j. (e) Randomly select a subset Z p of Z (t) from M (t) features wth probablty {ν (t) j } (Boosted- RANSAC). (f) Let x (t) +N/2 = R(Z p; x (t 1) ). 3. = N +1,...,N + Q, generate x (t) = G(X (t 1),ζ). x (t) 4. For =1,...,N+ Q, compute ω (t) and ω (t) = ω (t) / N+Q =1 ω (t). Fgure 4: The RANSAC-PF algorthm. and by Let = L(Z (t) ; x (t) ) 3, the probablty P s becomes 1/2 =(1/2 1/2+1/2 1/2). Smlarly, P s =(2/3 2/3 +1/3 1/3) for the cases that mode 2 or 3 s frst sampled; P s =(1/2 1/2+1/4 1/4+1/4 1/4) for the case that no moton mode s frst sampled. 5 X t -1 X t X t+1 X = f Z, Z, X ) p Z t X ) t ( t t-1 t-1 ( t Fgure 5: A graphcal representaton of RANSAC-PF where the new state X t s a functon of new observaton Z t, former observaton Z t 1 and state X t 1. next partcle s now even hgher. In the sequel, the partcle sampled from the reweghted mage features (Step 2f of Fgure 4) s called the dual of the current hypothess. We note that there are several other possble boostng algorthms. For example, n the current algorthm all partcles from RANSAC-PF are ndependently sampled, whle the boosted RANSAC-PF partcle set conssts of half RANSAC produced hypothess and half ther boosted dual partcles. An possble alternaton s to frst sample all RANSAC partcles accordng to ther weghts (lkelhood), then sample ther dual partcles. For more complex densty model, a cascaded boostng scheme may be used [31] Comparson of mult-modal densty trackng In Fgure 6, we show the results of smulatng mult-modal densty trackng results for wth four dfferent algorthms. The underlyng moton densty functon has three modes wth 1/2, 1/4 and 1/4 supportng features, respectvely. We synthesze a sequence wth 8 frames and assume that the postons of feature ponts are detected wthout errors. We compare the four trackers performance on densty functon approxmaton accuracy n terms of keepng all the three modes durng trackng. For better vsualzaton, we convert the weghted partcle representaton of the densty functon nto a hstogram model usng Parzon wndow ntegraton [5]. From Fgure 6 (a) and (b), our proposed RANSAC- PF and boosted RANSAC-PF successfully track the three modes very accurately wthout any pror knowledge of the underlyng densty functon. Boosted RANSAC-PF further ncreases the samplng effcency by havng more partcle weghts concentrated on the three moton modes. By comparson, a vanlla partcle flterng algorthm can theoretcally track mult-modal functons, but t normally requres a very large number of partcles n the absence of a well tuned dynamcal model. In order to generate partcles coverng the three modes, we use a constant velocty

7 (a) (b) (c) (d) Fgure 6: The comparson of mult-modal densty trackng on n-plane rotaton. There are three modes n the underlyng densty functon. The syntheszed sequence contans 8 frames. 6 partcles are used for trackng, and the densty functon s vsualzed as a hstogram representaton of 68 bns n each frame. (a) (Condtonal ndependent) RANSAC partcle flterng (b) Condtonal dependent) boosted RANSAC partcle flterng (c) Partcle flterng wth one dffused dynamcs (d) Partcle flterng wth three swtched dynamcs. dynamc model wth large dffuson. The trackng result n Fgure 6 (c) demonstrates ts nablty to represent the mult-modal densty wth 6 partcles. Assumng we know a pror of the three densty modes, we can also desgn a swtched model of three dynamcs wth tght dffusons for each mode [22]. In Fgure 6 (d), the densty functons converge to have three modes after about 4 frames. However, the results are no better than boosted RANSAC-PF, whch has no pror knowledge. 4 Samplng Effcency and Trackng Evaluaton 4.1 Samplng Effcency The key of the success of partcle flterng s to adaptvely concentrate partcles n regons of hgh posteror/lkelhood probablty by resamplng [2], whle stll guaranteeng far samplng of the space. In Fgure 7, we show the partcles lkelhood values (weghts n factored samplng [13]) extracted from a face vdeo sequence. For comparson, red crcles represent partcles drven by RANSAC-PF and blue stars are partcles propagated through a second order Markov dynamcs. In Fgure 7 (a), there are a few hgh weght blue stars appearng n the cloud of red dots. As tme passes, both red and blue partcles ntally decrease ther weghts n (b), then stablze ther weghts at a reasonable level. (c) depcts a hard-to-track frame wth very poor object appearance 2 resultng n even lower weghts. However, the clear recovery s found n (d) where the partcles lkelhood values return to the same level as (b). The lkelhood dstrbuton of blue partcles s normally a very few hgh stars wth mostly low ones, whle the red partcles mantaned by RANSAC-PF have an opposte dstrbuton. 2 The subject s face s turnng down deeply, so the face regon s small and hghly tlted. It causes dffcultes for any face tracker. 6

8 Lkelhood Value Lkelhood Value 4 3 Lkelhood Value 15 Lkelhood Value Partcle Number (a) Frame 3 Partcle Number (b) Frame 15 Partcle Number (c) Frame 36 Partcle Number (d) Frame 425 Fgure 7: The lkelhood dstrbuton of partcles n a vdeo sequence. There are 1 (blue star) dynamcs drven partcles and 1 (red crcle) RANSAC-PF guded partcles).) 4.2 An Entropy-Based Crteron for Trackng Whle the mean of weghted partcles can serve as a representatve or estmate of the set of partcles, t s the entre set that does the trackng. To evaluate how well t s dong, we ntroduce an entropy based crteron. The entropy of a probablty dstrbuton p(x) s defned as H = p(x) log 2 p(x)dx (5) x Snce we are trackng only one object confguraton, a sngle sharp peak of the posteror dstrbuton s deal, whle a broad peak probably means poor or lost trackng. Entropy can be used as a scale to dscrmnate between these two condtons. Low entropy means less trackng uncertanty, thus better performance. Nevertheless, the weghted partcles {x,ω } N =1 are only a set of samples from a probablty dstrbuton p(x), not the dstrbuton tself. There are a number of ways to estmate the entropy of underlyng dstrbuton. The smplest method s to compute the entropy drectly from weghts of dscrete samples: N N H = ω log 2 p(x )= ω log 2 ω (6) =1 =1 H converges to H when N approaches nfnty, but they may have a sgnfcant dfference when N s small. An alternatve n ths case s to nclude a wndow functon to spread the support of a partcle lke a kernel. We have performed numercal evaluatons that suggest there s no sgnfcant dfference between these two methods wth 5 2 partcles. Whle the entropy tself s a good ndcator, we sometmes need to better dscrmnate between unmodal and mult-modal dstrbutons. To do so, we artfcally merge any par of partcles nto one super-partcle provded they are near enough n state space. In ths way, we further lower the outputs of entropy-estmate functons for sngle mode dstrbutons; thus promotng them. 5 Experments on 3-D face trackng The dagram of our face trackng system s shown n Fgure 8 (a). We use a generc trangle based face model, whch s hghly parameterzed and can be easly manpulated wth geometrc modelng software. Dfferent from [3], the approxmate 3D face models are suffcent to acheve the reasonable good trackng results n our experments. When ntatng trackng, we regster the generc 3D model to the frst vdeo frame by manually pckng 6 fducal (mouth and eye) corners n the face mage. A two-vew geometrc estmate [6, 33] s then computed for the face pose on the next frame, followed by a Gaussan dffuson. Consequently, x (1) and x (2) are obtaned as the state vectors that encode the face pose (three components for rotaton and three components for translaton). Trackng n subsequent frames proceeds as descrbed below. 5.1 Feature Detecton and Random Projecton In our face trackng applcaton, we frst detect Harrs-lke [1] mage corner features n two frames. Then, a cross correlaton process for feature matchng and a rough feature clusterng algorthm based on eppolar geometry are performed to form an ntal set of correspondng feature pars. To compute a relatve pose change, we spatally and unformly sample 9 matched mage features between two frames by RANSAC, llustrated n Fgure 9 (c). We now, m (t) j )}, where are a par of (possble non-perfectly) obtan a set of feature matches {(m (t 1) j m (t 1) j and m (t) j matched ponts n two successve mages. For each pont n the reference mage, we cast a 3D ray from the camera center through that pont, and compute the ntersecton Z j of that ray wth the face mesh model, usng a resampled pose state x (t 1) at frame (t 1). The relatve ( ) ˆR pose ˆT = ˆt T can then be computed accordng to 1 m t 1 j 7

9 Image Frame: t-1 Image Frame: t Face Pose PDF at Image Frame: t-1 Select 9 pars of mage feature by RANSAC Select 9 pars of mage feature by RANSAC Face Pose PDF at Image Frame: t P( X ) t -1 Partcle Importance Resamplng X t-1 t-1 x 3D Face Model 3D Facal Features Partcle Geometrcal Projecton t x t P( X ) t X Partcle Flterng Fgure 8: Dagram of RANSAC-PF as appled to 3D face pose trackng. (b) The graphcal representaton of RANSAC-PF where the new state X t s a functon of new observaton Z t, former observaton Z t 1 and state X t 1. (a) (b) (c) (d) Fgure 9: The ntalzaton process of face trackng. (a) The ntal frame s manually algned wth a 3D face model usng 6 fducal corners. (b) The next frame s tracked through the two-vew moton estmaton. The RANSAC-PF trackng begns from the thrd frame. We show the Maxmum A Posteror (MAP) result wth a red color reprojected face mesh overlad on the mages, whle the mean of weghted partcles (MWP) wth a black color. (c)map and MWP are dfferent at the begnnng frames of the RANSAC-PF trackng. (d) MAP and MWP converge together quckly. the followng equaton AP ˆT Zj = λ m t j (7) where Z j =(Z T j, 1)T and m j =(m T j, 1)T. The ntrnsc matrx A, the standard projecton matrx P, Z j and m t j are known. Each of the above equatons gves two constrants on ˆT. We compute ˆT wth a lnear least-squares technque 3 descrbed n [6]. A par of ( ˆR, ˆt ) corresponds to a certan partcle as X (t). Therefore, ths lnear geometrc projecton behaves as a brdge between the propagaton of state partcles from X (t 1) to X (t), =1,...,N p on frame t. We call ths process random projecton (RP). An llustraton of 3 We use 9 as the number of mage features for the random projecton n our algorthm. In theory, 3 s the mnmal possble number to compute the 3D object pose. By consderng the sub-pxel matchng errors, too few (e, 3) features can not provde stable geometrc estmates normally. On the contrary, too many features lose the advantage of robustness by random samplng. We emprcally fnd 9 s a good number for the trade-off. More theoretcal and expermental analyss wll be explored for future work. the random projecton procedure s demonstrated n Fgure 1 (a,b,c). 5.2 Dynamcs and Image lkelhood Image observatons are modelled as a Gaussan process. Wth each x (t) and ts former state hstory x (t 1), we can project the poston of mage pont features at mage (t 1) to mage (t). The reprojecton errors are the 2D Eucldean dstances d 2 m between mage features (u (t) m,v m (t) ) at frame t and reprojected mage features (ũ (t) m, ṽ m (t) ). d 2 m =(u (t) m ũ m (t) ) 2 +(v m (t) ṽ m (t) ) 2 (8) Then the condtonal probablty for lkelhood s p(z x) 1 e d 2 m 2σ 2 (9) 2πσ where the standard dervaton σ can be estmated from the set of all feature reprojecton dstances {d m } for each par m 8

10 (a) (b) (c) (d) Fgure 1: (a) A partcle of head pose s overlad n current frame. (b) Some facal mage feature correspondences between the current and next frame are establshed. (c) The partcle s projected nto the next frame va random selected mage features. (d) Image features are selected spatally and unformly va RANSAC. of x (t 1) and x (t). In the experments, we set σ to 2.5 pxels for smplcty. No apparent mprovement was found when estmatng σ from data. 5.3 Trackng Results We use a smple constant velocty model to gude the temporal evoluton of partcle flterng. In Fgure 9, we show a short face trackng vdeo (Comparson.av) 4 wth large out-of-plane rotatons. In ths case, a subject s face s consdered as a rgd object wthout facal expressons. From ths fgure, reasonable trackng accuracy 5 s acheved, although the generc face model s not very accurate for the gven subject and feature msmatchng does exst. After few frames, the estmates of MAP and MWP estmates converge together. For the convenence of comparson, we generalze our RANSAC-PF algorthm wth partcles gudng by a second order Markov (constant velocty) dynamcs n parallel (see the dashed lne n Fgure 8). The results on the above short sequence (Comparson.av) s shown n Fgure 11. We name the partcles drven by RANSAC-PF RP (random projecton) partcles, and the others as DP (dynamc propagaton) partcles. Note that the constant velocty dynamcs can be consdered as a reasonable assumpton for ths smple yawng vdeo sequence. Nevertheless, the trackng n Fgure 11 (c) s quckly lost due to the relatvely small number of partcles accordng to the 6-DOFs requred by 3D tasks. On the other hand, our algorthm performs better wth the same or smaller number of total partcles. From Fgure 11 (a) and (Comparson.av), good trackng results are obtaned wth 1 RP partcles and 1 DP ones. When reducng the RP partcles to 1 n Fgure 11 (d), slght trackng accuracy s lost for MWP and the MAP results become to flcker around MWP estmates. It means that the computed 4 All the vdeo can be accessed from lelu/ransacpf.htm 5 Snce we do not have the ground truth for trackng, no explct numercal comparson s provded. The valdty s shown by overlayng the 3D face mesh model to mages. MWP s stable and MWP s not. Here 1 can be thought of as a lower bound for the number of RP partcles. The decrease of DP partcles (comparng (b) to (a) n fgure 11) does not apparently nfluence trackng qualty. We also tested our algorthms on trackng people faces from dfferent races. Two vdeo sequences (cher.av, donald.av) are lnked n author s webste. (Cher.av) has moderate expresson changes and results n better trackng, compared to (donald.av), where ntensve expressonal deformatons occur. Both of the vdeos (tracked wth 8 partcles) have long rotaton ranges over 2 3 seconds, and subjects move ther face arbtrarly. Automatc recoveres from poorly tracked frames can also be found. To test the robustness to msalgnments, we manually algn the frst frame n the trackng sequence wth some moderate errors. Our algorthm shows the remarkable stablty from Fgure 12. The ntal regstraton errors do not ncrease wth tme, and a sgnfcant accumulaton of trackng errors s not observed. A general 3D face model s used for trackng though partcular adjustments of face model to a subject may mprove the trackng. For quanttatve evaluaton of the trackng results, we capture 2 vdeo sequences of a subject movng hs head wth moton capture devce. Sx optcal markers n the subject s helmet are tracked durng head movng (for nstance, n Fgure. 1 (a,b,c)), and the subject s head poses are then computed as the ground truth. As shown n fgure 13, our method can keep track of the subject s head poses over the large ranges. The average trackng errors of yaw, ptch and roll are o, o and o n the frst vdeo sequence (rgd face motons only) and o, o and o n the second vdeo sequence (moderate facal deformatons). The relatve moton between successve frames s not requred to be very smooth n our experments. We have concluded that random projecton s most successful when handlng rotatons of o to 5 o degrees. One way to test ths robustness s to smply leave frames out. In experments, mages used for trackng can be sub-sampled every 3 to 1 9

11 (a) (b) (c) (d) Fgure 11: The trackng result comparson of RANSAC-PF under dfferent confguratons. (a) 1 RP partcles and 1 DP partcles (b) 1 RP partcles and 1 DP partcles (c) 2 DP partcles (d) 1 RP partcles and 1 DP partcles (a) (b) (c) (d) (e) (f) (g) (h) Fgure 12: Robustness testng for two msalgned vdeo sequences. (a) The ntal frame wth the vsble algnment error of subject 1 (b) Frame 195 (c) Frame 628 (d) Frame 948 (e) The ntal frame wth the vsble algnment error of subject 2 (f) Frame 185 (g) Frame 255 (h) Frame 285. frames and we do not observe the evdent degradng trackng results. In our experments, the entropy curve s very stable most of the tme as expected, ndcatng the stable trackng performance. For the extreme cases (Fgure 7 (c)), the entropy value does ncrease. In Fgure 14, we show three entropy curves computed from equaton 6 based on the dscrete dstrbuton representaton (wth a varaton of mergng the close partcles nto a super partcle also) and the contnuous dstrbuton approxmaton by Parzon wndow [5]. The testng mages are from cher.av. 6 greatly mproves the effcency of the samples used by the algorthm. Ths algorthm presents several drectons for further research. Most mportantly, we hope to offer a convergence proof for ths algorthm along the lnes suggested n [4, 2]. In partcular, we hope to show that some form of boosted RANSAC s formally convergent. The latter shows great promse at allevatng some of the problems dentfed n [12] when trackng mult-modal denstes. Further mprovng the boostng method s another drecton that offers great promse. We also ntend to mprove and extend our work to multface trackng. Our local feature matchng algorthm s expected to dstngush features from dfferent faces by appearance and spatal neghborhood constrants. Ths step can help RANSAC generate proposals from each person s matched feature set respectvely, whle boostng wll ensure the multple modes are mantaned. Fnally, fndng sutable methods to compute mportance Summary and Future Work In ths paper, we have presented a stochastc technque for full 3D face trackng wth a small number of partcles and a weak dynamcal model. The man feature of ths algorthm s a RANSAC-based mage feature selecton whch 1

12 Comparson of ground truth and trackng result of Yaw Ground truth Trackng result 3 Comparson of ground truth and trackng result of Ptch Ground truth Trackng result 6 Comparson of ground truth and trackng result of Roll Ground truth Trackng result Rotaton angle (degree) 2 Rotaton angle (degree) 1 2 Rotaton angle (degree) Image frame number Image frame number Image frame number (a) (b) (c) Comparson of ground truth and trackng result of Yaw Ground truth Trackng result Comparson of ground truth and trackng result of Ptch Ground truth Trackng result 5 4 Comparson of ground truth and trackng result of Roll Ground truth Trackng result Rotaton angle (degree) 2 Rotaton angle (degree) 5 1 Rotaton angle (degree) Image frame number Image frame number Image frame number (d) (e) (f) Fgure 13: Comparson of the tracked rotatons and the ground truth. The frst row s a vdeo sequence contanng the rgd face motons only; the second row s a vdeo sequence contanng some moderate facal deformatons. (a) Yaw (b) Ptch (c) Roll (d) Yaw (e) Ptch 2 (f) Roll. proposal probabltes for Monte Carlo-style algorthms s another area of future work. References [1] J. Xao, S. Baker, I. Matthews, and T. Kanade, Real-Tme Combned 2D+3D Actve Appearance Models, IEEE Conf. on Computer Vson and Pattern Recognton, June, 24. [2] D. Crsan and A. Doucet, A Survey of Convergence Results on Partcle Flterng for Practtoners, IEEE Trans. Sgnal Processng, vol. 5, no. 3, pp , 22. [3] J. Deutscher, A. Blake and I. Red, Artculated Body Moton Capture by Annealed Partcle Flterng. CVPR. [4] D. Crsan and A. Doucet, Convergence of Sequental Monte Carlo Methods. CUED/F-INFENG/TR381, 2. [5] R. Duda, P. Hart and D. Stork, Pattern Classfcaton (2nd), Wley Interscence, 22. [6] O. Faugeras, Three-Dmensonal Computer Vson: a Geometrc Vewpont, MIT Press, [7] D. Fox, KLD-Samplng: Adaptve Partcle Flters. NIPS 1. [8] M.A. Fschler and R.C. Bolles, Random Sample Consensus: A Paradgm for Model Fttng wth Applcaton to Image Analyss and Automated Cartography. Commun. Assoc. Comp. Mach., vol. 24:381-95, [9] G. Hager and P. Belhumeur, Effcent Regon Trackng Wth Parametrc Models of Geometry and Illumnaton. IEEE Trans. PAMI, 2:1, [1] C. Harrs and M. Stephens, A combned corner and edge detector. 4th Alvey Vson Conf., pp , [11] T. Haste and R. Tbshran, Dscrmnant Analyss by Gaussan Mxtures. Journal of Royal Statstcal Socety Seres B, 58(1): [12] O. Kng and D. Forsyth, How does CONDENSATION behave wth a fnte number of samples? ECCV, pp [13] M. Isard and A. Blake, CONDENSATION condtonal densty propagaton for vsual trackng. IJCV 29:1, pp. 5-28, [14] M. Isard and A. Blake, ICONDENSATION: Unfyng lowlevel and hgh-level trackng n a stochastc framework. ECCV 98, pp [15] M. La Casca, S. Sclaroff, and V. Athtsos, Fast, Relable Head Trackng under Varyng Illumnaton: An Approach Based on Robust Regstraton of Texture-Mapped 3D Models. IEEE Trans. PAMI, 22:4, Aprl, 2. [16] J.S. Lu and R. Chen. Sequental Monte Carlo for Dynamc System. Journal of the Amercan Statstcal Assocaton 93, pp [17] J.S. Lu, R. Chen, and T. Logvnenko. A Theoretcal Framework for Sequental Importance Samplng and Resamplng. 11

13 7 Entropes wthn the Trackng Procedure Entropy Value Entropy wth Mergng Entropy wth Dscrete Dstrbuton Entropy wth Parzon Wndow Frame Number Fgure 14: The entropy curves based on mergng, dscrete dstrbuton and Parzon wndows [5] durng trackng. Sequental Monte Carlo n Practce, A. Doucet, N. defretas, and N. Gordon, Eds., New York: Sprnger-Verlag, 2. [18] H. Moon, R. Chellappa, and A. Rosenfeld, 3D Object Trackng usng Shape-Encoded Partcle Propagaton. ICCV 1. [19] L. Morency, A. Rahm and T. Darrell, Adaptve Vew- Based Appearance Models. CVPR 3. [2] B. North, A. Blake, M. Isard, and J. Rttscher, Learnng and classfcaton of complex dynamcs. IEEE Trans. PAMI, 22:9, pp , Sep., 2. [21] K. Okuma, A. Taleghan, N. De Fretas, J. Lttle, D. Lowe, A boosted partcle flter: multtarget detecton and trackng, European Conference on Computer Vson, May 24. [22] V. Pavlovc, J. Rehg and J. MacCormck, Ttle Learnng Swtchng Lnear Models of Human Moton, In Neural Informaton Processng Systems, 2. [23] V. Pavlovc4, Model-based moton clusterng usng boosted mxture modelng, CVPR, 24. [24] J. Sullvan and J. Rttscher, Gudng random partcles by determnstc search. ICCV, I:323-33, 21. [25] P.H. Torr and A. Zsserman, MLESAC: A New Robust Estmator wth Applcaton to Estmatng Image Geometry. CVIU, 78: , 2. [26] P.H. Torr and C. Davdson, IMPSAC: Synthess of Importance Samplng and Random Sample Consensus. IEEE Trans. PAMI, 25:3, March, 23. [27] K. Toyama and G. Hager, Incremental focus of attenton for robust vson-based trackng, Internatonal Journal of Computer Vson, 35(1):45-63, Nov [28] K. Toyama and A. Blake, Probablstc Trackng n a Metrx Space. ICCV 1. [29] Z. Tu and S.C. Zhu, Image Segmentaton by Data-Drven Markov Chan Monte Carlo. IEEE Trans. PAMI, 24:5, May 22. [3] L. Vacchett, V. Lepett, P. Fua, Fusng Onlne and Offlne Informaton for Stable 3D Trackng n Real-Tme. CVPR 3. [31] P. Vola, M. Jones, Rapd object detecton usng a boosted cascade of smple features, IEEE Conference on Computer Vson and Pattern Recognton, 21. [32] J. Xao, T. Kanade, and J. Cohn, Robust Full-Moton Recovery of Head by Dynamc Templates and Re-regstraton Technques. FG 2. [33] Z. Zhang, et al., A Robust Technque for Matchng Two Uncalbrated Images Through the Recovery of the Unknown Eppolar Geometry, Artfcal Intellgence J., 78 pp: , Oct

14 * Abstract Partcle flterng s a very popular technque for sequental state estmaton. However, n hgh-dmensonal cases where the state dynamcs are complex or poorly modeled, thousands of partcles are usually requred for real applcatons. Ths paper presents a hybrd samplng soluton that combnes RANSAC and partcle flterng. In ths approach, RANSAC provdes proposal partcles that, wth hgh probablty, represent the observaton lkelhood. Both condtonally ndependent RANSAC samplng and boostng-lke condtonally dependent RANSAC samplng are explored. We show that the use of RANSAC-guded samplng reduces the necessary number of partcles to dozens for a full 3D trackng problem. Ths s method s partcularly advantageous when state dynamcs are poorly modeled. We show emprcally that the samplng effcency (n terms of lkelhood) s much hgher wth the use of RANSAC. The algorthm has been appled to the problem of 3D face pose trackng wth changng expresson. We demonstrate the valdty of our approach wth several vdeo sequences acqured n an unstructured envronment.