Visual Hand Tracking Using Nonparametric Belief Propagation

Transcription

1 MIT LABORATORY FOR INFORMATION & DECISION SYSTEMS TECHNICAL REPORT P-2603, MAY 2004 PRESENTED AT THE 2004 IEEE CVPR WORKSHOP ON GENERATIVE MODEL BASED VISION Vsual Hand Trackng Usng Nonparametrc Belef Propagaton Erk B. Sudderth, Mchael I. Mandel, Wllam T. Freeman, and Alan S. Wllsky Department of Electrcal Engneerng and Computer Scence Massachusetts Insttute of Technology Abstract Ths paper develops probablstc methods for vsual trackng of a three-dmensonal geometrc hand model from monocular mage sequences. We consder a redundant representaton n whch each model component s descrbed by ts poston and orentaton n the world coordnate frame. A pror model s then defned whch enforces the knematc constrants mpled by the model s jonts. We show that ths pror has a local structure, and s n fact a parwse Markov random feld. Furthermore, our redundant representaton allows color and edge-based lkelhood measures, such as the Chamfer dstance, to be smlarly decomposed n cases where there s no self occluson. Gven ths graphcal model of hand knematcs, we may track the hand s moton usng the recently proposed nonparametrc belef propagaton (NBP) algorthm. Lke partcle flters, NBP approxmates the posteror dstrbuton over hand confguratons as a collecton of samples. However, NBP uses the graphcal structure to greatly reduce the dmensonalty of these dstrbutons, provdng mproved robustness. Several methods are used to mprove NBP s computatonal effcency, ncludng a novel KD-tree based method for fast Chamfer dstance evaluaton. We provde smulatons showng that NBP may be used to refne naccurate model ntalzatons, as well as track hand moton through extended mage sequences. I. INTRODUCTION Accurate vsual detecton and trackng of three dmensonal artculated objects s a challengng problem wth applcatons n human computer nterfaces, moton capture, and scene understandng [25]. In ths paper, we develop a probablstc method for trackng a geometrc hand model from monocular mage sequences. Because artculated hand models have many (roughly 26) degrees of freedom, whch are only ndrectly related to the observed mages, exact representaton of the posteror dstrbuton over model confguratons s ntractable. Extended and unscented Kalman flters [12, 14, 19] approxmate the posteror by a sngle Gaussan, and update these approxmatons va a lnearzaton of the measurement process. However, because many dfferent hand confguratons may approxmately match a gven mage, the true posteror s often multmodal, makng lnear approxmatons neffectve. Gven the ambgutes nherent n vsual trackng problems, many authors have consdered nonparametrc densty representatons. For example, partcle flters [6] approxmate the posteror dstrbuton by a set of representatve elements, and use Monte Carlo mportance samplng rules to update these partcles. However, due to the large number of degrees of freedom n hand trackng problems, partcle flters cannot hope to accurately represent the true posteror. Instead, partcles tend to concentrate n only a few of the most sgnfcant modes, and the tracker can suffer catastrophc falures. Ths problem has motvated prevous authors to consder smplfed models whch only allow a lmted range of object motons [11], as well as sophstcated pror models whch better predct the object s dynamcs [16, 26]. Determnstc nonparametrc approxmatons are also possble, as demonstrated by a recently proposed tree based estmator [20] whch defnes a multscale dscretzaton of the state space. Ths approach acheves computatonal savngs by approxmatng mage lkelhoods as pecewse constant at coarse scales of the dscretzaton, and only recursvely evaluatng those cells who s probablty s above a predefned threshold. However, ths approach s only effectve when the tree structure s constructed usng pror nformaton whch strongly constrans the hand s confguraton. If the pror s unnformatve, such prunng rules are very lkely to mss mportant hand confguratons. Gven the dffcultes n approxmatng hgh dmensonal dstrbutons, some authors have proposed replacng trackng by classfcaton [1, 15], where classes correspond to some dscretzaton of allowable hand confguratons. These methods are most approprate n applcatons such as sgn language recognton, where only a small set of poses s of prmary nterest. Also, as these methods are based on precomputed mages of the hand from all possble confguratons, they requre large amounts of storage space. A recently proposed method for nterpolatng between classes [22] makes no use of the mage data durng the nterpolaton, and thus assumes that the transton between any par of hand pose classes s hghly predctable. An alternatve way to address the hgh dmensonalty of artculated trackng problems s to dentfy statstcal structure wthn the posteror dstrbuton. Ths structure can be descrbed usng a graphcal model, or Bayesan network. Graphcal models have been used to track vew based human body representatons [13], contour models of restrcted hand confguratons [4], vew based 2.5D cardboard models of hands and people [24], and a full 3D knematc human body model [17]. Because the varables n these graphcal models are contnuous, and dscretzaton s ntractable for three dmensonal models, most tradtonal graphcal nference algorthms are napplcable. Instead, these trackers are based on recently proposed extensons of partcle flters to general graphs: mean feld Monte Carlo n [24], and nonparametrc belef propagaton (NBP) [8, 21] n [17]. In ths paper, we show that NBP may be used to track a three dmensonal geometrc model of the hand. To derve 1

2 Fg. 1. Projected edges (top row) and slhouettes (bottom row) for two confguratons (left and rght blocks) of the 3D structural hand model. To ad vsualzaton, the model jont angles are set to match the mages (left), and then also projected followng rotatons by 35 (center) and 70 (rght) about the vertcal axs. a graphcal model for the trackng problem, we consder a redundant local representaton n whch each hand component s descrbed by ts own three dmensonal poston and orentaton. We show that the model s knematc constrants, ncludng self ntersecton constrants not captured by jont angle representatons, take a smple form n ths local representaton. Furthermore, n cases where model components do not sgnfcantly occlude each other, standard edge and color based lkelhood measures may be smlarly decomposed. We descrbe the mplementaton of NBP on ths model, as well as several methods for mprovng computatonal effcency. These nclude a novel method for fast orentaton based Chamfer dstance evaluaton usng KD trees [2]. We conclude wth smulatons demonstratng that NBP can refne nosy ntalzatons n sngle frames, as well as track hand moton over two extended sequences. A. Structural Model II. GEOMETRIC HAND MODELING Structurally, the hand s composed of sxteen approxmately rgd components: three phalanges or lnks for each fnger and thumb, as well as the palm [25]. As proposed by [14, 19], we model each rgd body by one or more truncated quadrcs (ellpsods, cones, and cylnders). These geometrc prmtves are well matched to the true geometry of the hand, and n contrast to 2.5 dmensonal cardboard models [24, 26], allow trackng from arbtrary vewng orentatons. In addton, because the perspectve projecton of a quadrc surface s a conc, one can effcently determne the mage ponts lyng on the boundary or slhouette of the projecton of any three dmensonal model confguraton [3, 19]. Fgure 1 shows the edges and slhouettes correspondng to two dfferent confguratons of the hand model, each of whch s seen from three dfferent vewponts. Because our model s desgned for estmaton, not vsualzaton, precse modelng of all parts of the hand s unnecessary. As our trackng results demonstrate, t s suffcent to capture the coarse structural features whch are most relevant to the observaton model descrbed n Sec. III. Note also that we do not consder model self occluson when fndng edges. See Sec. III-A for further dscusson of ths approxmaton. B. Knematc Model The knematc constrants between dfferent hand model components are well descrbed by revolute jonts [25]. Fgure 2(a) shows a graph descrbng ths knematc structure, n whch nodes correspond to rgd bodes and edges to jonts. The two jonts connectng the phalanges of each fnger and thumb have a sngle rotatonal degree of freedom, whle the jonts connectng the base of each fnger to the palm have two degrees of freedom (correspondng to graspng and spreadng motons). Thus, twenty jont angles are requred to descrbe the relatve postons of all hand parts. The full confguraton of the hand s descrbed by these angles along wth the palm s global poston and orentaton, gvng a total of 26 degrees of freedom. Gven mage measurements, calculaton of a model confguraton s lkelhood generally requres the global poston and orentaton of each component. Ths forward knematcs problem s easly solved va a seres of transformatons derved from the poston and orentaton of each jont axs, along wth the correspondng jont angles (see, for example, [12] for detals). C. Redundant Local State Representaton Most model based hand trackers parameterze the model state n terms of the twenty jont angles descrbed above, along wth the palm s global poston and orentaton. In ths paper, we nstead explore a redundant representaton n whch the th rgd body s descrbed by ts poston q and orentaton r (a unt quaternon). Let x = (q, r ) denote ths local descrpton of each hand component s confguraton, and x = {x 1,..., x 16 } the confguraton of the entre hand. Clearly, there are dependences among the elements of x mpled by the knematc constrants. Let E K be the set of all 2

3 (a) Fg. 2. Graphs descrbng the hand model s physcal constrants, where nodes correspond to dfferent hand components. (a) Knematc constrants correspondng to the revolute jonts between neghborng components. (b) Structural constrants whch prevent the ntersecton of hand components n three dmensonal space. pars of rgd bodes whch are connected by jonts, or equvalently the edges n the knematc graph of Fg. 2(a). For each jont (, j) E K, defne an ndcator functon ψ,j K (x, x j ) whch s equal to one f the par (x, x j ) are vald rgd body confguratons assocated wth some settng of the angles of jont (, j), and zero otherwse. Vewng the component confguratons x as random varables to be estmated, the followng pror model explctly enforces all of the constrants mpled by the orgnal jont angle representaton: p K (x) ψ,j K (x, x j ) (1) (,j) E K The structure of eq. (1) shows that p K (x) s a graphcal model (n partcular, a parwse Markov random feld). The graph descrbng the knematc structure (Fg. 2(a)) s the same as the graph descrbng the Markov structure of p K (x). Intutvely, ths graph expresses the fact that condtoned on the confguraton of the palm, the poston and orentaton of each fnger s descrbed by an ndependent set of jont angles, and s thus statstcally ndependent. At frst glance, the local representaton descrbed n ths secton may seem unattractve: the state dmenson has ncreased from 26 to 96, and nference algorthms must now explctly deal wth the pror constrants descrbed by p K (x). However, as we show n the followng sectons, local encodng of the model state greatly smplfes many other aspects of the trackng problem. D. Structural Constrants In realty, the jont angles descrbng hand confguraton are not ndependent because dfferent fngers can never occupy the same physcal volume. The constrants that ths places on jont angles are a complex functon of the hand s geometry, and are dffcult to express compactly. However, n the local representaton of the prevous secton, these structural constrants take a smple form: the poston and orentaton of (b) t t+1 Fg. 3. Graphcal model of the dynamcs relatng two consecutve tme steps. For clarty, edges correspondng to structural potentals are not shown. every par of rgd bodes must be such that ther component quadrc surfaces do not ntersect. For computatonal effcency, our trackng algorthm approxmates ths deal constrant n two ways. Frst, we only explctly constran those pars of rgd bodes whch are most lkely to ntersect, correspondng to the edges E S of the graph n Fg. 2(b). Furthermore, because the relatve orentaton of each fnger s quadrcs s mplctly constraned by the knematc pror p K (x), we may detect most ntersectons based on the dstance between object centrods: { ψ,j S 1 q q (x, x j ) = j > δ,j (2) 0 otherwse Here, δ,j s a threshold determned from the rad of the cones or cylnders defnng rgd bodes and j. As for the knematc constrants, we defne a pror model whch ensures that the structural constrants are not volated: p S (x) ψ,j S (x, x j ) (3) (,j) E S We have found ths constrant to be mportant n our smulatons to prevent dfferent fngers from attemptng to track the same mage data. E. Temporal Constrants Thus far, our dscusson has focused on the hand constrants present at a sngle pont n tme. In order to track hand moton, we must have some model of the hand s dynamcs. Let x t, denote the poston and orentaton of the th hand component at tme t, and x t = {x t,1,..., x t,16 }. For each component at tme t, our dynamcal model adds a Gaussan potental connectng t to the correspondng component at the prevous tme step: 16 p T (x t x t 1 ) = N (x t 1, x t, ; 0, Λ ) (4) A graphcal representaton of these potentals s gven n Fg. 3. Although ths temporal model s factorzed, the knematc constrants at the followng tme step mplctly couple the 3

4 (a) Fg. 4. Image evdence used for trackng. (a) Intensty edges detected by a thresholded gradent operator. (b) Lkelhood ratos at each pxel for a color based skn detector. correspondng random walks. These dynamcs can be justfed as the maxmum entropy model gven observatons of the nodes margnal varances Λ. III. OBSERVATION MODEL Our hand trackng system s based on a set of effcently computed edge and color cues. For notatonal smplcty, we focus on a sngle vdeo frame for the remander of ths secton. (b) A. Edge Matchng Usng the Chamfer Dstance As a hand s moved n front of a camera, t obscures the background scene and thus tends to produce ntensty edges along the boundares of ts projecton n the mage plane (see Fg. 4(a)). Ths edge cue s used by vrtually all model based hand trackng systems [11, 14, 19, 20, 24, 26]. Followng [20], we use the Chamfer dstance to measure dscrepances between projected model edges and mage edges detected by a smple gradent operator. To mprove accuracy, we measure dstance n terms of both edge poston and orentaton. Let Π(x) denote the set of edges n the projecton of three dmensonal model confguraton x, and (y) the output of an edge detector on the mage y. The Chamfer dstance d E (Π(x), (y)) s then gven by d 2 E(Π(x), (y)) = [ ] mn v (y) g2 (u, v) (5) u Π(x) Here, g(u, v) determnes the metrc by whch errors n edge matches are measured. Lettng u = (u p, u θ ) denote the poston u p and orentaton u θ of edge u, we defne ( g 2 up v p 2 ) (u, v) = mn σ 2 + d 2 π(u θ, v θ ), g 0 (6) where d π (u θ, v θ ) measures absolute dfferences n orentaton modulo π, and g 0 adds robustness to edge detecton falures. Fnally, we assocate ths dstance wth a lkelhood functon as follows: p E (y x) exp { λ E d 2 E(Π(x), (y)) } (7) For a dscusson of the generatve model underlyng ths lkelhood functon, see [23]. B. Slhouette Matchng Usng Skn Color Statstcs Skn colored pxels are well known to have predctable statstcs [9], and thus provde a powerful cue for hand trackng. We model the color dstrbuton p skn of skn pxels by a sngle Gaussan n RGB space, wth mean and covarance estmated from hand selected tranng patches. We assume that non skn pxels have a unform color dstrbuton p bkgd. Let Ω(x) denote the set of pxels n the slhouette of projected hand model confguraton x, and Υ the set of all mage pxels. Assumng each pxel s ndependent, the lkelhood of an mage y s p C (y x) = p skn (u) p bkgd (v) u Ω(x) u Ω(x) p skn (u) p bkgd (u) v Υ\Ω(x) The second equaton follows by neglectng the proportonalty constant v Υ p bkgd(v), whch s ndependent of x [4]. Note that we must only evaluate the lkelhood rato over the slhouette regon Ω(x). Fgure 4(b) plots these lkelhood ratos for a sample hand mage. C. Local Decomposton of Lkelhoods Suppose that the hand model s n a three dmensonal confguraton for whch there s no self occluson. In ths case, each hand component wll project to a dsjont subset of the mage pxels, and the Chamfer dstance (eq. (5)) decomposes as 16 d 2 E(Π(x), (y)) = d 2 E(Π(x ), (y)) (9) Ths n turn mples that the edge based lkelhood (eq. (7)) factorzes nto a product of terms whch provde ndependent, local evdence for each component: (8) 16 p E (y x) p E (y x ) (10) Smlarly, the skn color lkelhood (eq. (8)) decomposes as 16 p C (y x) p C (y x ) (11) Note that ths statstcal decomposton does not hold for the orgnal jont angle representaton, and s heavly dependent on our choce of a state representaton n whch the relatonshp between model parameters and mage coordnates s local. In cases where there s self occluson, the local decomposton of eq. (10, 11) wll not hold. Nevertheless, we beleve that ths decomposton wll often provde a good approxmaton. In partcular, because occluson reasonng can only reduce the number of projected model edges, the local decomposton of eq. (10) wll always provde an upper bound on the true edge lkelhood p E (y x). 4

5 D. Fast Lkelhood Computaton Because the nonparametrc belef propagaton algorthm proposed n ths paper must evaluate many dfferent hypotheses for each model component, t s mportant that the evaluatons of our lkelhood functons be computatonally effcent. For the skn color term (eq. (8)), we precompute the cumulatve sum of the log lkelhood ratos along each row of pxels. We may then quckly ntegrate the lkelhood of each hypotheszed slhouette regon, gven only the boundares of that slhouette. For the Chamfer dstance, our ncluson of orentaton nformaton makes t dffcult to use standard dstance transform methods. We nstead use KD trees [2] to explot the geometrc structure underlyng our detected edges. For low dmensonal collectons of ponts, KD trees may be effcently constructed, and then used to fnd nearest neghbors n logarthmc tme. Gven a set of detected edges, we precompute a KD tree representaton of the three dmensonal vectors correspondng to each edge s poston and orentaton. To account for the fact that orentaton dstance must be measured modulo π, we also nclude a second, approprately rotated copy of each pont. Then, for each hypotheszed model confguraton, the mnmzaton step of the Chamfer dstance computaton (eq. (5)) can be performed va effcent nearest neghbor search n the KD tree. Usng KD trees, we acheve very fast Chamfer dstance computaton wthout requrng excess storage or sufferng from dscretzaton artfacts. IV. NONPARAMETRIC BELIEF PROPAGATION A. Graphcal Models and Belef Propagaton In the prevous sectons, we have shown that a redundant, local representaton of the geometrc hand model s confguraton x t allows p (x t y t ), the posteror dstrbuton of the hand model at tme t gven mage observatons y t, to be wrtten as [ 16 ] p(x t y t ) p K (x t )p S (x t ) p E (y t x t, )p C (y t x t, ) (12) where p K (x t ) and p S (x t ) are knematc and structural pror models correspondng to the graphs of Fg. 2. Ths expresson s exact when there s no self occluson, and a potentally useful approxmaton more generally. When T vdeo frames are observed, the overall posteror dstrbuton s gven by T p (x y) p (x t y t ) p T (x t x t 1 ) (13) t=1 Equaton (13) s an example of a parwse Markov random feld, whch can more generally be wrtten as p(x y) ψ (x, y) (14) (,j) E ψ,j (x, x j ) V Here, V s a set of nodes, correspondng to the sxteen components of the hand model at each tme step, and E s a set of edges specfyng ther statstcal dependences. Gven our analyss, hand trackng can be seen as a specal example of nference n a graphcal model. In ths paper, we consder belef propagaton (BP) [27], a method for solvng nference problems va local message passng. At each teraton of the BP algorthm, some node V calculates a message m j (x j ) to be sent to some neghborng node j Γ() {j (, j) E}: m n j (x j ) = α ψ j, (x j, x ) ψ (x, y) x m n 1 k (x ) dx (15) k Γ()\j Here, α denotes an arbtrary proportonalty constant. At any teraton, each node can produce an approxmaton ˆp(x y) to the margnal dstrbuton p (x y) by combnng the ncomng messages wth the local observaton: ˆp n (x y) = αψ (x, y ) m n j (x ) (16) j Γ() For tree structured graphs, the approxmate margnals, or belefs, ˆp n (x y) wll converge to the true margnals p (x y) once the messages from each node have propagated to every other node n the graph. On graphs wth cycles, the margnal dstrbutons estmated by BP are only approxmate, but these approxmatons are often hghly accurate [27]. B. Nonparametrc Representatons For the hand trackng problem, the varables x take on contnuous values. Because accurate dscretzaton of the sx degrees of freedom at each node s ntractable, and the BP message update (eq. (15)) has no closed form for the potentals underlyng hand trackng, exact mplementaton of BP s nfeasble. Instead, we explore nonparametrc, partcle based approxmatons to these messages usng the nonparametrc belef propagaton (NBP) algorthm [21]. In NBP, each message s represented usng ether a sample based densty estmate (a mxture of Gaussans) or an analytc functon. Both types of messages are needed for hand trackng, as we dscuss below. Each NBP message update nvolves two stages: samplng from the estmated margnal, followed by Monte Carlo approxmaton of the outgong message. For the general form of these updates, see [21]. In the followng sectons, we gve a hgh level overvew focusng on the unque features of the hand trackng applcaton. The hand trackng applcaton s complcated by the fact that the orentaton component r of x = (q, r ) s an element of the rotaton group SO(3). Followng [5, 17], we represent orentatons as unt quaternons, and use a lnearzed approxmaton when constructng densty estmates. Any sampled orentatons may be projected back to SO(3) by normalzng the correspondng four dmensonal vector. Ths approxmaton s most approprate for denstes wth tghtly concentrated rotatonal components. C. Margnal Computaton From eq. (16), we see that the BP estmate of the local margnal dstrbuton ˆp(x y) s equal to the product of the 5

6 Gven nput messages m j (x ) from knematc neghbors Γ K(), structural neghbors Γ S(), and temporal neghbors Γ T (): 1) Draw M ndependent samples {x (l) } M l=1 from the product x (l) m j (x ) m k (x ) j Γ T () k Γ K () usng the multscale samplng methods of [7]. 2) For each x (l) = (q (l), r (l) ), normalze the orentaton r (l) 3) Compute an mportance weght for each sample x (l) w (l) p E(y x (l) )p C(y x (l) ) j Γ S () : m j(x (l) ) 4) Use a bandwdth selecton method (see [18]) to construct a kernel densty estmate ˆp(x y) from {x (l), w (l) } M l=1. Alg. 1.. NBP update of the estmated margnal dstrbuton ˆp(x y). ncomng messages from neghborng nodes wth the local observaton potental. Lke partcle flters, NBP uses mportance samplng to approxmate ths product. As we descrbe n the followng secton, our NBP hand tracker employs Gaussan mxtures for some messages (along knematc and temporal edges), and analytc functons for others (structural edges). The mage lkelhood p E (y x )p C (y x ) s an analytc functon whch can be effcently evaluated at any canddate x usng the methods of Sec. III-D. The mportance samplng update of the margnal estmate ˆp(x y) s summarzed n Alg. 1. Frst, M samples {x (l) } M l=1 are drawn drectly from the product of the knematc and temporal Gaussan mxture messages. Note that ths samplng problem s nontrval: gven d mxtures of M Gaussans, ther product s a mxture of M d Gaussans. However, n ths paper we use a recently proposed multscale Gbbs sampler [7] to effcently draw accurate, albet approxmate, samples. Followng normalzaton of the rotatonal component, each sample x (l) s assgned a weght w (l) equal to the product of the color and edge lkelhoods wth any messages along structural edges. Fnally, the computatonally effcent rule of thumb heurstc [18] s used to set the bandwdth of Gaussan smoothng kernels placed around each sample, producng an estmate of the desred margnal dstrbuton. The prevous procedure assumes that at least one of the ncomng messages s a Gaussan mxture. For the hand tracker, ths s true except for the ntal message updates on the frst frame, when the only ncomng message s the local analytc lkelhood functon. For the smulatons presented n ths paper, we ntalzed the tracker by hand specfyng a hgh varance Gaussan proposal dstrbuton centered roughly around the true startng hand confguraton. In the future, we hope to replace ths manual ntalzaton by automatc mage based feature detectors. D. Message Propagaton and Schedulng To derve the message propagaton rule, as suggested by [10] we rewrte the message update equaton (15) n terms of the Gven M weghted samples {x (l), w (l) } M l=1 from ˆp(x y), and the ncomng message m j (x ) used to construct ˆp(x y): 1) Reweght each sample x (l) KINEMATIC EDGES: 2) Draw M samples { x (l) as w (l) w (l) /m j(x (l) ). } M l=1 wth replacement from the dscrete dstrbuton defned by the weghts { w (l) } M l=1. 3) For each x (l), sample unformly from the allowable angles for jont (, j). Determne x (l) j va forward knematcs. 4) Use a bandwdth selecton method to construct a kernel densty estmate m j (x j) from the unweghted samples {x (l) j } M l=1. TEMPORAL EDGES: 2) Construct a kernel densty estmate m j (x j) wth centers {x (l) } M l=1, weghts { w (l) } M l=1, and unform bandwdths Λ. STRUCTURAL EDGES: 2) For any x j = (q j, r j), let L = {l q (l) 3) Calculate m j(x j) = l L w(l). q j > δ,j}. Alg. 2. NBP update of the nonparametrc message m j (x j ) sent from node to node j as n eq. (17), for each of the three potental types. margnal dstrbuton ˆp(x y): m n j (x j ) = α ψ j, (x j, x ) ˆpn 1 (x y) x m n 1 j (x ) dx (17) Our explct use of the current margnal estmate ˆp n 1 (x y) helps focus the Monte Carlo approxmaton on the most mportant regons of the state space. Consder frst the case where (, j) E K, so that ψj, K corresponds to a knematc constrant. The message propagaton step makes drect use of the partcles {x (l) } M l=1 sampled durng the last margnal estmate. We reweght each partcle x (l) by 1/m j (x (l) ), and then resample to get M unweghted partcles { x (l) } M l=1 (see Alg. 2). We must then sample canddate x j confguratons from the condtonal dstrbuton ψj, K (x j, x (l) ). Because ψj, K s an ndcator potental, ths samplng has a partcularly appealng form: frst sample unformly among allowable jont angles, and then use forward knmatcs to fnd the x (l) j correspondng to each x (l). Fnally, the rule of thumb bandwdth selecton method [18] s used to construct the outgong Gaussan mxture message. Because the temporal constrant potentals are Gaussan, the samplng assocated wth knematc message updates s unnecessary. Instead, as suggested by [8], we smply adjust the bandwdths of the current margnal estmate ˆp(x y) to match the temporal covarance Λ (see Alg. 2). Ths update mplctly assumes that the bandwdth of ˆp(x y) s much smaller than Λ, whch wll hold for suffcently large M. For structural constrant edges E S, a dfferent approach s needed. In partcular, from eq. (2) we see that the parwse potental s one for all state confguratons outsde some ball, and therefore the outgong message wll not be fntely ntegrable. For structural edges, messages must then take the form of analytc functons. In prncple, at some pont x j the message m j (x j ) should equal the ntegral of ˆp(x y) /m j (x ) over all confguratons outsde some ball centered at q j. We 6

7 V. SIMULATIONS In ths secton, we examne the emprcal performance of the NBP hand tracker. All results are based on mages (or vdeo sequences) recorded by a calbrated camera. The physcal dmensons of the quadrcs composng the hand model were measured offlne. All messages were represented by M = 200 partcles, and the result fgures show the projectons of the fnal densty estmates fve largest modes. Fg. 5. Schedulng of the knematc constrant message updates for NBP: messages are frst passed from fngertps to the palm, and then back to the fngertps. Structural constrant messages (not shown) are updated as needed. approxmate ths quantty by the sum of the weghts of all kernels n ˆp(x y) outsde that ball (see Alg. 2). For NBP, the message update order effects the outcome of each local Monte Carlo approxmaton, and may thus effect the qualty of the fnal margnal estmates. Gven a sngle frame, we terate the tree based message schedule of Fg. 5, n whch messages are passed from fngertps to the palm, and then back to the fngertps. The structural messages, whch for clarty are not shown, are also updated whenever the source node s belef changes. For vdeo, we process the frames n sequence, updatng the temporal messages to the next frame followng a fxed number of knematc message sweeps. However, the tracker could be easly extended to ncorporate nformaton from future vdeo frames usng reverse tme messages. E. Related Work The NBP algorthm has also recently been used to develop a three dmensonal person tracker [17]. However, ths person tracker uses a loose lmbed formulaton of the knematc constrants whch dffers sgnfcantly from our hand tracker. In partcular, the loose lmbed tracker represents the condtonal dstrbuton of each lmb s locaton gven ts neghbor va a Gaussan mxture estmated from tranng data. For each jont, the two needed condtonal denstes (for example, upper arm gven lower arm and lower arm gven upper arm) are learned ndependently. In general, however, there may be no parwse clque potental whch s consstent wth these condtonals. Thus, there may be no globally consstent generatve model underlyng ther results, makng the standard theoretcal justfcatons of belef propagaton napplcable. The two dmensonal trackng results of [8, 24] are also based on explct (and sometmes nconsstent) relaxatons of the true knematc constrants. In contrast, we have shown that an NBP tracker may be bult around the local structure of the true knematc constrants. Conceptually, ths has the advantage of provdng a clearly specfed, globally consstent generatve model whose propertes can be analyzed. Practcally, our formulaton avods the need to explctly approxmate the knematc constrants, and allows us to buld a functonal tracker wthout the need for tranng data. A. Refnement of Coarse Intalzatons Gven a sngle mage, NBP may be used to progressvely refne a coarse, user suppled ntalzaton nto an accurate estmaton of the hand s confguraton. See Fg. 6 for two examples of such a refnement. In the second example, note that the ntal fnger postons are not only msalgned, but the user has suppled no nformaton about the graspng confguraton of the hand. By the fourth NBP teraton, however, the system has algned all of the jonts properly. In both mages, a poorly algned palm s eventually attracted to the proper locaton by well ft fngers. For these examples, each NBP teraton (a complete update of all messages n the graph) requres about 1 mnute on a Pentum IV workstaton. B. Temporal Trackng Two vdeo sequences demonstratng the NBP hand tracker are avalable at Total computaton tme for each vdeo sequence, ncludng all lkelhood calculatons, s approxmately 4 mnutes per frame. The frst shows the hand rgdly movng n three dmensonal space. The extrema of ths moton are shown n Fg. 7. The NBP estmates closely track the hand throughout the sequence, but are nosest when the fngers pont towards the camera because the sharp projecton angle reduces the amount of mage evdence. Note, however, that the estmates quckly lock back onto the true hand confguraton when the hand rotates away from the camera. The second vdeo sequence exercses the hand model s jonts, contanng both ndvdual fnger motons and combned graspng motons (see Fg. 8). Our model supports all of these degrees of freedom, and mantans accurate estmates even when the rng fnger s partally occluded by the mddle fnger (bottom row of Fg. 8). Ths robustness to moderate occlusons comes from our use of structural potentals to prevent self ntersecton, and s only relable when the hand s moton s well predcted by the dynamcal model. VI. DISCUSSION We have demonstrated that the geometrc models commonly used for hand trackng naturally have a graphcal structure, and exploted ths fact to buld an effectve hand trackng algorthm usng nonparametrc belef propagaton. We are currently nvestgatng more challengng test sequences, as well as a rgorous comparson of our algorthm to exstng methods. Prelmnary results ndcate that accurate trackng through sgnfcant self occluson wll requre a more sophstcated local lkelhood approxmaton, as well as rcher 7

8 Fg. 6. Two examples of refnement of a coarse hand model ntalzaton usng NBP. We show results followng 1, 2, and 4 teratons of the message schedule n Fg. 5. Plots show the projectons of the most sgnfcant margnal modes. Fg. 7. Four frames showng extrema of a hand s rgd moton, and the poston estmates produced by the NBP tracker. dynamcal models. In addton, we hope to use local hand feature detectors to mprove our method s robustness. ACKNOWLEDGMENTS The authors thank C. Maro Chrstoudas and Mchael Sracusa for ther help wth vdeo data collecton. We would also lke to thank Mchael Black, Alexander Ihler, Mchael Isard, and Leond Sgal for helpful conversatons. Ths research was supported n part by AFOSR Grant F REFERENCES [1] V. Athtsos and S. Sclaroff, Estmatng 3D hand pose from a cluttered mage, n CVPR, vol. 2, 2003, pp [2] J. L. Bentley, Multdmensonal bnary search trees used for assocatve searchng, Comm. ACM, vol. 18, no. 9, pp , Sept [3] J. Blnn, The algebrac propertes of second order surfaces, n Introducton to Implct Surfaces, J. Bloomenthal, Ed. Morgan Kaufmann, 1997, pp [4] J. M. Coughlan and S. J. Ferrera, Fndng deformable shapes usng loopy belef propagaton, n ECCV, vol. 3, 2002, pp [5] J. Deutscher, M. Isard, and J. MacCormck, Automatc camera calbraton from a sngle Manhattan mage, n ECCV, vol. 4, 2002, pp [6] A. Doucet, N. de Fretas, and N. Gordon, Eds., Sequental Monte Carlo Methods n Practce. New York: Sprnger-Verlag, [7] A. T. Ihler, E. B. Sudderth, W. T. Freeman, and A. S. Wllsky, Effcent multscale samplng from products of Gaussan mxtures, n NIPS, [8] M. Isard, PAMPAS: Real valued graphcal models for computer vson, n CVPR, vol. 1, 2003, pp [9] M. J. Jones and J. M. Rehg, Statstcal color models wth applcaton to skn detecton, IJCV, vol. 46, no. 1, pp , [10] D. Koller, U. Lerner, and D. Angelov, A general algorthm for approxmate nference and ts applcaton to hybrd Bayes nets, n UAI 15, 1999, pp [11] J. MacCormck and M. Isard, Parttoned samplng, artculated objects, and nterface qualty hand trackng, n ECCV, vol. 2, 2000, pp [12] I. Mkc, M. Trved, E. Hunter, and P. Cosman, Human body model acquston and trackng usng voxel data, IJCV, vol. 53, no. 3, pp , [13] D. Ramanan and D. A. Forsyth, Fndng and trackng people from the bottom up, n CVPR, vol. 2, 2003, pp [14] J. M. Rehg and T. Kanade, DgtEyes: Vson based hand trackng for human computer nteracton, n Proc. IEEE Workshop on Non Rgd and Artculated Objects, [15] G. Shakhnarovch, P. Vola, and T. Darrell, Fast pose estmaton wth parameter senstve hashng, n ICCV, 2003, pp [16] H. Sdenbladh, M. J. Black, and L. Sgal, Implct probablstc models of human moton for synthess and trackng, n ECCV, vol. 1, 2002, pp [17] L. Sgal, M. Isard, B. H. Sgelman, and M. J. Black, Attractve people: Assemblng loose lmbed models usng nonparametrc belef propagaton, n NIPS, [18] B. W. Slverman, Densty Estmaton for Statstcs and Data Analyss. London: Chapman & Hall, [19] B. Stenger, P. R. S. Mendonca, and R. Cpolla, Model based 3D trackng of an artculated hand, n CVPR, vol. 2, 2001, pp [20] B. Stenger, A. Thayananthan, P. H. S. Torr, and R. Cpolla, Flterng usng a tree based estmator, n ICCV, 2003, pp [21] E. B. Sudderth, A. T. Ihler, W. T. Freeman, and A. S. Wllsky, Nonparametrc belef propagaton, n CVPR, vol. 1, 2003, pp

9 Fg. 8. Eght frames from a trackng sequence n whch the hand makes graspng motons and ndvdual fnger movements. Note that the rng fnger s accurately tracked even through a partal occluson by the mddle fnger (bottom row). [22] C. Tomas, S. Petrov, and A. Sastry, 3D Trackng = Classfcaton + Interpolaton, n ICCV, 2003, pp [23] K. Toyama and A. Blake, Probablstc trackng wth exemplars n a metrc space, IJCV, vol. 48, no. 1, pp. 9 19, [24] Y. Wu, G. Hua, and T. Yu, Trackng artculated body by dynamc Markov network, n ICCV, 2003, pp [25] Y. Wu and T. S. Huang, Hand modelng, analyss, and recognton, IEEE Sgnal Proc. Mag., pp , May [26] Y. Wu, J. Y. Ln, and T. S. Huang, Capturng natural hand artculaton, n ICCV, [27] J. S. Yedda, W. T. Freeman, and Y. Wess, Constructng free energy approxmatons and generalzed belef propagaton algorthms, MERL TR