Image and Vision Computing

Transcription

1 Image and Vision Computing 29 (2011) Contents lists available at SieneDiret Image and Vision Computing journal homepage: Towards human motion apture from a amera mounted on a mobile robot Paulo Menezes a, Frédéri Lerasle b,,, Jorge Dias a a ISR-University of Coimbra, Polo II, Coimbra, Portugal b CNRS; LAAS; 7 avenue du Colonel Rohe, Toulouse Cedex 4, Frane Université de Toulouse; UPS, INSA, INP, ISAE; UT1, UTM, LAAS; Toulouse Cedex 4, Frane artile info abstrat Artile history: Reeived 14 Otober 2009 Reeived in revised form 9 July 2010 Aepted 3 January 2011 Keywords: Computer vision Data fusion Partile filtering Human motion apture Assistant robot This artile desribes a multiple feature data fusion applied to a partile filter for marker-less human motion apture (HMC) by using a single amera devoted to an assistant mobile robot. Partile filters have proved to be well suited to this roboti ontext. Like numerous approahes, the priniple relies on the projetion of the model's silhouette of the traked human limbs and appearane features loated on the model surfae, to validate the partiles (assoiated onfigurations) whih orrespond to the best model-to-image fits. Our partile filter based HMC system is improved and extended in two ways. First, our estimation proess is based on the so-alled AUXILIARY sheme whih has been surprisingly seldom exploited for traking purpose. This sheme is shown to outperform onventional partile filters as it limits drastially the well-known burst in term of partiles when onsidering high dimensional state-spae. The seond line of investigation onerns data fusion. Data fusion is onsidered both in the importane and measurement funtions with some degree of adaptability depending on the urrent human posture and the environmental ontext enountered by the robot. Implementation and experiments on indoor sequenes aquired by an assistant mobile robot highlight the relevane and versatility of our HMC system. Extensions are finally disussed Elsevier B.V. All rights reserved. 1. Introdution and framework A major hallenge of Robotis is undoubtedly the assistant robot, with the perspetive having suh an autonomous mobile platform to serve humans in their daily life. Embedding human motion apture (HMC) systems, based on onventional ameras, on the robot would give it the ability to at in a soial and human aware way and, (ii) interat with humans in a natural and rih way. While the Vision ommunity proposes a plethora of remarkable works on HMC from eiling-mounted and wide-angle ameras (see a survey in [24,30]), only a few of them address roboti appliations [2,19]. 3D traking from a mobile platform is arguably a hallenging task, whih imposes several requirements. First, the embedded amera overs a narrow field of view omparatively to ambient multi-oular systems. As the robot's operation takes plae on a wide variety of environmental onditions, bakground modelling tehniques [5,34,35] are preluded, while the traker is inevitably faed with ambiguous data. In these situations, several hypotheses must be handled simultaneously, and a robust integration of multiple visual ues is required. Finally, on-board proessing power is limited, thus are must be taken to design omputationally effiient algorithms. Like many researhers of the This paper has been reommended for aeptane by Ian Reid. Corresponding author at: CNRS; LAAS; 7 avenue du Colonel Rohe, Toulouse Cedex 4, Frane. Tel.: ; fax: address: lerasle@laas.fr (F. Lerasle). Vision ommunity, we aim at investigating marker-less HMC systems based on embedded vision tehniques. In brief, our framework differs learly from pedestrian detetion [10,11,21] whih aims at estimate the global person pose, and (ii) HMC from multiple wide-angle ameras [13,35,40] in home environment surveillane appliations. Most of the existing approahes in our ontext have onentrated on 3D artiulated models of the traked human limbs in order to make the problem more tratable. These approahes essentially differ in the assoiated data proessing (namely 3D reonstrution versus appearane based approahes) and the estimation framework (namely deterministi vs. Bayesian approahes). Reonstrution-based approahes try to fit the model to the 3D-point loud issued from 3D-sensor system e.g. a stereo head [3,40,42] or a Swiss Ranger [19]. Besides, the appearane-based approahes infer the model onfiguration from its projetions in the video stream [5,20,31,35,41]. These strategies enable the use of abundant appearane information whih an be extrated from the image ontents. We perform the integration of all those visual ues in a rigorous probabilisti way, sine we onsider a Bayesian formulation for posture estimation. For this, we use the partile filtering framework [7] whih has beome popular for traking, sine it was pioneered by Isard et al. in the form of CONDENSATION [15]. As it makes no restritive assumptions on the probability distributions to be propagated, partile filtering is well suited to the above roboti requirements. The main drawbak of using onventional partiles filters remains on the number of partiles that are required to effiiently sample the high dimensional state spae. Searh spae deomposition tehniques /$ see front matter 2011 Elsevier B.V. All rights reserved. doi: /j.imavis

2 P. Menezes et al. / Image and Vision Computing 29 (2011) inorporates several visual ues and geometri onstraints with appropriate degrees of adaptivity depending on the human posture and the environmental ontext enountered by the robot while navigating during in indoor human-entred environments. The rest of this paper is organised as follows. Setion 2 fouses on the modelling of the 3D struture, and presents a variant from the generi tehnique for projeting and handling self-olusions between parts. Setion 3 briefly outlines the well-known partile filtering formalism, its limitations, and the auxiliary sheme whih permits a multilevel integration of multiple ues. Setion 4 presents the different likelihoods involved in our AUXILIARY sheme. Implementation and experiments on two-arm gestures traking are presented in Setion 5. Setion 6 summarises our ontribution and opens the disussion for future extensions. 2. From the 3D model to its appearane Fig. 1. An interative task requiring HMC. [5] undoubtedly enable to takle this problem, yet global view strategies 1 are often favoured to determine the orret human posture. To limit the required number of partiles (and so the omputational load) we need an alternative to the widely used CONDENSATION algorithm for HMC [2,20,33] that enables the design of a better proposal distribution. Indeed, samples in the CONDENSA- TION sheme are predited blindly i.e. thanks to an importane funtion whih depends only on dynami models. The dynami models, typially based on Gaussian random walk, are quite diffuse and poor, beause inter-frame human motions are diffiult to haraterise. Consequently, very large partile sets are required in order to ahieve aeptable performane. Besides, alternative sampling shemes have been suggested to steer sampling towards state spae regions of high likelihood by inorporating the urrent observation. Unsented partile filters [32] have shown to outperform the CONDENSATION sheme but they still onstrain the importane funtion to be Gaussian, whih leads to ineffiient sampling in the presene of multi-modal distributions. In the same vein, the ICONDENSATION framework [16], by positioning samples aording to visual detetors i.e. independently of the past state, addresses also this problem but this may lead to be ontraditing with the proess history and so to an ineffiient filtering (see [39] for explanations). Reent approahes like [9] use an approah that mixes the priniples of partile filtering with global optimisation apabilities of simulated annealing with good results, but still requiring 4 ameras and about 61 s of proessing time per frame. Pitt et al. in [29] have suggested the so-alled AUXILIARY sheme whih is shown to overome this problem and to propose an optimal importane funtion (see [8] for theoretial developments). The proposal distribution is an appropriately defined mixture that depends both on the past state and the urrent observation. This genuine strategy has been surprisingly seldom exploited for traking purpose [12,18]. Conventionally, this strategy involves the evaluations of data-driven likelihoods both in the predition and weighting stages. Consequently, the omputational burden when applied to visual traking may be pretty high [39], espeially for triky traking in high state spae dimension. Our view is to differentiate learly the datadriven likelihoods involved in these two stages. For the proposal density, we propose a basi and oarse likelihood in order to plae samples in the relevant regions of the state spae without greatly inreasing the global time onsumption of the filter. The likelihood funtion involved in the weighting stage is more elaborated as it 1 i.e. involving all the human parts in a single step. Performing 3D traking from a single amera requires the use of a model that ompensates for the lak of depth information. Although reent works, like [9], use polygonal meshes to reate height fidelity models, we use simpler models based on trunated quadris whih are perfetly adequate for our purpose. Quadris are quite popular geometri primitives for use in human body traking [3,4,37]. This is due to fat that are easily handled, they an be ombined to reate omplex shapes, and their projetions are oni setions that an be obtained in losed form. This setion starts by realling some basis on quadris and their projetive models, followed by the presentation of the model, it ends with the details of generation of the appearane model whih is used in the measuring stage of the estimation proess. One ould argue that modern graphis ards an be used to obtain the model projetion, solving all the visibility issues that we address here. Although this hypothesis was onsidered we found that would introdue another level of omplexity for the following reasons: 1) this would require the reation of OpenGL projetive ameras that orrespond exatly to our physial amera. This would introdue some problems espeially for low ost ameras that normally present onsiderable deviations from the ideal amera in terms of the prinipal point (the intersetion of the lens axis with the retina) whih is frequently away from the image entre, and skew fators frequently different from 0; 2) the seond problem omes from the fat that the model projetion obtained with graphis ards are textures and not lists of line segments as required by our approah Basis on projetive geometry modelling A quadrati surfae or quadri Q is a seond degree impliit surfae in 3D spae. It an be represented in vetorial form using homogeneous oordinates as X T QX =0; X Q, wherex is a 4 1 vetor and Q is a 4 4 symmetri matrix. To employ quadris for modelling more general shapes, it is neessary to trunate them, and eventually ombine them, in our ase we used trunated ylinders and ones to approximate the upper limbs shapes. For a given quadri Q, the trunated ounterpart is defined by the set of points X that verify n X T QX =0 X T ΠX 0 where Π is a matrix representing the pair of lipping planes whih delimit the quadri Q. Finally, it should be noted that a given quadri Q resulting from the appliation of an Eulidian transformation H to the points of the surfae X T QX=0 is X TQ X =0 where X = HX and Q = H T QH 1. ð1þ

3 384 P. Menezes et al. / Image and Vision Computing 29 (2011) image plane Intersetion with a trunating plane projetion entre X 2 (s) X 1 (s) O C Q Intersetion with a trunating plane Fig. 2. Trunating projeted segments Desription of the human limbs model Considering N p parts in the 3D model, the latter is built using a set of rigid trunated quadris Q i i 1; ; N p to represent approximatively the shape of the human limbs. The trunated quadris are onneted between them by artiulations where eah one an ontain one or more degrees of freedom (DOF). In the urrent work, eight DOF, i.e. four per arm (3 rotations on the shoulder and 1 rotation on the elbow), are atually traked by this system. The arm ends, as well as their joints are here represented by spheres in order to improve the visual effet, but they are not being onsidered as model parts for traking purposes Generation of the model projetion Projetion of a quadri Let us start with the projetion of a quadri in a normalised amera. The assoiated projetion matrix is P = ½I 3 ñ 2 j0 3 ñ 1 Š.Considering a pinhole amera model, the amera entre and an image point x define a projetive ray whih ontains the points given by X=[xs] T,as illustrated on Fig. 2. The depth of a 3D point, situated on this ray, is determined by the salar s. The expression of the quadri X T QX =0 an then be rewritten as h i X T Ax +2sb T x + s 2 =0; where Q = A b T b : ð2þ This expression an be onsidered as an equation of seond degree in s. Then, when the ray represented by XðÞis s tangent to Q, there is a single solution for Eq. (2) i.e. its disriminant is zero, so: x T (bb T A) x=0.this expression orresponds to a oni C in the image plane, with C = A bb T = A bb T, whih therefore orresponds to the projetion of the quadri Q. For any x belonging to C, the orresponding 3D point X is fully defined by finding s 0 whih is given by s 0 = b T x/. This formula an be extended to arbitrary projetive amera with P=K[R t]. Defining a 4 4 matrix H suh that PH = ½Ij0Š, and then x = ½Ij0ŠH 1 X Projetion of our model's omponents Eah of the quadris, omposing the model, is said to be degenerate beause its matrix, Q, is singular. The image of suh a degenerated quadri, obtained by projetive projetion, is a degenerated oni. Being our model omposed of ones and ylinders, and depending on the point of view, its image an be a pair of parallel lines, a pair of onurrent lines, or even a single point. These three ases are easy to identify if matrix C is diagonal. A diagonal matrix C orresponds to a oni aligned with the x- and y-axis and entred at the origin. To ahieve this diagonal form, Stenger et al. in [36] use eigen-deomposition while we propose a more intuitive method. Our strategy is to define analytially a 3 3 matrix B that brings the oni into this partiular onfiguration, thene making the C matrix a diagonal one. For the more general ases, that orrespond to having two projeted lines, we an easily determine two points (denoted x 1 and x 2 )oneahof O p m r A B Pm Fig. 3. Hidden segment testing. Fig. 4. Top Row: Illustration of a projetion before (left) and after (right) hidden line removal. Bottom tow: examples of the appliation of the algorithm to a hand model.

4 P. Menezes et al. / Image and Vision Computing 29 (2011) Table 1 Time onsumption during model projetion. Time onsumption (s) Quadris transformation Contours projetion Hidden parts removal Total the lines. These rendered points are then transformed bak into their original onfiguration through transformation B 1, haraterising then eah projeted line l in the original frame. The next step is to trunate the projetions of the ones. Eah 3D line segment L, on the quadris surfae, orresponding to a projeted line l segment, is haraterised by two points X i whih are assoiated with h image points x i. These 3D points are omputed by X i = X i bt x i where i=1,2,where b and are bloks of the quadri's matrix. The 3D line defined by these two points orresponds to the visibility frontier of the quadri surfae. A generi point on this line is then given by X = X 1 + λx 2 ; X L: The intersetions that happen between this line and the two lipping planes are given by ðx 1 + λx 2 Þ T ΠðX 1 + λx 2 Þ = 0. This is in fat a seond degree equation in λ as all the other involved points are known. Solving this equation gives two values for λ that, one substituted bak into Eq. (3), produe the required 3D (intersetion) points. These ones, are then re-projeted onto the image plane to obtain the extremities of the line segments that orrespond to the one's projetion. This proedure is illustrated on Fig Self-olusion handling The final step is the handling of self-olusion. There are several algorithms available in the literature to manage the hidden parts of a model [25,38]. Most of them are omputationally heavy or inadequate for the urrent problem. One example is a reent algorithm presented in [36] that, although being quite adequate to the problem, presents a omplexity whih depends both on the size of the projeted parts and on the required preision. In the urrent work, the use of quadris of oni or ylindri type enables the use of a method whose omplexity depends only on the number of projeted parts and not on their size or preision. This algorithm, that will be desribed hereafter, requires, onsequently, less omputational power than the former ones. The algorithm starts with the omputation of all the strit intersetions amongst the whole set of projeted segments. Strit intersetion between two segments is defined as the ase where two segments interset and the intersetion point is not an extremity of either segment. The omputation of these intersetions an benefit i T ð3þ from the appliation of the sweeping line algorithm [26], espeially if the number of projeted segments is large, as this method presents a omplexity that is linear with the number of segments while the usual brute fore method presents a quadrati one. For eah intersetion point, the two impliated segments are setioned at this point. At the end there will be a list of (smaller) segments that do not exhibit any strit intersetion between them. The next step is to use the middle point, X m, of eah segment, and the amera entre to define a projetive line. Lets reall that every point on this line projets onto X m, as show by Fig. 3. The omputation of the set of intersetions between this line and the whole set of quadris enable us to verify if the 3D point that originated X m is in front of all the other quadris or is hidden by any of them, the same happening to the segment that ontains the point. For eah of the segments of the new set, its middle point p m is omputed, whih in onjuntion with O (the amera entre) defines a projetive ray. A searh is then performed to find possible intersetions between this projetive ray and any of the quadris that ompose the 3D struture. The intersetion points are then ordered using their distane to the amera entre. The segment is then marked as visible or invisible, whether the first point of intersetion in the list belongs or not to the quadri that originated it (through projetion). Fig. 3 illustrates this idea, p m is the middle point of an image segment, r is the projetive line that passes by p m and O, and A, B and P m the intersetion points with the two quadris. From this test it results that the projeted segment is not visible as both the points A and B are loser to O than P whih is the one that is on the surfae of the one that originated this segment. This is done by defining that any point on the projetive ray that passes through p m will have projetive oordinates of the form X p = h i p m s where s is a salar. Then replaing X p in the equation of eah quadri, a seond degree equation in s will be obtained. Then for eah obtained s the resulting point X p is tested to see if it is situated between the orresponding one trunating planes, in other words, if the point verify the inequality Xp T ΠXp 0 then it belongs to the trunated one, otherwise it belongs to the one but it is out of the zone delimited by the two lipping planes. It should be noted that omparing distanes between the intersetion points an be done by just omparing the respetive perspetive oordinates, s, whereas the larger this value, the smaller the distane from the onsidered point to the amera entre. Contrary to other methods that test the visibility of every projeted point [36], a single test on the segment's middle point is enough to infer about the visibility of a segment. Table 2 Generi partile filtering algorithm (SIR). [{x k,w N k }] i =1 =SIR([{x k 1 N,w k 1,}] i =1,z k ) 1: IF k=0, THEN Draw x (1) 0,, x 0,,x (N) 0 i.i.d. aording to p(x 0 ), and set w ðþ i 0 = 1 END IF N N 2: IF k 1 THEN [{x k 1,w k 1 }] i =1 being a partile desription of p(x k 1 z 1:k 1 ) } 3: FOR i=1,,n, DO 4: Propagate the partile x k 1 by independently sampling x k q(x k x k 1, z k ) 5: Update the weight w k assoiated to x k aording to w ðþ i k w ðþ i p ð z j x ðþ i k k Þpðx ðþ i j x ðþ i k k 1Þ, prior to a normalisation step so that k 1 qðx ðþ i j x ðþ i ;z k k 1 kþ i w k =1 6: END FOR N 7: Compute the onditional mean of any funtion of x k, e.g. the MMSE estimate E p(xk z1:k)[x k ], from the approximation i =1 w k δ(x k x k ) of the posterior p(x k z 1:k ) 8: At any time or depending on an effiieny riterion, resample the desription [{x k,w N k }] i =1 of p(x k z 1:k ) into the equivalent evenly weighted partiles set ð x sðþ ; 1 N by sampling in {1,,N} the indexes s k N i =1, (1),, s (N) aording to P(s =j)=w k(j) ; set x k and w k with x k(s) and 1 N 9: END IF

5 386 P. Menezes et al. / Image and Vision Computing 29 (2011) Table 3 Auxiliary Partile Filter (AUXILIARY). [{x k, w N k }] i =1 =AUXILIARY([{x k 1, w k 1 N }] i =1 (1) 1: IF k=0, THEN Draw x 0, (N),x0,,x0 i.i.d. aording to p(x 0 ), and set w ðþ i 0 = 1 END IF hn oi N 2: IF k 1 THEN x ðþ i k 1 ; w ðþ N i being a partile desription of p(x k 1 k 1 z 1:k 1 i =1 ) g,z k ) 3: FOR i=1,,n, DO 4: From the approximation ^p z k jx i = p z k 1 k jμ i e.g. with μ k k p(xk x k 1 )orμ k =E p(xk x k 1 )[x k ] ompute the auxiliary weights λ i k w k 1^p i z k jx i, prior to a k 1 normalisation step so that i λ i =1 k 5: END FOR hn oi N 6: Resample x i k 1 ; λ i k i =1 s(1),,s (N) of the partiles at time k 1 aording to P(s (j) =j)=λ k in order to get the ð equivalent evenly weighted partiles set x sðþ k 1 ; 1 N ; both N i =1 N k δ x k 1 x i and 1 k 1 i =1 N N i =1 δ x ð k 1 x sðþ approximate the smoothing pdf p(x k 1 k 1 z 1:k ) 7: FOR i=1,,n, DO 8: Propagate the partiles by independently drawing x (s k p(x k x k 1 ) 9: ðþ p z Update the weights, prior to their normalisation, by setting w i k jx i p z k k k jx i k = ð ^p z k jx sðþ ð p z k 1 k jμ sðþ k 10: Compute the onditional mean of any funtion of x k, e.g. the MMSE estimate E p(xk z1:k)[x k ], from the approximation N i =1 w i k δ x k x i k of the posterior p(x k z 1:k ) 11: END FOR 12: END IF Fig. 4 (tow row) illustrates the intersetion points between projetions of the segments and the result after hidden segments removal. The bottom row of the same figure shows the results of the appliation of this method to the projetions of a hand model. A simple performane evaluation was done when handling a 3D model omposed of the trunk/two arms and so 5 quadris and 10 trunating planes. The time onsumption is reported in Table 1 for the following operations: transform the whole set of quadris (ones and trunating ones), projet them, and haraterise their oluding ontours. To stress the performane gain relatively to other methods, it should be noted that this one only performs a visibility test for a single point on eah projeted segment whereas the method presented in [36] has to perform the visibility test for eah projeted point. Therefore to omputing time for the hidden parts removal shown on Table 1 orresponds to the verifiation of the visibility a small number of points for all projeted parts. This time should inrease linearly with the number of points verified 3. Partile filtering algorithms for data fusion 3.1. Basis strategies and assoiated limitations Partile filters are sequential Monte Carlo simulation methods for the state vetor estimation of any Markovian dynami system subjet to possibly non-gaussian random inputs [1]. Their aim is to reursively approximate the posterior density funtion (pdf) p(x k z 1:k ) of the state vetor x k at time k onditioned on the set of measurements z 1:k =z 1,, z k. A linear point-mass ombination px ð k jz 1:k Þ i k δ x k x ðþ i N k ; w ðþ i i =1 w ðþ i k =1: ð4þ is determined with δ(.) the Dira distribution whih expresses the seletion of a value or partile x k with probability or weight w k,i=1,,n. An approximation of the onditional expetation of any funtion of x k, suh as the minimum mean square error (MMSE) estimate E p (xk z1:k)[x k ], then follows. The generi partile filter or Sampling Importane Resampling SIR is shown on Table 2. The partiles x k evolve stohastially over the time, being sampled from an importane density q(.) whih aims at adaptively exploring relevant areas of the state spae. Their weights w k are updated thanks to p(x k x k 1 ) and p(z k x k ), resp. the state dynamis and measurement funtions, so as to guarantee the onsisteny of the approximation (Eq. (4)). In order to limit the degeneray phenomenon, whih says that after few instants all but one partile weights tend to zero, step 8 inserts a resampling proess. Another solution to limit this effet in addition to re-sampling, is the hoie of a good importane density. The CONDENSATION algorithm is instaned from the SIR as q(x k x k 1, z k )=p(x k x k 1 ). Another differene relative to the SIR algorithm presented is that the re-sampling step 8 is applied on every yle. Resampling by itself annot effiiently limit the degeneray phenomenon as the state-spae is blindly explored without any knowledge of the observations. On the other side, the ICONDENSATION algorithm [16], onsider importane density q(.) whih lassially relates to importane funtion π(x k z k )defined from the urrent image. However, if a partile drawn exlusively from the image is inonsistent with its predeessor in terms of state dynamis, the update formula leads to a small weight [39] Towards the optimal ase: the Auxiliary Partile Filter It an be shown [7] that the most effiient reursive sheme, i.e. the one whih best limits the degeneray phenomenon, must define q*(x k x k 1,z k ) p(x k x k 1,z k ) and thus w * k w * k 1 p(z k x k 1 ) in the SIR algorithm Table 2.This optimal strategy notieably affets eah partile x k aweightw * k whih solely depends on x k 1, w k 1 and z k through the preditive likelihood p(z k x k 1 ) so that w * k an be omputed prior to drawing x k. Further, to enhane the algorithm effiieny, the resampling step an be shifted just before the propagation through the optimal importane funtion q*(x k x k 1,z k ) of the weighted partiles set [{x k 1,w * N k }] i =1, whih in fat represents the smoother pdf p(x k 1 z 1:k ). Unfortunately, exept in very partiular ases, the above formulae Fig. 5. Two self-ollision examples.

6 P. Menezes et al. / Image and Vision Computing 29 (2011) Fig. 6. Likelihoods for our 3D traker: a) p(z k S x k ) ontour related likelihood obtained by sweeping the parameter spae for 2DoF of the struture; b) p(z k C x k ) olour related likelihood obtained by sweeping the image with the olour pattern; ) p(z k MS, z k C x k ) ombined likelihood of the former two showing on the image the projetion of the model that orresponds to the peak. are of limited pratial interest, for it is often impossible to samplefrom p (x k x k 1,z k )nortoomputep z k jx ðþ i = pðz k 1 k jx k Þp x k jx ðþ i dx k 1 k. Still, it remains possible to mimi the optimal strategy even if an importane funtion π(x k x k 1,z k )isdefined instead of p(x k x k 1,z k ) and if only an approximation ^p z k jx ðþ i of the preditive likelihood p k 1 (z k x k 1 ) an be omputed from the urrent measurement z k. First, an auxiliary weight λ ðþ i k w ðþ k 1^p i z k jx ðþ i is assoiated to eah partile k 1 x k 1. Then, the approximation [{x k 1,λ N k }] i =1 of p(x k 1 z 1:k ) is hn oi resampled into the evenly weighted set x si ðþ N k 1 ; 1 N, whih is i =1 s further propagated until time k through π(x k x k 1,z k ). Finally, the weights of the resulting partiles x k must be orreted in order to take into aount of the distane between λ k and w * k,aswellasofthe dissimilarity between the seleted and optimal importane funtions π(x s k x k 1,z k )andp(x s k x k 1,z k ). To this end, one must set. w ðþ i k ðþ i p z k jx p x ðþ i k k jx s ðþ ð i Þ k 1 ð ^p z k jx sðþ π x ðþ i k 1 k jx s ðþ ð k 1 ; z k Table 4 Parameter values used in our upper human body traker. Symbol Meaning Value N Number of partiles 400 λ st Fator in model prior p st (x) 0.5 λ o Fator in model prior p o (x) 0.5 K Penalty in likelihood pz MS k jx k 0.5 σ s Standard deviation in pz MS k jx k 1 N R Number of pathes in olour-based likelihood pz C k jx k 6 σ Standard deviation in pz C k jx k 0.3 N bi Number of olour bins per hannel involved in pz C k jx k 32 The Auxiliary Partile Filter (AUXILIARY) was developed in this vein by [29] ontemporarily to ICONDENSATION. It approximates the preditive likelihood by ^p z k jx ðþ i = p z k 1 k jμ ðþ i, where μ k k haraterises the distribution of x k onditioned on x k 1 e.g. μ k =E pðx k jx k 1 Þ [x k ] or μ k p(x k x k 1 ) and its importane funtion follows the system dynamis, i.e. π(x k x k 1,z k )=p(x k x k 1 ), see Table 3. The partiles loud an thus be steered towards relevant areas of the state spae. In the visual traking ontext, the approximate preditive likelihood an rely on visual ues whih are different from those involved in the omputation of the final-stage likelihoods p(z k x k ). In pratie, the AUXILIARY sheme runs slightly slower than the CONDENSATION as we need to perform two weighted bootstraps rather than one. However, the improvement in sampling will usually dominate these small effets. By making proposals whih have high onditional likelihoods, we redue the osts of sampling many times from partiles whih have very low likelihoods and so will not be resampled at the seond proess stage. This improves the statistial effiieny of the sampling proedure and means that we an redue substantially the partiles number. 4. Likelihoods entailed in the AUXILIARY sheme The next subsetions desribe the different likelihoods entailed in the proposal distribution and in the measurement funtion. The latter is based on multiple visual ues and model priors to enode physial properties of the 3D model Preditive likelihood for the proposal distribution The partiles sampled in step 6 (Table 3) are drawn from a likelihood whih enodes the similarity between skin-like image ROIs and virtual pathes related to the two hands projetion given a model onfiguration. Let h skin and h μ be two N bi -bin normalised histograms in hannel {R, G, B}, respetively orresponding to a skin olour distribution and to a region B μ parameterized by the state μ. The olour likelihood p(z μ) in Table 3 must favour andidate olour

7 388 P. Menezes et al. / Image and Vision Computing 29 (2011) Fig. 7. From top-left to bottom-right: snapshots from a simple 3 d.o.f. arm traking sequene showing the superimposition of the projeted model over the input image and the orresponding model onfiguration. histograms h μ, p for the two hands lose to the skin olour distribution i.e. h skin! pzjμ ð Þ exp D2 ; D = 1 2σ p =1 D B h u;p ; h skin ; ð5þ provided that D B terms the Bhattaharyya distane [27] between the two histograms h μ, p and h skin and σ is the standard deviation being determined a priori. Training images from the Compaq database [17] theta 1 theta 2 theta Fig. 8. Evolution of the estimated parameters (joint angles) of the model versus the input frame number, for the single arm sequene. enables to model the skin olour distribution h skin hands Measurement sub-funtions dediated to the Shape ue Using this ue requires the 3D model projetion with hidden parts removing. The shape-based likelihood p(z S x) is omputed using the sum of the squared distanes between model points and the nearest image edges [15]. The measurement points are hosen to be uniformly distributed along the model projeted segments. In our implementation, the edge image is onverted into a Distane Transform image, noted I DT, whih is used to approximate the distane values. The advantage of mathing our model ontours against a DT image rather than using diretly the edges image is that the resulting similarity measure is a smoother funtion of the model parameters. In addition, the DT image redues the involved omputations as it needs to be generated only one whatever the number of partiles involved in the filter. The likelihood p(z S x) is given by! p z S D jx 2 exp λ s ; D = 1 N p 2σs 2 N p j =0 I DT ðþ; j where j indexes the N p model points uniformly distributed along eah visible model projeted segments, I DT (j) is the assoiated value in the ð6þ

8 P. Menezes et al. / Image and Vision Computing 29 (2011) Fig. 9. From top-left to bottom right: snapshots from the two arm traking sequene with a low luttered bakground: λ s = 10,λ p, =0.1. DT image, σ s is the standard deviation being determined a priori, and λ s is a weighting fator disussed later Motion ue Considering a stati amera, it is highly possible that the targeted subjet be moving, at least intermittently in some H/R situations. To ope with bakground lutter, we thus favour the moving edges by ombining motion and shape ues into the definition of the likelihood p(z MS x) of eah partile x. This is aomplished by using two DT images, noted I DT and I DT, where the new one is obtained by filtering out the stati edges, based on fðzðþ j Þ the onventional optial flow vetor at pixel z (j) assuming the onstant brightness assumption i.e. onsidering that the brightness of a pixel will not hange between two suessive image frames [14]. The likelihood p(z MS x) has a form similar to (6) with assoiated parameters σ m and λ m, provided that D is given by D = 1 N p N p j =0 min I DT ðþ; j K:I 0 DTðÞ j ; with weight values K 1 make moving edges more attrative. This edge-based likelihood favours the moving edges without moving the stati ones. Consequently, the assoiated traker will prefer to stik with the moving edges but in their absene its fallbaks is to use the stati ones Colour ue Using disriminant olour (or texture) distributions related to lothes is also of great interest in the weighting stage. By assuming onditional independene of the olour measurements, the likelihood (5) beomes for N r ROIs! pz D 2 jx exp λp; ; D = 1 2σ 2 N r N r p =1 D B h x;p ; h ref ;p ; ð7þ where λ p, are weighting fators disussed in Setion 5. Eah referene histograms h ref, p for the p-th path is learnt on the first sequene image assuming the 3D model has been beforehand superimposed. From Eq. (7), we ould also define a likelihood p(z T k x) and assoiated weights λ p, t relative to textured pathes based on the intensity omponent Model priors Non-observable parameters stabilisation A ertain pose onfiguration leads to observation ambiguities as partiular DOFs sometimes annot be inferred from the urrent image observations. For instane, when one arm is straight and the likelihood p(z S x) is used, rotations around the arm's own axis annot be observed with a oarse, and symmetri body part model. The oluding ontour hanges little when this limb rotates around its own axis. Leaving this DOF unonstrained would often lead to ambiguities that produe nearly flat ost surfaes. Intuitively, adding texture/olour pathes on the arms allows this rotation to be observed. A seond and potentially deeper issue introdued in [35] is to ontrol all these hard-to-estimate parameters with a stiky prior stabilisers p st (x) exp( λ st x def x 2 ) that reahes its minimum on a predefined resting onfiguration x def. This prior only depends on the struture parameters and the fator λ st will be hosen in a way that the stabilising effet will be negligible for the whole onfiguration spae with the exeption of the regions where the visual

9 390 P. Menezes et al. / Image and Vision Computing 29 (2011) Fig. 10. From top-left to bottom right: snapshots of traking sequene involving deiti gestures: λ s =10,λ p, =0.1. measurement funtions are onstant. In the absene of strong observations, the parameters are onstrained to lie near their default values x def whereas strong observations unstik the parameters values from these default onfigurations Body parts ollision detetion Physial onstraints impose that the body parts do not interpenetrate eah other. The admissible volume of the parameter spae is smaller than initially designed beause ertain regions are not physially reahable, and so many false hypothesis may be pruned. Handling suh onstraints in a ontinuous optimisation-based framework would be far from trivial. Disrete stohasti sampling framework is learly more suitable even if some samples may apriorifall inside the non admissible parameter spae (see examples in Fig. 5). Our strategy is to simply rejet these hypothesis thanks to ollision detetion mehanism. The ollision model prior for a state x is thus p oll (x) exp( λ o f o ) with f o (x)=0 (resp. 1) whether no ollision (resp. in ollision). This funtion, although being disontinuous for some points of the onfiguration spae and onstant for all the remaining, is still usable in a Dira partile filter ontext. The advantage of its use is twofold, first it avoids the exploration of the filter to zones of no interest, and seond it avoids wasting time in performing the measuring step for unaeptable hypothesis as they an be immediately rejeted Unified measurement funtion Fusing multiple visual ues enables the traker to better benefit from M distint measurements (z 1,,z M ). Assuming that these are mutually independent onditioned on the state, and given L model priors p 1 (x),,p L (x), the unified measurement funtion thus fatorizes as p z 1 ; ; z M jx M p z i jx : L p j ðxþ: i =1 j =1 5. Implementation and experiments In its atual form, the system traks two-arms gestures under an 8- DOF model, i.e. four per arm. We assume therefore that the torso is oarsely fronto-parallel with respet to the amera while the position of the shoulders are dedued from the position of the fae given by dediated traker [22]. All the DOFs are aounted for in the state vetor x k related to the k-th frame. Kinemati onstraints require that the values of these joint angles evolve within anatomially onsistent intervals. Samples (issued from the proposal) falling outside the admissible joint parameter range are enfored to the hard limits and not rejeted as this ould lead to asades of rejeted moves that slow down the sampler. Reall that the p(x k X k 1 ) pdf enodes information about the dynamis of the targeted human limbs. These are desribed by an Auto-Regressive model with the following form X k =Ax k 1 +w k where X k =[X k,x k 1 ] and w k defines the proess noise. In the urrent implementation, these dynamis orrespond to a onstant veloity model. We find this AR model gives empirially better results than usual random walk model [28,33]. A set of pathes are distributed on the surfae model and their possible olusions are managed during the traking proess. Our approah is different from the traditional marker-based ones beause we do not use artifiial but natural olour or texture-based markers ð8þ

10 P. Menezes et al. / Image and Vision Computing 29 (2011) Fig. 11. From top-left to bottom-right: snapshots of traking sequene involving heavy lutter: λ s =1,λ p, =1. e.g. the two hands and ROIs on the lothes. We adopt the AUXILIARY sheme (Table 3) whih allows to use some low ost measure or a priori knowledge to guide the partile plaement, therefore onentrating them on the regions of interest of the state spae. The measurement strategy is as follows: (1) partiles are firstly loated in good plaes of the onfiguration spae aording to the initial sampling based on the likelihood (5) in step 6, (2) partiles' weights are fine-tuned by ombining shape and motion ues, multiple pathes per arm as well as model priors thanks to (8) in step 9. Fig. 6-left shows the plot of the shape-based likelihood (6) obtained by sweeping a subspae of the onfiguration spae formed by the orientation of the right arm model involving moderate bakground lutter. In luttered bakground, shape ue is not suffiiently disriminant as multiple peaks are present. Fig. 6-middle plots the olour-based likelihood (7) for a single path orresponding to the marked hand. This is extremely sharp but shows false positive as soon as spurious skin olour like regions are deteted. Fig. 6-right plots the likelihood p(z MS,z C x) issued from (8) when fusing shape, motion and olour ues. This plot brings out that this funtion is more disriminant than the ones involving a single ue. Clearly, mixing all these ues into the measurement funtion of the underlying estimation sheme helps the traker to work under a wide range of onditions enountered by our robot during its normal operation. A seond and important line of investigation onerns the inorporation of appropriate degrees of adaptivity into these multiple ues based likelihoods depending on the target appearane and environmental onditions. Therefore, some heuristis allow to weight the strength of eah visual ue in the unified likelihood (8). An a priori onfidene riterion of a given oloured or textured path relative to lothes an be easily derived from the assoiated likelihood funtions t where the p-th olour referene histogram h ref, p (h ref, p for the texturerelated one) is uniform and so given by h j;ref = 1 N bi ; j =1; ; N bi where index p has been omitted for ompatness reasons. Typially, uniform oloured pathes produe low likelihood values, whereas higher likelihood values haraterise onfident pathes beause their assoiated olour distributions are disriminant and ensure non ambiguous mathings. As stated before, parameters λ p, and λ p, t weight the strength of the p-th marker in the likelihood (8). In the same way, the parameter λ s weights the edges' density ontribution and is fixed from the first DT image of the sequene. Due to the effiieny of the importane density and the relatively low dimensionality of the state-spae, traking results are ahieved with a reasonably small number of partiles i.e. N=400 partiles. In our unoptimised implementation, a PentiumIV-3 GHz runs the two arm traking proess at about 1fps, being most of the time spent in evaluating the observation funtion. To ompare, lassi systems take a few seonds per frame to proess a single arm traking. The fixed parameters involved in the likelihoods, proposal and state dynamis of our upper human body traker are reported in Table 4. The above desribed approah has been implemented and evaluated over monoular image sequenes. Fig. 7 show an initial test result of traking a single arm using a 3 degree of freedom model while Fig. 8 shows the estimated joint angles values (radians) versus the input frame number. To illustrate our approah, we show and omment snapshots from typial sequenes aquired from the robot Jido (Fig. 1) in different situations to highlight our flexible data fusion strategy. The full images as well as other sequenes an be found at the URL ~paulo/hri. The first sequene (Fig. 10) was shot against a white and unevenly illuminated bakground. Here although using loose fitting lothes, the traker an follow deiti gestures that demonstrate its ability to work even for poses out of the fronto-parallel plane. The seond sequene (Fig. 11) involves oarse fronto-parallel motions

11 392 P. Menezes et al. / Image and Vision Computing 29 (2011) Fig. 12. From top-left to bottom-right: snapshots of traking sequene and animation of HRP2 using the estimated parameters: λ s =5,λ p, =0.5. over a heavy luttered bakground. For eah snapshot, the right subfigures show the model onfiguration for the MMSE estimate, while the left ones show its orresponding estimated onfiguration orresponding to the posterior pdf p(x k z 1:k ). The first traking senario (Fig. 9) shows the estimation of the arms onfigurations in the presene of a low luttered bakground. It an be seen that even with a single amera the system is able to estimate non-planar poses. The seond sequene (Fig. 9) involves pointing gestures. The target ontours are prominent and are weakly disturbed by the bakground lutter. The high onfident ontours ue ensure the traking suess. The pathes on the uniform sweater are here of little help as their enlosed olour or texture distributions are quite uniform. The flexibility introdued in system by the use of tuneable parameters (λ (.) ), allows to give these pathes a weak strength in the unified likelihood ost (λ p, =0.1 against λ s =10). Although they do not introdue any improvement with respet to their position on the arm, they are not ompletely disarded, as they still ontribute with a form of an inside/outside information, whih omplements the ontours, espeially when they fail. This permitted the traking of the arms even when they got out of the fronto-parallel plane thanks to all the pathes (Fig. 10). For the seond senario (Fig. 11), the traker deals with signifiantly more omplex sene but traks also the full sequene without failure. This senario takes learly benefit from the introdution of disriminant pathes as their olour distributions are far from uniform ones. This leads to higher values of onfidene dediated to the likelihood p(z k C x k ), namely λ p, =1. In this hallenging operating onditions, two heuristis allow jointly to release from distrating lutter that might partly resemble human body parts (for instane the upboard pillar, whose olour is lose to skin olour). On the one hand, estimating the edges density in the first frame highlights that shape ue is not a onfident one in this ontext, so its onfidene level in the global ost (8) is redued aordingly during the traking proess i.e. λ s =1. On the other hand, optial flow weights the importane relative to the foreground and bakground ontours thanks to the likelihood pz MS k jx k. If onsidering only ontour ues in the likelihood, the traker would attah itself to luttered bakground zones and onsequently lose the target. Sine algorithms orrespond to a stohasti framework, we ompare multiple runs of the CONDENSATION and AUXILIARYfilters on these two sequenes with idential starting state. For a fair empirial omparison, the number of partiles used with eah filter was hosen so that the number of likelihood evaluations per frame was equal. Speifially, 800 partiles were used for CONDENSATION and 400 partiles for AUXILIARY. The CONDENSATION filter with non flexible data fusion fails in the great majority of runs. In only 10% runs did it provide a orret estimate in terms of the MMSE. The AUXILIARY gave reasonable estimates in 80% runs. Moreover, the AUXILIARY is shown to redue the variane of the estimate along onseutive trials. Beyond the aforementioned assistant robot paradigm, another envisaged appliation onerns the animation of a humanoid robot [23]. This last senario (Fig. 12) with moderate lutter explores 3D estimation behaviour with respet to problemati motions i.e. non fronto-parallel ones, elbow end-stops and observation ambiguities. The left olumn represents the input images and the projetion of the model ontours superimposed while the right olumn represents the animation of the HRP2 using the estimated parameters. The first frames involve both elbow end-stops and observation ambiguities. These partiular onfigurations are easily dealt with in our partile filtering framework. When elbow end-stop ours, the sampler is able to maintain the elbow angle within its predefined hard limits. Observation ambiguity arises when the arm is straight. The twist

12 P. Menezes et al. / Image and Vision Computing 29 (2011) parameter is temporary unobservable but remains stable thanks to the likelihood p st ðx k Þ. As highlighted in [6], Kalman filtering is quite unable to trak suh partiular arm onfigurations. Some frames later in Fig. 12, the left arm bends slightly towards the amera. Thanks to the pathes on the hands, the traker manages to follow this temporary unobservable motion, although it signifiantly misestimates the rotation during this motion. The entire HRP2 video is available at 6. Conlusion This artile presents an innovative partile filtering framework for 3D traking the upper human body parts using a single amera mounted on an assistant mobile robot. Like numerous approahes, the priniple relies on the model image projetion and model-image mathing ost metri to infer the model onfiguration in 3D. Our partile filter based HMC system differs from onventional partile filters as follows. The genuine AUXILIARY strategy limits drastially the well-known burst in terms of partiles when onsidering high dimensional state-spae. This nie property is due to both dynami and image data driven proposal density. Data fusion is also onsidered in the measurement funtion. The weighting stage relies on a new model-image mathing ost metri, whih ombines, in a flexible way, extrated edges (weighted by flow), olour (or texture)-based pathes, kinemati joint limits, and non self-intersetion onstraints. Experiments on monoular sequenes aquired from the robot, show that the proposed framework is suitable for traking human motions, and that the flexible integration of multiple ues improve the traker versatility. Moreover, our traker is applied in a quasi-real-time proess and so requires less omputational power than most of the existing approahes. Combined with today's powerful off-the-shelf PCs, suh quasi real-time HMC approah devoted to mobile robot nearly beomes a reality and would have a great number of roboti appliations. Several diretions are studied regarding our visual modalities. Firstly, we are urrently working on the idea of ontinuously learn the appearane of new pathes, distributed all over the surfae of the model, during the traking proess. Next, our measurement funtion will be enrihed with sparse 3D reonstrution-based data. Sparse 3D-point loud will aid reovery from transient traking failures and will free the traker from the lassial ad ho initialisation. Further evaluations will be also performed using a motion apture testbed that provides more aurate ground truth from a ommerial HMC system, suh as VICON, that will be synhronised with the video streams. Referenes [1] S. Arulampalam, S. Maskell, N. Gordon, T. Clapp, A tutorial on partile filters for online non-linear/non-gaussian Bayesian traking, Trans Signal Proessing 2 (50) (2002) [2] P. Azad, A. Ude, T. Asfour, R. Dillmann, Stereo-based markerless human motion apture for humanoid robot systems, Int. Conf. on Robotis and Automation (ICRA'07), April 2007, pp , Roma, Italy. [3] Q. Delamarre, O. Faugeras, 3D artiulated models and multi-view traking with physial fores, International Journal Computer Vision and Image Understanding (CVIU'01) 81 (2001) [4] J. Deutsher, A. Blake, I. Reid, Artiulated body motion apture by annealed partile filtering, Int. Conf. on Computer Vision and Pattern Reognition (CVPR'00), Hilton Head, USA, June 2000, pp [5] J. Deutsher, A. Davison, I. Reid, Automati partitioning of high dimensional searh spaes assoiated with artiulated body motion apture, Int. Conf. on Computer Vision and Pattern Reognition (CVPR'01), 2001, pp [6] J. Deutsher, B. North, B. Basle, A. Blake, Traking trough singularities and disontinuities by random sampling, Int. Conf. on Computer Vision (ICCV'99), Corfu, Greee, September [7] A. Douet, N. De Freitas, N.J. Gordon, Sequential Monte Carlo Methods in Pratie, Series Statistis For Engineering and Information Siene, Springer-Verlag, New York, [8] A. Douet, S.J. Godsill, C. Andrieu, On sequential Monte Carlo sampling methods for Bayesian filtering, Statistis and Computing 10 (3) (2000) [9] J. Gall, B. Rosenhahn, H.-P. Seidel, An introdution to interating simulated annealing, in: R. Klette, D. Metaxas, B. Rosenhahn (Eds.), Human Motion Understanding, Modeling, Capture and Animation, Computational Imaging and Vision, 36, Springer-Verlag, 2008, pp [10] D.M. Gavrila, Multi-ue pedestrian detetion and traking from a moving vehile, International Journal of Computer Vision (IJCV'07) 73 (1) (2007) [11] D. Geronimo, A.M. Lopez, A. Sappa, T. Graf, Survey of pedestrian detetion for advaned driver assistane systems, Trans on Pattern Analysis Mahine Intelligene (PAMI'10) 28 (6) (2010) [12] F. Gustafsson, F. Gunnarsson, N. Bergman, U. Forssell, J. Janson, R. Karlsson, P.-J. Nordlund, Partile filters for positioning, navigation and traking, Trans. on Signal Proessing 50 (2) (2002) [13] N. Hasler, B. Rosenhahn, T. Thormahlen, W. Wand, J. Gall, H.-P. Seide, Markerless motion apture with unsynhronized moving ameras, Int. Conf. on Computer Vision and Pattern Reognition (CVPR'09), Miami, USA, June [14] B. Horn, B. Shunk, Determining optial flow, Artifiial Intelligene 17 (1981) [15] M. Isard, A. Blake, Contour traking by stohasti propagation of onditional density, European Conf. on Computer Vision (ECCV'96), Cambridge, UK, April 1996, pp [16] M. Isard, A. Blake, I-CONDENSATION: unifying low-level and high-level traking in a stohasti framework, European Conf. on Computer Vision (ECCV'98), Freiburg, Germany, June 1998, pp [17] M. Jones, J. Rehg, Color Detetion, Tehnial report, Compaq Cambridge Researh Lab, 1998, p. 11. [18] J. Kenna, H. Nait-Charif, Traking human motion using auxiliary partile filters and iterated likelihood weighting, International Journal Image and Vision Computing (IVC'07) 25 (6) (2007) [19] S. Knoop, S. Vaek, R. Dilman, Sensor fusion for 3D human body traking with an artiulated 3D body model, Int. Conf. on Robotis and Automation (ICRA'06), Orlando, USA, May 2006, pp [20] M.W. Lee, I. Cohen, S.K. Jung, Partile filter with analytial inferene for human body traking, Workshop on Motion and Video Computing (MOTION'02), Orlando, USA, Deember [21] B. Leibe, K. Shindler, N. Cornelis, L.V. Gool, Coupled objet detetion and traking from stati ameras and moving vehiles, International Journal Trans. on Pattern Analysis and Mahine Intelligene (PAMI'08) 30 (10) (2008) [22] P. Menezes, J.C. Barreto, J. Dias, Fae traking based on haar-like features and eigenfaes, IFAC Symp. on Intelligent Autonomous Vehiles, Lisbon, July [23] P. Menezes, F. Lerasle, J. Dias, R. Chatila, A single amera motion apture system dediated to gestures imitation, Int. Conf. on Humanoid Robots (HUMANOID'05), Tsukuba, Japan, 2005, pp [24] T. Moeslund, A. Hilton, V. Kruger, A survey of advanes in vision-based human motion apture and analysis, International Journal Computer Vision and Image Understanding (CVIU'06) 104 (2006) [25] T.M. Mutali. Effiient Hidden-Surfae Removal in Theory and in Pratie. PhD thesis, Dept. of Computer Siene, Brown University, 1998 [26] J. Nierverget, F.P. Preparata, Plane sweeping algorithms for interseting geometrial figures, Communiations of the ACM 25 (1982) [27] K. Nummiaro, E. Koller-Meier, L. Van Gool, An adaptative olor-based partile filter, International Journal Image and Vision Computing (IVC'03) 21 (90) (2003) [28] P. Pérez, J. Vermaak, A. Blake, Data fusion for visual traking with partiles, International Journal IEEE 92 (3) (2004) [29] M.K. Pitt, N. Shephard, Filtering via simulation: auxiliary partile filters, Journal of the Amerian Statistial Assoiation 94 (446) (1999). [30] R. Pope, Vision-based human motion analysis: an overview, Internatioanl Journal Computer Vision and Image Understanding (CVIU'07) 108 (2007) [31] D. Ramanan, D.A. Forsyth, A. Zisserman, Strike a pose: traking people by finding stylized poses, Int. Conf. on Computer Vision and Pattern Reognition (CVPR'05), San Diego, USA, June [32] Y. Rui, Y. Chen, Better proposal distributions: objet traking using unsented partile filter, Int. Conf. on Computer Vision and Pattern Reognition (CVPR'01), Hawai, Deember 2001, pp [33] H. Sidenbladh, M.J. Blak, D.J. Fleet, Stohasti traking of 3D human figures using 2D image motion, European Conf. on Computer Vision (ECCV'00), Dublin, Ireland, July 2000, pp [34] L. Sigal, S. Bhatia, S. Roth, J. Blak, M. Isard, Traking loose-limbed people, Int. Conf. on Computer Vision and Pattern Reognition (CVPR'04), Washington, USA, June [35] C. Sminhisesu, B. Triggs, Estimating artiulated human motion with ovariane saled sampling, International Journal on Roboti Researh (IJRR'03) 6 (22) (2003) [36] B. Stenger, P.R.S. Mendonça, R. Cipolla, Model-based hand traking using an unsented kalman filter, British Mahine Vision Conf. (BMVC'01), Manhester, UK, 1, September 2001, pp [37] B. Stenger, A. Thayananthan, P.H.S. Torr, R. Cipolla, Filtering using a tree-based estimator, Int. Conf. on Computer Vision (ICCV'03), Nie, Frane, Otober 2003, pp [38] I.E. Sutherland, R.F. Sproul, R.A. Shumaker, A haraterization of ten hiddensurfae algorithms, ACM Computing Survey 1 (6) (1974) [39] P. Torma, C. Szepesvari, Sequential importane sampling for visual traking reonsidered, AI and Statistis, 2003, pp [40] R. Urtasun, P. Fua, 3D human body traking using deterministi temporal motion models, European Conf. on Computer Vision (ECCV'04), Prague, Czeh Republi, May [41] X. Xu, B. Li, Learning motion orrelation for traking artiulated human body with a rao-blakwellised partile filter, Int. Conf. on Computer Vision (ICCV'07), Rio de Janeiro, Brazil, Otober [42] J. Ziegler, K. Nikel, R. Stiefehagen, Traking of the artiulated upper body on multi-view stereo image sequenes, Int. Conf. on Computer Vision and Pattern Reognition (CVPR'06), New York, USA, June 2006.