FACIAL ACTION TRACKING USING PARTICLE FILTERS AND ACTIVE APPEARANCE MODELS. Soumya Hamlaoui, Franck Davoine

Transcription

1 FACIAL ACTION TRACKING USING PARTICLE FILTERS AND ACTIVE APPEARANCE MODELS Soumya Hamlaoui, Franck Davoine HEUDIASYC - CNRS / Université de Technologie de Compiègne Soc-Eusai Our objective Tracking a near- frontal view face in video sequences: Global motion: 2D pose (position, scale, orientation) Local motion: facial features (appearance variations) Soc-Eusai

2 Proposed scheme Stochastic tracking system based on the CONDENSATION algorithm (particle filtering): The unobserved state includes pose and appearance parameters. The observations distribution is derived froman Active Appearance Model (AAM) and uses a robust distance measure. The dynamic distribution and the particle number are adaptive. Soc-Eusai State space & observations Unobserved state: poset x t = ct pose t = p t = (t x, t y, δ, θ) t T c t : first four modes of the appearance variation (92 % of appearance variations) Observed data: z t = image texture = g image Soc-Eusai

3 Active Appearance Models [Cootes & al 01] PCA/ Shape: s = s m + f s b s PCA/ Texture: g = g m + f g b g Coupling the two s: b = W s b s b g PCA / concatenated shape and texture parameters b: b = f c c Shape Instance:s (c) = s m + Q s c Texture Instance: g (c) = g m + Q g c Match a target face in a given image (iterative gradient search): Minimize a texture residual: r( c, p) = g( c) gimage (, c p) Find the optimal correction (δc, δp) to apply in order to minimize r (c, p ) δc = -R c r(c, p) δc = -R p r(c, p) R p, R p Matrices precomputedfromtraining data Soc-Eusai (Condensation algorithm [Isard & Blake 98]) Unobserved states x 0 x 1 x t-1 x t To infer Dynamic z 1 Observed data z t-1 z t Observation Tracking recursive evaluation of the posterior density of the target state conditionally to the history of observations P(x t z 1:t ) by means of the empirical distribution of a system of particles : - Dynamic : P(x t x t-1 ) - Observation : P(z t x t ) Soc-Eusai

4 Description of the Condensation algorithm Dynamic Observation Soc-Eusai Dynamic (1/2) Propagates the particles systemthrough time: x = xˆ + v + Su t t 1 t t : estimate ofthe state vectorat the previous time step. ˆ -1 x t v t = ( p, c) T : predicted shift in pose/appearance obtained by an AAM searchin the current frame. u : randomvariates having zero meanand unit variance. S t = diag ( ( t x ) ( s,, 4 ) t s c t ) : standard deviations for each pose/appearance parameter. Soc-Eusai

5 Dynamic (2/2) Adaptive standard deviations [Zhou & al 04]: [ σ,..., σ ] = diag( R,..., R )[ σ,..., σ ] ( t x) ( c4 ) T ( t x) ( c4 ) ( t x) ( c4 ) T t t t t 0 0 () i Rt ε : texture error averaged over the L pixels of the textures: L l l 2 g gimage( x% t) εt = ρ L l = 1 σ l Adaptive particle number(substantial gain in computing time): N 0 : initial fixedparticle number 8 ε Nt = N R 0 i = 1 () i t Soc-Eusai Description of the Condensation algorithm Dynamic Observation Soc-Eusai

6 Observation (1/2) Consists of the likelihoodp(z t x t ) according to whichthe particles are weighted: Based on the difference between: Image texture : g image (p t, c t ) sampled at the hypothesized pose and shape Model texture : g (c t ) given by the AAM d[ g ; g ] image Pz ( x ) = Ce t C: Normalizingconstant of this distribution t Soc-Eusai Observation (2/2) The texture distance d(,) is an error measure summed over all L pixels of bothtextures: L gl gl ' dgg (, ') = ρ l= 1 σl ρ() is a robust error function in order to reduce the influence of occluded pixels: 1 2 γ 2 ργ ( ) = 1 h γ h 2 2 if if γ h γ > h h: fixedthreshold above which the difference γ is consideredto be an outlier Soc-Eusai

7 Experimental results The proposed method was implemented in C++ and tested on a PC running WinXP at 2.4 GHz with512 Mb of RAM. N 0 = 500 N t evolves between about 20 and 80 and increase when change in pose and/or appearance is rapid 2 frames per second Soc-Eusai Particle number evolution Frame 920 Nearly full occlusion Frame 1445 Partial occlusion Soc-Eusai

8 But The effectiveness of the appearance remains conditioned by the fact that the tracked appearance must be beforehand learned and led. This ling is sensitive to the recording conditions of the training images. Replace the appearance by an adaptive appearance estimated on-line Soc-Eusai New texture g on-line : initialized manually using the face texture in the first frame Updated: gon-line () t = αgon-line ( t 1) + (1 α) gimage(, tx% t 1) α: forgetting factor determining the update importance : the current image texture estimated to the state hypothesis at t-1 gimage(, t x% t 1) The hidden state space encodes the pose and the first four modes of the shape parameters obtained from the face : x t = (p t, b s ) T Soc-Eusai

9 Some perspectives Recognition of facial actions and behavior, by analysing the state trajectories (applications: HCI, surveillance, etc.). 3D pose tracking. Soc-Eusai Thank you for your attention [Cootes & al 01] : T.F. Cootes, G.J. Edwards and C.J. Taylor. Active Appearance Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp , June [Hamlaoui & Davoine 05] : S. Hamlaoui, F. Davoine. Facial action tracking using an AAMbased Condensation approach. IEEE Int. Conf. on Acoustic, speech and Signal Processing, March [Isard & Blake 98] : M. Isard, A. Blake. Condensation Conditional Density Propagation for Visual Tracking. Int. Journal of Computer Vision, pp. 5-28, [Zhou & al 04] : S. Zhou, R. Chellappa, and B. Moghaddam. Visual tracking and recognition using appearance-adaptive s in particle filters. IEEE Trans. on Image Processing, November Soc-Eusai