Particle Filters for Visual Tracking

Particle Filters for Visual Tracking T. Chateau, Pascal Institute, Clermont-Ferrand 1

Content Particle filtering: a probabilistic framework SIR particle filter MCMC particle filter RJMCMC particle filter Particle Filters for Visual Tracking T. Chateau 2

Content Particle filtering: a probabilistic framework SIR particle filter MCMC particle filter RJMCMC particle filter Particle Filters for Visual Tracking T. Chateau 3

Visual Tracking State (hidden variable) y bird view x Observation y x Particle Filters for Visual Tracking T. Chateau 4

Definitions Dynamic Bayesian Network representation x k 3 x k 2 x k 1 x k x k+1 x k+2 z k 3 z k 2 z k 1 z k z k+1 z k+2 States X. = {x k } k=1,...,k Observations Z. = {z k } k=1,...,k Particle Filters for Visual Tracking T. Chateau 5

Online Tracking Online Tracking Observation z k 3 z k 2 z k 1 z k z k+1 z k+2 State x k 1 x k Particle Filters for Visual Tracking T. Chateau 6

Online Tracking Probabilistic framework p(z k X k ) Observation z k 3 z k 2 z k 1 z k z k+1 z k+2 State x k 1 x k p(x k 1 Z 0:k 1 ) p(x k Z 0:k ) Particle Filters for Visual Tracking T. Chateau 7

Sequential Monte-Carlo Inference p(x k 1 z k 1 ) Prediction (Chapman Kolmogorov) p(x k z k 1 ) Update (Bayes) p(x k z k ) p(x k x k 1 ) p(z k x k ) Dynamics Likelihood Particle Filters for Visual Tracking T. Chateau 8

Sequential Monte-Carlo Inference p(x k 1 z k 1 ) Prediction (Chapman Kolmogorov) p(x k z k 1 ) Update (Bayes) p(x k z k ) p(x k x k 1 ) p(z k x k ) Dynamics Likelihood Stochastic representation of distributions Particle Filters for Visual Tracking T. Chateau 9

Stochastic representation of distributions With a set of particles p(x) 1 N NX n=1 (X X n ) With a set of weighted particles p(x) NX n (X X n ) n=1 Particle Filters for Visual Tracking T. Chateau 10

Visual Tracking State (hidden variable) y bird view x Observation y x Particle Filters for Visual Tracking T. Chateau 11

Content Particle filtering: a probabilistic framework SIR particle filter MCMC particle filter RJMCMC particle filter Particle Filters for Visual Tracking T. Chateau 12

SIR Particle Filter Sampling Importance Resampling y x State distribution at time t-1 Resampling y x Particle Filters for Visual Tracking T. Chateau 13

SIR Particle Filter State distribution at time t-1 Prediction Particle Filters for Visual Tracking T. Chateau 14

SIR Particle Filter Predicted distribution at time t Update Posterior S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. A tutorial on particle filters for on-line non-linear/ non-gaussian bayesian tracking. IEEE Transactions on Signal Processing, 50(2):174 188, Feb. 2002. Particle Filters for Visual Tracking T. Chateau 15

SIR Particle Filter t=1 t=2 t t=3 Target Temporal Filtering t=4 Particle Filters for Visual Tracking T. Chateau 16

SIR Particle Filter: some examples State vector: 2D position and scale (image reference frame) Dynamics: random step Observation model: max. of gradients set of points T. Chateau and J. Lapresté. Robust real time tracking of a vehicle by image processing. In IEEE Intelligent Vehicles Symposium,, Parma, Italy, June 2004. Particle Filters for Visual Tracking T. Chateau 17

SIR Particle Filter: some examples State vector: 2D position and scale (image reference frame) Dynamics: random step Observation model: offline learning based model (Haar wavelets) T. Chateau, V. Gay-Belille, F. Chausse, and J. T. Lapresté. Real-time tracking with classifiers. In WDV - WDV Workshop on Dynamical Vision at ECCV2006, Grazz, Austria, May 2006. Particle Filters for Visual Tracking T. Chateau 18

SIR Particle Filter: some examples State vector: 3D position, orientation, steering angle and velocity Dynamics: driven by a bicycle model Observation model: background/foreground subtraction, camera and laser range finder Collaboration with LCPC Nantes Y. Goyat, T. Chateau, and L. Trassoudaine. Tracking of vehicle trajectory by combining a camera and a laser rangefinder. Springer MVA : Machine Vision and Application, online, March 2009. Particle Filters for Visual Tracking T. Chateau 19

SIR Particle Filter: some examples Particle Filters for Visual Tracking T. Chateau 20

Efficient implementation of SIR Particle Filters Particle Filters for Visual Tracking T. Chateau 21

Efficient implementation of Particle Filters SIR Particle Filter N / e d N: number of particles d : size of the state vector Particle Filters for Visual Tracking T. Chateau 22

Content Particle filtering: a probabilistic framework SIR particle filter MCMC particle filter RJMCMC particle filter Particle Filters for Visual Tracking T. Chateau 23

MCMC Particle Filter p(x k 1 Z k 1 ) Particle Filters for Visual Tracking T. Chateau 24

MCMC Particle Filter p(x k 1 Z k 1 ) Draw Particle Filters for Visual Tracking T. Chateau 24

MCMC Particle Filter p(x k 1 Z k 1 ) Draw Move X 0 Particle Filters for Visual Tracking T. Chateau 24

MCMC Particle Filter Markov Chain Monte Carlo X 0 Particle Filters for Visual Tracking T. Chateau 25

MCMC Particle Filter Markov Chain Monte Carlo Draw X 0 Particle Filters for Visual Tracking T. Chateau 25

MCMC Particle Filter Markov Chain Monte Carlo X Move Draw X 0 Particle Filters for Visual Tracking T. Chateau 25

MCMC Particle Filter Markov Chain Monte Carlo X Move Draw X 0? Compare Particle Filters for Visual Tracking T. Chateau 25

MCMC Particle Filter Markov Chain Monte Carlo X Move Draw X 0? Compare X 1 Accept X or copy X 0 Particle Filters for Visual Tracking T. Chateau 25

MCMC Particle Filter Markov Chain Monte Carlo X Move Draw X 0? Compare X 1 X N Accept X or copy X 0 Particle Filters for Visual Tracking T. Chateau 25

MCMC Particle Filter: marginal move Metropolis Hasting rule Particle Filters for Visual Tracking T. Chateau 26

MCMC Particle Filter: marginal move! =min 1,... t k... Metropolis Hasting rule Particle Filters for Visual Tracking T. Chateau 26

MCMC Particle Filter: example - State vector: 2D location, scale and combination parameters of observation modules (colour, texture, gradient,...) - Dynamics: random step - Observation function: learning based (Adaboost) - In collaboration with Teb-online Particle Filters for Visual Tracking T. Chateau 27

RJMCMC Particle Filter Used to track a varying number of objects X t. = {It, x 1 t,..., x I t t } The state is defined into a joint space Particle Filters for Visual Tracking T. Chateau 28

RJMCMC Particle Filter Reversible Jump Markov Chain Monte Carlo Approximate the distribution with a variable size A pair of new proposals X = {x 1,x 2,x 3 } X = {x 1,x 2 } Jump into a lower dimensional space X = {x 1,x 2,x 3,x 4 } Jump into a higher dimensional space Particle Filters for Visual Tracking T. Chateau 29

RJMCMC Particle Filter Proposals have to be add: - object position update - add one object - remove one object The state is defined into a joint space Data driven Particle Filters for Visual Tracking T. Chateau 30

RJMCMC Particle Filter Add an object: a data driven proposal Background/foreground hypothesis X New object position proposal map Background/foreground observation map Particle Filters for Visual Tracking T. Chateau 31

RJMCMC Particle Filter Real time pedestrian tracking Particle Filters for Visual Tracking T. Chateau 32

RJMCMC Particle Filter Real time vehicle tracking Particle Filters for Visual Tracking T. Chateau 33

Proposals: - update object position - add/remove one object - update object category RJMCMC Particle Filter Simultaneous tracking and categorisation One geometric and kinematic model for each category Particle Filters for Visual Tracking T. Chateau 34

Proposals: RJMCMC Particle Filter Simultaneous tracking and categorisation - update object position - add/remove one object - update object category category update proposal matrix Particle Filters for Visual Tracking T. Chateau 35

RJMCMC Particle Filter Simultaneous tracking and categorisation F. Bardet, T. Chateau, and D. Ramadasan. Unifying real-time multi-vehicle tracking and categorization. In Intelligent Vehicle Symposium, volume 1, 2009. Particle Filters for Visual Tracking T. Chateau 36

RJMCMC Particle Filter Simultaneous tracking, categorisation and context detection Proposals: - update object or sun position position - add/remove one object or sun - update object category Particle Filters for Visual Tracking T. Chateau 37

RJMCMC Particle Filter Simultaneous tracking and categorisation F. Bardet, T. Chateau, and J. Lapresté. Illumination aware mcmc particle filter for long-term outdoor multi-object simultaneous tracking and classification. In ICCV 2009, International Conference on Computer Vision, Tokyo, Japan, 09 2009. Particle Filters for Visual Tracking T. Chateau 38

RJMCMC Particle Filter Particle Filters for Visual Tracking T. Chateau 39

Efficient implementation of MCMC Particle Filters Multi Proposal MCMC Particle Filter X n 1 1 α π n 1 α X π X n π n X n 1 1 α π n 1 X1 π1 XP πp α X n π n X p π p p {1,..., P } (a): single proposal MCMC (b): P-proposal MCMC Particle Filters for Visual Tracking T. Chateau 40

t = 0 0.1 0.05 MCMC 1 PF Efficient implementation of Particle Filters 0.1 0.05 MCMC 2 PF 0.1 0.05 MCMC 4 PF t = 1 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 t = 2 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 t = 3 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 t = 4 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 0 0.1 0.05 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 Particle Filters for Visual Tracking T. Chateau 41

Conclusion Particle filters are widely used for temporal filtering applications They provide tools able to handle with non linear systems SIR particle filters have to be chosen for low dimensional problems MCMC particle filters with marginal proposal strategy are preferred for high dimensional problems RJMCMC particle filters can be used to manage states with a varying dimension Particle Filters for Visual Tracking T. Chateau 42