Extended target tracking using PHD filters

Transcription

1 Ulm University (35) With applications to video data and laser range data Division of Automatic Control Department of Electrical Engineering Linöping university Linöping, Sweden

2 Presentation outline 2(35) 1. Introduction to target tracing using PHD filters Single target Multiple targets PHD filters Illustrative example: Pedestrian tracing in video data 2. Tracing of extended/group targets Definition Gamma-Gaussian-inverse Wishart-model 3. Applications Video: group target tracing Laser: pedestrian and bicycle tracing

3 Introduction to target tracing using PHD filters Single target Multiple targets PHD filters Illustrative example: Pedestrian tracing in video data Publicly available data: PETS 2012 dataset S1: Person count and density estimation. Frame 50 Frame 100 Frame 150 3(35)

4 Single target tracing 4(35) Process sequence of detections z to maintain estimate of target state x. observation space z z+1 state space x target motion x+1 Above: Target state x is, e.g., position and velocity in image. Above: Red rectangles are the detections z.

5 Single target tracing 4(35) Process sequence of detections z to maintain estimate of target state x. observation space z z+1 state space x target motion x+1 Recursive Bayesian filter propagates full distribution:... p ( x z ) p p ( x+1 z ) c p ( x +1 z +1 )... where z = {z 0, z 1,..., z }

6 Single target tracing 4(35) Process sequence of detections z to maintain estimate of target state x. observation space z z+1 state space x target motion x+1 E.g. constant gain Kalman filter propagates 1 st order moment:... ˆx p ˆx+1 c ˆx where ˆx = E [ x z ]. Simpler, but computationally cheaper!

7 Multiple target tracing 5(35) Processing of multiple detections z (i) from multiple sources to maintain estimates of the targets states x (j). x (1) z (2) z (1) z (3) z (4) x (2) x (3) target motion z (1) +1 z (3) +1 x (1) +1 z (5) +1 z (2) +1 observation space x (2) +1 x (3) +1 x (4) +1 z (6) +1 z (4) +1 state space Above: Target states x (j) are, e.g., positions and velocities. Above: Red rectangles are the detections z (i).

8 Multiple target tracing 5(35) Processing of multiple detections z (i) from multiple sources to maintain estimates of the targets states x (j). x (1) z (2) z (1) z (3) z (4) x (2) x (3) target motion z (1) +1 z (3) +1 x (1) +1 z (5) +1 z (2) +1 observation space x (2) +1 x (3) +1 x (4) +1 z (6) +1 z (4) +1 state space Classic methods: Require solution to data association problem. 1. Which detection belong to which target? 2. Are there any false detections?

9 Multiple target tracing 5(35) Processing of multiple detections z (i) from multiple sources to maintain estimates of the targets states x (j). x (1) z (2) z (1) z (3) z (4) x (2) x (3) target motion z (1) +1 z (3) +1 x (1) +1 z (5) +1 z (2) +1 observation space x (2) +1 x (3) +1 x (4) +1 z (6) +1 z (4) +1 state space Alternative: Model with Random Finite Sets. No data association needed.

10 Random Finite Set (RFS) 6(35) A Random Finite Set X is a random variable Set cardinality X = N x, is a random variable. Each set memeber x (j) X 0 is a random variable. Often X 0 = R n x.

11 Random Finite Set (RFS) 6(35) A Random Finite Set X is a random variable Set cardinality X = N x, is a random variable. Each set memeber x (j) X 0 is a random variable. Often X 0 = R n x. Instantiations X = { X = x (j) X is without order, i.e. } Nx,, j=1 x(j) R n x for j = 1,..., N x, { } { } x (1), x (2) = x (2), x (1)

12 Multiple target tracing using RFS models 7(35) Processing of sequence of RFS of detections Z from set of sources to maintain estimate of RFS of target states X. observation space Z Z+1 state space X meta target motion X+1 RFS of targets X. Above: the pedestrians RFS of detections Z. Above: the detection windows Single target set, single detection set No data association needed

13 Recursive multi state Bayes filter 8(35) Propagate the full set distribution... p ( X Z ) p p ( X+1 Z ) c p ( X +1 Z +1 )... observation space observation space Z Z+1 z z+1 state space state space X meta target motion X+1 x target motion x+1 Multiple target equivalent to single target Bayes filter,... p ( x z ) p p ( x+1 z ) c p ( x +1 z +1 )...

14 Recursive multi state Bayes filter 8(35) Propagate the full set distribution... p ( X Z ) p p ( X+1 Z ) c p ( X +1 Z +1 )... observation space observation space Z Z+1 z z+1 state space state space X meta target motion X+1 x target motion x+1 Multiple target equivalent to single target Bayes filter,... p ( x z ) p p ( x+1 z ) c p ( x +1 z +1 )... Computationally intractable to implement due to set integrals. Alternative: Use PHD or CPHD filter instead.

15 The Probability Hypothesis Density 9(35) Expected value: 1 st order moment of single target pdf p ( x z ) The probability hypothesis density (PHD): 1 st order moment of multi-target pdf p ( X Z ). Defined on single target states x X 0, denoted D ( x Z ) The PHD is an intensity function, not a pdf.

16 The Probability Hypothesis Density 10(35) The peas in the intensity correspond to liely target locations. Expected number of targets in any region S X 0 is D (x ) Z dx S = X 0 gives xpected total number of targets S D (x) D +1 (x) x y [m] x [m] y [m] x [m]

17 The PHD filter 11(35) The PHD filter propagates the PHD in time... D (x Z) p D +1 (x Z) c D (x Z)... PHD filter is analogous to constant gain Kalman filter,... ˆx p ˆx+1 c ˆx Two types of implementations: 1. Sequential Monte Carlo PHD approximations. 2. Distribution Mixture PHD approximations, D (x Z ) = J w (j) j=1 ( Common choice: p (j) (x Z) = N ( p(j) x x; ζ (j) ζ (j) ) ), ζ (j) ( = m (j) ), P(j)

18 Tracing of extended/group targets 12(35) 1. Introduction to target tracing using PHD filters Single target Multiple targets PHD filters Illustrative example: Pedestrian tracing in video data 2. Tracing of extended/group targets Definition Gamma-Gaussian-inverse Wishart-model 3. Applications Video: group target tracing Laser: pedestrian and bicycle tracing

19 Tracing of extended/group targets 13(35) Point target assumption: 1 detection per target per time step. Typical in early RADAR-airplane-tracing.

20 Tracing of extended/group targets 13(35) Point target assumption: 1 detection per target per time step. Typical in early RADAR-airplane-tracing. Modern sensors typically have higher resolution, and can give multiple detections per target. A group of targets will give multiple detections per group. Definition: Extended/group targets are targets that potentially give rise to more than one detection per time step.

21 Gamma-Gaussian-inverse Wishart-model 14(35) For each group we estimate the group state ξ = (x, X, γ ) Gaussian distributed random vector x, p (x ) Z = N ( ) x ; m, P

22 Gamma-Gaussian-inverse Wishart-model 14(35) For each group we estimate the group state ξ = (x, X, γ ) x : position and inematics Gaussian distributed random vector x, p (x ) Z = N ( ) x ; m, P

23 Gamma-Gaussian-inverse Wishart-model 14(35) For each group we estimate the group state ξ = (x, X, γ ) x : position and inematics X : spatial extension, i.e. size and shape Inverse Wishart distributed random matrix X (group shape = ellipse), p (X ) Z ( ) = IW d X ; v, V

24 Gamma-Gaussian-inverse Wishart-model 14(35) For each group we estimate the group state ξ = (x, X, γ ) x : position and inematics X : spatial extension, i.e. size and shape γ : number of detections, related to number of individuals in group 1 8 Gamma distributed random variable γ (Poisson rate), p (γ ) Z = G ( ) γ ; α, β

25 Gamma-Gaussian-inverse Wishart-model 14(35) For each group we estimate the group state ξ = (x, X, γ ) x : position and inematics X : spatial extension, i.e. size and shape γ : number of detections, related to number of individuals in group 1 8 Group state ξ is GGIW distributed, p (ξ ) Z = GGIW ( ) ξ ; ζ ζ = (m, P, v, V, α, β)

26 Group tracing using a GGIW-PHD filter 15(35) Problem: Trac pedestrian groups as they move along footpath. Estimate: position, size/shape, number of persons. GGIW-PHD filter: the PHD is approximated by a GGIW mixture, J D (ξ Z ) = (j) (j) GGIW w ξ ; ζ j=1

27 Applications 16(35) 1. Introduction to target tracing using PHD filters Single target Multiple targets PHD filters Illustrative example: Pedestrian tracing in video data 2. Tracing of extended/group targets Definition Gamma-Gaussian-inverse Wishart-model 3. Applications Video: group target tracing Laser: pedestrian and bicycle tracing

28 Group target tracing approach 17(35) 1: For each image, use pedestrian detector 2: Project each detection onto ground plane 3: Use GGIW-PHD filter to trac the groups 4: Results visualization: project filter output bac into images

29 Group target tracing approach 18(35) 1: For each image, use pedestrian detector

30 Group target tracing approach 18(35) 2: Project each detection onto ground plane 10 Road edge Field of view Detections 5 0 y [m] x [m]

31 Group target tracing approach 18(35) 3: Use Extended/Group target PHD filter to trac the groups 10 Road edge Field of view Detections y [m] x [m]

32 Group target tracing approach 18(35) 4: Visualization: project filter output bac into images

33 Group target tracing approach 18(35) 4: Visualization: projection is group foot-print

34 Group target tracing results movie 19(35)

35 Group target tracing results 20(35) 10 Ellipse is a good approximation of the group shape ˆγ gives a lower bound for the number of pedestrians in the group A model of occlusion from stationary objects, e.g. the lamp post, could improve results.

36 The laser range sensor 21(35) Consider an urban scene: cars, bies, pedestrians.

37 The laser range sensor 21(35) Consider an urban scene: cars, bies, pedestrians. Measures distance to closest object at given angles, relatively low sensor noise.

38 The laser range sensor 21(35) Consider an urban scene: cars, bies, pedestrians. Measures distance to closest object at given angles, relatively low sensor noise. Measurements z (red ) along the parts of the target surface that are visible to sensor.

39 The laser range sensor 21(35) Consider an urban scene: cars, bies, pedestrians. Measures distance to closest object at given angles, relatively low sensor noise. Measurements z (red ) along the parts of the target surface that are visible to sensor. Car rectangle. Pedestrian ellipse. Bie line.

40 The laser range sensor: challenges 22(35) Occlusion: objects behind other objects are invisible to the sensor. Fix: non-homogeneous detection probability.

41 The laser range sensor: challenges 22(35) Occlusion: objects behind other objects are invisible to the sensor. Fix: non-homogeneous detection probability. Time changing measurement appearance: depending on orientation measurement appearances is different. Fix: multiple measurement models.

42 Pedestrian tracing: data with occlusion 23(35) Experiment data from SICK laser sensor Multiple human targets, at most 3 at the same time. Measurements of stationary objects removed beforehand y [m] x [m] Occlusion handled with non-homogeneous detection probability.

43 Pedestrian tracing: variable detection probability 24(35) Occlusion handled with non-homogeneous detection probability. Low p D behind predictions. Gaussian smoothing on edges. y Similar idea used in x Reuter, Dietmayer, Pedestrian tracing using random finite sets, FUSION

44 Pedestrian tracing: GIW-PHD filter results 25(35) y [m] x [m] Sum of weights Time Occlusion handled by non-homogeneous detection probability. Extension estimate maes it possible to handle partial occlusion.

45 Pedestrian tracing: GIW-PHD filter results 26(35) y [m] x [m]

46 Time changing measurement appearance 27(35) Most extended targets have constant extensions (orientation may change over time). We consider constant extension and time changing appearance, especially abrupt changes. Observability of state variables may change with appearance.

47 Time changing measurement appearance 27(35) Most extended targets have constant extensions (orientation may change over time). We consider constant extension and time changing appearance, especially abrupt changes. Observability of state variables may change with appearance. Example: Bicycles measured by a laser mounted at pedal height

48 y Bicycle tracing: measurement appearance 28(35) Measurement appearance varies significantly x Along bie length

49 y y Bicycle tracing: measurement appearance 28(35) Measurement appearance varies significantly x Along bie length x Around pedals

50 y y y Bicycle tracing: measurement appearance 28(35) Measurement appearance varies significantly x Along bie length x In-between x Around pedals

51 y y y Bicycle tracing: measurement appearance 29(35) Measurement appearance varies significantly x x x Along bie length In-between Around pedals Two simple cases can be identified, 1. line shaped measurements, 2. point clusters, and ambiguous cases in-between the two. Assume geometric shape: bie line with constant length Use MM-PHD filter with a point model and a line model.

52 y y y Bicycle tracing: single target results 30(35) y y x Measurements x Proposed MM approach x MM approach x Only L model x Only P model Legend: CT, P model CV, P model CT, L model CV, L model

53 y y y y Bicycle tracing: single target results 31(35) x Measurements x MM approach Legend: CT, P model CV, P model CT, L model CV, L model x Only L model x Only P model

54 Bicycle tracing: multiple target results & Summary 32(35) y y x x Legend: CT, P model CV, P model CT, L model CV, L model

55 Bicycle tracing: multiple target results & Summary 32(35) y y x x In general: estimated mode corresponds to expectation/intuition. Only P model wors in most cases, however heading/orientation is more uncertain. Only L model sometimes fails during turns.

56 Bicycle tracing: multiple target results & Summary 32(35) y y x x In general: estimated mode corresponds to expectation/intuition. Only P model wors in most cases, however heading/orientation is more uncertain. Only L model sometimes fails during turns. Using both models generally superior. In MM case L model can aid in detecting CT maneuvers.

57 Bicycle tracing: MM-PHD filter movie 33(35)

58 Summary 34(35) Brief introduction to PHD filters. Extended/group objects appear in many modern sensors. PHD filter adaptations: Non-homogeneous detection probability Multiple models Experimental results using video and laser Groups of pedestrians Pedestrians Bicycles List of publications, with pdf:s, can be found at

59 Than you for listening! Any questions? List of publications, with pdf:s, can be found at