Graphical Models and Their Applications

Transcription

1 Graphical Models and Their Applications Tracking Nov 26, 2o14 Bjoern Andres & Bernt Schiele slides adapted from Stefan TU Darmstadt

2 Face Tracking Face tracking using color histograms and image gradients along contour: 2

3 Lane Tracking Lane tracking, e.g. for car navigation: 3

4 Ant Tracking Tracking is also very useful for facilitating behavioral research in animals. 4

5 Bee Tracking Tracking is also very useful for facilitating behavioral research in animals. 5

6 New Topic: Tracking Tracking is the problem of finding the motion of an object in an image sequence. Useful for: Animation & Interaction Navigation Recognition of objects Video surveillance Medical applications Computer assisted living Etc. 6

7 Tracking We typically distinguish 3 cases: Tracking rigid objects Tracking articulated objects, e.g. humans or animals Tracking fully non-rigid objects We will talk mostly about: Rigid objects and briefly touch upon: - Tracking articulated objects, specifically humans 7

8 Challenges Fast motions Changing appearance Changing object pose Dynamic backgrounds Lots of clutter Occlusions Poor image quality... 8

9 Simple Solution If the object we want to track is easy to find, we can just detect it in every frame ( Tracking by detection ). Example: Face tracking If there is only one person in the scene and is always in a frontal view, we could use a standard face detector to find it. But we don t want to search the entire image, because that is inefficient. Use the position from the previous frame to constrain search. [IBM Vision Group] 9

10 Gradient-based tracking To avoid the explicit search, we can also take a matching function (e.g. from SSD = sum of squared differences) and compute the gradient w.r.t. the object motion between frames. For example, SSD-matching against a simple template: Now we compute and (around the previous position). E SSD (x, y) = r,c I(x + c, y + r) T (c, r) 2 / x E SSD (x, y) / y E SSD (x, y) Then we can just go a step along the gradient (or solve for the motion explicitly) 10

11 Gradient-based tracking What is the hidden assumption behind all this? The object motion between frames is small. That is not a bad assumption in itself, but we don t have any explicit control over how large the motion may be. We need a more general way of incorporating the dynamics of the object 11

12 Typical Models of Dynamics Constant position: I.e. no real dynamics, but if the velocity of the object is sufficiently small, this can work. Constant velocity (possibly unknown): We assume that the velocity does not change over time. As long as the object does not quickly change velocity or direction, this is a quite reasonable model. Constant acceleration (possibly unknown): Also captures the acceleration of the object. This may include both the velocity, but also the directional acceleration. 12

13 Illustration Goal: Estimate car position at each time instant (say, of the red car). Observations: Image sequence and known background. [Michael Black] 13

14 Illustration Perform background subtraction. Obtain binary map of possible cars. But which one is the one we want to track? [Michael Black] 14

15 Bayesian Tracking Likelihood: noisy observation p(f G car = (x, y)) Prior: p(car = (x, y)) system state: car position observations: images Posterior: Bayesian update p(car = (x, y) F G) [Michael Black] 15

16 Notation x k R d : internal state at k-th frame (hidden random variable, e.g., position of the object in the image). X k = [x 1, x 2,..., x k ] T : history up to time step k z k R c : measurement at k-th frame (observable random variable, e.g. the given image). : history up to time step k Z k = [z 1, z 2,..., z k ] T [Michael Black] 16

17 Goal Estimating the posterior probability p(x k Z k ) How??? One idea: Recursion p(x k 1 Z k 1 ) p(x k Z k ) How to realize the recursion? What assumptions are necessary? [Michael Black] 17

18 Recursive Estimation p(x k Z k ) = p(x k z k, Z k 1 ) p(z k x k, Z k 1 ) p(x k Z k 1 ) p(z k x k ) p(x k Z k 1 ) p(z k x k ) p(x k, x k 1 Z k 1 ) dx k 1 Bayes rule: p(a b) = p(b a)p(a)/p(b) Assumption: p(z k x k, Z k 1 ) = p(z k x k ) Marginalization: p(a) = p(z k x k ) p(x k x k 1, Z k 1 ) p(x k 1 Z k 1 ) dx k 1 p(a, b) db p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 Assumption: p(x k x k 1, Z k 1 ) = p(x k x k 1 ) 18

19 Bayesian Formulation p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 p(x k Z k ) p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) posterior probability at current time step likelihood temporal prior posterior probability at previous time step normalizing term 19

20 Bayesian Graphical Model Hidden Markov model: x k 1 x k x k+1 z k 1 z k z k+1 Assumptions: p(z k x k, Z k 1 ) = p(z k x k ) p(x k x k 1, Z k 1 ) = p(x k x k 1 ) p(x k X k 1 ) = p(x k x k 1 ) 20

21 Estimators Assume the posterior probability p(x k Z k ) is known: posterior mean ˆx k = E(x k Z k ) maximum a posteriori (MAP) ˆx k = arg max xk p(x k Z k ) p(x k Z k ) [Michael Black] 21

22 General Model p(x k Z k ) can be an arbitrary, non-gaussian, multi-modal distribution. The recursive equation often has no explicit solution, but can be numerically approximated using Monte Carlo techniques. Special Case - Kalman filter [Kalman, 1960] If both likelihood and prior are Gaussian, the solution has closed form and the two estimators (posterior mean & MAP) are the same. The important restrictions of the Kalman filter are that it assumes linear state and output transformations, as well as Gaussian noise. There are many cases where this is inappropriate. We thus discuss only a more general version: Particle filtering more general recursive estimation technique. But also computationally much harder, and tricky to implement correctly... [Michael Black] 22

23 Multi-Modal Posteriors Measurement clutter in natural images causes likelihood functions to have multiple, local maxima. In a particular frame, the observation may be poor so that there are multiple promising looking locations. We cannot resolve these ambiguities until we have seen more data (additional frames). To do that, we have to allow for the posterior at each frame to be multi-modal. This rules out many parametric distributions, including the Gaussian. 23

24 Multi-Modal Posteriors posterior State (e.g. position) How can we represent the posterior at each time step in a flexible way that allows for: Multiple modes - To encode multiple promising locations. Varying number of modes - Modes may appear and disappear again when they are ruled out. 24

25 Non-Parametric Approximation We could sample at regular intervals. Instead of representing a continuous function, we approximate it using a discrete set of samples (or particles) each of which has a weight. S = We usually use normalized weights: N (x (i), w (i) ); i = 1,..., N i=1 w (i) = 1 25

26 Non-Parametric Approximation We could sample at regular intervals. Since there is no underlying parametric form, we call this a non-parametric representation or approximation. If needed, we can convert this back to a continuous density by assuming that each sample is represented by a small Gaussian mixture component: p(x) = w (i) N (x; x (i), 2 ) i - Note though that this is typically not necessary 26

27 Non-Parametric Approximation We could sample at regular intervals. Problems? Most samples have low weight wasted computation. How finely do we need to discretize? High dimensional space discretization impractical. 27

28 Non-Parametric Approximation Idea: Sample at irregular intervals and (optionally) weigh samples. weighted samples Weighted samples: S = (x (i), w (i) ); i = 1,..., N Normalized weights N i=1 w (i) = 1 28

29 Importance Sampling Approach: approximate expectation directly goal: E[f] = f(z)p(z) grid-sampling: discretize z-space into a uniform grid evaluate the integrand as a sum of the form: L E[f] f(z (l) )p(z (l) ) l=1 but: number of terms grows exponentially with number of dimensions 29

30 Importance Sampling Idea: use a proposal distribution q(z) from which it is easy to draw samples express expectation in the form of a finite sum over samples drawn from q(z) E[f] = 1 L with importance weights: f(z)p(z)dz = L l=1 p(z (l) ) q(z (l) ) f(z(l) ) r l = p(z(l) ) q(z (l) ) {z (l) } f(z) p(z) q(z) q(z)dz p(z) q(z) f(z) z 30

31 Importance Sampling typical setting: p(z) can be only evaluated up to a normalization constant (unknown): p(z) = p(z)/z p q(z) can be also treated in a similar fashion then: with: E[f] = Z q Z p 1 L r l = p(z(l) ) q(z (l) ) q(z) = q(z)/z q f(z)p(z)dz = Z q Z p L l=1 r l f(z (l) ) f(z) p(z) q(z) q(z)dz 31

32 Importance Sampling Ratio of normalization constants can be evaluated: Z p = 1 p(z) p(z)dz = Z q Z q q(z) q(z)dz 1 L and therefore: with: w l = E[f] r l m r m L l=1 = w l f(z (l) ) p(z (l) ) q(z (l) ) m p(z (m) ) q(z (m) ) L l=1 r l 32

33 How does this help us? Remember the filtering recursion: p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 We need to be able to compute integrals of the type: Monte-Carlo approximation: f(x) p(x) dx f(x) p(x) dx i f(x (i) ), x (i) p(x) 33

34 Monte-Carlo Approximation f(x) p(x) dx f(x (i) ), In other terms, the x (i) are a sample representation of the density p(x) What if we have a weighted sample representation? Just as easy... f(x) p(x) dx i x (i) p(x) Note however that in these cases the are usually not the same as before. i x (i) w (i) f(x (i) ) 34

35 Filtering Step-by-Step p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 We start with assuming that we have a weighted sample representation for the posterior p(x k 1 Z k 1 ) at the previous time step: S k 1 = (x (i) k 1, w(i) k 1 ); i = 1,..., N 35

36 Filtering Step-by-Step p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 We start with assuming that we have a weighted sample representation for the posterior p(x k 1 Z k 1 ) at the previous time step: S k 1 = (x (i) k 1, w(i) k 1 ); i = 1,..., N Use this to carry out Monte-Carlo integration: p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 i w (i) k 1 p(x k x (i) k 1 ) 36

37 Filtering Step-by-Step p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 Represent the approximation again using another particle set (we discuss in a minute how ): i w(i) k 1 p(x k x (i) k 1 ) Ŝ k 1 = (ˆx (i) k 1, ŵ(i) k 1 ); i = 1,..., N 37

38 Filtering Step-by-Step p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 Represent the approximation again using another particle set (we discuss in a minute how ): Ŝ k 1 = (ˆx (i) k 1, ŵ(i) k Take into account the likelihood by re-weighting the particles: x (i) k = ˆx (i) k 1 w (i) k = p(z k ˆx (i) k 1 )ŵ(i) k 1 S k = (x (i) k i w(i) k 1 ); i = 1,..., N 1 p(x k x (i) k 1 ), w(i) k ); 1,..., N 38

39 Filtering Step-by-Step p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 We obtain a weighted sample representation for the posterior at the current time step: p(x k Z k ) S k = (x (i) k, w(i) k ); 1,..., N 39

40 Filtering Step-by-Step p(x k Z k ) = p(z k x k ) p(x k x k 1 ) p(x k 1 Z k 1 ) dx k 1 We obtain a weighted sample representation for the posterior at the current time step: Remaining question: How do we represent using a sample set? i w(i) k S k = 1 p(x k x (i) k 1 ) (x (i) k p(x k Z k ), w(i) k ); 1,..., N 40

41 Temporal Propagation The simplest way to deal with the problem of representing the result of the Monte-Carlo integration is to propagate each sample independently according to the temporal dynamics and keeping the weight: i w(i) k Problem: Sample Impoverishment Solution: Resampling 1 p(x k x (i) k 1 ) ˆx (i) k 1 p(x k x (i) k 1 ) ŵ (i) k 1 = w (i) k 1 41

42 Resampling Given weighted sample set S k 1 = (x (i) k 1, w(i) k 1 ); i = 1,..., N Draw unweighted samples by sampling from the weight distribution: sample u U(0, 1) N Cumulative distribution of weights 42

43 Particle Filter Posterior p(x k 1 Z k 1 ) Isard & Blake 96 43

44 Particle Filter Posterior p(x k 1 Z k 1 ) resample Isard & Blake 96 44

45 Particle Filter Posterior p(x k 1 Z k 1 ) resample Apply temporal dynamics p(x k x k 1 ) Isard & Blake 96 45

46 Particle Filter Posterior p(x k 1 Z k 1 ) resample Apply temporal dynamics p(x k x k 1 ) reweight Likelihood p(z k x k ) Isard & Blake 96 46

47 Particle Filter Posterior p(x k 1 Z k 1 ) resample Apply temporal dynamics p(x k x k 1 ) reweight Likelihood p(z k x k ) normalize Posterior p(x k Z k ) Isard & Blake 96 47

48 [Michael Isard] 48

49 Some Properties It can be shown that in the infinite particle limit this converges to the correct solution [Isard & Blake]. In practice, we of course want to use a finite number. In low-dimensional spaces we might only need 100s of particles for the procedure to work well. In high-dimensional spaces sometimes 1000s or even 10000s particles are needed. There are many variants of this basic procedure, some of which are more efficient (e.g. need fewer particles) See e.g.: Arnaud Doucet, Simon Godsill, Christophe Andrieu: On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing, vol. 10, pp ,

50 Contour Tracking State: control points of splinebased contour representation Measurements: edge strength perpendicular to contour Dynamics: 2 nd order Markov (often learned) [Isard & Blake, Condensation - conditional density propagation for visual tracking. IJCV, 1998] 50

51 High-Dimensional State Spaces Tracking a hand (high-dimensional state space) [Michael Isard] 51

52 Tracking in Clutter Tracking a leaf that moves fast in a very cluttered scene: [Michael Isard] 52

53 Tracking the unpredictable They called it dance [Michael Isard] 53

54 Outlook: Articulated Tracking So far, we have seen relatively simple tracking applications where the state vector is often just an image position. Can we do more equipped with the tools we have learned? Articulated tracking is one such advanced example: Here, we want to track the position and the configuration of an articulated object, i.e. an object that consists of several parts that can move (somewhat) independently. Most prominent example: Human tracking. 54

55 Motion Capture - Mocap Cameras CMU Mocap lab

56 Motion Capture - Mocap Motion capture of human subjects has a very wide range of applications. Most prominent: Entertainment industry Capture humans to animate digital characters. But also: Health (e.g. analysis of gait) Sports Etc. 56

57 Examples Tom Hanks in The Polar Express From IGN 57

58 Examples Movie based on Mocap: Beowolf From NY Times 58

59 Marker-Based Mocap Put many reflective markers on the person to be tracked. Record person from different viewpoints simultaneously. Triangulate 3D position of each marker (like in stereo). Convert marker positions into a body model. There are robust (but expensive) commercial systems available: e.g. from Vicon. 59

60 Body Model: Kinematic Tree Cameras CMU Mocap lab Tree rooted at torso Global position and orientation Other body parts Relative orientation to parent

61 Marker-Less Mocap Problems with marker-based mocap: Tedious. Many markers have to be placed, they often fall off, etc. Only works in lab setting. Cannot easily be applied in everyday environments. Marker-less mocap: Use tracking techniques from computer vision to perform mocap without the need of markers. Not as robust yet as marker-based tracking, but getting there. Ultimate goal: only 1 camera This is very difficult. Most accurate approaches: Several (4-6) cameras 61

62 Where do we want to go (for now)? From Leonid Sigal 62

63 Particle Filter Approach In order to apply the particle filter to this problem, we need to define an observation likelihood and a dynamic model. Observation likelihood: Often based on background subtraction p(bg pixel limb location and orientation) p(edge filter response limb edge location and orientation) From Leonid Sigal after [Deutscher et al, 00] 63

64 Particle Filter (a) Approach Figure 2. Walking GPLVM: Learned from 1 gait cycle from each of 6 subjects. Plots show side and top views of the 3D latent space. Circles and arrows denote latent positions and temporal sequence. In order to apply the particle filter to this problem, we need to define an observation likelihood and a dynamic Speed model. Variation: Dynamical model Wide range from simple Gaussian priors to complex motion models for walking, golf swings, etc. (a) (c) Figure 3. Walking GPDM: Balanced GPDM learned from 1 gait cycle from 6 subjects. (a,b) Side and top views of 3D latent space. (c) Volumetric Graphical visualization Models of and reconstruction Their Applications variance. - Nov 25, (d) 2o14 Green (d) (b) (b) Figure 4. Speed Variation: 2D models learned fo jects. Each one walking at 9 speeds ranging from points are latent positions of training poses. Int tional to 2 ln σ y x,x,y, β, so brighter regions h reconstruction variance. The subject on the left that on the right has a knee pathology and walks Fig. 4 shows 2D GPDM two subjects, each of which walked four gai of 9 speeds between 3 and 7km/h (equispace latent trajectories are approximately circular by speed; the innermost and outermost tra spond to the slowest and fastest speeds resp estingly, the subject on the left is healthy w on right has a knee pathology. As the trea creases, the side of the body with the path the motion at slower speeds to avoid pain, a side of the gait cycle must speed up to main This explains the anisotropy of the latent sp 3.4. Prior over New Motions The GPDM also defines a smooth prob over new motions (Y, X ). That is, just multiple sequences above, we write the joi the concatenation of the sequences. The con of the new sequence is proportional to the jo with the training data and latent positions he Model of a human walking cycle from [Urtasun et al. 06] p(x, Y X, Y, ᾱ, β) p( [X, X ], [Y, Y This density can also be factored to provide 64

65 Some Results (Annealed) Particle filtering with a walking prior From Leonid Sigal after [Deutscher et al, 00] 65

66 Challenges Yet, the human tracking problem is far from being solved. There are many difficult challenges: Self-occlusion Occlusion by other objects or other people Fast motions Complex and cluttered backgrounds Etc. 66

67 Current Approaches Graphical models and belief propagation [Sigal et al.] And many many more ψ ψ 21 ψ 43 4 time 7 ψ Frame 7 Frame 12 Frame 17 Frame 22 Figure 7: Loose-limbed tracking. The body model whose automatic initialization is shown in Figure 6 is tracked for a further 25 frames. Four sample frames are shown from 2 of the 4 camera views used. [4] J. Deutscher, M. Isard and J. MacCormick. Automatic camera [16] Moghaddam B. and Pentland A., Probabilistic Visual Learn- calibration from a single manhattan image, ECCV, pp. ing for Object Representation, PAMI 19(7): , , [17] V. Pavolvić, J. Rehg, T-J. Cham and K. Murphy. A dynamic [5] J. Deutscher, A. Blake and I. Graphical Reid. Articulated Models body and mo- Their Applications Bayesian - Nov network 25, 2o14 approach to fi gure tracking using learned

68 Pose Prediction from a Single Frame Even more challenging... [Sminchisescu et al., 07] 68

69 Multi-People Tracking [Andriluka et al. 2010] 69

70 Summary Particle filtering is a very general tool for temporal inference that we can exploit for tracking. Nonetheless, it applies in a variety of other applications as well. It has problems in high-dimensional spaces, however, but there are a number of variants that alleviate some of these issues. Human tracking ( Marker-less mocap ) can be performed using particle filtering: Wide range of applications, especially in entertainment. Only a small part of the problems is solved to date. 70

71 For those interested in the Kalman filter: Greg Welch and Gary Bishop: An Introduction to the Kalman Filter Background for Particle Filtering: Simon Maskell and Neil Gordon: A Tutorial on Particle Filtering for On-Line Nonlinear/Non-Gaussian Bayesian Tracking 71