Methods in Computer Vision: Mixture Models and their Applications

Transcription

1 Methods in Computer Vision: Mixture Models and their Applications Oren Freifeld Computer Science, Ben-Gurion University May 7, 2017 May 7, / 40

2 1 Background Modeling Digression: Mixture Models GMM for Background Modeling 2 GMMs in Computer Vision Mixture Models for Optical Flow Spatio-intensity Model Mixtures over Surface Normals May 7, / 40

3 Background Models Background Modeling Background models are very useful for tasks such as object detection/segmentation and motion analysis. We will restrict discussion to a single stationary camera. May 7, / 40

4 Examples Background Modeling May 7, / 40

5 Examples Background Modeling May 7, / 40

6 Examples Background Modeling May 7, / 40

7 Examples Background Modeling May 7, / 40

8 Examples Background Modeling May 7, / 40

9 Examples Background Modeling May 7, / 40

10 Background Models Background Modeling Two main approaches: A global approach, e.g.: Linear models asuch as PCA and Robust PCA we already discussed this. A local (pixel-level) approach, e.g.: a pixel-wise temporal mean/median/robust-mean a pixel-wise probability distribution model, often using a mixture model May 7, / 40

11 Mixture Models Background Modeling Digression: Mixture Models More generally (than Computer Vision), mixture models are usually used for two types of statistical tasks: density 1 estimation and clustering. A (parametric) mixture model can be written as: p(x) = p(x; θ) = z p(x, z; θ) = K p(x, z = j; θ) j=1 K = p(x z = j; θ) Pr(z = j θ) }{{}}{{} j=1 component j w j K z {1,..., K} w j = 1 w j > 0 j {1,..., K} j=1 Conceptually, the generative process is sample z from the probability mass function, (w j ) K j=1, and then sample x p(x z = j; θ) 1 In effect, pdf May 7, / 40

12 Mixture Models Background Modeling Digression: Mixture Models A central example of mixture models is the Gaussian Mixture Model (GMM), AKA Mixture of Gaussians (MoG). In effect, p(x z = j; θ) = N (x; µ j, Σ j ): p(x) = p(x; θ) = z = K w j p(x z = j; θ) = j=1 p(x, z; θ) = K p(x, z = j; θ) j=1 K w j p(x z = j; θ j ) = j=1 K w j N (x; µ j, Σ j ) j=1 where θ j = (µ j, Σ j ) and θ = (θ 1,..., θ K, w 1,..., w K ) May 7, / 40

13 Mixture Models Background Modeling Digression: Mixture Models N: number of measurements K: number of components x i : measurement i z i {1,..., K}: the latent label (measurement-to-component association) Typically, it is assumed that the x i s are iid samples from the mixture. Given observations (x 1,..., x N ), the clustering task is to find z (z 1,..., z N ) Z {1,..., K} N while the density-estimation task is to find θ. Typically, regardless which task is of interest, most approaches alternate between inferring θ given z and inferring z given θ. May 7, / 40

14 GMM and K-Means Background Modeling Digression: Mixture Models We will see how the K-means algorithm can be derived as a particular inference algorithm under certain assumptions on the GMM parameters. In that setting, θ = (µ 1,..., µ K ), we only care about z, and the cost function is F (θ, z) = F (µ 1,..., µ K, z 1,..., z N ) = N K 1 zi =j x i µ j 2 (1) i=1 j=1 May 7, / 40

15 GMM and K-Means Background Modeling Digression: Mixture Models The K-means algorithm alternates between: and ˆθ = ( ˆµ 1,..., ˆµ K ) = arg min (µ 1,...,µ K ) = arg min K (µ 1,...,µ K ) j=1 i:ẑ i =j N K i=1 j=1 1ẑi =j x i µ j 2 (2) x i µ j 2 (3) ẑ = (ẑ 1,..., ẑ N ) = arg min z 1,...,z N = arg min z 1,...,z N N x i ˆµ zi 2 i=1 N i=1 j=1 K 1 zi =j x i ˆµ j 2 (4) May 7, / 40

16 GMM and K-Means Background Modeling Digression: Mixture Models Note that, equivalently, we can work on the smaller problems: For each cluster j = 1,..., K, compute: ˆµ j = arg min x i µ j 2 = arg min x i µ 2 µ j µ i:z i =j i:z i =j = 1 x i n j = {i : z i = j} (5) n j i:z i =j and for each data point x i, i = 1,..., N, compute: ẑ i = arg min x i ˆµ zi 2 = arg min x i ˆµ j 2 z i j (6) For nice demos, see May 7, / 40

17 Mixture Models Background Modeling Digression: Mixture Models For now, we will hold off with a more detailed discussion on how, given data, we can infer the parameters or the labels under the assumption of a mixture model; rather, we will now focus on applications, starting with background modeling. May 7, / 40

18 Background Modeling Pixelwise Adaptive GMM GMM for Background Modeling Model the intensity of a each pixel using a GMM. Adapt GMM parameters on the fly. May 7, / 40

19 Background Modeling GMM for Modeling Pixel Intensity GMM for Background Modeling Consider the gray-scale case. I(x, t): intensity of pixel x at frame t p(i(x, t)) = p(i(x, t); θ(x, t)) = w j (x, t) > 0 K w j (x, t)n (I(x, t); µ j (x, t), σ j (x, t)) j=1 K w j (x, t) = 1 µ j (x, t) R σ j (x, t) > 0 j=1 θ(x, t) = (µ j (x, t), σ j (x, t), w j (x, t)) K j=1 ( 1 N (x; µ, σ) exp 1 (x µ) 2 ) 2πσ 2 2 σ 2 May 7, / 40

20 Background Modeling GMM for Modeling Pixel Intensity GMM for Background Modeling For RGB images, I(x, t) = [ r(x, t) g(x, t) b(x, t) ] T, can either assume independent channels, p(i(x, t); θ(x, t) R, θ(x, t) G, θ(x, t) B ) = 1D GMM 1D GMM 1D GMM {}}{{}}{{}}{ p(r(x, t); θ R (x, t)) p(g(x, t); θ G (x, t)) p(b(x, t); θ B (x, t)) or, more generally, p(i(x, t); θ(x, t)) = K w j (x, t)n (I(x, t); µ j,t, Σ j,t ), j=1 May 7, / 40

21 Background Modeling [Stauffer and Grimson, CVPR 99] GMM for Background Modeling An often-effective heuristic that combines adaptive GMM parameter estimation with classification of the GMM components to background and non-background components. May 7, / 40

22 Background Modeling GMM for Background Modeling Stauffer and Grimson s Method in Broad Sweeps Need to decide which of the GMM are background (BG) components and which are foreground/noise. The suggested heuristic seeks BG components of high weight and low variance. If I(x, t) µ j (x, t) < 2.5σ j (x, t), declare Gaussian j as matched. 0 number of matched Gaussians K For a matched Gaussian: increase w j (x, t), adjust µ j (x, t) toward I(x, t), decrease σ j (x, t). E.g.: µ j (x, t) new = αµ j (x, t) + (1 α)i(x, t) where α controls the learning rate. For an unmatched Gaussian: decrease w j (x, t) If all Gaussians (at pixel x, at frame t) are unmatched: Mark I(x, t) as foreground pixel Pick the least probable Gaussian and: set its mean to I(x, t), set its variance to be high, assign low weight May 7, / 40

23 Background Modeling GMM for Background Modeling There are also other GMM-based (or, more generally, mixture-based) methods for BG modeling. May 7, / 40

24 OpenCV Examples Background Modeling GMM for Background Modeling Video frame For more details, cf. May 7, / 40

25 OpenCV Examples Background Modeling GMM for Background Modeling Result of BackgroundSubtractorMOG For more details, cf. May 7, / 40

26 OpenCV Examples Background Modeling GMM for Background Modeling Result of BackgroundSubtractorMOG2 (gray shadows) For more details, cf. May 7, / 40

27 OpenCV Examples Background Modeling GMM for Background Modeling Result of BackgroundSubtractorGMG (GMG=the authors initials) For more details, cf. May 7, / 40

28 GMMs in Computer Vision Mixture Models in Computer Vision Applications More generally (than BG modeling), there are numerous CV applications involving a GMM in particular, or mixture models in general. May 7, / 40

29 GMMs in Computer Vision Mixture Models for Optical Flow GMM for Optical Flow 3 v y Figure 2.1: Constraint lines from dierently moving surfaces within an aperture give rise to distinct \clusters" of constraint line intersections. intersections can be observed. If we assume a single translational motion over the region the Recall \optimal" the motion gradient-constraint estimate will lie somewhere equation: between the two actual motions and will result in a blurring of the ow eld at the motion boundary. A similar situation results in cases of multiple transparenti motions x (x i, t)u and+ fragmented I y (x i, t)v occlusion + I t (x i where, t) = no0 clear surface boundary exists. These situations result in exactly the same type of clusters of constraints. To cope with situations such as this we relax the single motion assumption in two ways. The First, authors we assumeuse thata agmm region may over contain the observed multiple coherent (I x (x i motions., t), I y (xwe i, can t), Ithink t (x, of t)) s. these multiple motions as corresponding layers [15] whose spatial extent may be the entire region. Figure from Jepson and Black, 1993 Each layer contains a single consistent motion and each layer may be described by dierent v x May 7, / 40

30 GMM for Optical Flow GMMs in Computer Vision Mixture Models for Optical Flow Besides application to motion segmentation, this method has nice 14 application for in images containing mirror reflections. a b c Figure 4.5: Where's Jim? Transparency sequence containing the reection of a mystery vision researcher (see text). 5 Conclusion We have examined the problems posed by multiple motions and outliers in the intergration of motion information over a spatial neighborhood. We relax the assumption of a single motion and, instead, view image regions as containing multiple layers corresponding to surfaces with dierent image motions. We also cope with outliers which can decease the accuracy of the recovered ow. To achieve this we introduced the idea of using mixture models May 7, / 40 Figure from Jepson and Black, 1993

31 Spatio-intensity GMM GMMs in Computer Vision Spatio-intensity Model N: number of pixels K: number of components x i = (x i, y i ): location of pixel i a i R d : some attribute of pixel i; e.g., color (d = 3). f i = (x i, a i ) R d+2 z i {1,..., K}: the latent label (pixel-to-component association) Usually there is some scaling parameter, which is either user-defined or estimated, between the spatial coordinates and the attributed. In effect, f i = (sx i, a i ). In what follows, our notation will assume s was already absorbed into the x i s. May 7, / 40

32 Spatio-intensity GMM GMMs in Computer Vision Spatio-intensity Model {f i } N i=1 are modeled as IID samples from a GMM: f i p(f i ; θ) = K p(f i, z i ; θ) = p(f i, z i = j; θ) z i = = j=1 j=1 K K Pr(z i = j; θ)p(f i z i = j; θ) = Pr(z i = j; θ)p(f i z i = j; θ j ) K w j N (f i ; µ j, Σ j ) j=1 where θ j = (µ j, Σ j ) and θ = (θ 1,..., θ K, w 1,..., w K ). Usually: [ µ a ] [ µ j = j Σ a ] Σ j = j 0 2 d µ x j j=1 0 d 2 Σ x j p(f i z i = j; θ j ) = N (x i ; µ x j, Σ x j )N (a i ; µ a j, Σ a j ). Important: x i a i z i but x i a i May 7, / 40

33 Spatio-intensity GMM GMMs in Computer Vision Spatio-intensity Model Given observations (f 1,..., f N ), the clustering task is to find z (z 1,..., z N ) Z {1,..., K} N while the density-estimation task is to find θ. Note that here, in the context of images, the clustering is in fact image segmentation, while the density estimation implies a low-dimensional probabilistic representation the image using a GMM. May 7, / 40

34 GMMs in Computer Vision Spatio-intensity Model Spatio-intensity GMM for Image Segmentation and Content- Based Image Indexing and Retrieval 5 Fig. 2. Sample image segmentations. Figure from [Carson et al., 1999.] May 7, / 40

35 GMMs in Computer Vision Spatio-intensity Model Spatio-intensity GMM for Image Segmentation Example: superpixels This method, Simple Linear Iterative Clustering (SLIC), is based on K-Means, followed by some morphological heuristics to eliminate holes. Figure from [Achanta et al., 2012] May 7, / 40

36 stitute of Technology Spatio-intensity GMM for Image Segmentation tificial Intelligence Laboratory GMMs in Computer Vision Spatio-intensity Model Example: superpixels ] s n Fig.Connectivity-Constrained 1: Example results using thethere proposed superpixel In this GMM is a twist: the labelsmethod are t identically distributed, but not independent (since the model explicitly - respects topological constraints) K = N/225 Image size: N = Image size: N = e Turbopixels 3 Figure Freifeld, Li and 10 Fisher, 2015 Veksler et al. 103 from 103 gslic May 7, / 40

37 GMMs in Computer Vision Spatio-intensity Model Spatio-intensity GMM for Medical Images A continuous and probabilistic framework for medical image respresentation and categorization Figure taken from Greenspan et al. May 7, / 40

38 GMMs in Computer Vision Spatio-intensity Model Combining Spatio-intensity GMM with Edge Cues The left column: original images. Second column: GMM segmentation. Third column: a competing algorithm. Fourth column: A GMM+edge segmentation. Figure taken from Rotem et al., 2007 May 7, / 40

39 GMMs in Computer Vision Mixtures over Surface Normals Mixtures over Surface Normals with Applications in Computer Vision and Robotics We explored two main types of mixtures over surface normals: with, and without orthogonality constraints. Orthogonality constraints form a Manhattan Frame (MF) The MMF = Mixture of Manhattan Frames. Both the sphere and the space of rotation matrices are nonlinear spaces. 11 MMF vmf MMF vmf RTMF JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007 (a) 1 MF (b) 1 MF (c) 2 MFs (d) 2 MFs (e) 2 MFs (f) 2 MFs (g) 3 MFs Fig. 11: Segmentation and inferred (M)MF of various indoor scenes partly taken from the NYU V2 depth dataset [40]. For single-mf scenes we color-code the assignment to MF axes and for MMF scenes the assignments to MFs. MMF Figure from Straub, Freifeld, Rosman, Leonard and Fisher, [CVPR 14, PAMI 17]. May 7, / 40

40 GMMs in Computer Vision Mixtures over Surface Normals Mixtures over Surface Normals with Applications in Computer Vision and Robotics Both types of models turned out to be useful for Semantically-Aware Aerial Reconstruction from Multi-Modal Data [Cabezas et al., ICCV 2015] Figure from Straub, Freifeld, Rosman, Leonard and Fisher, [CVPR 14, PAMI 17]. May 7, / 40