Energy Minimization for Segmentation in Computer Vision

Transcription

1 S * = arg S min E(S) Energy Minimization for Segmentation in Computer Vision Meng Tang, Dmitrii Marin, Ismail Ben Ayed, Yuri Boykov

2 Outline Clustering/segmentation methods K-means, GrabCut, Normalized Cut K-means for clustering/segmentation Probabilistic K-means Kernel K-means Our Kernel Cut: Normalized Cut+ MRF How to optimize Applications Motion, RGBD segmentation, image clustering etc.

3 K-means Clustering 0. Initialize cluster centers 1. Assign points to closest clusters 2. Re-compute means Repeat (1) and (2) until convergence

4 K-means Segmentation: Color Quantization

5 Graph Cut Segmentation Pr(I Fg) Pr(I Bg) E ( 1 S, 0, ) GrabCut [Rother et al. 2004] MRF regularization

6 Markov Random Field (MRF) (graphical models) Potts Model: edge alignment S c interactive segmentation [Boykov & Jolly, 2001] 6

7 Markov Random Field (MRF) (graphical models) Robust P n Potts: bin consistency S c semantic segmentation [Kohli & Torr 2009] [Gould 2014] 7

8 Normalized Cut (NC) eigenvectors discretization A pq [Arbelaez, Maire, Fowlkes & Malik, 2010] [Shi & Malik, 2000] [Ng, Jordan & Weiss, 2002] [Belkin & Niyogi, 2003] [Luxburg, 2007] 8

9 K-means minimizes sum of squared distance(ssd) Assume K=2 SSD + are in color space as

10 squared distance as log-likelihoods single Gaussian of fixed covariance single Gaussian

11 Probabilistic K-means with Descriptive Models single Gaussian of variable covariance model-fittings: ML estimation of

12 Probabilistic K-means with Descriptive Models Gaussian Mixture Models (GMM) Optimized in GrabCut (with smoothness term)

13 K-means probabilistic K-means make models more complex kernel K-means make data more complex embedding input space feature space kernel K-means = K-means in feature space

14 Why is it called kernel K-means? Cluster variance Replace norms by dot products kernel trick kernel, affinity, similarity, dot product explicit kernel κ implicit embedding φ

15 Why is it called kernel K-means? Cluster variance Replace norms by dot products kernel trick kernel, affinity, similarity, dot product explicit kernel κ implicit embedding φ

16 Normalized Cut is Kernel K-means (assuming constant degree d)

17 World Map of Segmentation geometric fitting GMM fitting pkm Probabilistic K-means (ML model fitting) histogram fitting gamma fitting weak kernel clustering (unary) Hilbertian distortion K-modes (mean-shift) basic K-means Gaussian kernel K-means kkm kernel K-means (average distortion AD or association AA) Normalized Cuts Average Cut Spectral clustering Gibbs fitting p.s.d. kernel distortion

18 (log-likelihoods) (normalized cut) Probabilistic K-means or Kernel K-means

19 Secrets of GrabCut typical MRF for segmentation: poor clustering (overfitting & local minima) ML term for θ k model fitting (e.g. GMM) GrabCut [Rother, Kolmogorov, Blake, 2004] color space clustering (probabilistic k-means [Kearns, Mansour & Ng, UAI 97]) 19

20 Normalized Cut clustering good clustering NC for colors Normalized Cut color space clustering 20

21 Our proposal: Normalized Cut + MRF balanced clustering regularization constraints 21

22 Why MRF for Normalized Cut? weak edge alignment semi-supervision is challenging previous NC approach: cannot-link [Yu & Shi 2004] [Eriksson et al. 2010] [Maji et al. 2011] [Chew et al. 2015] post-processing (e.g. [Arbelaez et al., 2011]) must-link reformulation of NC constrained eigen problem Our approach: NC + Potts MRF NC + Potts + seeds MRF 22

23 Why MRF for Normalized Cut? How to incorporate group priors? How many clusters? 8 clusters 3 clusters Our approach: NC + Robust P n Potts MRF NC + label costs MRF 23

24 Bound optimization, in general E(S) E(S t ) A t (S) A t+1 (S) E(S t+1 ) S t S t+1 guaranteed energy decrease 24

25 Bound for our joint energy unary bound for NC we propose kernel bound and spectral bound for NC 25

26 Kernel bound for NC Lemma 1 (concavity) Function e : R Ω R is concave over region S k > 0 given p.s.d. affinity matrix A := [A pq ]. (I) (II) a t (S) first-order Taylor expansion: e(s k ) a t+1 (S) equivalently kernel k-means for NC [Dhillon et al., 2004] 26

27 Our Kernel Cut algorithm unary bound for NC (Kernel Bound or Spectral Bound) iterate (move-making and graph cuts [Boykov, Veksler, Zabih, 2001]) 27

28 Experiments: MRF helps Normalized Cut using image tags (e.g. beach, car) to help image clustering NC + robust P n Potts + with knn kernel on deep features 28

29 Experiments: Normalized Cut helps MRF (a) Video frames (b) Optical flow (c) Our Kernel Cut (NC + Potts) Fig. motion segmentation using RGB, location (XY) and motion (M). +xy means with MRF 29

30 NC with increasing label cost

31 Robustness to smoothness term A seeds ground truth A B B C C GrabCut A B C Kernel Cut separating similar objects

32 More Experiments Potts model improves edge alignment Fig. 1. RGBD segmentation Spectral Clustering (no Potts) Our Kernel Cut (NC with Potts) Our Spectral Cut (NC with Potts) GrabCut Kernel Cut 32

33 Medical image segmentation (3D)

34 Conclusion + Probabilistic K-means vs Kernel K-means NC equivalent to Kernel K-means new unary kernel and spectral bounds for NC can combine NC with any MRF constraints can combine MRF with balanced clustering MRF with features of any dimension (RGBD, RGBM, RGBXYM, deep, ) 34

35 Take home code for