Multiple cosegmentation

Transcription

1 Armand Joulin, Francis Bach and Jean Ponce. INRIA -Ecole Normale Supérieure April 25, 2012

2 Segmentation Introduction Segmentation Supervised and weakly-supervised segmentation Cosegmentation Segmentation is classical and fundamental vision problem.

3 Segmentation Introduction Segmentation Supervised and weakly-supervised segmentation Cosegmentation Segmentation is classical and fundamental vision problem. Problem: Many possible solutions.

4 Existing solutions Introduction Segmentation Supervised and weakly-supervised segmentation Cosegmentation Supervised Segmentation: Need ground truth for every class of object Cannot deal with an unknown object. P. Krähenbühl and V. Koltun (NIPS 11) Interactive segmentation (scribbles or a bounding box) Need human interaction for each image. GrabCut

5 Cosegmentation Introduction Segmentation Supervised and weakly-supervised segmentation Cosegmentation Dividing one images.

6 Cosegmentation Introduction Segmentation Supervised and weakly-supervised segmentation Cosegmentation Dividing a set of images by using shared information.

7 Cosegmentation Introduction Segmentation Supervised and weakly-supervised segmentation Cosegmentation Dividing a set of images by using shared information. No prior information. But: common foreground and different background.

8 Cosegmentation Introduction Segmentation Supervised and weakly-supervised segmentation Cosegmentation Previous existing methods (Rother et al. 2006, Singh and Hochbaum 2009,...) only work with 2 images and the exact same object. The first presented method works on multiple images and on an object class. The second one extends it to multiple images and multiple object classes.

9 Segmentation Supervised and weakly-supervised segmentation Cosegmentation...Cosegmentation is also a ill-posed problem In natural images, objects are link with their environement...the background is also common to all the images.

10 Segmentation Supervised and weakly-supervised segmentation Cosegmentation...Cosegmentation is also a ill-posed problem In natural images, objects are link with their environement...the background is also common to all the images. Solutions: Use user interaction on some images,

11 Segmentation Supervised and weakly-supervised segmentation Cosegmentation...Cosegmentation is also a ill-posed problem In natural images, objects are link with their environement...the background is also common to all the images. Solutions: Use user interaction on some images, Segment the background into meaningful regions.

12 The goals of our approach Goal of our approach Notations Our method should: Handle multiple images.

13 The goals of our approach Goal of our approach Notations Our method should: Handle multiple images. Works on any kind of object/stuff.

14 The goals of our approach Goal of our approach Notations Our method should: Handle multiple images. Works on any kind of object/stuff. Segments the background into meaningful regions.

15 The goals of our approach Goal of our approach Notations Our method should: Handle multiple images. Works on any kind of object/stuff. Segments the background into meaningful regions. Uses no prior information but can be easily extended to interactive cosegmentation.

16 Method goals Introduction Goal of our approach Notations Local consistency Maximizing spatial consistency within a particular image. Figure: Image space.

17 Method goals Introduction Goal of our approach Notations Local consistency Maximizing spatial consistency within a particular image. Separation of the classes Maximizing the separability of K classes between different images Our framework: Unsupervised discriminative clustering. Figure: Image space. Figure: Feature space.

18 Problem Notations Introduction Goal of our approach Notations Each image i is reduced to a subsampled grid of pixels. For the n-th pixel, we denote by: x n its d-dimensional feature vector. y n the K-vector such as y nk = 1 if the n-th pixel is in the k-class and 0 otherwise.

19 Figure: Image space. Normalized Cut (Shi and Malik, 2000): The similarty between two pixels is mesured by the rbf distance between their position p n and their color c n.

20 Figure: Image space. Normalized Cut (Shi and Malik, 2000): The similarty between two pixels is mesured by the rbf distance between their position p n and their color c n. For an image i, our similarity matrix is: W i nm = exp( λ p p n p m 2 2 λ c c n c m 2 ).

21 Figure: Image space. Normalized Cut (Shi and Malik, 2000): The Laplacian matrix is L = I D 1/2 WD 1/2 where D the diagonal matrix composed of the row sums of W

22 Figure: Image space. Normalized Cut (Shi and Malik, 2000): The Laplacian matrix is L = I D 1/2 WD 1/2 where D the diagonal matrix composed of the row sums of W We thus have the following in our cost function: E B (y) = µ N tr(yt Ly).

23 Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Figure: Feature space. Discriminative classifier: given the labels y, we solve the following problem: 1 E U (y) = min A IR K d, N b IR K N n=1 l(y n,aφ(x n )+b)+ λ 2K A 2 F, Notations φ a non-linear mapping of the feature, l is a cost function.

24 Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Figure: Feature space. Discriminative classifier: given the labels y, we solve the following problem: 1 E U (y) = min A IR K d, N b IR K N n=1 l(y n,aφ(x n )+b)+ λ 2K A 2 F, Notations φ a non-linear mapping of the feature, l is a cost function.

25 Mapping approximation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Our discriminative clustering framework works with positive definite kernels

26 Mapping approximation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Our discriminative clustering framework works with positive definite kernels We use the χ 2 kernel matrix K: ( D K nm = exp λ h d=1 (x nd x md ) 2 ), x nd +x md

27 Mapping approximation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Our discriminative clustering framework works with positive definite kernels We use the χ 2 kernel matrix K: ( D K nm = exp λ h d=1 (x nd x md ) 2 ), x nd +x md Equivalent to apply a mapping φ from the feature space to a high-dimensional Hilbert space F, such that: K nm = φ(x n ), φ(x m )

28 Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Figure: Feature space. Discriminative classifier: given the labels y, we solve the following problem: 1 E U (y) = min A IR K d, N b IR K N n=1 l(y n,aφ(x n )+b)+ λ 2K A 2 F, Notations φ a non-linear mapping of the feature, l is a cost function.

29 Loss function Introduction Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation We choose the soft-max loss function because it is suited for multiclass and is related to probabilistic models: l(y n,aφ(x n )+b) = K k=1 ( exp(a T ) y nk log k φ(x n )+b k ) K, l=1 exp(at l φ(x n )+b l )

30 Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Find the set of labels y which leads to the best data separation into K classes: min y {0,1} N K, y1 K =1 N min E U(y,A,b) A IR K d,b IR K

31 Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Find the set of labels y which leads to the best data separation into K classes: min y {0,1} N K, y1 K =1 N min E U(y,A,b) A IR K d,b IR K Problem: Same label for all the pixels perfect separation

32 Cluster size balancing Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Two solutions: adding linear constraints on the number of elements per class Encourage the proportion of points per class to be uniform

33 Cluster size balancing Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Two solutions: adding linear constraints on the number of elements per class Encourage the proportion of points per class to be uniform We choose the second: No additional parameters and have a probabilistic interpretation. H(y) = i I K k=1 ( 1 N n N i y nk ) ( 1 log N n N i y nk where i is an image, and N i the number of pixels in i Note: In a weakly supervised setting (e.g., interactive segmentation), this term can be modify to take into account prior knowledge. ).

34 Overall problem Introduction Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Combining the unary and binary term with the class balancing term, we obtain the following problem: [ ] min y {0,1} N K, y1 K =1 N min E U (y,a,b) +E B (y) H(y). A IR d K, b IR K

35 Probabilistic interpretation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation We introduce t n in {0,1} I indicating to which image n belongs and z n in {1,...,M} giving for each pixel n some observable information The label y is a latent variable of the observable information z given x (x y z t) inducing an explain away phenomenon: the label y n and the variable t n compete to explain the observable information z n.

36 Probabilistic interpretation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation More precisely, we suppose a bilinear model: P(z nm = 1 t ni = 1, y nk = 1) = y nk G ik mt ni, where N m=1 Gik m = 1

37 Probabilistic interpretation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation More precisely, we suppose a bilinear model: P(z nm = 1 t ni = 1, y nk = 1) = y nk G ik mt ni, where N m=1 Gik m = 1 and a exponential family model for Y = (y 1,...,y N ) given X = (x 1,...,x N ) with unary parameters (A,b) and binary parameters L.

38 Probabilistic interpretation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Our cost function is the mean-field variational approximation of the following (regularized) negative conditional log-likelihood of Z = (z 1,...,z N ) given X and T = (t 1,...,t N ) for our model: min A IR d K,b IR K, G IR N K I, G T 1 N =1, G 0 1 N N n=1 log ( p(z n x n, t n ) ) + λ 2K A 2 2.

39 Probabilistic interpretation Formulation Mapping approximation Loss function Cluster size balancing Overall problem Probabilistic interpretation Our cost function is the mean-field variational approximation of the following (regularized) negative conditional log-likelihood of Z = (z 1,...,z N ) given X and T = (t 1,...,t N ) for our model: min A IR d K,b IR K, G IR N K I, G T 1 N =1, G 0 1 N N n=1 log ( p(z n x n, t n ) ) + λ 2K A 2 2. Z can encode must-link and must-not-link constraints between pixels (e.g., superpixels).

40 EM procedure Introduction [ min y {0,1} N K, y1 K =1 N min A IR d K, b IR K ] E U (y,a,b) +E B (y) H(y).

41 EM procedure Introduction [ min y {0,1} N K, y1 K =1 N min A IR d K, b IR K ] E U (y,a,b) +E B (y) H(y). This cost function is not jointly convex in y and (A,b).

42 EM procedure Introduction [ min y {0,1} N K, y1 K =1 N min A IR d K, b IR K ] E U (y,a,b) +E B (y) H(y). This cost function is not jointly convex in y and (A,b). However it is convex in both independently.

43 EM procedure Introduction [ min y {0,1} N K, y1 K =1 N min A IR d K, b IR K ] E U (y,a,b) +E B (y) H(y). This cost function is not jointly convex in y and (A,b). However it is convex in both independently. We alternatively optimize over each variable while fixing the other: We use L-BFGS for (A,b) We use a projected gradient descent for y.

44 The initialization Since our problem is not convex, a good initialization is crucial We propose a quadratic convex approximation related to Joulin et al. (CVPR 10). Quadratic function may lead to poor solutions, thus we also use random initializations.

45 Initialization: Quadratic approximation The second-order Taylor expansion of our cost function is: J(y) = K [ tr(yy T C)+ 2µ 2 NK tr(yyt L) 1 ] N tr(yyt Π I ), where C = 1 N Π N(I Φ(NλI K +Φ T Π N Φ) 1 Φ T )Π N is related to the reweighted ridge regression classifier (Joulin et al. CVPR 10).

46 Initialization: Quadratic approximation The second-order Taylor expansion of our cost function is: J(y) = K [ tr(yy T C)+ 2µ 2 NK tr(yyt L) 1 ] N tr(yyt Π I ), where C = 1 N Π N(I Φ(NλI K +Φ T Π N Φ) 1 Φ T )Π N is related to the reweighted ridge regression classifier (Joulin et al. CVPR 10). This is not convex because of the last term which can be replaced by the following linear constraints: y nk 0.9N i ; y nk 0.1(N N i ). n N i n N j j I\i we obtain a formulation similar to Joulin et al. (CVPR 10).

47 Introduction Binary segmentation (foreground/background) on MSRC: High variability in foreground and background, around 30 images per classes, We use SIFT features. Multiclass cosegmentation on icoseg: Low variability in the image, same illumination... around 10 images per classes, We use color histograms. Some extensions: Grabcut. weakly supervised problem video key frames.

48 Binary cosegmentation

49 Binary cosegmentation

50 Binary cosegmentation class Ours Kim et al. (ICCV 11) Joulin et al. (CVPR 10) Bike Bird Car Cat Chair Cow Dog Face Flower House Plane Sheep Sign Tree Average

51

52

53 class K Ours Joulin et al. CVPR 10 Kim et al ICCV 11 Baseball player Brown bear Elephant Ferrari Football player Helicopter Kite Panda Monk Panda Skating Stonehedge Plane Face Average

54 Extensions Introduction grabcut: Weakly supervised learning with image tags ({ plane, sheep, sky, grass}). Video shot segmentation:

55 Limitations Introduction Number of classes: Each class must be in each image (because of the entropy). Running time: About half an hour to one hour (MATLAB implementation).

56 Thank you.