Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

Transcription

1 Joint optimization of segmentation and appearance models Vicente, Kolmogorov, Rother Chris Russell November 26, 2009 Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

2 Grabcut Given an image and a bounding box Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

3 Grabcut Given an image and a bounding box Extract the object from the centre of the box. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

4 Image segmentation Given a cropped image, split it into two consistent classes: Foreground class a single object in the centre of the image. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

5 Image segmentation Given a cropped image, split it into two consistent classes: Foreground class a single object in the centre of the image. Background class everything else in the image. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

6 Image segmentation Given a cropped image, split it into two consistent classes: Foreground class a single object in the centre of the image. Background class everything else in the image.. E(x, θ) = ln(p(x i θ)) + i V (i,j) E w i,j x i x j (1) x the assignment of pixels and θ is the set of model parameters. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

7 Image segmentation Given a cropped image, split it into two consistent classes: Foreground class a single object in the centre of the image. Background class everything else in the image.. E(x, θ) = ln(p(x i θ)) + i V (i,j) E w i,j x i x j (1) x the assignment of pixels and θ is the set of model parameters. Submodular in x. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

8 Image segmentation Given a cropped image, split it into two consistent classes: Foreground class a single object in the centre of the image. Background class everything else in the image.. E(x, θ) = ln(p(x i θ)) + i V (i,j) E w i,j x i x j (1) x the assignment of pixels and θ is the set of model parameters. Submodular in x. Easy if θ is known. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

9 Coping with unknown θ Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

10 Coping with unknown θ Pretend θ is known. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

11 Coping with unknown θ Pretend θ is known. We bootstrap θ. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

12 Coping with unknown θ Pretend θ is known. We bootstrap θ. Assume that a central region is our foreground object and the rest is background. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

13 Coping with unknown θ Pretend θ is known. We bootstrap θ. Assume that a central region is our foreground object and the rest is background. We learn a model for foreground and background. (Either gmm, or histogram based) Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

14 Coping with unknown θ Pretend θ is known. We bootstrap θ. Assume that a central region is our foreground object and the rest is background. We learn a model for foreground and background. (Either gmm, or histogram based) Then run graph-cut to find a new central region. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

15 Coping with unknown θ Pretend θ is known. We bootstrap θ. Assume that a central region is our foreground object and the rest is background. We learn a model for foreground and background. (Either gmm, or histogram based) Then run graph-cut to find a new central region. Repeat till convergence. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

16 An example Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

17 What s it doing? Every move proposed maximises P(x θ) (2) Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

18 What s it doing? Every move proposed maximises Can be seen as an em or hill climbing approach. P(x θ) (2) Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

19 What s it doing? Every move proposed maximises P(x θ) (2) Can be seen as an em or hill climbing approach. Guaranteed to converge. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

20 Joint optimisation of segmentation and appearance models Is there a better way? Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

21 Joint optimisation of segmentation and appearance models Is there a better way? Let θ be a histogram based model. U(x, θ) = i V ln(p(x i θ)) = i V ln θ x i b i (3) Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

22 Joint optimisation of segmentation and appearance models Is there a better way? Let θ be a histogram based model. U(x, θ) = i V ln(p(x i θ)) = i V ln θ x i b i (3) where b B θ s b = 1 s {0, 1} (4) Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

23 Joint optimisation of segmentation and appearance models Is there a better way? Let θ be a histogram based model. U(x, θ) = i V ln(p(x i θ)) = i V ln θ x i b i (3) where b B θ s b = 1 s {0, 1} (4) We rephrase this. Let nk s be the number of pixels taking label s that lie in bin k. U(x, θ) = nk s ln θs k (5) s k Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

24 Joint optimisation of segmentation and appearance models The optimal value of θ s k is θ s k = ns k k ns k (6) Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

25 Joint optimisation of segmentation and appearance models The optimal value of θ s k is θ s k = ns k k ns k (6) Use this to eliminate θ E(x) = min E(x, θ) = θ k h k (n 1 k ) + h(n1 ) + P(x) (7) h k (n 1 k ) = g(n1 k ) g(n k n 1 k ) h(n 1 ) = g(n 1 ) + g(n n k ) g(z) = z ln z (8) Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

26 Nearly there... g(z) = z ln z is super-linear so... Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

27 Nearly there... g(z) = z ln z is super-linear so... h k (n 1 k ) = g(n1 k ) g(n k n 1 k ) is concave and symmetric about n k/2. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

28 Nearly there... g(z) = z ln z is super-linear so... h k (n 1 k ) = g(n1 k ) g(n k n 1 k ) is concave and symmetric about n k/2. Similarly, h(n 1 ) = g(n 1 ) + g(n n 1 ) is convex and symmetric about n/2. Joint optimization of segmentation and appearance models November Vicente, 26, Kolmogorov, / Rothe 14

29 New energy E(x) = E 1 (x) + E 2 (x) (9) Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 10 / Rothe 14

30 New energy E(x) = E 1 (x) + E 2 (x) (9) Submodular component E 1 (x) = k h k (nk 1 ) + P(x) (10) Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 10 / Rothe 14

31 New energy E(x) = E 1 (x) + E 2 (x) (9) Submodular component E 1 (x) = k h k (nk 1 ) + P(x) (10) very high order, can be solved using general submodular solver(slow), or a lazy variant of the robust P n model (fast). Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 10 / Rothe 14

32 New energy E(x) = E 1 (x) + E 2 (x) (9) Submodular component E 1 (x) = k h k (nk 1 ) + P(x) (10) very high order, can be solved using general submodular solver(slow), or a lazy variant of the robust P n model (fast). Super-modular component E 2 (x) = h(n 1 ) (11) Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 10 / Rothe 14

33 New energy E(x) = E 1 (x) + E 2 (x) (9) Submodular component E 1 (x) = k h k (nk 1 ) + P(x) (10) very high order, can be solved using general submodular solver(slow), or a lazy variant of the robust P n model (fast). Super-modular component E 2 (x) = h(n 1 ) (11) Convex, piecewise linear with 2n pieces. Solved by brute force. Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 10 / Rothe 14

34 Dual Decomposition We can solve E 1 and E 2 on their own, but not together. Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 11 / Rothe 14

35 Dual Decomposition We can solve E 1 and E 2 on their own, but not together. So we perturb E 1 and E 2 to try to enforce agreement. Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 11 / Rothe 14

36 Dual Decomposition We can solve E 1 and E 2 on their own, but not together. So we perturb E 1 and E 2 to try to enforce agreement. Create new energies E 1 (x) = E 1 (x) < y, x > (12) and E 2 (x) = E 2 (x)+ < y, x > (13) Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 11 / Rothe 14

37 Dual Decomposition We can solve E 1 and E 2 on their own, but not together. So we perturb E 1 and E 2 to try to enforce agreement. Create new energies E 1 (x) = E 1 (x) < y, x > (12) and E 2 (x) = E 2 (x)+ < y, x > (13) Then E(x) = E 1 (x) + E 2 (x) Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 11 / Rothe 14

38 Dual Decomposition We can solve E 1 and E 2 on their own, but not together. So we perturb E 1 and E 2 to try to enforce agreement. Create new energies E 1 (x) = E 1 (x) < y, x > (12) and E 2 (x) = E 2 (x)+ < y, x > (13) Then E(x) = E 1 (x) + E 2 (x) Try to pick y such that arg min E 1 (x) = arg min E 2 (x) (14) x x Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 11 / Rothe 14

39 Dual Decomposition We can solve E 1 and E 2 on their own, but not together. So we perturb E 1 and E 2 to try to enforce agreement. Create new energies E 1 (x) = E 1 (x) < y, x > (12) and E 2 (x) = E 2 (x)+ < y, x > (13) Then E(x) = E 1 (x) + E 2 (x) Try to pick y such that arg min E 1 (x) = arg min E 2 (x) (14) x x Such a y can be found 61% of the time. Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 11 / Rothe 14

40 Is it worth it? Sort of Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 12 / Rothe 14

41 Is it worth it? Sort of Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 12 / Rothe 14

42 Is it worth it? Sort of Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 12 / Rothe 14

43 Encouraging heterogeneity E 2 (x) = λh(n 1 ) (15) Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 13 / Rothe 14

44 Optimal Results Joint optimization of segmentation and appearance models November Vicente, 26, 2009 Kolmogorov, 14 / Rothe 14