Image Segmentation with a Bounding Box Prior Victor Lempitsky, Pushmeet Kohli, Carsten Rother, Toby Sharp Microsoft Research Cambridge

Transcription

1 Image Segmentation with a Bounding Box Prior Victor Lempitsky, Pushmeet Kohli, Carsten Rother, Toby Sharp Microsoft Research Cambridge Dylan Rhodes and Jasper Lin 1

2 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 2

3 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 3

4 Segmentation Problem How does one separate the foreground from the background with minimal user input? 4

5 Bounding Box Allows the algorithm to focus on subimage Desired segmentation is close to sides of bounding box 5

6 Bounding Box Allows the algorithm to focus on subimage Desired segmentation is close to sides of bounding box 6

7 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 7

8 Basic Formulation 8

9 Basic Formulation B is the set of pixels within the bounding box 9

10 Basic Formulation E is the set of adjacent pixels within the bounding box 10

11 Basic Formulation p and q are pixel indices 11

12 Basic Formulation x_p can take a label of 1 for foreground or 0 for background 12

13 Basic Formulation Unary potentials encode preference for foreground or background 13

14 Basic Formulation Pairwise potentials enforce smoothness of the solution 14

15 Related Work Nowozin and Lampert derived framework for segmentation under connectivity constraint Relax NP-hard integer problem and solve resulting LP 15

16 Nowozin and Lampert 16

17 Nowozin and Lampert 17

18 Nowozin and Lampert 18

19 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 19

20 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 20

21 Why tightness? 21

22 Tightness Definition 22

23 Corollary A shape x is strongly tight if and only if its intersection with the middle box has a connected component touching all four sides of the middle box 23

24 Energy Minimization Problem 24

25 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 25

26 Energy Minimization Problem 26

27 Energy Minimization Problem 27

28 Convex Continuous Relaxation 28

29 Convex Continuous Relaxation 29

30 Continuous Optimization 30

31 Continuous Optimization 31

32 Additional Approximation Intuition: Solve LP with a subset Γ' of the constraints in 3c activated 32

33 Calculating Γ' 1. Begin with Γ' = 2. Solve the LP 3. Pick a group of crossing paths from Γ \ Γ' which are violated by more than a small tolerance 4. Add these paths to Γ' 5. Repeat steps 2 through 4 until all paths in Γ are satisfied within the tolerance 33

34 Final Form 34

35 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 35

36 Pinpointing Algorithm Normally, output of LP is rounded to integer solution 36

37 Pinpointing Algorithm Pinpoint set Π contains pixels hard-assigned to foreground 37

38 Pinpoint Algorithm 38

39 Pinpoint Algorithm 39

40 Challenges Existing methods perform energy-driven shrinking over bounding box No guarantees optimization won t shrink excessively Stuck at poor local minima Discretization of approximate solution is noisy 40

41 Paper s Contributions Common methods initialize foreground region and perform energy-driven shrinking No guarantees optimization won t shrink excessively Solution: new tightness prior Stuck at poor local minima Discretization of approximate solution is noisy 41

42 Paper s Contributions Common methods initialize foreground region and perform energy-driven shrinking No guarantees optimization won t shrink excessively Solution: new tightness prior Stuck at poor local minima Solution: new approximation strategies Discretization of approximate solution is noisy 42

43 Paper s Contributions Common methods initialize foreground region and perform energy-driven shrinking No guarantees optimization won t shrink excessively Solution: new tightness prior Stuck at poor local minima Solution: new approximation strategy Discretization of approximate solution is noisy Solution: new pinpointing algorithm 43

44 Presentation Overview Segmentation problem description Background and Previous Work Problems and Proposed Solutions Formalizing tightness Defining tractable optimization problem for segmentation Discretizing continuous approximation of solution Experiments and Results 44

45 Experiments Evaluated over 50 image GrabCut dataset Each image comes with bounding box Comparison with competing methods and initialization strategies 45

46 GrabCut Dataset 50 natural images with bounding box annotations Includes background, outside strip, and foreground bounding boxes 46

47 GrabCut Dataset 50 natural images with bounding box annotations Includes background, outside strip, and foreground bounding boxes 47

48 Unary and Pairwise Terms Pairwise terms over 8-connected edge set 48

49 Relative Performance Error rate - mislabeled pixels inside bounding box Optimum Rank - average rank of energy of final integer program solutions 49

50 Relative Performance 50

51 Iterative Process Compare the following algorithms on the segmentation task: GrabCut with standard graph cut minimization for all segmentation steps GrabCut which enforces the tightness prior for all segmentation steps 5 iterations each 51

52 Initialization Strategies Compare two methods for initializing foreground/background GMMs: InitThirds = same as Experiment 1 (outside strip + best matches vs. poor matches) InitFullBox which sets background GMM to outside strip and foreground to whole interior of bounding box 52

53 Iterative Process 53

54 Effect of Margin Thickness Error rates as function of margin thickness 54

55 Strong vs. Weak Tightness Strong and weak tightness lead to similar error rates in general same error rate (3.7%) for best model (GrabCutPinpoint/InitThirds) 55

56 Iterative Process Comparisons 56

57 Conclusions New bounding-box based prior for interactive image segmentation 57

58 Conclusions New bounding-box based prior for interactive image segmentation Demonstrated segmentation tasks under this prior can be formulated as integer programs 58

59 Conclusions New bounding-box based prior for interactive image segmentation Demonstrated segmentation tasks under this prior can be formulated as integer programs Developed new optimization approaches for approximate solution of these NP-hard problems Can be applied to other computer vision problems e.g. other image segmentation or silhouettes in multi-view stereo 59

60 Thank you! 60