Image Segmentation. Marc Pollefeys. ETH Zurich. Slide credits: V. Ferrari, K. Grauman, B. Leibe, S. Lazebnik, S. Seitz,Y Boykov, W. Freeman, P.

Transcription

1 Image Segmentation Perceptual and Sensory Augmented Computing Computer Vision WS 0/09 Marc Pollefeys ETH Zurich Slide credits: V. Ferrari, K. Grauman, B. Leibe, S. Lazebnik, S. Seitz,Y Boykov, W. Freeman, P. Kohli

2 Topics of This Lecture Perceptual and Sensory Augmented Computing! Introduction!! Gestalt principles!! Image segmentation! Segmentation as clustering!! k-means!! Feature spaces!! Mixture of Gaussians, EM! Model-free clustering: Mean-Shift! Graph theoretic segmentation: Normalized Cuts! Interactive Segmentation with GraphCuts

3 Examples of Grouping in Vision Perceptual and Sensory Augmented Computing Determining image regions What things should be grouped? What cues indicate groups? Grouping video frames into shots Object-level grouping Figure-ground Slide credit: Kristen Grauman

4 Similarity in appearance Slide adapted from Kristen Grauman

5 Symmetry Slide credit: Kristen Grauman

6 Common Fate Image credit: Arthus-Bertrand (via F. Durand) Slide credit: Kristen Grauman

7 Proximity Slide credit: Kristen Grauman

8 The Gestalt School! Grouping is key to visual perception! Elements in a collection can have properties that result from relationships!! The whole is greater than the sum of its parts Illusory/subjective contours Slide credit: Svetlana Lazebnik Occlusion Familiar configuration Image source: Steve Lehar

9 Gestalt Theory! Gestalt: whole or group!! Whole is greater than sum of its parts!! Relationships among parts can yield new properties/features! Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) Slide credit: B. Leibe I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees. Max Wertheimer ( ) Untersuchungen zur Lehre von der Gestalt, Psychologische Forschung, Vol. 4, pp ,

10 Gestalt Factors These factors make intuitive sense, but are very difficult to translate into algorithms. Slide credit: B. Leibe Image source: Forsyth & Ponce

11 Continuity through Occlusion Cues Slide credit: B. Leibe

12 Continuity through Occlusion Cues Continuity, explanation by occlusion Slide credit: B. Leibe

13 Continuity through Occlusion Cues Slide credit: B. Leibe Image source: Forsyth & Ponce

14 Continuity through Occlusion Cues Slide credit: B. Leibe Image source: Forsyth & Ponce

15 Figure-Ground Discrimination Slide credit: B. Leibe

16 The Ultimate Gestalt test Slide adapted from B. Leibe

17 Image Segmentation! Goal: identify groups of pixels that go together Slide credit: Steve Seitz, Kristen Grauman

18 The Goals of Segmentation! Separate image into objects Image Human segmentation Slide credit: Svetlana Lazebnik

19 The Goals of Segmentation! Separate image into objects! Group together similar-looking pixels for efficiency of further processing superpixels X. Ren and J. Malik. Learning a classification model for segmentation. ICCV Slide credit: Svetlana Lazebnik

20 Topics of This Lecture! Introduction!! Gestalt principles!! Image segmentation! Segmentation as clustering!! k-means!! Feature spaces!! Mixture of Gaussians, EM! Model-free clustering: Mean-Shift! Graph theoretic segmentation: Normalized Cuts! Interactive Segmentation with GraphCuts

21 Image Segmentation: Toy Example input image black pixels gray pixels intensity! These intensities define the three groups.! We could label every pixel in the image according to which of these it is.!! i.e. segment the image based on the intensity feature.! What if the image isn t quite so simple? white pixels Slide credit: Kristen Grauman

22 Input image Input image Slide credit: Kristen Grauman Pixel count Pixel count Intensity Intensity

23 Input image Pixel count Intensity! Now how to determine the three main intensities that define our groups?! We need to cluster. Slide credit: Kristen Grauman

24 Intensity 1 2! Goal: choose three centers as the representative intensities, and label every pixel according to which of these centers it is nearest to.! Best cluster centers are those that minimize SSD between all points and their nearest cluster center c i : 3 Slide credit: Kristen Grauman

25 Clustering! With this objective, it is a chicken and egg problem:!! If we knew the cluster centers, we could allocate points to groups by assigning each to its closest center.!! If we knew the group memberships, we could get the centers by computing the mean per group. Slide credit: Kristen Grauman

26 K-Means Clustering! Basic idea: randomly initialize the k cluster centers, and iterate between the two steps we just saw. 1.! Randomly initialize the cluster centers, c 1,..., c K 2.! Given cluster centers, determine points in each cluster! For each point p, find the closest c i. Put p into cluster i 3.! Given points in each cluster, solve for c i! Set c i to be the mean of points in cluster i 4.! If c i have changed, repeat Step 2! Properties!! Will always converge to some solution!! Can be a local minimum! Does not always find the global minimum of objective function: Slide credit: Steve Seitz

27 Segmentation as Clustering img_as_col = double(im(:)); cluster_membs = kmeans(img_as_col, K); labelim = zeros(size(im)); for i=1:k inds = find(cluster_membs==i); meanval = mean(img_as_column(inds)); labelim(inds) = meanval; end K=2 K=3 Slide credit: Kristen Grauman

28 K-Means Clustering! Java demo:

29 Feature Space! Depending on what we choose as the feature space, we can group pixels in different ways.! Grouping pixels based on intensity similarity! Feature space: intensity value (1D) Slide credit: Kristen Grauman

30 Feature Space! Depending on what we choose as the feature space, we can group pixels in different ways.! Grouping pixels based on color similarity B! Feature space: color value (3D) Slide credit: Kristen Grauman R G R=15 G=189 B=2 R=3 G=12 B=2 R=255 G=200 B=250 R=245 G=220 B=248

31 Segmentation as Clustering! Depending on what we choose as the feature space, we can group pixels in different ways.! Grouping pixels based on texture similarity F 1 F 2 F 24! Feature space: filter bank responses (e.g. 24D) Slide credit: Kristen Grauman! Filter bank of 24 filters

32 Spatial coherence! Assign a cluster label per pixel " possible discontinuities Original! How can we ensure they are spatially smooth? Labeled by cluster center s intensity? Slide adapted from Kristen Grauman

33 Spatial coherence! Depending on what we choose as the feature space, we can group pixels in different ways.! Grouping pixels based on intensity+position similarity Intensity Y X! Way to encode both similarity and proximity. Slide adapted from Kristen Grauman

34 K-Means without spatial information! K-means clustering based on intensity or color is essentially vector quantization of the image attributes!! Clusters don t have to be spatially coherent Image Intensity-based clusters Color-based clusters Slide adapted from Svetlana Lazebnik Image source: Forsyth & Ponce

35 K-Means with spatial information! K-means clustering based on intensity or color is essentially vector quantization of the image attributes!! Clusters don t have to be spatially coherent! Clustering based on (r,g,b,x,y) values enforces more spatial coherence Slide adapted from Svetlana Lazebnik Image source: Forsyth & Ponce

36 Summary K-Means! Pros!! Simple, fast to compute!! Converges to local minimum of within-cluster squared error! Cons/issues!! Setting k?!! Sensitive to initial centers!! Sensitive to outliers!! Detects spherical clusters only!! Assuming means can be computed Slide credit: Kristen Grauman

37 Probabilistic Clustering! Basic questions!! What s the probability that a point x is in cluster m?!! What s the shape of each cluster?! K-means doesn t answer these questions.! Basic idea!! Instead of treating the data as a bunch of points, assume that they are all generated by sampling a continuous function.!! This function is called a generative model.!! Defined by a vector of parameters! Slide credit: Steve Seitz

38 Mixture of Gaussians! One generative model is a mixture of Gaussians (MoG)!! K Gaussian blobs with means µ b covariance matrices V b, dimension d! Blob b defined by:!! Blob b is selected with probability!! The likelihood of observing x is a weighted mixture of Gaussians, Slide adapted from Steve Seitz

39 Expectation Maximization (EM)! Goal!! Find blob parameters! that maximize the likelihood function overall all N datapoints! Approach: 1.! E-step: given current guess of blobs, compute probabilistic ownership of each point 2.! M-step: given ownership probabilities, update blobs to maximize likelihood function 3.! Repeat until convergence Slide adapted from Steve Seitz

40 EM Details! E-step!! Compute probability that point x is in blob b, given current guess of!! M-step!! Compute overall probability that blob b is selected!! Mean of blob b!! Covariance of blob b (N data points) Slide adapted from Steve Seitz

41 Applications of EM! Turns out this is useful for all sorts of problems!! Any clustering problem!! Any model estimation problem!! Missing data problems!! Finding outliers!! Segmentation problems! Segmentation based on color!!...! EM demo! Segmentation based on motion! Foreground/background separation!! Slide credit: Steve Seitz

42 Segmentation with EM Original image EM segmentation results k=2 k=3 k=4 k=5 Slide credit: B. Leibe Image source: Serge Belongie

43 Summary: Mixtures of Gaussians, EM! Pros!! Probabilistic interpretation!! Soft assignments between data points and clusters!! Generative model, can predict novel data points!! Relatively compact storage ( )! Cons!! Initialization! often a good idea to start from output of k-means!! Local minima!! Need to know number of components K! solutions: model selection (AIC, BIC), Dirichlet process mixture!! Need to choose generative model (math form of a cluster?)!! Numerical problems are often a nuisance Slide adapted from B. Leibe

44 Topics of This Lecture! Introduction!! Gestalt principles!! Image segmentation! Segmentation as clustering!! k-means!! Feature spaces!! Mixture of Gaussians, EM! Model-free clustering: Mean-Shift! Graph theoretic segmentation: Normalized Cuts! Interactive Segmentation with GraphCuts

45 Finding Modes in a Histogram! How many modes are there?!! Mode = local maximum of a given distribution!! Easy to see, hard to compute Slide adapted from Steve Seitz

46 Mean-Shift Segmentation! An advanced and versatile technique for clusteringbased segmentation D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI Slide credit: Svetlana Lazebnik

47 Mean-Shift Algorithm! Iterative Mode Search 1.! Initialize random seed center and window W 2.! Calculate center of gravity (the mean ) of W: 3.! Shift the search window to the mean 4.! Repeat steps 2+3 until convergence Slide adapted from Steve Seitz

48 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

49 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

50 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

51 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

52 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

53 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

54 Mean-Shift Region of interest Center of mass Slide by Y. Ukrainitz & B. Sarel

55 Real Modality Analysis Tessellate the space with windows Slide by Y. Ukrainitz & B. Sarel Run the procedure in parallel

56 Real Modality Analysis The blue data points were traversed by the windows towards the mode. Slide by Y. Ukrainitz & B. Sarel

57 Mean-Shift Clustering! Cluster: all data points in the attraction basin of a mode! Attraction basin: the region for which all trajectories lead to the same mode Slide by Y. Ukrainitz & B. Sarel

58 Mean-Shift Clustering/Segmentation!!!! Choose features (color, gradients, texture, etc) Initialize windows at individual pixel locations Start mean-shift from each window until convergence Merge windows that end up near the same peak or mode Slide adapted from Svetlana Lazebnik

59 Mean-Shift Segmentation Results Slide credit: Svetlana Lazebnik

60 More Results Slide credit: Svetlana Lazebnik

61 Summary Mean-Shift! Pros!! General, application-independent tool!! Model-free, does not assume any prior shape (spherical, elliptical, etc.) on data clusters!! Just a single parameter (window size h)! h has a physical meaning (unlike k-means) == scale of clustering!! Finds variable number of modes given the same h!! Robust to outliers! Cons!! Output depends on window size h!! Window size (bandwidth) selection is not trivial!! Computationally rather expensive!! Does not scale well with dimension of feature space Slide adapted from Svetlana Lazebnik

62 Topics of This Lecture! Introduction!! Gestalt principles!! Image segmentation! Segmentation as clustering!! k-means!! Feature spaces!! Mixture of Gaussians, EM! Model-free clustering: Mean-Shift! Graph theoretic segmentation: Normalized Cuts! Interactive Segmentation with GraphCuts

63 Images as Graphs q! Fully-connected graph p w pq!! Node (vertex) for every pixel!! Edge between every pair of pixels (p,q)!! Affinity weight w pq for each edge! w pq measures similarity! Similarity is inversely proportional to difference (in color, texture, position, ) w Slide adapted from Steve Seitz

64 Segmentation by Graph Cuts w A B C! Break Graph into Segments!! Delete edges crossing between segments!! Easiest to break edges with low similarity (low weight)! Similar pixels should be in the same segments! Dissimilar pixels should be in different segments Slide adapted from Steve Seitz

65 Measuring Affinity! Distance { 2 } 1 2! aff ( x, y) = exp " x " y 2 d! Intensity! Color! Texture { 2 } 1 2! aff ( x, y) = exp " I( x) " I( y) 2 d { ( ) } dist c x c y 2! aff ( x, y) = exp " ( ), ( ) d (some suitable color space distance) { 2 } 1 2! aff ( x, y) = exp " f ( x) " f ( y) 2 d (vectors of filter outputs) Source: Forsyth & Ponce

66 Scale Affects Affinity! Small!: group only nearby points! Large!: group far-away points small! large! Slide adapted from Svetlana Lazebnik data points Small " Medium " Large " Image Source: Forsyth & Ponce

67 Graph Cut (GC) A B! GC = edges whose removal partitions a graph in two! Cost of a cut!! Sum of weights of cut edges: cut( A, B)! A graph cut gives us a segmentation! w p, q p" A, q" B!! What is a good graph cut and how do we find one? = Slide adapted from Steve Seitz

68 Graph Cut Here, the cut is nicely defined by the block-diagonal structure of the affinity matrix.! How can this be generalized? Slide credit: B. Leibe Image Source: Forsyth & Ponce

69 Minimum Cut! We can do segmentation by finding the minimum cut in a graph!! Efficient algorithms exist for doing this! Drawback:!! Weight of cut proportional to number of edges in the cut!! Minimum cut tends to cut off very small, isolated components Ideal Cut Cuts with lesser weight than the ideal cut Slide credit: Khurram Hassan-Shafique

70 Normalized Cut (NCut)! Min-cut has bias toward partitioning out small segments! This can be fixed by normalizing for size of segments! The normalized cut cost is: cut(a,b) assoc(a,v ) + cut(a,b) assoc(b,v ) assoc(a,v) = sum of weights from A to all nodes to graph! The exact solution is NP-hard but an approximation can be computed by solving a generalized eigenvalue problem. J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000 Slide adapted from Svetlana Lazebnik

71 Interpretation as a Dynamical System! Treat the edges as springs and shake the system!! Elasticity proportional to cost!! Vibration modes correspond to segments! Can compute these by solving a generalized eigenvector problem Slide adapted from Steve Seitz

72 NCuts as a Generalized Eigenvector Problem! Definitions W: the affinity matrix, Wij (, ) = w ; $ D: the diag. matrix, Dii (,) = Wi (, j); N x: a vector in {1,! 1}, xi ( ) = 1 " i# A.! Rewriting Normalized Cut in matrix form: cut(a,b) NCut(A,B) = + assoc(a,v) cut(a,b) assoc(b,v) j i, j T T (1 + x) ( D! W)(1 + x) (1! x) ( D! W)(1! x) Dii (,) x 0 ; i > = + k = T T k1 D1 (1! k)1 D1 Dii (, ) =... " " i Slide credit: Jitendra Malik

73 Some More Math Slide credit: Jitendra Malik

74 NCuts as a Generalized Eigenvalue Problem! After simplification, we get T y ( D! W) y T NCut( A, B) =, with yi "{1,! b}, y D1 = 0. T y Dy! This is a Rayleigh Quotient!! Solution given by the generalized eigenvalue problem!! Solved by converting to standard eigenvalue problem! Subtleties ( D " W ) y =! Dy with!! Smallest eigenvector is with eigenvalue 0 (and )!! Optimal solution is second smallest eigenvector!! Gives continuous result must convert into discrete values of y This is hard, as y is discrete! Relaxation: y is continuous. Slide adapted from Alyosha Efros

75 NCuts Example NCuts segments Smallest eigenvectors Slide credit: B. Leibe Image source: Shi & Malik

76 NCuts: Overall Procedure 1.! Construct a weighted graph G=(V,E) from an image. 2.! Connect each pair of pixels, and assign graph edge weights, Wij (, ) = Prob. that iand jbelong to the same region. 3.! Solve ( D " W ) y =! Dy for the smallest few eigenvectors. This yields a continuous solution. 4.! Threshold eigenvectors to get a discrete cut!! This is where the approximation is made (we re not solving NP). 5.! Recursively subdivide if NCut value is below a prespecified value. NCuts Matlab code available at Slide credit: Jitendra Malik

77 Color Image Segmentation with NCuts Slide credit: Steve Seitz Image Source: Shi & Malik

78 Results with Color & Texture

79 Summary: Normalized Cuts! Pros:!! Generic framework, flexible to choice of function that computes weights ( affinities ) between nodes!! Does not require any model of the data distribution! Cons:!! Time and memory complexity can be high! Dense, highly connected graphs! many affinity computations! Solving eigenvalue problem!! Preference for balanced partitions! If a region is uniform, NCuts will find the modes of vibration of the image dimensions Slide credit: Kristen Grauman

80 Segmentation: Caveats! We ve looked at bottom-up ways to segment an image into regions, yet finding meaningful segments is intertwined with the recognition problem.! Often want to avoid making hard decisions too soon! Difficult to evaluate; when is a segmentation successful? Slide credit: Kristen Grauman

81 Topics of This Lecture! Introduction!! Gestalt principles!! Image segmentation! Segmentation as clustering!! k-means!! Feature spaces!! Mixture of Gaussians, EM! Model-free clustering: Mean-Shift! Graph theoretic segmentation: Normalized Cuts! Interactive Segmentation with GraphCuts

82 Markov Random Fields! Allow rich probabilistic models for images! But built in a local, modular way!! Learn local effects, get global effects out Observed evidence Hidden true states Neighborhood relations Slide credit: William Freeman

83 MRF Nodes as Pixels (or Patches) Image pixels!( x, y ) i i!( x, x ) i j Image states (e.g. foreground/background) Slide adapted from William Freeman

84 Network Joint Probability Pxy (, ) =!( x, y) "( x, x) states Image # # i Image-state compatibility function i i i j i, j Local observations state-state compatibility function Neighboring nodes Slide adapted from William Freeman

85 Energy Formulation! Joint probability Pxy (, ) =!( x, y) "( x, x) # # i i i i j i, j! Maximizing the joint probability is the same as minimizing the log % % log Pxy (, ) = log #( x, y) + log $ ( x, x) i % % i i i j i, j Exy (, ) =!( x, y) + "( x, x) i i i i j i, j! This is similar to free-energy problems in statistical mechanics (spin glass theory). We therefore draw the analogy and call E an energy function.!! and " are called potentials. Slide credit: B. Leibe

86 Energy Formulation! Energy function Exy (, ) =!( x, y) + "( x, x)! Unary potentials!!! Encode local information about the given pixel/patch!! How likely is a pixel/patch to be in a certain state? (e.g. foreground/background)?! Pairwise potentials " i i i j i, j Unary potentials!! Encode neighborhood information Pairwise potentials!( x, y )!! How different is a pixel/patch s label from that of its neighbor? (e.g. here independent of image data, but later based on intensity/color/texture difference) Slide adapted from B. Leibe # # i i i! ( x, x ) i j

87 Energy Minimization! Goal:!! Infer the optimal labeling of the MRF.!( x, y )! Many inference algorithms are available, e.g.!! Gibbs sampling, simulated annealing!! Iterated conditional modes (ICM)!! Variational methods!! Belief propagation!! Graph cuts! Recently, Graph Cuts have become a popular tool!! Only suitable for a certain class of energy functions!! But the solution can be obtained very fast for typical vision problems (~1MPixel/sec). i i! ( x, x ) i j Slide credit: B. Leibe

88 Graph Cuts for Optimal Boundary Detection! Idea: convert MRF into source-sink graph Minimum cost cut can be computed in polynomial time (max-flow/min-cut algorithms) hard constraint n-links t s a cut hard constraint Slide adapted from Yuri Boykov [Boykov & Jolly, ICCV 01]

89 Simple Example of Energy!! E( L) = D ( L ) + w $ % ( L D p (t) t s Regional term p p t-links p a cut D p (s) pq" N Boundary term pq n-links p w! # L q ) L p!{ s, t} & ' I exp %( $ 2) pq pq = 2!I pq (binary segmentation) # "! Slide credit: Yuri Boykov

90 Adding Regional Properties D p (t) n-links t a cut Regional bias example I s Suppose are given expected intensities of object and background t-link w pq s t-link D p (s) t and I ( s 2 2 D ( s) # exp " I " I / 2! ) NOTE: hard constrains are not required, in general. p p ( t 2 2 " I I / 2 ) D ( t) # exp "! p p Slide credit: Yuri Boykov [Boykov & Jolly, ICCV 01]

91 Adding Regional Properties D p (t) n-links t a cut expected intensities of object and background I s and can be re-estimated I t t-link EM-style optimization w pq s t-link D p (s) (" I p s 2 I / 2 2) (" I t 2 I / 2 2) D ( s) # exp "! p D ( t) # exp "! p p Slide credit: Yuri Boykov [Boykov & Jolly, ICCV 01]

92 Adding Regional Properties! More generally, regional bias can be based on any intensity models of object and background D p (s) D p (t) t s a cut D ( L ) =! log Pr( I L ) Pr( Pr( I p I p t) s) p p p p given object and background intensity histograms I p I Slide credit: Yuri Boykov [Boykov & Jolly, ICCV 01]

93 How to Set the Potentials? Some Examples! Color potentials!! e.g. modeled with a Mixture of Gaussians! Edge potentials!! e.g. a contrast sensitive Potts model where!( xi, yi; "!) = log $ "!( xi, kpk ) ( xi) N( yi; yk, # k)!( x, x, g ( y); " ) = $ " T g ( y) #( x % x ) i j ij!! ij i j g ( y) = e! ij " y " y i ( 2 )! = " # 2 avg yi y j! Parameters # $, # % need to be learned, too! j 2 k Slide credit: B. Leibe [Shotton & Winn, ECCV 06]

94 How Does it Work? The s-t-mincut Problem Source v 1 v Sink 4 Graph (V, E, C) Vertices V = {v 1, v 2... v n } Edges E = {(v 1, v 2 )...} Costs C = {c (1, 2)...} Slide credit: Pushmeet Kohli

95 The s-t-mincut Problem What is an st-cut? Source v 1 v Sink = 16 An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T Slide credit: Pushmeet Kohli

96 The s-t-mincut Problem What is an st-cut? Source v 1 v Sink = 7 An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost Slide credit: Pushmeet Kohli

97 History of Maxflow Algorithms Augmenting Path and Push-Relabel n: #nodes m: #edges U: maximum edge weight Algorithms assume nonnegative edge weights Slide credit: Andrew Goldberg

98 How to Compute the s-t-mincut? Source v 1 v Sink 4 Solve the dual maximum flow problem Compute the maximum flow between Source and Sink Constraints Edges: Flow < Capacity Nodes: Flow in = Flow out Min-cut/Max-flow Theorem In every network, the maximum flow equals the cost of the st-mincut Slide credit: Pushmeet Kohli

99 Maxflow Algorithms Source v 1 v 2 5 Flow = Sink Slide credit: Pushmeet Kohli 9 4 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

100 Maxflow Algorithms Source v 1 v Flow = Sink Slide credit: Pushmeet Kohli 9 4 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

101 Maxflow Algorithms Flow = Source v 1 v Sink Slide credit: Pushmeet Kohli 9 4 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

102 Maxflow Algorithms Source v 1 v Flow = Sink Slide credit: Pushmeet Kohli 9 4 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

103 Maxflow Algorithms Source v 1 v 2 3 Flow = Sink Slide credit: Pushmeet Kohli 9 4 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

104 Maxflow Algorithms Source v 1 v Flow = Sink Slide credit: Pushmeet Kohli 9 4 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

105 Maxflow Algorithms Flow = Source v 1 v Sink Slide credit: Pushmeet Kohli 5 0 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

106 Maxflow Algorithms Source v 1 v 2 3 Flow = Sink Slide credit: Pushmeet Kohli 5 0 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

107 Maxflow Algorithms Source v 1 v Flow = Sink Slide credit: Pushmeet Kohli 5 0 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

108 Maxflow Algorithms Flow = Source v 1 v Sink Slide credit: Pushmeet Kohli 4 0 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

109 Maxflow Algorithms Source v 1 v Flow = Sink Slide credit: Pushmeet Kohli 5 0 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

110 Maxflow Algorithms Source v 1 v Flow = Sink Slide credit: Pushmeet Kohli 5 0 Augmenting Path Based Algorithms 1.! Find path from source to sink with positive capacity 2.! Push maximum possible flow through this path 3.! Repeat until no path can be found Algorithms assume non-negative capacity

111 Maxflow in Computer Vision! Specialized algorithms for vision problems!! Grid graphs!! Low connectivity (m ~ O(n))! Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004]!! Finds approximate shortest augmenting paths efficiently!! High worst-case time complexity!! Empirically outperforms other algorithms on vision problems!! Efficient code available on the web Slide credit: Pushmeet Kohli

112 When Can s-t Graph Cuts Be Applied? E( L) = Regional term!! s-t graph cuts can only globally minimize binary energies that are submodular. [Boros & Hummer, 2002, Kolmogorov & Zabih, 2004]! Non-submodular cases can still be addressed with some optimality guarantees.!! Current research topic p E E(L) can be minimized by s-t graph cuts p ( L t-links p )! +! pq" N Boundary term E( L p n-links, L q ) L p!{ s, t} E ( s, s) + E( t, t)! E( s, t) + E( t, s) Submodularity ( convexity ) Slide credit: B. Leibe

113 Dealing with Non-Binary Cases! For image segmentation, the limitation to binary energies is a nuisance.! Binary segmentation only! We would like to solve also multi-label problems.!! NP-hard problem with 3 or more labels! There exist some approximation algorithms which extend graph cuts to the multi-label case!! &-Expansion!! &'-Swap! They are no longer guaranteed to return the globally optimal result.!! But &-Expansion has a guaranteed approximation quality (2- approx) and converges in a few iterations. Slide credit: B. Leibe

114 #-Expansion Move! Basic idea:!! Break multi-way cut computation into a sequence of binary s-t cuts. & other labels Slide credit: Yuri Boykov

115 #-Expansion Algorithm 1.! Start with any initial solution 2.! For each label # in any order 1.! Compute optimal #-expansion move (s-t graph cuts): set the label of each node to alpha or leave to current label (so s = alpha and t = current) 2.! Decline the move if there is no energy decrease 3.! iterate to 2.! Stop when no expansion move would decrease energy! why good? " each move is optimal within a very large set of possible segmentations (2 N ) Slide adapted from Yuri Boykov

116 #-Expansion Moves! In each a-expansion a given label # grabs space from other labels initial solution -expansion -expansion -expansion -expansion -expansion -expansion -expansion For each move we choose the expansion that gives the largest decrease in the energy: binary optimization problem Slide credit: Yuri Boykov

117 GraphCut Applications: GrabCut! Interactive Image Segmentation [Boykov & Jolly, ICCV 01]!! Rough region cues sufficient!! Segmentation boundary can be extracted from edges! Procedure!! User marks foreground and background regions with a brush " get initial segmentation " correct by additional brush strokes User segmentation cues Additional segmentation cues Slide adapted from Matthieu Bray

118 GrabCut: Data Model Foreground color Background color! Obtained from interactive user input!! User marks foreground and background regions with a brush!! Alternatively, user can specify a bounding box Global optimum of the unary energy Slide adapted from Carsten Rother

119 GrabCut: Coherence Model! An object is a coherent set of pixels: How to choose (? Slide credit: Carsten Rother ( mn, )& C [ ] %! ym% yn "( xy, ) = # ( $ xn ' xm e 25 2 Error (%) over training set:

120 Iterated Graph Cuts R Foreground & Foreground Background Result Background Background G Color model (Mixture of Gaussians) Energy after each iteration Slide credit: Carsten Rother

121 GrabCut: Example Results

122 Improving Efficiency of Segmentation! Problem: Images contain many pixels!! Even with efficient graph cuts, an MRF formulation has too many nodes for interactive results.! Efficiency trick: Superpixels!! Group together similar-looking pixels for efficiency of further processing.!! Cheap, local oversegmentation!! Important to ensure that superpixels do not cross boundaries! Several different approaches possible!! Superpixel code available here!! Slide credit: B. Leibe Image source: Greg Mori

123 Superpixels for Pre-Segmentation Slide credit: B. Leibe Graph structure Speedup

124 Summary: Graph Cuts Segmentation! Pros!! Powerful technique, based on probabilistic model (MRF).!! Applicable for a wide range of problems.!! Very efficient algorithms available for vision problems.!! Becoming a de-facto standard for many segmentation tasks.! Cons/Issues!! Graph cuts can only solve a limited class of models! Submodular energy functions! Can capture only part of the expressiveness of MRFs!! Only approximate algorithms available for multi-label case Slide credit: B. Leibe