Unit 3 : Image Segmentation

Transcription

1 Unit 3 : Image Segmentation K-means Clustering Mean Shift Segmentation Active Contour Models Snakes Normalized Cut Segmentation CS

2 Histogram-based segmentation Goal Break the image into K regions (segments) Solve this by reducing the number of colors to K and mapping each pixel to the closest color CS

3 Histogram-based segmentation Goal Break the image into K regions (segments) Solve this by reducing the number of colors to K and mapping each pixel to the closest color Here s what it looks like if we use two colors CS

4 Clustering How to choose the representative colors? This is a clustering problem! G G Objective R Each point should be as close as possible to a cluster center Minimize sum squared distance of each point to closest center R CS

5 Break it down into subproblems Suppose I tell you the cluster centers c i Q: how to determine which points to associate with each c i? A: for each point p, choose closest c i Suppose I tell you the points in each cluster Q: how to determine the cluster centers? A: choose c i to be the mean of all points in the cluster CS

6 K-means clustering K-means clustering algorithm 1. Randomly initialize the cluster centers, c 1,..., c K 2. Given cluster centers, determine points in each cluster For each point p, find the closest c i. Put p into cluster i 3. Given points in each cluster, update c i to be the mean of all points in cluster i 4. If c i have changed, repeat Step 2 Properties Will always converge to some solution Can be a local minimum does not always find the global minimum of objective function: CS

7 K-Means++ Can we prevent arbitrarily bad local minima? 1.Randomly choose first center. 2.Pick new center with prob. proportional to: (contribution of p to total error) 3.Repeat until k centers. CS

8 Probabilistic clustering Basic questions what s the probability that a point x is in cluster m? what s the shape of each cluster? K-means doesn t answer these questions Basic idea instead of treating the data as a bunch of points, assume that they are all generated by sampling a continuous function This function is called a generative model defined by a vector of parameters θ CS

9 Mixture of Gaussians One generative model is a mixture of Gaussians (MOG) K Gaussian blobs with means μ b covariance matrices V b, dimension d blob b defined by blob b is selected with probability the likelihood of observing x is a weighted mixture of Gaussians where CS

10 Expectation maximization (EM) Goal find blob parameters θ that maximize the likelihood function: Approach: 1. E step: given current guess of blobs, compute ownership of each point 2. M step: given ownership probabilities, update blobs to maximize likelihood function 3. repeat until convergence CS

11 E-step EM details compute probability that point x is in blob i, given current guess of θ M-step compute probability that blob b is selected N data points mean of blob b covariance of blob b CS

12 Applications of EM Turns out this is useful for all sorts of problems CS 6550 any clustering problem any model estimation problem missing data problems finding outliers segmentation problems... segmentation based on color segmentation based on motion foreground/background separation 11

13 Finding Modes in a Histogram How Many Modes Are There? Easy to see, hard to compute CS

14 Mean Shift [Comaniciu & Meer] CS 6550 Iterative Mode Search 1. Initialize random seed, and window W 2. Calculate center of gravity (the mean ) of W: 3. Translate the search window to the mean 4. Repeat Step 2 until convergence 13

15 Mean-Shift Approach Initialize a window around each point See where it shifts this determines which segment it s in Multiple points will shift to the same segment CS

16 Mean-shift for image segmentation Useful to take into account spatial information instead of (R, G, B), run in (R, G, B, x, y) space D. Comaniciu, P. Meer, Mean shift analysis and applications, 7th International Conference on Computer Vision, Kerkyra, Greece, September 1999, CS 6550 More Examples: 15

17 What is Mean Shift? A tool for: Finding modes in a set of data samples, manifesting an underlying probability density function (PDF) in R N Non-parametric Density Estimation Discrete PDF Representation Data Non-parametric Density GRADIENT Estimation (Mean Shift) PDF Analysis CS

18 Non-Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Data point density implies PDF value! Assumed Underlying PDF Real Data Samples CS

19 Non-Parametric Density Estimation Assumed Underlying PDF Real Data Samples CS

20 Non-Parametric? Density Estimation Assumed Underlying PDF Real Data Samples CS

21 Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF PDF( x) = i c e i ( x-μ ) 2 i 2 i 2 Estimate Assumed Underlying PDF Real Data Samples CS

22 Kernel Density Estimation Parzen Windows - Function Forms n 1 P( x) K( x - xi) A function of some finite number of data points n i 1 x 1 x n In practice one uses the forms: Data d K( ) ck( xi ) x or K( x) ck x i1 Same function on each dimension Function of vector length only CS

23 Kernel Density Estimation Various Kernels n 1 P( x) K( x - xi) A function of some finite number of data points n i 1 x 1 x n Examples: Epanechnikov Kernel K E ( x) 2 c x x otherwise Data Uniform Kernel K U ( x) c x 1 0 otherwise Normal Kernel KN 1 ( x) cexp 2 x 2 CS

24 Kernel Density Estimation Gradient n 1 P( x) K( x - x ) n i 1 i Give up estimating the PDF! Estimate ONLY the gradient Using the Kernel form: We get : 2 x - x i K( x - xi ) ck h Kernel bandwidth n ig n n i c c x i1 P( x) ki gi n n i1 n i1 gi i1 x g( x) k( x) CS

25 CS ( ) n i i n n i i i n i i i i g c c P k g n n g x x x Computing The Mean Shift Yet another Kernel density estimation! Simple Mean Shift procedure: Compute mean shift vector Translate the Kernel window by m(x) ( ) n i i i n i i g h g h x - x x m x x x - x g( ) ( ) k x x

26 CS

27 Mean Shift Properties Automatic convergence speed the mean shift vector size depends on the gradient itself. Near maxima, the steps are small and refined Adaptive Gradient Ascent Convergence is guaranteed for infinitesimal steps only infinitely convergent, (therefore set a lower bound) For Uniform Kernel ( ), convergence is achieved in a finite number of steps Normal Kernel ( ) exhibits a smooth trajectory, but is slower than Uniform Kernel ( ). CS

28 Real Modality Analysis An example Window tracks signify the steepest ascent directions CS

29 Mean Shift Strengths & Weaknesses Strengths : Application independent tool Suitable for real data analysis Does not assume any prior shape (e.g. elliptical) on data clusters Can handle arbitrary feature spaces Only ONE parameter to choose Weaknesses : The window size (bandwidth selection) is not trivial Inappropriate window size can cause modes to be merged, or generate additional shallow modes Use adaptive window size h (kernel bandwidth) has a physical meaning, unlike K-Means CS

30 Mean Shift Applications Clustering Image segmentation Discontinuity-preserving filtering Mean-shift tracking CS

31 Clustering Cluster : All data points in the attraction basin of a mode Attraction basin : the region for which all trajectories lead to the same mode CS Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer

32 Clustering Synthetic Examples Simple Modal Structures Complex Modal Structures CS

33 Feature space: L*u*v representation Clustering Real Example Initial window centers Modes found Modes after pruning Final clusters CS

34 Clustering Real Example L*u*v space representation CS

35 Clustering Real Example 2D (L*u) space representation Final clusters Not all trajectories in the attraction basin reach the same mode CS

36 Discontinuity Preserving Smoothing Feature space : Joint domain = spatial coordinates + color space s r x x K( x) C ks kr h s h r Meaning : treat the image as data points in the spatial and gray level domain Image Data (slice) Mean Shift vectors Smoothing result CS Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer

37 Mean-shift Filtering Example CS

38 Discontinuity Preserving Smoothing The effect of window size in spatial and range spaces CS

39 Discontinuity Preserving Smoothing Example CS

40 Discontinuity Preserving Smoothing Example CS

41 Segmentation Algorithm: Run Filtering (discontinuity preserving smoothing) Cluster the clusters which are closer than window size Image Data (slice) Mean Shift vectors Smoothing result Segmentation result Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer CS

42 CS

43 Segmentation Example when feature space is only gray levels CS

44 Segmentation Example CS

45 Segmentation Example CS

46 Segmentation Example CS

47 Segmentation Example CS

48 Active Contour Model - Snake Energy-minimizing spline guided by external constraint guided by external constraint forces and pulled by image forces toward features. CS

49 Introduction First an initial spline (snake) is placed on the image, and then its energy is minimized. Local minima of this energy correspond to desired image properties. CS

50 Snake Behavior A snake falls into the closest local energy minimum. The local minima of the snake energy comprise the set of alternative solutions A higher level knowledge is needed to choose the correct one from these solutions High level reasoning User interaction These high-level methods can interact with the contour model by pushing it toward an appropriate local minimum CS

51 Snake Behavior They rely on other mechanisms to place them near the desired contour. The existence of such an initializer is application dependent. Even in the case of manual initialization, snakes are quite powerful in refining the user s input. Basically, snakes are trying to match a deformable model to an image by means of energy minimization. CS

52 Snake Algorithm The snake is defined parametrically as v(s)=[x(s),y(s)], where s[0,1] is the normalized arc length along the contour. The energy functional to be minimized may be written as * snake 1 0 E E ( v( s)) ds snake int 0 image 0 E ( v( s)) ds E ( v( s)) ds E ( v( s)) ds forces E int : internal energy of the spline due to bending. E image : image forces pushing the snake toward image features, such as edges. E forces : external constraints forces responsible for putting the snake near the desired local minimum. It may come from: Higher level interpretation or User interaction, etc CS

53 Internal Energy The internal spline energy can be written as dv E ( s) ( s) int ds 2 d d 2 2 v s 2 where (s), (s) specify the elasticity and stiffness of the snake. CS

54 Internal Energy The snake is a controlled continuity spline Regularizes the problem The first order derivative Vs(s) makes the spline act like a membrane ( elasticity ). The second order derivative Vss(s) makes it act like a thin-plate ( rigidity ). α(s) and β(s) controls the relative importance of membrane and thin-plate terms Setting β(s)=0 for a point allows the snake to become second-order discontinuous and develop a corner. CS

55 Image Forces The image forces E image are derived from the image data over which the snake lies. Three important features the snake can be attracted to are line, edge and termination functions. The total image energy can be expressed as a weighted combination of the three. E image line E line edge E edge term E term The simplest useful image functional is the image intensity E line = I(x, y) Depending on line the snake is attracted to dark or light lines. CS

56 Edge based and Termination Functional The edge-based functional 2 2 E I ( x, y) or I ( x, y) edge attracts the snake to contours with large image gradients, that is, to locations of strong edges. The termination functional can be obtained by a function checking the curvature of level lines in a slightly smoothed image. CS

57 Scale-Space Minimization Minimization by scale continuation: 1. Spatial smooting the edge or line functional Eedge = (G σ 2 I) 2, where, where G σ is a Gaussian standard deviation σ Minima lie on zero crossings of G σ 2 I (~edges) 2. Snake comes to equilibrium on a blurry energy 3. Slowly reduce the blurring CS

58 Motion Tracking Once a snake finds a feature, it locks on. If the feature begins to move, the snake will track the same local minimum. Fast motion could cause the snake to flip into a different minimum. CS

59 Snake Energy Minimization A contour is defined to lie in the position in which the snake reaches a local energy minimum. The functional to be minimized is 2 2 * E E ( v( s)) ds snake snake The spline v(s) which minimizes E* snake must satisfy d ds E 2 d x 2 ds E 2 d y 2 ds E v E Solution is rather complex. Several methods exist, e.g. dynamic programming, neural nets. Problems with numerical instability. CS d ds s v 0

60 Snake Energy Minimization When α(s) and β(s) are constant, we get two independent Euler-Lagrange equations. When α(s) and β(s) are not constant then it is simpler to use a discrete formulation: CS

61 Snake Energy Minimization Let f x (i) = E ext / x i where derivatives are approximated by finite differences if they cannot be computed analitically. The corresponding Euler equations are In matrix form where A is a is a pentadiagonal banded matrix: CS

62 Snake Energy Minimization γ is the step size. Taking into account the derivatives requires changing A at each iteration. Speed up: at each iteration. We assume that f x and and f y are constant during a time step, i.e. explicit Euler method w.r.t the external forces. internal forces are specified by A, thus we can evaluate the time derivative at t rather than t-1 CS

63 Snake Energy Minimization At equilibrium, the time derivative vanishes. The Euler equations can be solved by matrix inversion: The inverse can be calculated by LU decomposition in O(n) time. CS

64 Balloon Snake CS

65 Examples Hand People CS 6550 Examples from Julien Jomier 66

66 Examples Highway Heart CS 6550 Examples from Julien Jomier 67

67 Problems with snakes Snakes sometimes degenerate in shape by shrinking and flattening. Stability and convergence of the contour deformation process may be unpredictable. Solution: Add some constraints: External forces or Physical properties Initialization is not straightforward. Solution: Manual and Statistics CS

68 Weakness of traditional snakes Extremely sensitive to parameters. Small capture range. No external force acts on points which are far away from the boundary. Convergence is dependent on initial position. CS

69 Weakness of traditional snakes (continued) Fails to detect concave boundaries. External force can t pull control points into boundary concavity. CS

70 Gradient Vector Flow (GVF) The GVF field is defined to be a vector field V(x,y) =(u(x,y), v(x,y)) V(x,y) is defined such that it minimizes the energy functional f(x,y) is the edge map of the image. CS

71 GVF CS

72 CS

73 Results CS

74 CS

75 GVF Snake Example CS

76 CS

77 GVF Snake Example CS

78 Shape Priors Snakes sometimes exhibit too many degrees of freedom, making it more likely to be trapped in local minima during their evolution. One solution to this problem is to control the snake with fewer degrees of freedom through the use of B-spline approximation, i.e. B-snake CS

79 Point Distribution Model Resistor shapes Control points Distribution of point locations (after alignment) CS

80 Shape Prior Constraint Covariance matrix C is determined from the training data Using eigenvalue analysis, i.e. Principal Component Analysis (PCA), the covariance matrix can be written as The resulting point distribution model can be written as Quadratic penalty for shape constraint CS

81 Graph-Based Image Segmentation Represent tokens using a weighted graph. affinity matrix Cut up this graph to get subgraphs with strong interior links CS

82 Normalized-Cut Algorithm CS

83 Images as graphs q p C pq c Fully-connected graph node for every pixel link between every pair of pixels, p,q cost c pq for each link c pq measures dissimilarity dissimilarity: difference in color and position CS

84 Segmentation by Graph Cuts w A B C Break Graph into Segments Delete links that cross between segments Easiest to break links that have high cost similar pixels should be in the same segments dissimilar pixels should be in different segments CS

85 CS

86 Measuring Affinity Intensity Distance affx, y exp i affx, y exp d Ix Iy 2 x y 2 Texture affx, y exp t cx cy 2 CS

87 Scale affects affinity CS

88 Problem with Minimum Cut CS

89 Cuts in a graph Link Cut A set of links whose removal makes a graph disconnected cost of a cut: Normalized Cut a cut penalizes large segments fix by normalizing for size of segments B CS

90 Eigenvectors and cuts Simplest idea: we want a vector a giving the association between each element and a cluster We want elements within this cluster to, on the whole, have strong affinity with one another We could maximize But need the constraint a T Aa a T a 1 This is an eigenvalue problem - choose the eigenvector of A with largest eigenvalue CS

91 Example eigenvector points eigenvector matrix CS

92 More than two segments Two options Recursively split each side to get a tree, continuing till the eigenvalues are too small Use the other eigenvectors CS

93 Normalized cuts Current criterion evaluates within cluster similarity, but not across cluster difference Instead, we d like to maximize the within-cluster similarity compared to the across cluster difference Write graph as V, one cluster as A and the other as B Maximize assoc(a, A) assoc(b,b) assoc(a,v ) assoc(b,v ) i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph CS

94 Normalized cuts Write a vector y whose elements are 1 if item is in A, -b if it s in B Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, 1 is the vector with all ones. Criterion becomes y T D W y min y y T Dy and we have a constraint y T D1 0 This is hard to do, because y s values are quantized CS

95 Normalized cuts Instead, solve the generalized eigenvalue problem max min y y T D Wy subject to y T Dy 1 which gives D W y Dy Now look for a quantization threshold that maximizes the criterion --- i.e all components of y above that threshold go to one, all below go to -b CS

96 Normalize Cut in Matrix Form W is the cost matrix : W ( i, j ) w i, D is the sum of costs from node i : D ( i, i ) After lots of math, we get: j ; W ( i, j ); T y (D W)y T Ncut ( A, B), with y {1, }, y D1 T i b y Dy Solution given by generalized eigenvalue problem: (D W)y λdy Solved by converting to standard eigenvalue problem: D (D W)D z λ z, where z D y j 1 0. optimal solution corresponds to second smallest eigenvector for more details, see J. Shi and J. Malik, Normalized Cuts and Image Segmentation, IEEE Conf. Computer Vision and Pattern Recognition(CVPR), 1997 CS

97 Recursive two-way Ncut grouping algorithm CS

98 Simultaneous K-way cut CS

99 Example: brightness image CS

100 Experiments on Synthetic Images CS

101 Figure from Image and video segmentation: the normalised cut framework, by Shi and Malik, copyright IEEE, 1998 CS

102 F igure from Normalized cuts and image segmentation, Shi and Malik, copyright IEEE, 2000 CS

103 Color Image Segmentation CS

104 1. N-Cuts can also handle: Texture segmentation Motion segmentation but it needs proper similarity definition 2. General problem of defining feature similarity incorporating many cues/features is non-trivial CS

105 Information Fusion for Image Segmentation CS

106 Summary for Normalized Cut Segmentation Normalized cut presents a new optimality criterion for partitioning a graph into clusters. Ncut is normalized measure of disassociation and minimizing it is equivalent to maximizing association. The discrete problem corresponding to Min Ncut is NP-Complete. We solve an approximate version of the MinNcut problem by converting it into a generalized eigenvector problem. Nice results in image segmentation CS