Segmentation (continued)

Transcription

1 Segmentation (continued) Lecture 05 Computer Vision

2 Material Citations Dr George Stockman Professor Emeritus, Michigan State University Dr Mubarak Shah Professor, University of Central Florida The Robotics Institute Carnegie Mellon University Dr David A. Forsyth Professor, University of Illinois at Urbana- Champaign Dr Kristen Grauman Associate Professor, University of Texas at Austin

3 Suggested Readings Chapter 10 Linda G. Shapiro and George Stockman, Computer Vision, Upper Saddle River, NJ, Prentice Hall, Chapter 15 David A. Forsyth and Jean Ponce, Computer Vision A Modern Approach, 2 nd edition, Prentice Hall, Inc., Chapter 3 Mubarak Shah, Fundamentals of Computer Vision.

4 Clustering Main idea: Cluster together (pixels, tokens, etc.) that belong together Tokens: Whatever we need to group (pixels, points, surface elements, etc.)

5 Segmentation by Clustering R 1 R2 R 4 R 3 R 6 R 5

6 Segmentation by Clustering Main idea: Cluster together (pixels, tokens, etc.) that belong together Simple Methods Divisive clustering Split cluster along best boundary Repeat Agglomerative clustering Attach token to cluster it is closest to Repeat Inter-Cluster distance criteria Single-link clustering Distance between closest elements of clusters Complete-link clustering Maximum distance between elements of clusters Group-average clustering Average distance between elements of clusters

7 Segmentation by Clustering Dendrograms A representation of structure of hierarchy of clusters Dataset Dendogram

8 Segmentation by Clustering K Means Objective function Assuming we know there are k clusters and k is known Each cluster has center Center of i th cluster is c i The j th element to be clustered is described by feature vector x j (could be any set of features) The objective function can be given as: Φ (clusters, data) = Properties i clusters j elements of i'th cluster function that it is desired to maximize or minimize. x j i 2 Will always converge to some solution Can be a local minimum ; doesn't always find the global minimum The k-means problem is finding the least-squares assignment to centroids.

9 Segmentation by Clustering K Means Desirables Choose a fixed number of clusters Choose cluster centers Point-cluster allocations to minimize error Algorithm Fix cluster centers; allocate points to closest Fix allocation; compute best cluster centers

10 Segmentation by Clustering K Means Andrew Moore

11 Segmentation by Clustering K Means Andrew Moore

12 Segmentation by Clustering K Means Andrew Moore

13 Segmentation by Clustering K Means Andrew Moore

14 Segmentation by Clustering K Means Andrew Moore

15 Segmentation by Clustering K Means Select a feature vector for every pixel (color, texture, position, or combination of these etc.) Define a similarity measure between feature vectors (Usually Euclidean Distance) Apply K-Means Algorithm Apply Connected Components Algorithm Merge any components of size less than some threshold to an adjacent component that is most similar to it

16 Segmentation by Clustering K Means Image Clusters on intensity Clusters on color K-means clustering using intensity alone and color alone No requirement that clusters be spatially localized and they re not

17 Example - Segmenting Kristen Grauman, University of Texas at Austin How to determine the three main intensities? Kristen Grauman, University of Texas at Austin

18 Segmentation by Clustering K Means Depending on what we choose as the feature space, we can group pixels in different ways. Grouping pixels based on intensity similarity Feature space: intensity value (1-d)

19 Segmentation by Clustering K Means K=2 K=3 quantization of the feature space; segmentation label map

20 Segmentation by Clustering K Means Depending on what we choose as the feature space, we can group pixels in different ways. Grouping pixels based on color similarity B G R=255 G=200 B=250 R=245 G=220 B=248 Feature space: color value (3-d) Kristen Grauman R R=15 G=189 B=2 R=3 G=12 B=2

21 Segmentation by Clustering K Means Depending on what we choose as the feature space, we can group pixels in different ways. Grouping pixels based on intensity similarity Clusters based on intensity similarity don t have to be spatially coherent. Kristen Grauman

22 Segmentation by Clustering K Means Depending on what we choose as the feature space, we can group pixels in different ways. Grouping pixels based on intensity + position similarity Intensity Kristen Grauman Y X Both regions are black, but if we also include position (x,y), then we could group the two into distinct segments; way to encode both similarity & proximity.

23 Segmentation by Clustering K Means Color, brightness, position alone are not enough to distinguish all regions Kristen Grauman

24 Segmentation by Clustering K Means Depending on what we choose as the feature space, we can group pixels in different ways. Grouping pixels based on texture similarity F 1 F 2 Filter bank of 24 filters F 24 Feature space: filter bank responses (e.g., 24-d) Kristen Grauman

25 Dimension 2 (mean d/dy value) Segmentation by Clustering K Means Texture representation example Windows with primarily horizontal edges Both Dimension 1 (mean d/dx value) Kristen Grauman Windows with small gradient in both directions Windows with primarily vertical edges

26 Count Count Count Segmentation by Clustering K Means Segmentation with texture features Find textons by clustering vectors of filter bank outputs Describe texture in a window based on texton histogram Image Texton map Texton index Malik, Belongie, Leung and Shi. IJCV Adapted from Lana Lazebnik Texton index Texton index

27 Segmentation by Clustering K Means Kristen Grauman

28 Segmentation by Clustering K Means Pros Simple, fast to compute Converges to local minimum of within-cluster squared error Cons/issues Setting k? Sensitive to initial centers Sensitive to outliers Detects spherical clusters Assuming means can be computed Sample Code

29 Segmentation by Clustering Mean Shift The mean shift algorithm seeks modes or local maxima of density in the feature space image Feature space (L*u*v* color values)

30 Segmentation by Clustering Mean Shift Mean Shift Algorithm Choose a search window size. Choose the initial location of the search window. Compute the mean location (centroid of the data) in the search window. Center the search window at the mean location computed in Step 3. Repeat Steps 3 and 4 until convergence.

31 Segmentation by Clustering Mean Shift Search window Center of mass Slide by Y. Ukrainitz & B. Sarel Mean Shift vector

32 Segmentation by Clustering Mean Shift Search window Center of mass Slide by Y. Ukrainitz & B. Sarel Mean Shift vector

33 Segmentation by Clustering Mean Shift Search window Center of mass Slide by Y. Ukrainitz & B. Sarel Mean Shift vector

34 Segmentation by Clustering Mean Shift Search window Center of mass Slide by Y. Ukrainitz & B. Sarel Mean Shift vector

35 Segmentation by Clustering Mean Shift Search window Center of mass Slide by Y. Ukrainitz & B. Sarel Mean Shift vector

36 Segmentation by Clustering Mean Shift Search window Center of mass Slide by Y. Ukrainitz & B. Sarel Mean Shift vector

37 Segmentation by Clustering Mean Shift Search window Center of mass Slide by Y. Ukrainitz & B. Sarel

38 Segmentation by Clustering Mean Shift Cluster: all data points in the attraction basin of a mode Attraction basin: the region for which all trajectories lead to the same mode Slide by Y. Ukrainitz & B. Sarel

39 Segmentation by Clustering Mean Shift Mean Shift Segmentation Algorithm Convert the image into tokens (via color, gradients, texture measures etc). Choose initial search window locations uniformly in the data. Compute the mean shift window location for each initial position. Merge windows that end up on the same peak or mode. The data these merged windows traversed are clustered together.

40 Segmentation by Clustering Mean Shift Find features (color, gradients, texture, etc) Initialize windows at individual feature points Perform mean shift for each window until convergence Merge windows that end up near the same peak or mode Szeliski Book

41 Segmentation by Clustering Mean Shift

42 Segmentation by Clustering Mean Shift

43 Segmentation by Clustering Mean Shift Pros: Does not assume shape on clusters One parameter choice (window size) Generic technique Find multiple modes Cons: Selection of window size Does not scale well with dimension of feature space Kristen Grauman

44 Segmentation by Graph theoretic clustering Graph Theory A graph is a non-empty finite set of vertices V along with a set E of 2- element subsets of V. The elements of V are called Vertices. The elements of E are called edges.

45 Segmentation by Graph theoretic clustering Represent tokens using A weighted undirected graph G = (V,E) Nodes are points in the feature space Fully connected graph The weight associated are generally called affinity measures and Edge weight w(i,j) is a function of the similarity between nodes i and j. The graphs can be represented by weight or affinity matrices Task: Cut up this graph to get sub-graphs with strong interior links, i.e. Partition the set V into disjoint sets V 1,..,V n, s.t. similarity among nodes in V i is high and similarity across V i and V j is low.

46 Segmentation by Graph theoretic clustering Segmentation by Graph Cuts p w pq q w A B C Break Graph into Segments Want to delete links that cross between segments Easiest to break links that have low similarity (low weight) Similar pixels should be in the same segments Dissimilar pixels should be in different segments Source: Steve Seitz

47 Graph Representations a a b c d e b c d e V = {a,b,c,d,e} E = {(a,b), (a,e), (c,e), (d,e)} Adjacency Matrix Cardinality: No of vertices G = 5 Degree of vertex: Nodes connecting to a vertex deg(a) = 2

48 Weighted Graphs and Their Representations a a b c d e b c d e

49 Measuring Affinity Distance aff x, y exp d x y 2 Intensity Texture aff x, y exp i aff x, y exp t I x I y 2 c x c y 2

50 Minimum Cut A cut of a graph G is the set of edges S such that removal of S from G disconnects G Given a graph G=(V,E), the sets A and B are a disjoint partition of V Cut A B (, ) wpq, p A, q B Cut(A,B) is a measure of similarity between the two groups. Cut(A,B) is the sum of weights of all edges in V that have one end in A and the other in B Minimum cut is the cut of minimum weight, where weight of Cut <A,B> is given as A B There can be more than one minimum cut in a given graph

51 Minimum Cut

52 Minimum Cut Problem with minimum cut: Weight of cut proportional to number of edges in the cut; tends to produce small, isolated components. Cuts with lesser weight than the ideal cut Ideal Cut

53 Normalized Cut We d like to maximize the within cluster similarity compared to the across cluster difference In other words the approach is to cut the graph into two connected components such that the cost of the cut is a small fraction of the total affinity within each group

54 Normalized Cut Normalized cut is defined as: Cut( A, B) Cut( A, B) Assoc( A, V ) Assoc( B, V ) i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph Cut(A,B) = Sum of weights of all edges in V that touch both A and B Assoc(A,V) = Sum of weights of all edges that touch A Ncut value small when we get two clusters with many edges with high weights, and few edges of low weight between them Approximate solution for minimizing the Ncut value is Generalized eigenvalue problem det( A I) 0

55 Finding Minimum Normalized Cut

56 Finding Minimum Normalized Cut

57 Normalized Cut - Example results Berkley Segmentation Engine

58 Normalized Cut Pros: Generic framework, flexible to choice of function that computes weights ( affinities ) between nodes Does not require model of the data distribution Cons: Time complexity can be high Bias towards partitioning into equal segments Sample student project: