Content-based Image and Video Retrieval Vorlesung, SS 2011 Image Segmentation 2.5.2011 / 9.5.2011
Image Segmentation One of the key problem in computer vision Identification of homogenous region in the image Partition an image into meaningful regions with respect to a particular application The segmentation is based on measurements taken from the image and might be greylevel, colour, texture, depth or motion (in video) Image segmentation and descriptors 2
Different Examples Search in image collections Find representations that make sense to the user and is related to picture content Video summarization / shot boundary detection Find similar frames, represent subsequences by key frame Finding people Specific detectors, part-based detectors Finding buildings Finding machine parts Background subtraction Image segmentation and descriptors 3
Motivation Before high-level reasoning on image, it can be broken down into it major structural components Necessary for extracting reasonable local features (color, texture, etc.) Simplify or change image representation into more meaningful one for ease of analysis Image segmentation and descriptors 4
Difficulties What is correct segmentation? No single correct answer Interpretation depends on prior world knowledge World knowledge is difficult to represent Alternative segmentations Image segmentation and descriptors 5
Correct Segmentation Image segmentation and descriptors 6
Good Segmentation? Typical assumptions (inspired from human vision): Intensity / color coherence Texture coherence Motion coherence Image segmentation and descriptors 7
Image Segmentation Categories: Pixel-based Segmentation Region-based Segmentation (Edge-based Segmentation) Graph-based Segmentation Superpixels Image segmentation and descriptors 8
Pixel-based Segmentation Thresholding Clustering Image segmentation and descriptors 9
Thresholding Determine the best threshold given a histogram of intensities Automatic thresholding P-tile method Mode method Local adaptive method Limitation of thresholding Use global information Ignore spatial relationships among pixels Image segmentation and descriptors 10
Thresholding Determine the best threshold given a histogram of intensities Simplest way to segment an image: separate light and dark regions T Image segmentation and descriptors 11
P-tile method Use the a priori knowledge about the size of the object: assume an object with size p Choose the threshold such that p% of the overall histogram is determined p% 100% Obviously limited use Image segmentation and descriptors 12
Mode-Method Find the peaks (modes) of the histogram and the local minimum between them Set threshold to the pixel value of the local minimum T 255 Not trivial to find peaks and local minimum on a noisy histogram Ignore local peaks Maximize peakiness Image segmentation and descriptors 13
Adaptive-Method One single global threshold does not work for uneven illumination Local adaptive method Divide an image into mxm subimages and determine a threshold for each subimage Image segmentation and descriptors 14
Double Threshoulding-Method Also called hysteresis thresholding Starting from a conservative initial threshold, determine the core parts of the object Continuing from this core part, grow this object by including neighboring pixels which are between and Image segmentation and descriptors 15
Clustering Process of partitioning a set of patterns into clusters find subsets of points which are close together Cluster pixels based on Intensity values Color properties Motion/optical flow properties Texture measurements etc. Input: set of measurements. Output: set of clusters and their centers Image segmentation and descriptors 16
Simple Clustering Approaches Agglomerative Clustering (Merging) 1. Make each point (pixel) a separate cluster 2. Merge clusters with smallest inter-cluster distance until clustering is satisfactory Divisive Clustering (Splitting) 1. Constract a single cluster using all points 2. Split clusters with largest inter-cluster distance until clustering is satisfactory Difficulties: Choice of inter-cluster distances Stopping criterion (how many cluster are there?) Image segmentation and descriptors 17
Segmentation by k-means Simple clustering methods use greedy approaches Alternative: Formulate an objective function that should be optimized Assuming that we know that there should be k-clusters, a good objective function would be Where x j is a point coordinate, c j is a cluster center If allocation of points to clusters where known, centers could be easily computed But this is not the case Image segmentation and descriptors 18
k-means algorithm Define iterative algorithm: Assume the cluster centers are known and allocate each point to closest cluster Assume allocation is known and choose new set of cluster centers. Each center is the mean of the points allocated to the that cluster Algorithm: Choose initial mean values for k regions Classify n pixels by assigning them to closest mean Recompute the means as the average of samples in their (new) classes Continue till there is no change in mean values Image segmentation and descriptors 19
Color Clustering Examples Clustering in RGB space Original images Segmented images 9 clusters 5 clusters 4 clusters Image segmentation and descriptors 20
Region-based Segmentation The main idea in region-based segmentation techniques is to identify different regions in an image that have similar features (gray level, colour, texture, etc.). There are two main region-based image segmentation techniques: Region merging Region splitting Image segmentation and descriptors 21
Formulation of Region-based Segmentation Let represent the entire image. Segmentation can be viewed as a process that partitions in to disjoint regions,, such that: a) b) is a connected region, c) for all d) for e) for where is a logical predicate over points in set and is the empty set. Image segmentation and descriptors 22
Region Merging Merge two adjacent regions if they have similar properties according to some criterion. What does similar mean? Examples: similar average values : small spread of gray values : Note: non-transitiv A similar to B, and B similar to C does not imply that A is similar to C. Image segmentation and descriptors 23
Region Merging Start with an initial segmentation e.g. By thresholding Form the Region Adjacency Graph (RAG) Regions are the nodes Adjacency relations are the links Initial segmentation RAG Image segmentation and descriptors 24
Region Merging For each region in the image do: Consider its adjacent regions and test if they are similar If they are similar, merge them and update the RAG Repeat the merging steps until there are no more merges. merge Image segmentation and descriptors 25
Region Splitting Quad-tree decomposition: Subdivide the entire image successively into smaller and smaller quadrant regions until for any region, The subdivision is represented with quad tree Image segmentation and descriptors 26
The Quadtree Representation Quadtrees: Trees where nodes have 4 children Build quadtree: Nodes represent regions Every time a region is split, it s node give birth to 4 children Leaves are nodes for uniform regions R Image segmentation and descriptors 27
Region Splitting & Merging Splitting only results in adjacent regions with identical properties The final result can be obtained through merging the quadtree Siblings that are similar can be merged Image segmentation and descriptors 28
Edge-based Segmentations Based on detection of discontinuities, and segment the image along the discontinuities 3 basic types of gray-level discontinuities: points, lines, edges Edge detection is the most common approach for detecting meaningful discontinuities Image segmentation and descriptors 29
Edge Linking and Boundary Detection From intensity discontinuities to more general segmentation For example, from edge pixels to line segments Local processing Analysis of small neighborhood Strength and direction of the gradient of edge pixels Global processing Analysis of the whole image Global relationships between pixels Hough Transform Image segmentation and descriptors 30
Visual descriptors 31
Graph-Based Segmentation Analysis images by graph theory, image is mapped onto a graph Each pixel (sub region) in the image correspond to a vertex Typical connection between the nodes is 4 (or 8) connections. Vertices and edges have weights Pixel gray level value is usually assigned to vertex weight Weights associated with each link are based on some property of the connected pixels, e.g. intensity differences. Pixel Vertex Link Four-way connected graph Image segmentation and descriptors 32
Graph-Based Segmentation Insertion of links between similar intensities creates subtrees for these objects Every sub tree represents a region of the image Region 1 Methods: Spanning tree Graph cuts Markov random fields Region 2 Image segmentation and descriptors 33
Recursive Shortest Spanning Tree (RSST) Methods which does not depend on global information will not achieve good segmentation (Morris et al. 1986) => Dynamic construction of image tree Algorithm: Image is mapped onto the graph Four way connected graph is used Absolute values of the gray level differences between vertices are assigned to link weights Link weight = 70 Image segmentation and descriptors 34
Recursive Shortest Spanning Tree (RSST) Nodes, whose link yields the smallest value, are merged Chosen link is saved as part of the tree Remove duplicated links during merging Update the changed weights of links that are touching to newly formed region Change link weights Before merging Smallest weighted link After merging Remove duplicated link Image segmentation and descriptors 35
Recursive Shortest Spanning Tree Beginning from the original image, where each pixel represents a different region Most similar neighboring regions are merged iteratively until a user-defined region number is left Image segmentation and descriptors 36
Graph-based Clustering Represents tokens (nodes) with a weighted graph Affinity matrix, A (similar nodes have higher entries) Cut up the graph to get subgraphs with strong links (Graph Cut) Affinity matrix Image segmentation and descriptors 37
Measuring Affinity Intensity Distance Color Image segmentation and descriptors 38
Scale Affects Affinity Image segmentation and descriptors 39
Solution via Eigenvectors Idea: We want a vector w giving the association between each element and a cluster We want elements within this cluster to, on the whole, have strong affinity with one another Maximize, with extra constraint Optimize by method of Lagrange multiplier : Solution is an eigenvalue problem Choose the eigenvector of A with the largest eigenvalue Image segmentation and descriptors 40
Solution via Eigenvectors points Affinity matrix Image segmentation and descriptors 41
Normalized Cuts Previous criterion: maximize within cluster similarity Normalized graph cuts: maximize within cluster similarity while maximizing cross cluster difference Given graph G=(V, E), sets A and B are a disjoint partition of the vertices set V Cut(A,B) is a measure of similarity between the two sets w corresponds to the weight of the link Image segmentation and descriptors 42
Normalized Cuts Cut is a measure of association Minimize cut = maximize disassociation A B Normalized cuts : to avoid bias for partitioning out small sets of points, normalize the cut with the total edge connections to all the nodes in the graph, e.g. A B Image segmentation and descriptors 43
Normalized Cuts Segmentation criterion: Finding the minimal normalized cut is NPcomplete, so computationally infeasible An approximation can be obtained by solving a generalized eigenvalue problem Approximation is very good in practice For details see: Jianbo Shi and Jitendra Malik, Normalized Cuts and Image Segmentation, PAMI, vol. 22, No. 8, 2000 Image segmentation and descriptors 44
Criterion becomes Normalized Cuts is the generalized Rayleigh quotient, which can be optimized by solving the generalized eigenvalue problem for the second smallest generalized eigenvector Image segmentation and descriptors 45
Segmentation Results Image segmentation and descriptors 46
Segmentation Results Image segmentation and descriptors 47
Results on color segmentation Image segmentation and descriptors 48
Superpixel Segmentation 49
From pixels to superpixels Superpixels are homogenous image regions that align well with object boundaries Reduce image complexity From 307,200 pixels (VGA) down to 1,000 superpixels Belong to class of oversegmentation algorithms Make more segments than there are objects Segments can then be used as building blocks 50
Example algorithms I Normalized cuts (Mori 2004, Mori et al. 2005) Tubopixels (Levinshtein et al. 2009) Placing superpixel seeds in the image Growing the seeds based on geometric flows 51
Example algorithms II Superpixel Lattices (Moore et al. 2008) Graph cuts with topological constraints SLIC (Achanta et al. 2010) Clustering approach based on k-means 52
SLIC Superpixels I Introduced by Achanta et al. (2010) Iterative algorithm based on k-means clustering with linear time complexity Pixels are assigned to superpixels based on a combination of the color distance in LABspace and Euclidean distance in image space l,a,b: LAB values, x,y: position, S: size of superpixels, m: weight of distance term 53
SLIC Superpixels II Initialization: Superpixel centers are distributed across the image Clusters (i.e. the superpixels) are initialized with the color of the pixel in its center (LAB-space) Iteration: For each superpixel, get all pixels in an area twice the size of the superpixel around its center and compute the distance of each pixel to the superpixel Assign each pixel to the best matching superpixel Recompute the superpixel characteristics (e.g. color mean, center position) Repeat until convergence 54
Example applications Mitochondria segmentation on EM-images (Achanta et al. 2010) Mincut-algorithm on superpixels Trail detection for robot navigation (Rasmussen and Scott 2008) Group superpixels based on color similarity Match triangles to boundaries of the groups to detect the trails 55
Visual descriptors 56
Example System: BlobWorld The problem: query images (e.g. from the WEB) using image information The solution: segment images into roughly uniform regions and search based on feature vectors The features: Color Texture Location (i.e. spatial compactness) The segmentation algorithm: Expectation Maximization (EM), similar in conception to K- means clustering algorithm Image segmentation and descriptors 57
Example segmentations Image segmentation and descriptors 58
Querying User selects a weighting of color vs. texture giving a diagonal weight matrix Given a blob with feature vector, compared to another vector using Mahalanobis distance: Image segmentation and descriptors 59
Image segmentation and descriptors 60
Image segmentation and descriptors 61
Image segmentation and descriptors 62
Image segmentation and descriptors 63
Visual descriptors 64
References Jianbo shi and Jitendra Malik, Normalized Cuts and Image Segmentation, PAMI, vol. 22, No. 8, 2000 C. Carson, S. Belongie, H. Greenspan, J. Marlik, Blobworld: image segmentation using expectationmaximization and its application to image querying, PAMI, vol. 24, issue. 8, pp. 1024-1038 Image segmentation and descriptors 65
References SLIC: R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and, S. Süsstrunk. SLIC Superpixels. EPFL Technical Report 149300, June, 2010. Normalized cuts implementation: G. Mori. Guiding Model Search Using Segmentation. In Proc. International Conference on Computer Vision, pages 1417-1423, 2005. G. Mori, X. Ren, A. A. Efros, and J. Malik. Recovering Human Body Configurations: Combining Segmentation and Recognition. In Proc. Computer Vision and Pattern Recognition, pages 326-333, 2004. Superpixel Lattices: A. P. Moore, S. J. D. Prince, J. Warrell, U. Mohammed, and G. Jones. Superpixel Lattices. In Proc. Computer Vision and Pattern Recognition, pages 1-8, 2008. TurboPixels: A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, S. J. Dickinson, and K. Siddiqi. TurboPixels: Fast Superpixels Using Geometric Flows. Transactions on Pattern Analysis and Machine Intelligence, 31(12):2290-2297, 2009. C. Rasmussen and D. Scott. Shape-Guided Superpixel Grouping for Trail Detection and Tracking. In Proc. International Conference on Intelligent Robots and Systems, pages 4092 4097, 2008. 66