Image Segmentation. Lecture14: Image Segmentation. Sample Segmentation Results. Use of Image Segmentation

Transcription

1 Image Segmentation CSED441:Introduction to Computer Vision (2015S) Lecture14: Image Segmentation What is image segmentation? Process of partitioning an image into multiple homogeneous segments Process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics Compact representation for image data in terms of a set of components Bohyung Han CSE, POSTECH bhhan@postech.ac.kr General framework Tokens: whatever we need to group (pixels, points, surface elements) Bottom up segmentation: Tokens belong together because they are locally coherent Top down segmentation: Tokens belong together because they lie on the same object Some materials for this lecture are the courtesy of Prof. S. Lazebnik in UIUC and Prof. S. Savarese in Michigan. 2 Sample Segmentation Results Use of Image Segmentation Primitives for other tasks Group together similar looking pixels for efficiency of further processing Unsupervised and bottom up process 3 4

2 Use of Image Segmentation High level understanding of object or scene Separate an image into coherent objects or region Semantic segmentation Object detection Use of Image Segmentation Image manipulation (in computer graphics) 5 6 Challenges Challenges Image segmentation is an inherently difficult task due to Subjectivity of problem Weak and inconsistent features Requirement of high level inferences Subjectivity of problem Each person has a different concept of segment in an image. Even a person has different concepts of segment depending on the situation Input image Annotations by human 7 8

3 Challenges Challenges Weak and inconsistent features Requirement of high level inferences Unclear object boundaries Image noises and artifacts Background clutter Recovering missing boundaries Considering scene context Handling repetitive patterns 9 10 Evaluation How to evaluate image segmentation algorithm? We need to construct datasets and evaluation protocols. Datasets Berkeley segmentation dataset Microsoft COCO Evaluation methods Precision and recall of region boundaries Other measures for clustering algorithm evaluation (Adjusted) RAND index Normalized mutual information Datasets and Benchmarks 11 12

4 Berkeley Segmentation Dataset and Benchmark Berkeley Segmentation Dataset and Benchmark BSD300 Original dataset 200 training and 100 testing images 12,000 hand labeled segmentations of 1,000 Corel dataset images from 30 human subjects. Related paper D. Martin and C. Fowlkes and D. Tal and J. Malik. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. ICCV 2001 An example Training image # annotations #1107: 20 segments #1109: 13 segments #1113: 12 segments #1119: 10 segments #1121: 6 segments #1129: 7 segments Berkeley Segmentation Dataset and Benchmark BSD500 An extended version: 200 additional testing images New images were segmented by five different subjects on average. Performance is evaluated by measuring precision/recall on detected boundaries and three additional region based metrics. resources.html Related paper P. Arbelaez, M. Maire, C. Fowlkes and J. Malik. Contour Detection and Hierarchical Image Segmentation. TPAMI Downloads Pre compiled Matlabpackage Source code (for Linux/Mac, 32/64 bits) Pre computed results on the BSD500 and PASCAL VOC 2012 What is Microsoft COCO? Microsoft COCO Common Objects in COntext A new image recognition and segmentation dataset that was released in Summer Resources Paper Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C. Lawrence Zitnick: Microsoft COCO: Common Objects in Context. ECCV (5) 2014: arxiv link:

5 Characteristics More than 70 categories Object segmentation Recognition in context Multiple objects per image More than 300,000 images More than 2 Million instances Microsoft CoCo Tools Microsoft COCO In Python and Matlab (not yet ready though) Download and set up package Common API in Python and Matlab with minor differences for reading and visualizing COCO Annotation Instance annotation Instances: storing an array of instance object that contains category id and segmentation of an instance Categories: mapping of category id to category name Sentence annotation An array of sentence annotation that describes an image Each image has at least five sentences (few has more than five) Some Example Images Microsoft COCO

6 Superpixel Segmentation Superpixel Segmentation Concept Clustering closely related pixels in the feature space Typically used as a preprocessing step for various tasks Being very simple and fast Famous algorithms SLIC (Simple Linear Iterative Clustering) ERS (Entropy Rate Superpixel Segmentation) R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, SLIC Superpixels Compared to State of the art Superpixel Methods, TPAMI 34(11), 2012 M. Y Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, Entropy Rate Superpixel Segmentation, CVPR SLIC Similar to k means clustering except that It limits the search space for distance computation to a region proportional to the superpixel size It employs a weighted distance measure based on color and spatial proximity It provides control over the size and compactness of the superpixels Basic setup # of superpixels: need to be given by human Initial segmentation Regular grid The centers are moved to seed locations corresponding to the lowest gradient position in a 3x3 neighborhood: preventing initial centers from being located at edges Feature space: color+location,,,, Search space SLIC Approximate superpixel size: Each pixel computes the distance to the cluster centers if the pixel is within 2 2 area from the centers. Distance measure Post processing for orphaned pixels Assigned the label of the nearest cluster center using a connected components algorithm 23 24

7 SLIC ERS Some results Entropy Rate Superpixel (ERS) segmentation Formulating clustering as a graph partitioning problem Searching for a graph topology that has K connected components and maximizes the objective function Graph construction: Each vertex represents a pixel. Every vertex has a self loop. Each edge denotes the pairwise similarity between pixels. When an edge is not used to merge two components, we increase the edge weight of the self loop of the associated vertices ERS ERS Objective function Given a graph, Two criteria max subject to and : entropy rate of the random walk on the graph Entropy rate: quantifies the uncertainty of a stochastic process It prefers compact and homogeneous clusters. : balance of superpixel sizes It encourages clusters with similar sizes. Objective function Entropy rate: Stationary distribution max subject to and, log, and,,, :, 1, if if 27 28

8 Objective function Balancing: ERS max subject to and log, where 1,, Prefers superpixels with similar sizes ERS Characteristics of objective function A monotonically increasing submodular function Submodularity:, Monotonicity: for all In general, maximization of submodular functions leads to NP hard problems. Optimization A very simple greedy heuristic 0.5 approximation bound by exploiting matroid structure Matroid,, where is cycle free Performance ERS Nice identification of region boundaries Irregular shape of superpixels Relatively slow Use of Superpixels for Image Segmentation As a preprocessing The number of superpixels is extremely smaller than the number of pixels in an image. Each superpixel is very homogeneous. The features in superpixels are richer than pixels. There is a significant benefit in terms of computational complexity. Examples Superpixel segmentation + mean shift Superpixel segmentation + normalized cut 31 32

9 Demo: Superpixel Segmentaton %% Parameters cluster_size = 61; num_of_clusters = 50; % cluster_size = 43; num_of_clusters = 100; % cluster_size = 29; num_of_clusters = 200; % cluster_size = 20; num_of_clusters = 400; %% SLIC [Y_slic, time_slic, bmap_slic, K_slic, out_slic, out2_slic]... = slic_superpixel(fname, cluster_size); Image Segmentation by Other Methods %% ERS [Y_ers, time_ers, bmap_ers, K_ers, out_ers, out2_ers]... = ers_superpixel(fname, num_of_clusters); Image Segmentation Algorithms Segmentation as grouping k means clustering Mean shift Segmentation as graph partitioning Spectral clustering Normalized cut Segmentation by other methods Boundary detection: Watershed Labeling: Graph cut Main idea Watershed Algorithm Image as a topographic relief: the grey level of a pixel is interpreted as its altitude in the relief. The watershed of a relief correspond to the limits of the adjacent catchment basins of the drops of water. Note: This categorization is not exclusive

10 Watershed Segmentation seg = watershed(bnd_im) Procedure Procedure Choose local minima as region seeds Add neighbors to priority queue, sorted by value Take top priority pixel from queue If all labeled neighbors have same label, assign to pixel Add all non marked neighbors Repeat step 3 until finished Image Gradient Watershed boundaries Results Characteristics Pros Fast (< 1 sec for 512x512 image) Among best methods for hierarchical segmentation Cons Only as good as the soft boundaries Not easy to get variety of regions for multiple segmentations No top down information Usage Preferred algorithm for hierarchical segmentation 39 40

11 Graph Cut Image Representation by Graph Cut A very general tool that can be applied to Representation as a graph Stereo depth reconstruction Texture synthesis Video synthesis Image denoising Foreground/background segmentation A node is created for each pixel. An edge represents the relationship between adjacent pixels. Two special nodes: source () and sink () s General characteristics Formulated using a graph Used for binary labeling problem Solved by energy minimization Large search space # of nodes for labeling: the number of all possible binary segmentations: 2 This is intractable. t Image Representation by Graph Cut Energy Minimization Problem Representation as a graph A node is created for each pixel. An edge represents the relationship between adjacent pixels. Two special nodes: source () and sink () Large search space # of nodes for labeling: the number of all possible binary segmentations: 2 This is intractable. FS Bt Energy function data term smoothness term Data term How similar is each labeled pixel to the foreground or background? Probability that this color belongs to foreground (resp. background) Smoothness term (a.k.a. regularization term) Penalty for having different label Encourage spatially coherent segments Penalty is downweighted if the two pixel colors are very different

12 Two Terms Solution Data term Approach Defined for each node Smoothness term, if, FG likelihood if, BG likelihood Defined for each edge : visual similarity between two nodes (pixels), and We should minimize energy by cutting the graph into two partitions. Solved by max flow/min cut algorithm Max flow/min cut algorithm Energy optimization is equivalent to graph min cut. Cut: remove edges to disconnect two regions Minimum: minimize sum of cut edge weight F B cut