Computational Modeling of Visual Perception. The Image Segmentation Problem

Transcription

1 Computational Modeling of Visual Perception The Image Segmentation Problem Francisco J. Estrada, Feb., 2006 Questions? Comments? elderlab.yorku.ca

2 Suppose I give you the following task: Divide the image into pieces, where each piece represents a distinguished thing in the image. It is important that all of the pieces have approximately equal importance. The number of things in the image is up to you. Something between 2 and 20 should be reasonable Instructions for segmenting images for the Berkeley Segmentation n Database, see Martin et. al. [MFTM 01]

3 Results from a few observers who performed this task:

4 Image Segmentation (in very general terms!): - Partitioning the image into salient regions - Salient regions (at least for human observers) tend to represent t individual objects, object parts, or individual surfaces - But note: Humans use all sorts of tricks to perform this task! Could we segment this image 2 without knowing what it is we re Looking at? 2- Prof. Richard L. Gregory s famous Dalmatian Image

5 Bottom-up Image Segmentation: - Partitioning the image into regions that have homogeneous appearance - Intended as a pre-processing processing stage, so we can t use object recognition. (in fact, we would like to use the segmentation to help with recognition!) r - Assume little or no knowledge about the contents of the image Input Segmentation 3 Input Segmentation 3 3- Produced by the Mean-Shift algorithm (see Comaniciu & Meer,, 2002 [CM02])

6 Why do we think this is useful? - Good segmentation should help with recognition, tracking, image database retrieval, and image compression among other high-level vision tasks. Detect and recognize objects in this image very hard! Unless, of course, you ve seen THIS particular tiger before, in a similar pose, but we ll get to that when we talk about object recognition What is the object represented by the pixels labeled in red? somewhat more manageable Though this is an idealized example. In practice we re likely to get several regions that we hope can be joined to form the tiger s outline.

7 What we re doing today: - Define the bottom-up segmentation problem (we just did!) - Talk about low-level level visual similarity: How do we determine that two pixels should be in the same region? - How do we encode pixel similarity? - Review of segmentation methods * Early techniques (thresholding( thresholding,, region growing) * Feature space methods * Graph Theoretic methods - How do we evaluate the quality of a segmentation?

8 The segmentation task segmented! It is useful to make a distinction between two sub-problems that must be addressed by bottom-up segmentation algorithms: 1.- Determining a suitable measure of similarity between image features. 2.- Developing a clustering (or grouping) algorithm that generates regions using the proposed similarity measure. In general, the clustering part can be made to work with different ferent similarity measures, and it is in this part that we will spend most m of the lecture, but first a few comments on similarity measures

9 Similarity Measures. What cues are at hand? - Image brightness - Colour - Edge energy - Texture - Image Position (spatial info) - Motion - Stereo disparity And now, the trick is deciding how to combine these cues. For some insight see Malik et al. [MBLS01]

10 Segmenting Images early techniques Gray-level thresholding - Groups of similar pixels appear as bumps in the brightness histogram - Split the histogram at local-minima - Label pixels w.r.t. which bump they belong in

11 Segmenting Images early techniques Gray-level thresholding - Notice bleeding problem. Also, regions are not connected - How many thresholds to use? - Simple, but can be useful in suitably constrained environments

12 Segmenting Images early techniques Region growing - Start with a set of seed regions (perhaps individual pixels) - Progressively grow regions by adding to them neighboring pixels whose similarity is above some threshold. - Stop when all the image has been labeled - See Beveridge et al. [BGKHR89], Pal and Pal [PP93]

13 Brief aside: over- and under-segmentation Two common terms used to describe general problems with segmentations tions 4 Undersegmentation (Mean-Shift) Good segmentation (human) 4 - Notice that under- and over segmentation are related, but not identical to the problem of deciding how much detail should be in a segmentation Oversegmentation (Mean-Shift)

14 Segmenting Images feature space methods - Select a set of image features, say position {x,y}, colour {R,G,B}, and a set of filter responses {f 1 (x,y) f k (x,y)} - For each pixel p i (x i,y i ) form a feature vector v i xi yi R i Gi B i = f1( xi, yi)... f2( xi, yi) - This vector represents a point in an N-dimensional feature space. N is the number of features - The feature vectors for similar pixels should occupy nearby locations in this feature space - Thus, homogeneous image regions become dense clouds of feature vectors in feature space.

15 Segmenting Images feature space methods - Quick example, let s look at an RGB feature space for a simple image: - The feature space has dimension 3. Notice that the principal image regions generate dense clusters in feature space.

16 Segmenting Images feature space methods K-means clustering 5 - Compute the feature space vectors - Randomly select K cluster centers in feature space (they don t have to correspond to any of the feature vectors, but see Fischler & Bolles [FB81]). - Iterate until convergence: * Assign feature vectors to the closest cluster center * Re-compute the cluster centers as a (weighted) mean of the feature vectors assigned to each cluster - Label pixels according to the cluster their feature vectors belong to. 5 See Forsyth & Ponce, Ch. 14

17 Segmenting Images feature space methods K-means clustering - 8 iterations of the K-means K procedure, K=5

18 Segmenting Images feature space methods K-means clustering - Effect of random initialization, K=5 - Effect of the choice of K K=3 K=5 K=8 K=15

19 Segmenting Images feature space methods Mixture of Gaussians Model (see Carson et al., Blobworld [CBGM02]) - Model the distribution of feature vectors as a mixture of Gaussians K pv ( M) = π Nv ( µ, Σ ), k k = 1 π k k = 1 = 1 k k k - Where µ k and Σ k are the mean and covariance matrix of Gaussians, and π k are the mixing proportions. - Model parameters can be fit using maximum-likelihood (the E-ME algorithm).

20 Segmenting Images feature space methods Mixture of Gaussians Model - Pixels are assigned to the Gaussian component that has the maximum ownership probability label i π knv ( i µ k, Σk) = arg max k pv ( i M) - Blobworld uses colour,, texture, and position as features See [CBGM02] and elib.cs.berkeley.edu/blobworld/

21 Segmenting Images feature space methods Mean-Shift algorithm (Comaniciu( & Meer,, see [CM02]) v i - Starting at each feature vector, initialize a hypothetical cluster center c = v 0 i c - Iteratively update t to be the (weighted) mean of all feature e vectors within a small search window S( ) centered at c t c t c t+ 1 = v S( c ) j v S( c ) j t t wc ( v) v t j j wc ( v) t j - The search window is usually a (hyper) sphere, ellipsoid, or cube in R d. The label of pixel i is set to the value of at convergence. c i

22 Segmenting Images feature space methods Mean-Shift algorithm - The mean-shift iteration converges to a region of locally maximal density in feature space.

23 Segmenting Images feature space methods Mean-Shift algorithm - A point of locally maximal density forms a basin of attraction for nearby feature vectors 6 - These domains of convergence define the segments produced by mean-shift 6 But notice that the range of attraction of this basin depends on the size and shape of the search window

24 Segmenting Images feature space methods Mean-Shift algorithm - Sample segmentations 7 Input image smaller search window larger search window 7 Using the EDISON system, see [EDISON]

25 Segmenting Images graph-theoretic methods Images as graphs - An image I(x,y) is equivalent to a graph G(V,E) Original image I(x,y) - V is a set of vertices or nodes, each node represents one image element (e.g. individual pixels) Graph G(V,E) - E is a set of edges linking neighboring nodes together. The weight or strength of the edge is proportional to the similarity between the vertices s it joins together.

26 Segmenting Images graph-theoretic methods Images as graphs - The image elements can be individual pixels, small regions, or other types of image features. - Usually, only elements within a small neighborhood are connected. This provides spatial coherence and has computational advantages. - Edge weight is also called pairwise similarity,, or pairwise affinity. - In general, given a graph G(V,E) graph-based segmentation methods attempt to find groups of nodes in G that are strongly connected to one another, but weakly connected to the rest of the graph.

27 Segmenting Images graph-theoretic methods Minimal spanning trees (MSTs( MSTs) - Tree that connects all vertices of the graph with the minimum total weight (sum of link weight in the tree) Can be constructed efficiently using Kruskal s algorithm: Start with a disconnected graph Add edges in increasing order of weight as long as doing so doesn t introduce a cycle Stop when all the vertices are connected 2 Minimal Spanning Tree for this graph

28 Segmenting Images graph-theoretic methods The Local Variation algorithm (see Felzenszwalb & Huttenlocher [FH04]) - Partition the image so that for any pair of regions the variation between the regions should be larger than the variation within the regions. - Extension of Kruskal s algorithm 8 : Add edges one at a time in order of increasing weight. Maintain a list of disconnected MSTs For each MST C i compute a threshold TC ( i) = wc ( i) + k/ Ci, where w(c i ) is the maximum weight in the spanning tree, C i is the number of pixels in C i, and k>0 is a user-defined constant) If the next edge to be added joins two separate MSTs,, the MSTs are merged only if wx (, x) min( TC ( ), TC ( )) k l i j 8 See also Jepson [Jep04] for more details

29 Segmenting Images graph-theoretic methods The Local Variation algorithm - Sample segmentations 9 Input image smaller k larger k 9 Generated using the implementation provided by the authors of [FH04],[ see [FH_code]

30 Segmenting Images graph-theoretic methods Graph Cuts B - A cut through a graph is defined as the total weight of the links that must be removed to divide the graph into two separate components. A cut A B (, ) wi, j i A, j B = Graph G(V,E)

31 Segmenting Images graph-theoretic methods Minimum Cut method (see Wu & Leahy [WL93]) - Find the cut through the graph that has the overall minimum weight MinCut( A, B) = min( cut( A, B)) AB, - Should correspond to the subset of edges of least weight that can be removed to partition the graph.1.1 MinCut( A, B ) = 8*(.1) =.8 - Since weight encodes similarity, this should be equivalent to partitioning the graph along the boundary of least similarity

32 Segmenting Images graph-theoretic methods Minimum Cut method - Can be computed efficiently However, it has a preference for short boundaries. Sometimes picks a trivial partition cut( A, B ) = 8*(5) = 40

33 Segmenting Images graph-theoretic methods Minimum Cut method - Can be computed efficiently However, it has a preference for short boundaries. Sometimes picks a trivial partition We have to somehow constrain the solution to avoid such trivial cuts cut( A, B ) = 8*(5) = 40 MinCut( A, B ) = 2*() = 20

34 Segmenting Images graph-theoretic methods Normalized Cuts (see Shi & Malik [SM00]) - Find the cut through the graph that minimizes the normalized cut measure B NCut( A, B ) = cut ( A, B ) cut ( A, B ) + assoc ( A, V ) assoc ( B, V ) - Here the term assoc(a,v) is the total weight of the connections between the region A and the rest of the nodes in the graph assoc( A, V ) = i A, j V,( v, v ) E i j w i, j Graph G(V,E) A

35 Segmenting Images graph-theoretic methods Normalized Cuts cut( A, B ) = 2*() = 20 assoc( A, V ) = 2*() = assoc( B, V ) = 16* + 8*5 = NCut( A, B ) = = Trivial cuts such as this one don t minimize the normalized cut

36 Segmenting Images graph-theoretic methods Normalized Cuts cut( A, B ) = 8*(5) = 40 assoc( A, V ) = 12* + 8* 4 = assoc( B, V ) = 4* + 8*5 = NCut( A, B ) = = The normalization by assoc(r,v) should (in general) prevent the bias toward short boundaries

37 Segmenting Images graph-theoretic methods Normalized Cuts - Solving for the optimal NCut exactly is NP-complete - However, we can obtain an approximate solution. Start with an affinity matrix W s.t.w(i,j) contains the weight of the edge linking nodes i and j - Notice that if we have an image (n x m) pixels in size, W will be an (n*m) 2 matrix. Fortunately, it is sparse - Also, define a diagonal matrix D s.t. Dii (,) = wi, j j

38 Segmenting Images graph-theoretic methods Normalized Cuts - Shi and Malik show that an approximate solution is given by the eigenvector with the smallest eigenvalue of the system ( D W) y = λdy - Furthermore, succeeding eigenvectors can be used to split previous partitions - General NCut procedure (there are several variants!) Compute W and D Solve the above system for the k eigenvectors with smallest eigenvalues Threshold each eigenvector, from the 2 nd smallest on, to obtain a partition of the image The segmentation is the intersection of the k partitions generated in this way Meila and Shi [MS01] present a probabilistic interpretation of NCuts in terms of random walks

39 Segmenting Images graph-theoretic methods Normalized Cuts 2 nd eigenvector Input image 3 rd eigenvector Segmentation with 25 segments th th eigenvector 11 Generated using the implementation provided by at [NCut[ NCut_code]

40 Segmenting Images graph-theoretic methods S-T T Min-Cut (see Boykov & Kolmogorov [BK01], and Boykov & Jolly [BJ01]) - Reduce the problem of trivial cuts by introducing two special nodesn called source (S) and sink (T) S - S and T are linked to some image nodes by links of very large weight (so that they will never be selected in a cut) - Find the minimum cut that separates the source from the sink T - Notice that the problem is deciding how to connect S and T to the image nodes Min S-T S T Cut

41 Segmenting Images graph-theoretic methods S-T T Min-Cut User-selected selected pixels connected to S (red) and pixels connected to T (cyan) Minimum S-T S T cut - If we have a good guess (or a human observer) to tell us hot to link the source and sink to the image, we will get an optimal segmentation

42 Segmenting Images graph-theoretic methods S-T T Min-Cut Source pixels (purple) and sink pixels (yellow) Minimum S-T S T cut - With a bad guess, things don t go so well - And, how do we get this guess to begin with?

43 Segmenting Images graph-theoretic methods S-T T Min-Cut - Good choices for the sets of image nodes connected to S and T should satisfy: Each set should be sufficiently large (otherwise, we get a trivial cut that contains only the image nodes in either the set for S or the set for T) Both sets should be fully contained within natural segments (this is because the sets themselves will never be partitioned. Think about the shirt leaking into the wall in the segmentation on the previous s slide) Many different sets for S and T should be tried to identify most of the salient boundaries in the image

44 Segmenting Images graph-theoretic methods SE-MinCut (see Estrada, Jepson & Chennubhotla [EJC04]) - Use spectral clustering to identify small groups of similar pixels. These groups define seed regions for S-T S MinCut Input image Seed regions found through spectral clustering. Combinations of these are used as source and sink regions for S-T S MinCut Final segmentation combines the partitions generated with each separate S-T S T cut.

45 Segmenting Images graph-theoretic methods SE-MinCut - Sample segmentations Input image Seed regions Segmentation

46 Segmenting Images graph-theoretic methods SWA - Segmentation by Weighted Aggregation (see Galun et al. [GSBB03]) - The methods covered so far are mostly single-scale, scale, and local Locally, we may have uncertainty about the segmentation Global information removes or Reduces the uncertainty - Makes sense to analyze the image at multiple scales for segmentation

47 Segmenting Images graph-theoretic methods SWA - Segmentation by Weighted Aggregation - Very general description of SWA: Compute a fine-scale graph with nodes corresponding to individual pixels Progressively coarsen the graph by contracting similar nodes Each node above the individual pixel level contains statistics about intensity, texture, and shape of the elements that compose it at the next finer level For a region S the aggregation process minimizes ES ( ) ES ( ) = = NS ( ) v S, v S,( v, v ) E i j i j v, v S,( v, v ) E i j i j This is similar to the NCut measure. It normalizes the sum of weights along the region s boundary by the sum of internal weights w w i, j i, j

48 Segmenting Images graph-theoretic methods SWA - Segmentation by Weighted Aggregation - Schematic diagram of SWA From Ronen Basri s SWA webpage - weizmann.ac..ac.il/~ronen/segmentation.html

49 Segmenting Images graph-theoretic methods SWA - Segmentation by Weighted Aggregation - Sample segmentations also also from (12)

50 Segmenting Images quick recap - First: Choose a suitable measure of similarity - Choose your favourite segmentation method: Thresholding,, simple region growing K-means clustering Feature space clustering (Mean Shift algorithm) Graph-based methods (Local Variation, Normalized Cuts, SE-MinCut MinCut,, SWA) - Have a segmentation! now, what do we do with it?

51 Segmenting Images a couple of questions - Given a (likely imperfect) segmentation, how do we use it to improve recognition, tracking, and other vision tasks? - How do we put together segmentation and grouping (boundary detection) methods? Are they the same thing working on different data? - For that matter. How do we evaluate bottom-up segmentation results?

52 Segmentation evaluation - Columns show segmentations by different algorithms (each column corresponds to 1 method). Which segmentations are better? SE-MinCut Mean-Shift Local Variation NCuts

53 Segmentation evaluation - Visual evaluation can be tricky, how can we be sure? - Quantitative evaluation Task dependent (i.e. how much does a segmentation help in solving g some particular problem?) Evaluation w.r.t. a standard image database with ground-truth - We have a standard image database: the Berkeley Segmentation Database (BSD)

54 Segmentation evaluation training images, 0 testing images - At least 4 human ground-truth segmentations per image - Interesting observation: Human segmentations are consistent with one another. - Most variation concerns level of detail, notice that this is different from over- and under- segmentation

55 Segmentation evaluation - We can compare automatic segmentations against the ground truth, and see which methods work better overall - Need an appropriate comparison metric - Martin et al. (see [MFTM01]) propose region-based measures of segmentation consistency. These are either too harsh, or insensitive to over- and under-segmentation - We will use the region boundaries instead, and compute precision and recall (see Martin [Mar02], and Estrada & Jepson [EJ05])

56 Segmentation evaluation - Precision: Percentage of detected boundary pixels that correspond to ground-truth (human) boundary pixels - Recall: Percentage of ground-truth (human) boundary pixels that were detected in the automatic segmentation Original image Overlayed human boundaries Mean-shift over-segmented Mean-shift under-segmented

57 Segmentation evaluation - Take a segmentation algorithm - Run it on the 0 test images, and compute precision and recall w.r.t. the human segmentations - The median precision/recall values over the 0 test images give an indication of segmentation quality - Systematically vary the algorithm s parameters to create a curve that fully characterizes the performance of the algorithm - Compare the tuning curves of different algorithms directly!

58 Segmentation evaluation - Does this agree with our visual evaluation? - What s the deal with Canny edges? Given these results, why even bother doing segmentation? - What about this gap between human performance and automatic segmentation results? How do we get there? Tuning curves for several segmentation methods (see Estrada [Est05])

59 Segmentation evaluation SE-MinCut - Notice the large variation between segmentations at different points along the tuning curves Mean-shift Local variation - Usually, the better segmentations occur close to the middle of the tuning curve (a compromise between precision and recall) NCut Canny edges Highest recall Mid-curve Highest precision - Canny edges don t partition the image. We would need to group them (which is quite another problem) to generate image partitions

60 Segmentation evaluation - What about the gap in performance between human observers and automatic segmentation methods? - It could be that we just don t know how to exploit available low-level level cues appropriately - Perhaps more likely, human observers use high-level knowledge to complete the segmentation task when low-level level information is ambiguous or missing - We should think about this: How much can we expect to be able tot improve bottom-up segmentation methods? How good is good enough?

61 Segmentation evaluation - How well can we do on a purely perceptual problem? (i.e. where object recognition or high level knowledge are not required) - 7 regions of uniform brightness plus fractal noise with ground truth - Same evaluation procedure as with BSD images Input Ground-truth SE-MinCut

62 Segmentation evaluation Tuning curves on fractal images (see Estrada [Est05])

63 Segmentation a few final thoughts - Current segmentation algorithms seem reasonably capable - We don t (yet) have many direct applications of segmentation, howeverh it is quite popular in medical imaging - Recent research has focused on the use of more mid-level information (i.e. shape priors, segmentation of known object classes, etc. see for example Kumar et al. [KTZ05]) - Ultimately, achieving human performance is likely to require a combination of bottom-up and top-down processing. We don t know how this should work at the present time

64 References [BGKHR99] Beveridge, Griffith, Kohler, Hanson, and Riseman, Segmenting Images Using Localized Histograms and Region Merging, IJCV, 1989 [BJ01] Boykov and Jolly, Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images, ICCV, 2001 [BK01] Boykov and Kolmogorov, An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision, Lecture Notes in Computer Science, 2001 [CM02] Comaniciu and Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI, 2002 [CBGM02] Carson, Belongie, Greenspan, and Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, PAMI, 2002 [EDISON] Georgescu and Christoudias: The Edge Detection and Image SegmentatiON (EDISON) system, Robust Image Understanding Laboratory, Rutgers University, [EJC04] Estrada, Jepson, and Chennubhotla, Spectral Embedding and Min-Cut for Image Segmentation, BMVC, 2004 [EJ05] Estrada and Jepson, Quantitative Evaluation of a Novel Image Segmentation Algorithm, CVPR, 2005 [Est05] Estrada, Advances in Computational Image Segmentation and Grouping, Ph.D. Thesis, University of Toronto, 2005 [FB81] Fischler and Bolles, Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography, Comm. of the ACM, [FH04] Felzenszwalb and Huttenlocher, Efficient Graph-Based Image Segmentation, IJCV, [FH_code] Felzenszwalb and Huttenlocher: Image segmentation software, MIT A.I. Lab, [GSBB03] Galun, Sharon, Basri and Brandt, Segmentation by Multiscale Aggregation of Filter Responses and Shape Elements, ICCV, 2003 [Jep04] Jepson: Lecture notes on Image Segmentation, [KTZ05] Kumar, Torr, and Zisserman, OBJ CUT, CVPR, 2005

65 [Mar02] Martin, An Empirical Approach to Grouping and Segmentation, Ph.D. Thesis, UC Berkeley, 2002 [MBLS01] Malik, Belongie, Leung, and Shi, Contour and Texture Analysis for Image Segmentation, IJCV, 2001 [MFTM01] Martin, Fowlkes, Tal, and Malik, A Database of Human-segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics, ICCV, 2001 [MS02] Meila and Shi, Learning Segmentation by Random Walks, NIPS, 2002 [NCut_code] Cour, Yu, and Shi: Normalized Cuts Matlab code, Computer and Information Science, Penn State University, [PP93] Pal and Pal, A Review on Image Segmentation Techniques, Pattern Recognition, 1993 [SM00] Shi and Malik, Normalized Cuts and Image Segmentation, PAMI, 2000 [WL93] Wu and Leahy, An Optimal Graph-Theoretic Approach to Data Clustering: Theory and its Application to Image Segmentation, PAMI, 1993