Targil 12 : Image Segmentation. Image segmentation. Why do we need it? Image segmentation

Transcription

1 Targil : Image Segmentation Image segmentation Many slides from Steve Seitz Segment region of the image which: elongs to a single object. Looks uniform (gray levels, color ) Have the same attributes (texture ). Moves in a similar way (motion segmentation) Why do we need it? Image Segmentation The first step towards higher level vision (object recognition ) Treat differently different segments. (compression, histogram manipulations ) Use it to find regions which are related to a single model (motion computations, texture analysis ) Many more applications. 3 4 Image segmentation Correct segmentation How do you pick the right segmentation? There may not be a single correct answer. Gestalt Laws seek to formalize what is an object proximity, similarity, continuation, closure, common fate ottom up segmentation: - Tokens belong together because they are locally coherent. Top down segmentation: - Tokens grouped because they lie on the same object. 5 Segmentation can be thought of a partition problem Many approaches proposed cues: color, regions, contours, texture, motions automatic vs. user-guided no clear winner 6

2 Main pproaches Histogram-based segmentation Region-based segmentation Edge detection Region growing Region splitting and merging. Clustering K-means Graph ased Clustering Example: Splitting text and background Thresholding: define a threshold T such that each pixel x,y is text iff I(x,y)<T. Our visual system is proof that automatic methods are possible classical image segmentation methods are automatic How do we determine the threshold? Just choose 8 as a threshold (problematic for dark images) Use the median/mean (both are not good, as most of the paper is white) 7 8 Histogram-based segmentation Goal reak the image into K regions (segments) Solve this by reducing the number of gray levels / colors to K and mapping each pixel to the closest gray level / color Histogram-based segmentation Goal reak the image into K regions (segments) Solve this by reducing the number of gray levels / colors to K and mapping each pixel to the closest gray level / color Possible threshold minimum on histogram (between larger values) 9 Here s what it looks like if we use two gray levels Problems with that approach Noise is considered as text. Holes and discontinuities in the resulting text. Changes in the illumination Solutions:. Enhance the result using morphological operations. daptive threshold. 3. Use spatial information! daptive Threshold Use a running window.. ssume a constant illumination in each window. For each window: compute a threshold and use it to segment the pixel in the middle of the window. Problem: This scheme will find text in blank areas! Possible Solution: we can try to recognize that these regions are unimodal.

3 Segmentation as clustering Consider the image as a set of points in the N-dimensional space: Color Histograms: p=(r(x,y),g(x,y),(x,y)) in R 3. ( ignores the spatial information.) Gray level images: p=(x,y,i(x,y)) in R 3 Color images: p =(x,y,r(x,y),g(x,y),(x,y)) in R 5 Texture: p= (x,y,vector_of_fetures) From this stage, we forget the meaning of each coordinate. We deal with arbitrary set of points. Important: We need to find the appropriate space in which the distance between points is meaningful! 3 Segmentation using simple clustering model Two natural algorithms Clustering by splitting the entire data set is one cluster and we split it to get a good clustering Clustering by merging each point is a different cluster and we merge similar clusters Usually the simple methods use greedy algorithms Important issues : Define a proper distance between data points Define a proper distance between clusters How many clusters are there? Main difficulty Defining threshold : when should we stop the algorithm Very large number of pixels : hard to look for best split / merge 4 Segmentation as Clustering How to choose the representative colors? This is a clustering problem! G G reak it down into subproblems Suppose I tell you the cluster centers c i Q: how to determine which points to associate with each c i? : for each point p, choose closest c i R R Objective Each point should be as close as possible to a cluster center Minimize sum squared distance of each point to closest center Suppose I tell you the points in each cluster Q: how to determine the cluster centers? : choose c i to be the mean of all points in the cluster 5 6 K-means clustering K-means clustering algorithm. Randomly initialize the cluster centers, c,..., c K. Given cluster centers, determine points in each cluster For each point p, find the closest c i. Put p into cluster i 3. Given points in each cluster, solve for c i Set c i to be the mean of points in cluster i 4. If c i have changed, repeat Step Example: Segmentation using Color + Texture Java demo: Properties Will always converge to some solution Can be a local minimum does not always find the global minimum of objective function: 7 8 3

4 Real segmentation example Ideas? Steps in solving specific problem. Make the assumptions appropriate for this case Size of the warm is relatively constant Warm does not move much between successive frames Lighting is changing between frames and within the frame, but can be considered as constans on small window Warm is not the only dark area in the picture. Use your assumption to tailor a specific solution 9 Real segmentation example Real segmentation example Examples Using the same thresh for different frames How to remove wrong areas? Put additional condition on the selected area to be next to large gradient pixel Remove noise by using morphological operations Using adaptive thresh based on gradient values 3 4 4

5 Possible Solution Segmentation by min (s-t) cut [oykov ] Find the warm location in the first frame Use gradient value to determine threshold Remove from the selection pixels that are selected according to the threshold but do not have large gradient in their area Use morphological operation to remove noise min cut Using the warm location in the previous frame, search in the current frame using a simple threshold on window around the warm Update window location based on the new position of warm nd still in practice can know if this works only after trying. 5 t Graph node for each pixel, link between pixels specify a few pixels as foreground and background create an infinite cost link from each bg pixel to the t node create an infinite cost link from each fg pixel to the s node compute min cut that separates s from t how to define link cost between neighboring pixels? s 6 Segmentation by min (s-t) cut [oykov ] how to define link cost between neighboring pixels? Cuts in a graph ssign large weight on edges of pixels that you expect to be in the same group (body / background) F( i ) F( j ) σ I (, j) = e w i F(i)- gray level value in pixel I - normalization factor σ 7 Link Cut set of links whose removal makes a graph disconnected cost of a cut:, ) = w( i, j) Find minimum cut gives you a segmentation mincut (, ) = min i, j, u v, w( u, v) 8 Examples for user assistance Min-cut segmentation (example from Micusik. and Hanbury ) Minimum cut Criterion for partition: mincut (, ) = min What is the problem with the min cut?, u v, w( u, v) Problem! Weight of cut is directly proportional to the number of edges in the cut. 9 Ideal Cut Cuts with lesser weight than the ideal cut First proposed by Wu and Leahy 3 5

6 utomatic graph cut [Shi & Malik] i j w(i,j) Fully-connected graph node for every pixel link between every pair of pixels, i,j cost w(i,j) for each link w(i,j) measures similarity» similarity is inversely proportional to difference in color and position w 3 utomatic graph cut [Shi & Malik] Construct a weighted graph from the image pixels G = (V,E) by taking each pixel and connecting each pair of pixels in an edge w( i, j) = e F e X ( i ) X ( j ) F ( i ) ( j ) * σ X σ if X i X I ( ) ( j ) F(i)- gray level value in pixel i X(i)- pixel I position else < r 3 Segmentation by Graph Cuts C reak Graph into Segments Delete links that cross between segments Easiest to break links that have low cost (similarity) similar pixels should be in the same segments dissimilar pixels should be in different segments w 33 Normalized Cut Normalized cut or balanced cut: a cut penalizes large segments fix by normalizing for size of segments N, ) =, ) + ) ) vol () = sum of costs of all edges that touch Finds better cut 34 Normalized Cut Observations(I) Volume of set (or association): ) =, V ) = u, t V w( u, t) Define normalized cut: a fraction of the total edge connections to all the nodes in the graph :, ), ) Ncut (, ) = +, V ), V ) Define normalized association: how tightly on average nodes within the cluster are connected to each other, ), ) Nassoc (, ) = +, V ), V ) Maximizing Nassoc is the same as minimizing Ncut, since they are related: N, ) = N, ) How to minimize Ncut? Transform Ncut equation to a matricial form. Solve an eigen vector problem on appropriate matrix:

7 Graph-based Image Segmentation Graph-based Image Segmentation Image (I) Image (I) Intensity Color Edges Texture Graph ffinities (W) Intensity Color Edges Texture Graph ffinities (W) N, ) =, ) + ) ) Slide from Timothee Cour ( 37 Slide from Timothee Cour ( 38 Graph-based Image Segmentation Graph-based Image Segmentation Image (I) Eigenvector X(W) Image (I) Eigenvector X(W) Discretization Intensity Color Edges Texture Graph ffinities (W) N, ) =, ) + ) ) ( D W) X = λdx if X ( i) = if i i Intensity Color Edges Texture Graph ffinities (W) N, ) =, ) + ) ) ( D W) X = λdx if X ( i) = if i i Slide from Timothee Cour ( 39 Slide from Timothee Cour ( 4 Graph-based Image Segmentation Graph terminology Similarity matrix: [ ] W = w i, j w i = e, j X( i ) X ( j ) σ X G = {V,E} V: graph nodes E: edges connection nodes Pixels Pixel similarity Slides from Jianbo Shi Slides from Jianbo Shi 4 4 7

8 ffinity matrix Graph terminology N pixels M pixels Warning the size of W is quadratic with the number of parameters! Reshape Similarity of image pixels to selected pixel righter means more similar Degree of node: = d i w i, j j N*M pixels N*M pixels Slides from Jianbo Shi Graph terminology Graph terminology Volume of set: Cuts in a graph: vol ( ) = d, V i i, ) = w i, j i, j Slides from Jianbo Shi Slides from Jianbo Shi Normalized Cut Volume of set (or association): ) =, V ) = u, t V w( u, t) Define normalized cut: a fraction of the total edge connections to all the nodes in the graph :, ), ) Ncut (, ) = +, V ), V ) Define normalized association: how tightly on average nodes within the cluster are connected to each other, ), ) Nassoc (, ) = +, V ), V ) Representation Partition matrix X: Degree matrix D: Laplacian matrix L: [ X ] X =,..., X K Pair-wise similarity matrix W: W ( i, j) = aff ( i, j) = D ( i, i) w j i, j L= D W segments pixels

9 Observations(I) Observations(II) Maximizing Nassoc is the same as minimizing Ncut, since they are related: How to minimize Ncut? Transform Ncut equation to a matricial form. fter simplifying: min N x) = min x Subject to: y T D= y T y ( D W) y T y Dy N, ) = N, ) Rayleigh quotient = D ( i, i) W ( i, j) j NP-Hard! y s values are quantized Instead, relax into the continuous domain by solving generalized eigenvalue system: ( ) subject to ( y T Dy= ) min max y y T ( D W)y Which gives: ( D W ) y= λdy Note that ( D W )= so, the first eigenvector is y = with eigenvalue. The second smallest eigenvector is the real valued solution to this problem!! y= f (x) 49 5 lgorithm Example (I) Define a similarity function between nodes. i.e.: F( i ) F( σ I σ X w i, j = e Compute affinity matrix (W) and degree matrix (D). Solve ( D W ) y= λdy Use the eigenvector with the second smallest eigenvalue to bipartition the graph. Decide if re-partition current partitions. j ) + X ( i ) X( j ) Eigenvectors Segments Note: W is very sparse and only few eigenvectors are required, the eigenvectors can be extracted very fast using Lanczos algorithm. 5 5 Example (II) More Slides spectral clustering Original Segments * Slide from Khurram Hassan-Shafique CP545 Computer Vision 3 9

10 Definitions Methods that use the spectrum of the affinity matrix to cluster are known as spectral clustering. Normalized cuts, verage cuts, verage association make use of the eigenvectors of the affinity matrix. Why these methods work? Spectral Clustering Data Similarities * Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University Eigenvectors and blocks Spectral Space lock matrices have block eigenvectors: λ = λ = λ 3 = Can put items into blocks by eigenvectors: e.7 λ 4 =..7 eigensolver e Near-block matrices have near-block eigenvectors:. -. eigensolver. -. λ =. λ =. λ 3 = -..7 λ 4 = e e Clusters clear regardless of row ordering: e e 57 e e 58 * Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University * Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University How do we extract a good cluster? Simplest idea: we want a vector x giving the association between each element and a cluster We want elements within this cluster to, on the whole, have strong affinity with one another We could maximize x T Wx ut need the constraint x T x= This is an eigenvalue problem - choose the eigenvector of W with largest eigenvalue. 59