EE 701 ROBOT VISION. Segmentation

Transcription

1 EE 701 ROBOT VISION Regions and Image Segmentation Histogram-based Segmentation Automatic Thresholding K-means Clustering Spatial Coherence Merging and Splitting Graph Theoretic Segmentation Region Growing Segmentation Segmentation : Dividing images into (semantically meaningful) regions that appear to be images of different surfaces Two major approaches : Histogram-based segmentation Segmentation based on spatial coherence Divide (split) or merge existing regions Growing regions from seed points Reliable segmentation is possible with a priori info, which is not available in most cases 2 1

2 Histogram-based Segmentation Segmentation : Gray level Binary mask using an unknown threshold How to find an automatic threshold? considering variations in illumination & surface One popular approach : histogramming Gray-level histogram gives the number of picture cells having a particular gray-level 3 Histogram-based Segmentation (1/2) Ideally, object & background have constant different brightness inside their regions put a threshold between peak values in histogram In practice, due to measurement noise non-uniform illumination non-uniform reflection from the surfaces brightness is not constant; there is some spread 4 2

3 Histogram-based Segmentation (2/2) Difficulties in histogramming non-constant intensity (color) variations on the objects prevents successful segmentation existence of intermediate gray levels of edge pixels; this problem increases by utilization of averaging against noise and decreases with high-resolution images Size differences between object and background may lead neglecting one with respect to other 5 Image Thresholding : General Main directions [Sankur 2002] : 1. Histogram shape-based Analyze peaks, valleys, curvatures of smoothed histogram 2. Clustering-based Iteratively, finding a threshold, clustering based this threshold 3. Entropy-based Choosing a threshold which max the info content in histogram 4. Attribute-based Similarities between edge, etc of image & its thresholded vers. 5. Spatial thresholding (higher order stats) Threshold selection on higher order statistics of spatial neighb. 6. Local thresholding Finding threshold values at each neighborhood using local stats 6 3

4 Automatic Thresholding Methods(1/4) 1. P-tile method Use the a priori knowledge about the size of the object : assume an object with size p Choose the threshold such that %p of the overall histogram is determined T P% 100% Obviously, very limited use 7 Automatic Thresholding Methods(2/4) 2. Mode method Find the peaks and valleys of the histogram Set threshold to the pixel value of the valley T 255 Non-trivial to find peaks/valleys : ignore local peaks; choose peaks at a distance; find the valley between those peaks Maximize peakiness (difference btw peaks & valleys) to find the threshold as valley 8 4

5 Automatic Thresholding Methods(3/4) 3. Iterative threshold selection Starting with an approximate threshold, refine it iteratively, taking into account some goodness measure e.g. T=(μ 1 +μ 2 )/2 where μ ι is the mean gray value of previous segmented region i 4. Adaptive Thresholding In case of uneven illumination, global threshold has no use One approach is to divide an image into mxm subimages and determine a threshold for each subimage 9 Automatic Thresholding Methods(4/4) 5. Double Thresholding Starting from a conservative initial threshold T 1, determine the core parts of the object Continuing from this core part, grow this object by including neighboring pixels which are between T 1 and T 2 T1 T2 10 5

6 K-means Clustering It is usually desired to segment an image into more than two regions Histograms can still be utilized to determine the regions on an image with different intensity content μ 1 μ 1 T μ 2 μ 3 μ μ 11 μ 2 T 2 T 2 μ 3 T 3 T 3 μ 4 T 4 Tμ 4 5 T 5 μ 6 6 K-means (or ISODATA) algorithm : 1) Choose initial mean values for k (or c) region 2) Classify n pixels by assigning them to closest mean 3) Recompute the means as the average of samples in their (new) classes 4) Continue till there is no change in mean values 11 K-means Clustering K-means by using color (11 segments) Original Image Clusters on color 12 6

7 Spatial Coherence Histogram-based methods totally neglect the dependency between neighboring pixels Neglecting this dependency may cause salt-npepper noise in the resulting binary image If spatial coherence between pels are taken into account, such noise can be eliminated by some preprocessing Such an approach decreases the error-rate but obviously does not guarantee being error-free Dependency between neighboring pixels or regions could be represented in various ways 13 Spatial Coherence Region Representation Regions can be represented in alternative forms : Array representations : masks Hierarchical representations : quad-trees Symbolic representations : bounding box, centroid, moments, Euler number Array representation Hierarchical representation Quad-tree 14 7

8 Spatial Coherence Data Structures for Segmentation Some data structures allowing easy manipulation of region characteristics Region Adjacency Graphs (RAG) Picture Trees a P b c d e f g h i Super Grid 15 Merging & Splitting : The output of any segmentation method can be improved by simply merging similar neighboring regions together Similarity can be measured by A simple threshold A geometrical attribute, such as common boundary length More sophisticated methods based on statistics Similarly, rather than merging, splitting can be required due to geometrical attributes 16 8

9 Region Merge: A general region merge algorithm Beginning from an initial segmentation, prepare an initial RAG; for each region check whether its neighboring regions are similar, if so, merge these regions & modify RAG For region similarity : Compare their mean intensities check with a predetermined threshold Compare their statistical distributions check whether such a merge represents observed values better Check weakness of the common boundary weak boundary: intensities on two sides differ less than a threshold Merge two regions if W/S > τ where W=length of weak boundary 1) S = min{s 1,S 2 } : minimum of two boundaries 2) S : common boundary 17 Graph Theory In Image Segmentation To analyze images by graph theory image is mapped onto a graph Each pixel (sub region) in the image corresponds to a vertex Typical connection between the nodes is 4 (or 8) connections. Vertices and edges have weights Pixel grey level value is usually assigned to vertex weight Weights associated with each link are based on some property of the pixels that it connects, such as their intensity differences. Pixel Vertex Link Four-way Connected Graph 18 Slides modified from Neslihan Yalçın Bayramoğlu Thesis Presentaton 9

10 Graph Theory In Image Segmentation Insertion of links between similar intensities creates subtrees for these objects Every sub tree represents a region of the image Region 1 Region 2 19 Slides modified from Neslihan Yalçın Bayramoğlu Thesis Presentaton Recursive Shortest Spanning Tree (RSST) Morris et al. (1986) stated that methods which does not depend on global information will not achieve good segmentation. He proposed dynamic construction of the image tree. Algorithm: Image is mapped onto the graph Four way connected graph is used Absolute values of the grey level differences between vertices are assigned to link weights Link weight=70 Slides modified from Neslihan Yalçın Bayramoğlu Thesis Presentaton 20 10

11 Recursive Shortest Spanning Tree (RSST) Nodes, whose link yields the smallest value, are merged Chosen link is saved as a part of the tree If duplicated links are obtained due to merging, they are removed If weights of links that are touching to newly formed region are changed they are updated Link weigths are changed Before Merging Smallest weighted link Chose, merge, save and delete from the heap After Merging one of them is duplicated link remove one 21 Slides modified from Neslihan Yalçın Bayramoğlu Thesis Presentaton Recursive Shortest Spanning Tree (RSST) Beginning from the original image, where each pixel represents a different region, most similar neighboring regions are merged iteratively until a user-defined region number is left 22 11

12 Graph theoretic clustering In Graph-theoretic clustering, tokens (nodes) are represented by using a weighted graph. affinity matrix, A (similar nodes have higher entries) Cut up this graph to get subgraphs with strong links 23 Measuring Affinity Intensity Distance Texture aff ( x, y)= exp 1 2 2σ i Ix ( () Iy ) ()2 ( ) aff ( x, y)= exp 1 2 2σ d x y 2 aff ( x, y)= exp 1 2 2σ t cx ( () cy ) ()

13 Effect of scale (σ) on the affinity ( ) aff ( x, y)= exp 1 2 2σ i I () x Iy ()2 aff( x, y)= exp 1 2 2σ d ( x y 2 ) aff( x, y)= exp 1 2 2σ t cx ( () cy ) ()2 25 T w A w = Solution via Eigenvectors Idea: Find vector, w, giving the association between each node and a cluster Elements within a cluster should have strong affinity with each other Maximize the following relation: w T Aw (A : affinity matrix) = i, j w a w i i, j j assoc. of nodei to cluster similarity nodei & node j assoc. of node jto cluster w i ai, j w j Above relation maximizes, in case all 3 terms are non-zero (i.e. not small) There should be an extra constraint, as w T w=1 Optimize by method of Lagrange multiplier : max { w T Aw + λ (w T w-1) } Solution is an eigenvalue problem Choose the eigenvector of A with the largest eigenvalue 26 13

14 Solution via Eigenvectors points eigenvector matrix 27 Normalized cuts Previous criterion only evaluates within cluster similarity, but not across cluster difference Instead, one would like to maximize within cluster similarity compared to the across cluster difference Write graph V, one cluster as A and the other as B Minimize Normalized Cut cut( A, B) cut( A, B) + assoc( A, V ) assoc( B, V ) cut(a,b) : sum of edges between A&B assoc(a,v) : sum of edges only in A i.e. construct A, B such that their within cluster similarity is high, compared to their association with the rest of the graph 28 14

15 Normalized cuts Write a vector y, whose elements are 1, if item is in A, -b, if item is in B Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, y T ( D W )y Criterion becomes min y y T Dy and we have a constraint (1 is a vector with all ones) y T D1 = 0 (Read proof in the distributed notes) 29 Normalized cuts Instead, solve the generalized eigenvalue problem which gives max y ( y T ( D W )y) subject to ( y T Dy = 1) ( D W )y = λdy Now look for a quantization threshold that maximizes the criterion --- i.e all components of y above that threshold go to one, all below go to -b 30 15

16 Figure from Image and video segmentation: the normalized cut framework, by Shi and Malik, copyright IEEE, Region Split : If some property of a region is NOT constant split Typical properties : variance of the intensities, error between the intensities and a fitted surface If decide on spitting, how to split, so that new regions will have constant values with this property? Try equal size splitting modified quad-tree representation Split and Merge Starting from a presegmentation, Find a region that can be splitted split into four regions If any two or more neighboring subregions are similar merge all these regions into a single region 32 16

17 Region Growing : Starting from seed regions (small regions with some homogeneity which is based on surface fitting) Find compatible neighboring points which fit to the model (surface) of the seed region grow the region Refit the surface taking into account these new points Check the difference between the new and the old goodness fit for the surface; if no improvement stop growing 33 Final Words on Segmentation Many of the assumptions do not apply to many real situations : Objects may have uniform reflectance but not uniform brightness Different objects may have the same intensities In order to obtain correct segmentation, we usually require extra information, such as user interaction, depth field, motion field Image segmentation problem is still an unsolved problem due to a requirement for a final intelligent decision 34 Graph-theoretic approaches are quite promising 17