Advanced Digital Image Processing and Others Xiaojun Qi -- REU Site Program in CVMA (0 Summer)
Segmentation Outline Strategies and Data Structures Overview of Algorithms Region Splitting Region Merging Region Growing K-Means Algorithm Based Segmentation
Importance of Segmentation Segmentation is generally the first stage in any attempt to analyze or interpret an image automatically. Segmentation bridges the gap between low-level image processing and high-level image processing. Some kinds of segmentation technique e will be found in any application involving the detection, recognition, and measurement of objects in images. 3
Importance of Segmentation (Cont.) The role of segmentation is crucial in most tasks requiring image analysis. The success or failure of the task is often a direct consequence of the success or failure of segmentation. However, a reliable and accurate segmentation of an image is, in general, very difficult to achieve by purely automatic means. 4
Image Segmentation Example 5
Image Segmentation -- Descriptive e Definition Segmentation subdivides an image into its constituent regions or objects. That is, it partitions an image into distinct regions that are meant to correlate strongly with objects or features of interest in the image. Segmentation can also be regarded as a process of grouping together pixels that have similar attributes. The level to which the subdivision is carried depends on the problem being solved. That is, segmentation should stop when the objects of interest in an application have been isolated. There is no point in carrying segmentation past the level of detail required to identify those elements. 6
7
8
Image Segmentation -- A Math Oriented Descriptive Definition It is the process that partitions the image pixels into non-overlapping regions such that:. Each region is homogeneous (i.e., uniform in terms of the pixel attributes such as intensity, color, range, or texture, and etc.) and connected.. The union of adjacent regions is not homogeneous. 9
Image Segmentation -- Explanation. All pixels must be assigned to regions.. Each pixel must belong to a single region only. 3. Each region must be uniform. 4. Any merged pair of adjacent regions must be non-uniform. 5. Each region must be a connected set of pixels. 0
How many regions do these two pictures have?
How many regions do these two pictures have?
Image Segmentation Strategies Image segmentation algorithms generally are based on one of two basic properties of intensity values: discontinuity and similarity. Discontinuity it based approach: Partition an image based on abrupt changes in intensity. Similarity il it based approach: Partition an image based on regions that are similar according to a set of predefined criteria. Thresholding Region splitting and merging Region growing 3
Discontinuity vs. Similarity Techniques based on discontinuity attempt to partition the image by detecting abrupt changes in gray level. l Point, line, and edge detectors. t Techniques based on similarity attempt to create the uniform regions by grouping together th connected pixels that satisfy predefined similarity criteria. Therefore, the results of segmentation may depend critically on these criteria and on the definition of connectivity. The approaches based on discontinuity and similarity mirror one another in the sense that completion of a boundary is equivalent to breaking 4 one region into two.
This histogrambased approach assumes that different features in an image give rise to distinct peaks in its histogram. Similarity Based Approach -- Thresholding How to choose threshold in this case? Case A: The image is composed of one light object on a dark background, in such a way that object and background pixels have gray levels els grouped into two dominant modes. Case B: Two types of light objects on a dark background. 5
Thresholding Ex 6
Ex Importance of accurate threshold selection. (a) Input Image (b) Correct choice of threshold (T = 90) (c) Threshold too low (T = 40) (d) Threshold too high (T = 5) 7
Determination of Threshold T -- An Iterative Approach. Select an initial estimate for T.. Segment the image using T. This will produce two groups of pixels: G consisting of all pixels with gray level values > T and G consisting of pixels with gray level values <=T. 3. Compute the average gray levels µ and µ for the pixels in regions G and G. 4. Compute a new threshold value: T = (µ+ µ)/. 5. Repeat step through 4 until the difference in Ts in successive iterations is smaller than a predefined parameter T 0. How do you select an initial estimate of T? 8
Similarity Based Approach -- Region Splitting Basic Idea: Divide an image into smaller and smaller regions until all the pixels in the different regions satisfy the predefined uniformity predicate or uniformity measure for that region. Steps: It begins with the entire image in one region. The region are then split to form sub-regions which satisfy the basic segmentation criterion using a suitable uniformity predicate or uniformity measure. 9
Special Data Structures -- Picture Tree in Region Splitting A picture tree is a graph structure in the form of a tree that has an arc between two nodes (i.e., R and R, where they respectively represent two regions) of the graph, if R is contained in R (i.e., R is the parent node of R). It indicates that R is a sub-region of the region R. R R R3 R4 0
Special Data Structures -- Quad Picture Tree (QPT) in Region Splitting Quad picture tree (QPT): It is a picture tree with the original image as the start region and progressively divides each region into four (square) sub-regions with an equal number of pixels. The QPT can be used to guide the search for regions with uniform gray-levels for gray-level images by developing a measure of uniformity (homogeneity) to test the regions as candidates to be split.
QPT Example Quad Picture Tree R R R 3 R 4 R R R 3 R 4 Image Data R R R 3 R 4 0 3 0 3 K 0 3 P 0 3 E 0 3 J L M N O Partition into uniform regions A B C D F G H I 00(B), 00(C), 003(D), 0x(E), 00(F), 0(G), 03(I), 03x(J), XX(K), X(M), 3X(O)
One Common Uniformity (Similarity) Measure Determine whether the region should be split Consider the case of region R split into the subregions R, R, R 3, and R 4. Let µ be the mean of R and µ i be the mean of R i. U i= ( R) = 4 4 What is the formula to calculate l ( μi μ) the variance? i= σ i L i= 0 R R R 3 R 4 μ ( r) = σ ( r) = ( r m) p( ). If U(R) is large, then R is not uniform and should be split into the four sub-regions. It indicates that the sub-regions R i differ from R but are themselves uniform. i r i 3
One Example 4 4 µ = 3, µ =, µ = 3, µ3 =, µ4 = 5. 0 ) ) ( ( p g R p 4 4 5 5 3) ) ( ( 0; 4 + + + = = p g R p σ 5 5 3 3 5 5 ) ) ( ( ; 4 4 = + + + = = p g R p σ 5) ) ( ( ; 4 4 ) ) ( ( 3 3 = + + + = = p g R p σ R R R3 R4 0. 4 5) ) ( ( 4 4 = = p g R p σ R3 R4 3 4 6 0 0 3) (5 3) ( 3) (3 3) ( ) ( = = + + + + + + = R U
Similarity Based Approach -- Region Merging. Define a splitting method to segment the image into small atomic regions satisfying the basic segmentation criterion using a suitable uniformity it predicate.. Define a method for merging adjacent regions. That is, merge two adjacent regions which satisfy the merging conditions and the basic segmentation criterion (i.e., the uniform predicate for the union of these two adjacent regions is true). 3. Repeat the merging procedure. If no regions can be merged, then stop. 5
Special Data Structures -- Region Adjacency Graph (RAG) in Region Merging Region adjacency graph gives the region adjacency relations. It is a structure often used with the QPT. That is, region R is adjacent to region R if there is a pixel in R with a 4-neighbor, or 8-neighbor, or m- neighbor in R. 0 0 0 0 0 0 0 4 4 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 E 0 3 4 Region Adjacency Graph 6
Similarity-Based Approach -- Region Growing Region growing is a bottom-up procedure that t starts t with a set of seed pixels. The aim is to grow a uniform, connected region from each seed. A pixel is added to a region if and only if It has not been assigned to any other region It is a neighbor of that region The new region created by addition of the pixel is still uniform. In general, it starts with a single pixel (seed) and add new pixels slowly. 7
Seeded Region Growing Basic Ideas: -- Non-Math Perspective. Choose the seed pixels.. Check the neighboring i pixels and add them to the region if they are similar to the seed by using a certain predicate. 3. Repeat step for each of the newly added pixels; stop if no more pixels can be added. 8
Seeded Region Growing Notations Let T be the set of pixels that are neighbors to some atomic region. That is: T = { p p U A I U i and ( N ( p) ( A i )) φ} i i where ee N(p) is sthe e8- or 4 or m-neighbor region of pixel p. The set of A i are initially the atomic regions. If pixel p is in T, then for every atomic region A i such that A N p) Φ, then letδ ( p) = g( p) mean{ g( A )} i ( i i This is the difference between p and A i, which is computed as the difference between the gray-level of p and the average gray-level l in A i. 9
Seeded Region Growing Notations The minimum of these differences over A i is defined as δ ( p) min{ δ ( p) N( p) A φ} = i i The point we are interested in is: p ' = { p T δ ( p )is a minimum ii } 30
Seeded Region Growing Algorithm -- Math Perspective. Label the pixels in the atomic regions with their initial labeling.. Find the pixels in T. Assign each pixel in T to a temporary label based on its neighboring region. Put these pixels in a linear list, which is sorted in an ascending order according to δ i ( p). 3. Remove the first pixel p from the linear list while the list is not empty. 3. Set p to the same label as its neighboring region. 3U 3. Update the mean of the affected (i.e., expanded) d) region. 3.3 If the new neighbors of the expanded region already have temporary labels, label these neighbors as boundary pixels. These boundary pixels will not be carried on to the further process. 3.4 Otherwise, assign the new neighbors a temporary label based on their neighboring g region. 3.5 Recalculate the δ i ( p) for each temporarily labeled neighboring pixel of the expanded region. Order the linear list 3 in ascending order according to the new ( p). δ i
Ave =, Ave = 7 Ave =, Ave = 7 Ave =, Ave = 7 Ave =, Ave = 7 3 4 3 4 3 4 3 4 3 5 6 3 5 6 3 5 6 3 5 6 3 6 7 3 6 7 3 6 7 3 6 7 3 6 7 3 6 7 3 6 7 3 6 7 0 0 0 0 0 0 Ave =.4, Ave = 6.5 Ave =.5, Ave = 7 Ave =.4, Ave = 7 Ave =.4, Ave = 6.7 3 4 3 5 6 3 6 7 3 6 7 3 4 3 5 6 3 6 7 3 6 7 3 4 3 5 6 3 6 7 3 6 7 3 4 3 5 6 3 6 7 3 6 7 0.75.75.75.75.75.6.6.6.6 0.7 0.7.6 0.5.5 3
Region-Merging-Based Image Segmentation Illustration 33
Region-Merging-Based Segmentation Using Color Features Divide the image into x non-overlapping overlapping blocks. Extract a color feature vector (i.e., the mean color of the block) for each block by using the Luv color space. Apply an unsupervised K-means algorithm to cluster the color features. The segmentation process adaptively increases the number of regions C (initially set as ) until a termination ti criterion i is satisfied. 34
Cluster Seeking -- K-Means Algorithm The K-means algorithm is based on the minimization of a performance index which is defined as the sum of the squared distances from all points in a cluster domain to the cluster center. 35
Cluster Seeking -- K-Means Algorithm (Cont.) Step : Choose K initial cluster centers z(), ( z(), (, zk(). These are arbitrary and are usually selected as the first K samples of the given sample set. Step : At the kth iterative step, distribute the samples {x} among the K cluster domains, using the relation, x S ( k) if x z ( k) < x z ( k) () j For all i =,,..., K, i j, where Sj(k) denotes the set of samples whose cluster center is zj(k). Ties in expression () are resolved arbitrarily. j i 36
Step 3: From the results of Step, compute the new cluster centers zj(k+), j =,,..., K, such that the sum of fthe squared ddistances from all points in Sj(k) to the new cluster center is minimized. In other words, the new cluster center zj(k+) is computed so that the performance index J j x S ( k ) = x z ( k + ), j j j =,, K, K is minimized. The zj(k+), which minimizes this performance index is simply the sample means of Sj(k). Therefore, the new cluster center is given by z j ( k + ) = x, N j x S j ( k ) j =,, K, K Where Nj is the number of samples in Sj(k). The name K-means is obviously derived from the manner 37 in which cluster centers are sequentially updated.
Step 4: If zj(k+) = zj(k) for j =,,..., K, the algorithm has converged and the procedure is terminated. Otherwise go to Step. The behavior of the K-means algorithm is influenced by: The number of cluster centers specified; The choice of initial cluster centers; The order in which the samples are taken; The geometrical properties of the data. In most practical cases the application of this algorithm will require experimenting with various values of K as well as different choices of starting ti configurations. 38
Cluster Seeking -- K-Means Algorithm Example 39
Step. Let K = and choose z() = x = (0, 0), z() = x = (, 0). Step. Since x ( z ) < x z () and i x z( ) < x3 z i (), i = 3 = S ( ) = { x, 3} we have that. Similarly, the remaining patterns are closer to z(), so S x ( ) = { x, x 4, x 5, K, 0} x 40
Step 3. Update the cluster centers: = ) ( ) ( x z x S N + = ) ( 3 x x = 5. 0 0. 0 = ) ( ) ( x z x S N + + + = 67 5 ) ( 8 0 4 x x x L 4 = 33. 5 67. 5
Step 4. Since zj() zj(), for j =,, we return to Step. Step. With the new cluster centers we obtain x l for and z l ( ) < xl z () =,, K,8 z ( ) < xl z () l = 9,0, K, Therefore: for 0 x l S( ) = { x, x, K, x8} and S ( ) = { x9, x0, K, 0} x 4
Step 3. Update the cluster centers: z (3) = x N z x S () = ( x + x + Lx 8.5 =.3 (3) = x N x S () = ( x9 + x0 + L+ x 7.67 = 7.33 8 ) 0 ) 43
Step 4. Since zj(3) zj(), for j =,, we return to Step. Step yields the same results as in the previous iteration: S ( 4) = S (3) and S (4) = S Step 3 also yields the same results. (3) Step 4. Since zj(4) = zj(3) for j =,, the algorithm has converged, yielding these cluster centers:.5 7.67 z =, z =.33 7. 33 These results agree with what we would expect from inspection of the given data. 44