Institute of Informatics Eötvös Loránd University Budapest, Hungary Basic Algorithms for Digital Image Analysis: a course Dmitrij Csetverikov with help of Attila Lerch, Judit Verestóy, Zoltán Megyesi, Zsolt Jankó and Levente Hajder http://visual.ipan.sztaki.hu
Lecture 11: Image segmentation Discussion of grey-level thresholding Examples of thresholding Imroving the histogram for better peak separation Thresholding versus edge detection Limits of thresholding Region-oriented segmentation Definition Region growing and merging Quadtrees Split-and-merge algorithms Other segmentation methods 2
Examples of thresholding images Otsu results Gaussian results Here, both methods give acceptable results. The Gaussian algorithm sets lower thresholds in both cases. Fits object contours better than Otsu. 3
fingerprint image image histogram Otsu T = 158 Gauss T = 199 Here, both methods still give satisfactory results. The Otsu algorithm sets threshold in valley of histogram. Fingerprint lines are well-separated. The Gaussian algorithm sets slightly high threshold. Some fingerprint lines touch. 4
low-quality image image histogram Otsu T = 159 Gauss T = 201 Here, only Otsu gives a satisfactory result. The Otsu algorithm finds the small class of pixels (dark discs). The Gaussian algorithm tries to separate two high peaks formed by the background. Noisy valley is selected because the true class is too small too far 5
image histogram Otsu result Here, only Otsu algorithm produces results. Gaussian algorithm gives no results at all. Upper row: unimodal histogram, no approximation obtained Lower row: an approximation obtained, but the threshold equation has no real root. 6
Imroving the histogram for better peak separation Use of gradient to improve histogram: Combine intensity and gradient information for better separation of objects and background. Pixels close to edges have high gradients and medium intensities. Pixels of object and background have low gradient and low or high intensities. To better separate objects from background, discard high gradient pixels when computing the histogram. T edge object P(i) original background improved T i Principle of histogram peak separation. 7
Thresholding versus edge detection Thresholding with a constant threshold is a global operation. Advantage: Closed contours guaranteed Drawback: Not applicable to images with uneven illumination Edge detection is a local operation Advantage: Applicable to images with uneven illumination Drawback: Closed contours not guaranteed thresholds f(x) f (x) x Signal with varying level that cannot be thresholded. Edges can be detected. 8
Limits of thresholding stone crack Otsu method Another method Merit (quality) of thresholding is task-dependent. The merit may include geometric properties. Image histogram does not account for geometry. The crack is detected as set of bright pixels independently of the crack shape. 9
Limits of thresholding: stone Otsu method Another method In this example, the merit of thresholding is uncertain. No geometric informaton is taken into account. Compact and connected regions are not guaranteed. Select threshold, then arbitrarily interchange pixels in image, select again same threshold Solution: Combine intensity and geometry using region-oriented methods. 10
Region-oriented segmentation Goal: Obtain homogeneous connected regions. Basic formulation of segmentation: Partition the image R into n subregions R 1, R 2,..., R n so that (a) n i=1 R i = R (b) R i is a connected region, i = 1, 2,..., n (c) R i R j = for all i and j, i j (d) P (R i ) = T RUE for i = 1, 2,..., n (e) P (R i R j ) = F ALSE for i j where P (R i ) is a logical homogeneity predicate over the points in region R i : P (R i ) = T RUE means that all pixels in R i have similar properties, that is, R i is homogeneous. 11
Examples of homogeneity predicates for a region: Difference between max and min greyvalues is small. Difference between any pixel and mean greyvalue is small. Meaning of conditions (a e): (a) Completeness: Each pixel is assigned to a region. (b) Connectedness: Points in each region are connected. (c) Regions are disjoint. (d) Each region is homogeneous. (e) Any union of regions is inhomogeneous: Minimise number of regions. 12
Region growing and region merging Region growing: Group pixels or subregions into larger regions while the homogeneity criterion is satisfied. A simple form of region growing is pixel aggregation: Select seed points and a homogeneity criterion. Grow regions around each seed point by appending those neighboring pixels that have similar properties. Pixel aggregation is usually followed by region merging. Region merging is a form of region growing. It can be applied as a separate operation, or be incorporated into an iterative segmentation algorithm. 13
Algorithm 1: Region growing by pixel aggregation 1. Initialisation: Select N seed pixels S i, i = 1, 2,..., N, and a threshold T. Initialise N regions R (0) i as the seed pixels S i. Initialise N region means M (0) i as the greyvalues of S i. 2. At each iteration k, find border pixels of each region R (k) i. 3. Search the 8-neighbourhood of each border pixel of each region. Assign a pixel p(x, y) to R (k) i if p(x, y) is a 8-neighbour of a border pixel of R (k) i and p(x, y) has not been assigned yet and greyvalue of p is sufficiently close to region mean: f(x, y) M (k) i T 4. Stop if no region growing was possible. Otherwise, update M (k) i going to step 2. and iterate by 14
Different homogeneity criteria can be used. In the example below: the criterion is the maximum allowed absolute difference T within region; region merging is applied after growing. 0 0 5 6 7 1 1 5 8 7 0 1 6 7 7 2 0 7 6 6 0 1 5 6 5 a a b b b a a b b b a a b b b a a b b b a a b b b a a a a a a a a a a a a a a a a a a a a a a a a a (a) (b) (c) Example of pixel aggregation followed by region merging: (a) Original image matrix with two seeds indicated. (b) Result for T = 4. (c) Result for T = 8. Major problem with pixel aggregation: How to select seeds? Result depends heavily on choice and order of seeds. 15
Region merging Region merging is a form of region growing. This operation is applied when adjacent regions have similar properties. For example, this happens in pixel aggregation when several seeds are given for one region. Different merging criteria: Merge two adjacent regions R i and R j if P agg (R i R j ) = T RUE, where P agg is the criterion used for aggregation, or P (R i R j ) = T RUE, where P is another homogeneity predicate, or border between R i and R j has no strong edges (gradient > T high ), or border between R i and R j has many weak locations (gradient < T low ), 16
Region growing by merging Merging and growing can be combined with image division (splitting). This can be done in different ways. The goal is to avoid interactive specification of seeds. Basic idea of region growing by merging: Divide image into atomic regions of constant greyevel, or other local propery. Merge similar adjacent regions sequentially until the adjacent regions become sufficiently different. weak boundary R 1 R 2 R 3 merge R m 1 =R 1 UR 2 UR 4 R 3 R 5 R 4 R5 Region growing by merging. 17
Advantages of region growing: Connected regions guaranteed. Summary of region growing Possibility to incorporate priori knowledge about regions: shape, size, texture, spatial relations, etc. Regions obtained have desired properties. Drawbacks: Prior information may be needed: seeds, starting points. Heuristics needed: for example, merging rules ( weak borders, etc.) Intrinsically sequential: no parallel implementation possible. Order-dependence: Result depends on the order in which pixels are examined. Possible solutions: Look-ahead, back-tracking. 18
Split-and-merge method and quadtrees The split-and-merge algorithm uses a homogeneity predicate P and has two stages: Top-down: Split image into homogeneous quadrant regions Bottom-up: Merge similar adjacent regions Algorithm 2: Split-and-merge algorithm 1. Top-down: Successively subdivide image and regions into smaller quadrant regions until P (R i ) = T RUE for each R i. Obtain a quadtree structure. 2. Bottom-up: At each level, merge any adjacent regions R i and R j for which P (R i R j ) = T RUE. 3. Repeat steps 1 and 2 until no further splitting/merging is possible. 19
1 A Split D 4 2 B A C D A D C Merge B C 3 stages of segmentation R 2 R B 1 R 1 R 2 1 2 3 A B D C A B C D A B C D quadtree split and merge 4 1C,2D,3A 1 (A,B,D) 2 (A,B,C) 3 (B,C,D) 4 segmented regions Segmentation by split-and-merge approach. 20
Summary of split-and-merge Advantages of split-and-merge: Connected regions guaranteed Some (limited) possibility to incorporate geometric knowledge Drawbacks: Result may depend on position and orientation of image with respect to raster Imposes rectangular structure on segmented image Relatively large computational load and storage requirements Simple quadtree only supports vertical dataflow Adjacent regions to be merged can be far away in quadtree Quadtrees are widely used in GIS (Geographic Information Systems). 21
Comparison of a growing and a merging algorithm image with seeds grow T = 20 grow T = 30 grow T = 40 When T = 20, face regions with large variation cannot grow. Dark hair region with small variation spreads and occupies large area When T = 30, face regions can grow and occupy larger areas. Dark hair region shrinks Result depends on order and positions of seeds. 22
merge T = 20 merge T = 30 merge T = 40 merge T = 20 merge T = 40 grow T = 20 grow T = 40 As T increases, number of regions decreases, merging segmentation becomes coarser. Dark hair region extends. In region growing, the effect is just the opposite: dark region shinks. 23
Comparison of split-and-merge and region merging cell image SaM T = 40 SaM T = 50 SaM T = 60 merge T = 40 merge T = 50 merge T = 60 Rectangular structure of splitting is visible in split-and-merge images. Merging segmentation looks finer. 24
Other segmentation methods Model-based segmentation: Partition image into segments of known shape and other properties. Edge-based segmentation: Find object contours by edge tracing. Texture-based segmentation: Apply region-based techniques to textural properties of regions. Motion-based segmentation: In an image sequence, locate objects based on their motion. Rule-based segmentation: Expert system with general-purpose knowledge about images and grouping criteria. 25