Integrating Intensity and Texture in Markov Random Fields Segmentation Amer Dawoud and Anton Netchaev {amer.dawoud*, anton.netchaev}@usm.edu School of Computing, University of Southern Mississippi 118 College Dr. Box#5106, Hattiesburg MS 39402 USA Abstract This paper proposes an algorithm that fuses visual cues of intensity and texture in Markov Random Fields (MRF) region growing texture image segmentation. The idea is to segment the image in a way that takes EdgeFlow edges into consideration, which provides a single framework for identifying objects boundaries based on texture and intensity descriptors. This is achieved by modifying the energy minimization process so that it would penalize merging regions that have EdgeFlow edges in the boundary between them. Experimental results confirm the hypothesis that the integration of edge information increases the precision of the segmentation by ensuring the conservation of the objects contours during the region growing process. Keywords: Segmentation, Markov Random Fields, EdgeFlow, Energy Function Minimization. 1. Introduction Image segmentation is a process that decomposes an image into disjoint regions and is a fundamental step for many image-processing tasks such as image understanding [1]. In general, image segmentation aims at producing regions that are homogeneous with respect to the extracted features, such as gray level or texture, and have significant different feature values across region boundaries [2]. Many algorithms were proposed over the years. Feature-based approaches such as thresholding [3] and clustering [4] would usually produce noisy results. Many other methods utilize spatial context information explicitly or implicitly. Edge-based segmentation [5], [6], [7] are efficient in describing local behavior but inefficient in producing global meaningful results. Region splitting and merging algorithms [8], [9] have problem with merging and stopping criteria that would usually cause the result to be either over-segmented or under-segmented. Model-based approaches, such as curve evolution [10], [11], [12] and random fields [13] [14] [15] [16], have established mathematical foundation but they require accurate model and the optimization process to be able to converge to a reasonable solution, which are difficult to achieve. In this paper, image segmentation algorithm is presented, which is an extension to our work [1]. The motivation is that the algorithm [1], which fuses Canny edges in MRF region growing, could not handle texture images. In this algorithm, visual cues of texture and intensity are fused in MRF region growing segmentation. The algorithm can be characterized by two aspects: 1) it uses region growing technique in search for optimal solution. 2) it fuses EdgeFlow intensity and texture edge information in the energy minimization process in a way that penalizes merging region with such edges in the boundary between them. Next section describes our algorithm in detail, and section 3 shows experimental results demonstrating that our algorithm increases the precision of the segmentation by ensuring the conservation of the objects contours during the region growing. 2. Problem Statement and Related Work This work is an extension to our work [1], so it is necessary to distinguish our contribution by briefly introducing our previous work first and then clearly describing our contribution, which is the fusion of EdgeFlow edges in MRF region growing image segmentation. 2.1 MRF-Based Formulation of Image Segmentation Let S denote the discrete rectangular lattice on which images are defined. Suppose there are n different classes in the image to be segmented. is a set of discrete valued random variables constituting a random field on S, with each variable X s taking a value in {1,,n} representing the class to which the site s belongs. is another random field somehow related to X and the observed image is a realization from Y. Let and denote the realizations of X and Y, respectively. The image segmentation task can be formulated as a maximum a posterior (MAP) problem for which maximizing the posterior P(x y) gives a solution. By the Bayes rule, this is equivalent to maximizing p(y x) P(x). Two models are used for analytically representing p(y x) (the feature model) and P(x) (the spatial context model). With both the
feature model and the spatial context model defined, the MAP formulation of the segmentation task is transformed into minimizing energy where (1) where a and b are neighboring sites forming a pair-site clique and is a positive number. Such a model makes the prior P(x) large if the local neighborhood is dominated by one single class and small otherwise and, hence, is effective in suppressing noisy configurations of class labels. R is the set of all cliques on the entire lattice S., (3) where and are the mean and variance of the pixel values in class i. So the image segmentation problem is formulated as follow as where is the Kronecker delta function. Yu and Clausi's [2] discussed selection in the literature. They said that needs to be set a priori using an experimentally satisfactory value and the general rule for selecting is that it should be set to a large value for simple scenes and small value for complex scenes. 2.2 Extending to Region-Based Segmentation To simplify the complexity of the problem, the MRF can be defined on irregular graphs rather than the regular image lattice. This allows the image segmentation problem formulated by (4) to be based on a set of interconnected groups of pixels, with the MRF spatial context model based on a region adjacency graph (RAG) [29]. Here, the labeling is not on single pixels but on regions, where the regions are commonly obtained by a deliberate oversegmentation. Each node in the RAG represents a region and a link between the nodes represents the existence of a common boundary between the regions Defined on the RAG, the MRF models the behaviors of the regions in a similar way as for pixels. Let R i denote node i in the graph and let x i denote the label for all sites s R i. The feature model energy for R i can be defined as and the MRF pair site clique energy for two neighboring nodes R i. and R j is Summation of the above energies over the entire RAG gives exactly (4). A combinatorial optimization technique is then applied to RAG nodes instead of pixels. Such a region-based segmentation method is advantageous in computation speed as the number of RAG nodes is usually significantly less than the number of pixels. 3. Proposed Algorithm Figure 1 shows the flowchart of the proposed algorithm. The initial step is over-segmentation of input image using watershed [30], which is a well-established method that is based on the image topology. The magnitude of the gradient is interpreted as elevation information. With successive flooding, watersheds with adjacent catchment
basins are constructed. This operation results in image being over-segmented and regions separated by watershed lines as shown in Figure 2(A). Input Image EdgeFlow edge vectors Watershed oversegmentation Combining EdgeFlow lines with watershed lines Find Region Adjacency Graph (RAG) Calculate regions statistics Calculate MRF energy of merging all neighbouring regions in RAG Merge the two regions producing minimum energy No Max. number of iterations reached? Yes Stop Figure 1: Algorithm's Flowchart 3.1. EdgeFlow Edges EdgeFlow [31] is a scheme that utilizes predictive coding model to identify the direction of change in intensity and texture to construct edge flow vectors. We include the following description of the algorithm which is necessary to understand our contribution. Intensity and texture energies and their corresponding probabilities obtained from different image attributes are combined together to form a single edge flow field for boundary detection. where and represent the energy and probability of the edge flow computed from image attribute intensity and texture} at pixel site orientation angle. is the weighting coefficient associated with image attribute. The flow direction is estimated at each location site in the image are computed. Then a continuous range of flow directions which maximizes the sum of probabilities in a corresponding half plane is identified: The edge flow vector is then defined to be the following vector sum: =, (10) where is a complex number with its magnitude representing the resulting edge energy and angle representing the flow direction. After the edge flow vector of an image is computed, boundary detection can be performed by propagating the edge flow vector and identifying the locations where two opposite direction of flows encounter
each other. At each location, the edge flow energy is transmitted to its neighbor in the direction of flow if the neighbor also has a similar flow direction. Figure 2: A) Watershed over-segmentation: watershed lines separating regions (total number of regions is 5). B) EdgeFlow lines: white 2-pixel thick line of an EdgeFlow line. C) Combining the EdgeFlow line (total number of regions is 7). The next step is to combine the Watershed line with EdgeFlow lines, as shown in Figure 2. The purpose of this step is to make sure that the EdgeFlow lines are coinciding with the boundary of some regions. As we will see in the next steps of the algorithm, this is necessary to integrate the edge information in the MRF clique energy. 3.2. Fusion of Edge Information in MRF Energy Minimization is a set of edge random field on S, with each variable D s taking a value of {0,1} representing a nonedge and an edge site in s, respectively. Let denote the realization of D. The MRF pair site clique energy for two neighboring nodes R i. and R j is modified from (6) to The difference between (6) and (11) is the introduction of, since is similar to in (6). Eq. (11) means that for a clique of two neighboring sites to contribute to then they must belong to two different classes. And if one and only one of the sites is an edge site then that clique adds to, and if both sites are either edge sites or non-edge site then that clique adds to. The effect of introducing which is negative, is that it should reduce the MRF pair site clique energy instead of increasing it as with the case of, which is positive. With regard to region growing (to be discussed in next sub-section), introducing will inhibit merging regions with EdgeFlow lines passing through the boundaries between them. Experimentally, best results were achieved when is set to -5 < < -2. Ω 1 Ω 2 Ω 3 Cliques contributing β1 to E Cliques contributing β2 to E Pixel sites with opposing EdgeFlow vectors Figure 3: Example showing 3 classes and cliques contributing and based on EdgeFlow location.
Figure 3 shows an example of three classes, with edge sites are marked with X. This configuration results in cliques contributing and to E, marked by blue and red links, respectively. As a result of being positive and being negative, merging with will be easier than merging with or with. 3.3. Iterative Region Growing Image Segmentation The algorithm starts merging image regions with the aim of minimizing the overall energy, which is the summation of feature model energy (6) and MRF pair site clique energy (11). This is achieved, as shown in flowchart of Fig. 1, by building the RAG. Each node in the RAG represents a region (an independent spatial grouping of pixels), and each link represents the common boundary between the regions. Then we calculate the statistics of all regions, and. Then the differences in energy associated with merging each two neighboring regions, which are represented as link in the RAG, are calculated as follows: Suppose two classes, and, are being investigated. Let denote the class obtained by merging the two classes. The energy difference = The two regions producing the lowest in the RAG are merged. This process is repeated iteratively and regions are merged sequentially, till number of desired classes is reached. In the next iteration, there is no need to do the whole calculations from scratch. The only calculations needed to update RAG and will be related to the new region formed by the merging in pervious iteration, which makes the program run faster. The number of desired classes is a parameter that the user has to provide in advance. There are many applications where this number is known in advance, like brain MRI segmentation and SAR classification. 4. Results Figures 4 and 5 show experimental results comparing our algorithm with the algorithms [1] and [2]. The segmentation results at various iterations or number of classes (N = 50, 10, and 6 for Figure 4, N = 10, 5 for Figure 5) are captured. The first row of Figure 4 shows EdgeFlow edges used by our algorithm. First three columns of the figure show the segmentation results (in color) using our proposed algorithm with different and settings. Last two columns show the segmentation results of algorithm [1] which integrates normal edges (Canny edges), and algorithm [2] without the integrating any edges. Our algorithm performed better in terms of segmenting the image along the edge lines, which confirms the hypothesis that the fusion of the visual cues of intensity and texture edges increases the precision of the segmentation by ensuring the conservation of the objects during the region growing. This is a direct result for introducing that neqative in the MRF energy function, which makes it difficult to megre regions separated by real edge line. Figure 5 shows the case of a textured image, and although the edge image was not accurate in outlining the five testured classes, it was enough to keep them separated, which demonstrates the flexability of our algorithm in utilizing the EdgeFlow edge information. 5. Conclusions We proposed an algorithm that fuses EdgeFlow intensity and texture edges into MRF region growing based image segmentation. EdgeFlow provides a single framework for identifying objects boundaries based on intensity and texture descriptors. The idea is to segment the image in a way that takes these edges into consideration. We achieved that by modifying the energy function minimization process so that it would penalize merging regions that have real edges in the boundary between them. Experimental results confirm the hypothesis that the fusion of the intensity and texture information increases the precision of the segmentation by ensuring the conservation of the objects during the region growing.
Figure 4: Segmentation results for the image shown in Figure 2 at various N (number of regions). First three columns show the segmentation results using our proposed algorithm with different and settings. Last two columns show the segmentation results without the fusion of EdgeFlow information (algorithm [1] and [2]) with different and settings. Figure 5: Segmentation of synthetic noisy image. First row shows the input image and the EdgeFlow edge image. Left column shows the segmentation results using our proposed algorithm with = 1, and = -5.
Right column shows the segmentation results (algorithm [1]) without the fusion of edge information with = 1, and = -5. References [1] A. Dawoud and A. Netchaev, "Preserving objects in Markov Random Fields region growing image segmentation," Pattern Analysis and Applications, accepted for publication.. [2] Q. Yu and D.A. Clausi, "IRGS: Image Segmentation Using Edge Penalties and Region Growing," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.30, pp. 2126-2139, 2008. [3] N. Otsu, A Threshold Selection Method from Gray-Level Histograms, IEEE Trans. Systems, Man, and Cybernetics, vol. 9, pp. 62-66, 1979. [4] A.K. Jain and R.C. Dubes, Algorithms for Clustering Data. Prentice Hall, 1988. [5] W.Y. Ma and B.S. Manjunath, Edgeflow: A Technique for Boundary Detection and Image Segmentation, IEEE Trans. Image Processing, vol. 9, pp. 1375-1388, 2000. [6] H.T. Nguyen, M. Worring, and R. van den Boomgaard, Watersnakes: Energy-Driven Watershed Segmentation, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, 2003. [7] L. Vincent and P. Soille, Watershed in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 6, pp. 583-598, 1991. [8] R. Adams and L. Bischof, Seeded Region Growing, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, June 1994. [9] K. Haris, S.N. Efstratiadis, N. Maglaveras, and A.K. Katsaggelos, Hybrid Image Segmentation Using Watersheds and Fast Region Merging, IEEE Trans. Image Processing, vol. 7, pp. 1684-1699, 1998. [10] T.F. Cootes, C.J. Taylor, D.H. Cooper, and J. Graham, Active Shape Models Their Training and Application, Computer Vision and Image Understanding, vol. 61, pp. 38-59, 1995. [11] M. Kass, A. Witkin, and D. Terzopoulos, Snakes: Active Contour Models, Int l J. Computer Vision, vol. 1, no. 4, pp. 321-331, 1987. [12] J.A. Yezzi, A. Tsai, and A. Willsky, A Fully Global Approach to Image Segmentation via Coupled Curve Evolution Equations, J. Visual Comm. and Image Representation, vol. 13, pp. 195-216, 2002. [13] H. Derin and H. Elliott, Modeling and Segmentation of Noisy and Textured Images Using Gibbs Random Fields, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, pp. 39-55, 1987. [14] S. Geman and D. Geman, Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, pp. 721-741, 1984. [15] S. Kumar and M. Hebert, Discriminative Random Fields, Int l J. Computer Vision, vol. 68, no. 2, pp. 179-202, 2006. [16] C.S. Won and H. Derin, Unsupervised Segmentation of Noisy and Textured Images Using Markov Random Fields, CVGIP: Graphical Models and Image Processing, vol. 54, no. 4, pp. 308-328, 1992.