CHAPTER 6 QUANTITATIVE PERFORMANCE ANALYSIS OF THE PROPOSED COLOR TEXTURE SEGMENTATION ALGORITHMS

Size: px

Start display at page:

Download "CHAPTER 6 QUANTITATIVE PERFORMANCE ANALYSIS OF THE PROPOSED COLOR TEXTURE SEGMENTATION ALGORITHMS"

Norah Charles
5 years ago
Views:

1 145 CHAPTER 6 QUANTITATIVE PERFORMANCE ANALYSIS OF THE PROPOSED COLOR TEXTURE SEGMENTATION ALGORITHMS 6.1 INTRODUCTION This chapter analyzes the performance of the three proposed colortexture segmentation algorithms quantitatively. A large number of experiments had been carried out to assess the reliability of the proposed color-texture segmentation algorithms. These tests were conducted on both natural and synthetic image datasets and the results were quantitatively evaluated. The first set of experiments was performed on the Berkeley database (Martin et al 2001) that is composed of natural images characterized by various degrees of complexity with respect to color and texture information such as non-uniform textures, unclear borders and low image contrast. Berkeley image segmentation database also contains benchmark segmentation results obtained from human subjects (between 4 and 7). The Berkeley image segmentation benchmark set is available at address: h/. The purpose of this investigation was to obtain a comprehensive quantitative performance evaluation of the proposed algorithms with respect to the correct identification of perceptual regions in the image and the level of image detail.

2 146 The second set of tests was performed on a dataset of 33 mosaic images (given in chapter 1) and the segmentation results were evaluated by analyzing the errors obtained by computing the displacement between the border pixels of the segmented regions and the region borders in the ground truth data. In order to illustrate the validity of the proposed scheme, the results of proposed algorithms have been compared against those obtained by the well-established J measure based SEGmentation (JSEG), Compression-based Texture Merging (CTM), Fusion of Clustering Results (FCR) and Color- Texture coherence (CTex) algorithms Implementation details of State-of-art Algorithms Three popular algorithms (JSEG, FCR and CTex) for which the source code and/or the results are made available by the authors are taken for comparison. The implementation details of these state of art algorithms are provided in this subsection. JSEG JSEG algorithm is a standard color-texture segmentation benchmark developed by Deng and Manjunath (2001) and the implementation is made available online by the authors ( /segmentation/jseg/software/). JSEG involves three parameters that have to be specified by the user (the color quantization threshold, the scale and the merge threshold) and in this study the parameter values are set (255, 1.0 and 0.4 respectively ) as suggested by the authors. CTM Compression-based texture merging (CTM) algorithm considers natural-image segmentation as a problem of clustering texure features as

3 147 multivariate mixed data. The distribution of the texture features are modeled using a mixture of Gaussian distributions. It was developed by Yang et al in The matlab source code is found at address: berkeley.edu/~yang/software/lossy_segmentation/. The authors also provide the segmentation results on Berkeley dataset with two different values for the parameter gamma (gamma=0.1and gamma=0.2). FCR FCR was proposed by Max Mignotte (2008) and it segments the images by fusion of histogram-based K-means clusters in different color spaces. The segmentation results on the entire berkeley database, their quantitative performance measures and the source code (in C++ language) are found at /ResearchMaterial. The user defined parameters of FCR are the number of classes of the segmentation to be fused (K 1 ), the resulting number of classes of the final fused segmentation map (K 2 ) and region merging threshold ( ). The author has provided the quantitative results for five different combinations of these parameter values on berkeley image database. They are [K 1 =13 K 2 =6 =0.135], [K 1 =6 K 2 =6 =0.130], [K 1 =13 K 2 =13 =0.145], [K 1 =9 K 2 =6 =0.140] and [K 1 =12 K 2 =4 =0.125]. CTex CTex was developed by Ilea and Whelan (2008) which is based on the adaptive inclusion of color and texture in the process of data partition. The quantitative and qualitative evaluation of CTex on synthetic mosaic dataset is provided in the literature (Ilea and Whelan 2008). But the source code is not available online. The authors have applied their CTex algorithm on Berkeley dataset and provided only the PRI measure. The other measures nameley VoI,GCE and BDE are not computed by the authors of CTex. This CTex

4 148 algorithm is taken for comparison only in the case of synthetic images because the same evaluation technique is followed in this thesis Quantitative Performance Measure used in Berkeley Dataset The quantitative performance measures used in the experiments performed on Berkeley dataset are: (i) The Probabilistic Rand Index (PRI) (Pantofaru, Hebert 2005) counts the fraction of pairs of pixels whose labellings are consistent between the computed segmentation and the ground truth, averaging across multiple ground truth segmentations to account for scale variation in human perception. (ii) The Variation of Information (VoI) metric (Meila, 2005) defines the distance between two segmentations as the average conditional entropy of one segmentation given the other, and thus roughly measures the amount of randomness in one segmentation which cannot be explained by the other. (iii) The Global Consistency Error (GCE) ( Martin et al 2001) measures the extent to which one segmentation can be viewed as a refinement of the other. Segmentations which are related in this manner are considered to be consistent, since they could represent the same natural image segmented at different scales. (iv) The Boundary Displacement Error (BDE) (Freixenet 2002) measures the average displacement error of boundary pixels between two segmented images. Particularly, it defines the error of one boundary pixel as the distance between the pixel and the closest pixel in the other boundary image.

5 149 Yang has kindly provided the Matlab source code to estimate these quantitative performance measures, which is available online at address The proposed segmentation algorithms have also been analyzed quantitatively in Table 6.1 using the mean and standard deviation of PR Index as in (Ilea and Whelan 2008). The PR index quantitatively evaluates the association between the obtained segmentation result and multiple ground-truth segmentations manually generated. The value of PR Index ranges between 0 and 1. The mean of PR Index indicates the overall performance of the system and the standard deviation of the PR Index points out the dependency of the algorithm with the color and texture properties of the input image. If the mean of PR Index is more the overall performance of the algorithm is better and if the standard deviation is less the algorithm behaves similarly with all images irrespective of their color contrast, homogeneity, complexity in textures etc Performance Measure on Synthetic Dataset Since the ground truth data associated with complex natural images is difficult to estimate and its extraction is highly influenced by the subjectivity of the human operator, the proposed algorithms are tested on synthetic data where the ground truth is unambiguous. The segmentation accuracy of the proposed algorithms and state of art algorithms discussed above is estimated by calculating the Euclidean distances between the pixels situated on the border of the regions present in the segmented results and the border pixels present in the ground truth data. To evaluate the segmentation errors numerically, the mean, standard deviation and r.m.s errors that measure the border displacement between the ground truth and the segmented results are calculated. The rest of

6 150 this chapter is organized as follows: Section 6.2 presents the quantitative results of proposed algorithms, where the experiments are performed on Berkeley segmentation dataset, Section 6.3 quantitatively analyzes the performance of proposed algorithms on synthetic images and this chapter is concluded in section EXPERIMENTS PERFORMED ON BERKELEY DATASET IMAGES Berkeley dataset consists of 300 color images of size 481x321. For each color image, a set of benchmark segmentation results, provided by human observers (between 4 and 7), is available and they are used to quantify the consistency of the proposed segmentation algorithms. Initially all color images from Berkeley dataset are resized to have the longest side equals to 320 pixels for convenience. The proposed algorithms are applied on all the 300 images and distribution of the different performance measure over the 300 images are computed and presented in the next subsections and finally the average performance of the three proposed algorithms are compared with those of the state of art algorithms Performance of the Proposed RB-Wavelet and SOM Color Texture Segmentation Algorithm Figure 6.1 presents the distribution of performance measures RI, VoI, GCE and BDE, when the proposed first approach was applied on all the 300 images in Berkeley dataset. Among 300 images less than 50 images have got lesser (less than 0.75) PRI and greater error measures (>0.35 GCE, >3 VoI, >15 BDE). On the average, the PRI, GCE, VoI and BDE measures obtained by this first approach are , , and respectively. The maximum and minimum PRI obtained by this approach are 0.95 and 0.35 respectively.

7 151 Figure 6.1 Distribution of the performance measures PRI, GCE,VoI, and BDE on all the 300 images from the Berkeley dataset for the proposed first algorithm Performance of the Proposed Pixon-Modified FMM Segmentation Algorithm In the proposed second approach color and texture information are evaluated in two levels. In the first level, the given color image is compressed by the pixon representation using color moments and consequently the size of the local color and texture distributions is automatically adjusted to the image content. In the second level the noisy pixons are identified by adaptively including the spatial information between them. Therefore the color- textures with different shapes (not necessarily square or rectangle), sizes are determined very effectively and this is the main advantage of the proposed system.

8 152 Figure 6.2 depicts the distribution of PRI, VoI, GCE and BDE performance measures, when the proposed second approach was applied on all the 300 images in Berkeley dataset. The average PRI, GCE, VoI and BDE measures obtained by this second approach are , , and respectively. This approach greatly reduces the boundary error and gives very good PRI value ( > 0.9) for most of the images. The maximum and minimum PRI obtained by this approach are 0.95 and 0.4 respectively. Figure 6.2 Distribution of the different performance measures when the proposed second approach is applied on all the 300 images of the Berkeley image database Performance of the Proposed Q-JSEG Segmentation Algorithm The main objective of this third approach is improve the JSEG through the application of global BQMP thresholding based quantization

9 153 technique on the uncorrelated, normally distributed dataset and increasing the number of scales in region growing step. Once the colors in the image are separated and quantized properly, the spatial segmentation becomes faster and perfect. The image details are identified by the scales used in the segmentation. Different texture models are identified by different scales. If the scales used in the spatial segmentation are appropriate and if their sizes coincide with the sizes of the texture patterns present in the image, the algorithm can preserve the image detail to the optimum level. But the time complexity of the algorithm is directly proportional to the number of scales used in the sytem. So there is a trade of between the accuracy and time. Figure 6.3 Distribution of the different performance measures when the Q-JSEG algorithm is applied on all the 300 images of the Berkeley image database

10 154 The time taken by the algorithm to segment the image with three scales [9x9], [5x5] and [3x3] is 64 seconds in an Intel Core 2 Duo processor T5800 (2 GHz), 1GB RAM and Windows XP operating system. Figure 6.3 illustrates the distribution of PRI, VoI, GCE and BDE performance measures over the 300 images in Berkeley dataset for the proposed third approach. The average PRI, GCE, VoI and BDE measures obtained by this third approach are , , and respectively. The maximum and minimum PRI obtained by this approach are 0.98 and 0.43 respectively Comparison of the Three Proposed Algorithms with State-of- Art Algorithms The measure PRI seems to be more highly correlated with human hand segmentations. Therefore realistic good segmentations give high score. For every segmentation (proposed and state of art ) method the PRI value is calculated for all images in the dataset. Their average and standard deviation are calculated for each method. The values obtained are presented in Table 6.1. PR Index mean gives the closeness factor of the segmentation results with the ground truth. If this factor is high then that particular segmentation is considered as good as human segmentations. PR Index standard_deviation provides the dependency factor of the segmentation with the input image. If the dependency factor of particular segmentation is more then that algorithm will behave differently for different kinds of images. Table 6.1 Performance Evaluation of the three proposed algorithms and state of art algorithms PR Index mean PR Index standard_deviation RB-Wavelet Pixon-ASFMM Q-JSEG JSEG CTM (gamma=0.2) FCR [ K1 13 K ] CTex

11 155 It can be observed from Table 6.1 that the proposed algorithms Pixon-ASFMM and Q-JSEG provide highest PRI (0.81). Therefore these algorithms outperform the recent state of art algorithms. Since PR Index standard_deviation attained by Pixon-ASFMM is lesser than Q-JSEG the former is more general than the later algorithm. Among all the segmentation algorithms presented in Table 6.1, FCR obtains minimum PR Index standard_deviation. Therefore FCR behaves similarly with all kinds of images. Table 6.2 Average performance of the three proposed algorithms, FCR and CTM PRI GCE VoI BDE RB-Wavelet Pixon-ASFMM Q-JSEG FCR FCR [ K1 13 K ] [ K1 13 K ] CTM (gamma=0.1) CTM (gamma=0.2) Table 6.2 presents the average performance of the proposed algorithms and state of art methods FCR and CTM. In addition to PRI this table shows the GCE, VoI and BDE error values. The BDE errors obtained by the proposed algorithms are lesser than the state of art algorithms. The minimum BDE error (7.9533) was obtained by Pixon-ASFMM. Among the proposed algorithms Q-JSEG obtains lesser VoI and GCE measures. Although the performance obtained by the proposed first system is not the best, it is competent with the other methods. GCE index obtained by this method is almost similar to the other proposed methods. Boundary error

12 156 is reduced comparatively with the state of art methods. Moreover the important properties of the first approach are its simplicity and completeness. It possess all the characteristics of a good segmentation. The second approach performs better in terms of BDE indices and it achieves equivalently good PRI measure as Q-JSEG. VoI and GCE values obtained by this approach are also competent with the other approaches. The proposed third approach improves the PR Index to 0.81 and also achieves optimal VoI, GCE and BDE indices. 6.3 EXPERIMENTS PERFORMED ON SYNTHETIC DATASET The synthetic database was generated by Ilea and Whelan (2008) inorder to quantify the segmentation accuracy of JSEG and their algorithm CTex. This dataset consists of 33 mosaic images of size 184x184. They were created by mixing textures from VisTex (Vision Texture) and photoshop databases. The mosaic images used in the experiment consists of various texture arrangements and the borders between different regions are irregular. The suite of 33 mosaic images is depicted in chapter 1 (Figure 1.4). These images are numbered from 01 to 33 starting from the upper left image in a raster scan manner. The proposed algorithms were applied on all the 33 images. The pixels situated on the border of the regions present in the segmented results are identified manually.then the Euclidean distance between border pixels present in the segmented results and those in the groundtruth data was computed. The segmentation errors are evaluated by the statistical measures such as mean, standard deviation and r.m.s errors. Table 6.3, Table 6.4 and Table 6.5 depict the average distance measure, standard deviation error and r.m.s error obtained by each synthetic image respectively.

13 157 Table 6.3 Mean distance between the border pixels of ground truth data and segmented results of proposed and state of art algorithms Image RB-Wavelet Pixon-ASFMM Q-JSEG JSEG CTex overall

14 158 Table 6.4 Standard deviation of distance between the border pixels of ground truth data and segmented results of proposed and state of art algorithms Image RB-Wavelet Pixon-ASFMM Q-JSEG JSEG CTex Overall

15 159 Table 6.5 R.M.S.error computed between the coordinates of the border pixels obtained from the ground truth data and segmentation results of proposed (RB-Wavelet, Pixon-ASFMM,Q-JSEG) and state of art algorithms (JSEG and CTex). Image RB-Wavelet Pixon-ASFMM Q-JSEG JSEG CTex overall

16 Discussion The experimental data depicted in Tables 6.3, 6.4 and 6.5 indicate that the overall mean errors calculated over the 33 images for the proposed algorithms are smaller than the overall mean errors calculated for CTex and JSEG. This shows the completeness of the proposed algorithms. All the approaches evaluate the color and texture information using explicit models and appropriately intergrate the color and texture features. The criterions are to assess the optimal number of components in the image. Although the proposed algorithms technically differ they adaptively adjust the size of the local color and texture distributions. Due to these reasons the proposed algorithms more or less give similar results in terms of mean, standard deviation and root mean square errors. The mean error is greatly reduced in the Pixon-ASFMM than the other proposed and state of art algorithms concerned. This shows the efficiency of second approach in segmenting complex color images, which is due to the effective representation of color, texture and their integration. 6.4 CONCLUSION Three new color texture segmentation algorithms namely RB- Wavelet and SOM, Pixon-ASFMM and Q-JSEG were proposed in this thesis. These algorithms were qualitatively analyzed in the previous chapters. A comprehensive quantitative analysis was performed in this chapter. This analysis consists of two kinds of tests. First test was carried out on natural images and the results were quantitatively analyzed using the measures PRI, VoI, GCE and BDE. The quantitative results proved that all the proposed algorithms are efficient and competent with the state of art algorithms especially Pixon-ASFMM greatly reduces the BDE error and Q-JSEG improves the PRI than the state of art algorithms.

17 161 Second test was performed on synthetic images generated from VisTex database. The segmentation accuracy of the proposed algorithms were evaluated by calculating the Euclidean distances between the pixels situated on the boundary of the regions present in the segmented results of proposed algorithms and the boundary pixels present in the ground truth data. The mean, standard deviation and r.m.s errors that measure the boundary displacement between the ground truth and the segmented results were calculated to statistically measure the segmentation errors. Minimum overall mean error was obtained by PIXON-ASFMM. RB-wavelet and SOM method gives minimum overall standard deviation and r.m.s errors. Moreover the overall errors (mean, std and r.m.s) obtained by all the three proposed algorithms are smaller than the errors generated by state of art algorithms JSEG and CTex.

A Multi-Objective Approach Based on TOPSIS to Solve the Image Segmentation Combination Problem

A Multi-Objective Approach Based on TOPSIS to Solve the Image Segmentation Combination Problem to Solve the Image Segmentation Combination Lazhar Khelifi, Max Mignotte Department of Computer Science and Operational Research (DIRO) University of Montreal 08 December 2016 1/40 Lazhar Khelifi, Max