Texture image retrieval and image segmentation using composite sub-band gradient vectors q

Transcription

1 J. Vis. Commun. Image R. 17 (2006) Texture image retrieval and image segmentation using composite sub-band gradient vectors q P.W. Huang a, *, S.K. Dai a, P.L. Lin b a Department of Computer Science, National Chung-Hsing University, Taichung 40227, Taiwan b Department of Computer Science and Information Management, Providence University, Shalu, Taiwan Received 15 March 2002; accepted 11 August 2005 Available online 19 June 2006 Abstract A new texture descriptor, called CSG vector, is proposed for image retrieval and image segmentation in this paper. The descriptor can be generated by composing the gradient vectors obtained from the sub-images through a wavelet decomposition of a texture image. By exercising a database containing 2400 images which were cropped from a set of 150 types of textures selected from the Brodatz Album, we demonstrated that 93% efficacy can be achieved in image retrieval. Moreover, using CSG vectors as the texture descriptor for image segmentation can generate very successful results for both synthesized and natural scene images. Ó 2006 Elsevier Inc. All rights reserved. Keywords: Texture descriptor; CSG vector; Image retrieval; Image segmentation; Wavelet decomposition 1. Introduction Texture is one of the most important attributes used in image analysis and pattern recognition. It provides surface characteristics for the analysis of many types of images including natural scenes, remotely sensed data, and biomedical modalities and plays an important role in the human visual system for recognition and interpretation. Although there is no formal definition for texture, intuitively this descriptor provides measures of properties such as smoothness, coarseness, and regularity. A common weakness of most of texture analysis schemes is that they analyze the image at one single scale. Beck et al. [1] found that visual cortex can be modeled as a set of independent channels, each with a particular orientation and frequency tuning. The multichannel processing is a strong motivation for multi scale texture analysis methods. Several multichannel texture analysis systems have been proposed [2 4]. In the last decade, wavelet theory has emerged and received the attention of the image processing society. Daubechies [5,6] provided the discretization method of wavelet transform. Mallat [7] established the q This research work was sponsored by National Science Council of ROC under contract No. NSC E * Corresponding author. Fax: addresses: powhei.huang@msa.hinet.net, huang@ amath.nchu.edu.tw (P.W. Huang), lan@pu.edu.tw (P.L. Lin) /$ - see front matter Ó 2006 Elsevier Inc. All rights reserved. doi: /j.jvcir

2 948 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) connection between wavelet transform and multi resolution theory. Since then, wavelet theory has become a mathematical framework which provides a formal and unified approach to multi scale (multi resolution) image analysis. The main reason of using multi scale methods for texture image analysis is based on the theory that big patterns can be better captured by lower resolution processes while small patterns can be better captured by higher resolution processes. However, big patterns are usually treated as objects rather than texture elements. Thus, to avoid such an ambiguity, we carefully define a texture image to be dealt with in this paper as the one having homogeneous texture elements which are small enough to be viewed as an integral texture surface rather than individual objects of the same pattern in regular placement. Based on the above assumption, we propose a new texture descriptor called Composite Sub-band Gradient Vector (or CSG vector) for the texture images containing homogeneous small texels. A CSG vector is formed by composing the gradient vectors generated from the sub-images of a wavelet transform of an image. We applied this new texture descriptor to image retrieval in image database systems and image segmentation to demonstrate its effectiveness. In the experiment of image segmentation, the CSG vector was used as a discriminatory feature to segment both synthesized and natural scene images. The results from segmenting both types of images were very successful. In image retrieval, the CSG vector was used as a texture descriptor to extract images that are similar in texture to the query image from the database. We established an image database containing 2400 texture images of size pixels which were cropped without overlapping from a set of 150 texture images of size pixels selected from the Brodatz Album [8]. Two images are regarded as similar only if they are cropped from the same original image. We adopt this rigorous criterion to prevent any possible influence due to subjective factors. Our experimental results showed that 93% efficacy can be achieved in image retrieval. The paper is organized as follows. In Section 2, we briefly review the theory of wavelet decomposition from which a CSG vector is derived. A new texture descriptor, the CSG vector, is introduced in Section 3. The results of image retrieval and image segmentation by using CSG vectors as the texture descriptor are presented in Sections 4 and 5, respectively. In Section 6 we summarize the results of the study and draw conclusions. 2. Wavelet decomposition The pyramidal wavelet transform uses a family of wavelet functions and the associated scaling functions to decompose the original signal into different sub-bands. The decomposition process is recursively applied to the low-frequency sub-band to generate the next level of the hierarchy. The wavelet and scaling filters are applied in both the horizontal and vertical directions, followed by a 2 1 sub-sampling of each output image. This generates three orientation selective detail images D j,k and a coarse or approximate image C j, where k = 1,2,3 and j represents the level of decomposition. The next level of resolution in the hierarchy is produced by repeating the same process. Thus, the hierarchical wavelet decomposition of an image can be described by the following equations: C j ¼½H x ½H y C j 1 Š #2;1 Š #1;2 ; D j;1 ¼½H x ½G y C j 1 Š #2;1 Š #1;2 ; D j;2 ¼½G x ½H y C j 1 Š #2;1 Š #1;2 ; D j;3 ¼½G x ½G y C j 1 Š #2;1 Š #1;2 ; where * denotes the convolution operator, # 2,1 denotes down-sampling at every other pixel along the x-direction, # 1,2 denotes down-sampling at every other pixel along the y-direction, and C 0 = I is the original image. H x and G x are a low and high-pass filter, respectively, along the x-direction. H y and G y are a low and high-pass filter, respectively, along the y-direction. The original image is thus represented by a set of sub-images at several scales {C J, D j,k } j=1,..., J; k = 1,2,3 which is a multi scale representation of depth J of the image I. In our prototype system, we choose DAUB4 as the wavelet basis because it has the best average performance [9].

3 3. Composite sub-band gradient vectors Features derived from gradient direction images can be used for texture analysis [10 12]. Gradient direction images generated by a gradient operator reflect the magnitude and direction of maximal gray-level change at each pixel of an input image. Such information provides important cues for human visual system. A number of gradient operators such as the popular Sobel operator [12,13] can be used for generating gradient direction images. Assume that there are 360 directions (0,1,..., 359 ). By summing up the magnitude value in the same direction at each pixel, a histogram of gradient directions with 360 bins is compiled. Such a histogram can be represented by a vector, called gradient vector, which allows us to analyze the texture of an image in terms of its edginess information. To reduce the length of a gradient vector and possible sensitivity due to a small change in imageõs orientation, every successive k directions can be grouped together to form one bin. Therefore, the total number of bins in a histogram of gradient directions will be 360/k. The length of a gradient vector is also 360/k. To measure the difference between two gradient vectors, methods such as Euclidean distance or weighted Euclidean distance can be easily applied. As discussed in Section 2, the wavelet decomposition can be implemented by using two channel filter banks composed of a low-pass (H) and a high-pass (G) filter and each filter bank is then sampled at the half rate of the previous frequency. As a consequence, the original image can be decomposed into four sub-images, namely: LL sub-image: both horizontal and vertical directions have low-frequencies. LH sub-image: the horizontal direction has low-frequencies and the vertical one has high-frequencies. HL sub-image: the horizontal direction has high-frequencies and the vertical one has low-frequencies. HH sub-image: both horizontal and vertical directions have high-frequencies. Thus, by applying a wavelet decomposition on an original image to obtain the LL, LH, HL, and HH subimages, we can construct a gradient vector for each sub-image. Let SGV 1, SGV 2, SGV 3, and SGV 4 be the gradient vectors associated with sub-images LL, LH, HL, and HH, respectively. Let CSGV = SGV 1 ksgv 2 k SGV 3 ksgv 4, where k is the append operator. Then CSGV is a texture descriptor that is more powerful than the gradient vector directly generated from the original image. For example, the textures in Figs. 1A and B are perceptually similar while the texture in Fig. 1C is different. The histograms of gradient directions for the texture images (panels A C) are shown in (panels D F), respectively. The Euclidean distance between panels (D and E) is while the Euclidean distance between panels (D and F) is only Thus using gradient vectors as the texture descriptor will result in a misjudgment in this example. On the other hand, we generated the histograms of gradient directions for the sub-band images associated with the same three images to obtain the CSG vectors as shown in Figs. 1G I. The Euclidean distance between panels (G and H) is while the Euclidean distance between panels (G and I) is This result is correct and consistent with human visual system. The above example shows that CSG vectors are more powerful than gradient vectors in discriminating textures. 4. Image retrieval using CSG vectors P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) In image retrieval, we used CSG vectors as the texture descriptor to discriminate images and extract from the database the images that seem similar in texture to the query image. The performance of image retrieval can be measured by a formula proposed by Kankanhalli et al. [14]: n=n if N 6 T g T ¼ n=t if N > T ; where n is the number of similar images retrieved, N is the total number of similar images in the database, and g T is the efficacy of retrieval for a given short-list of size T. IfN 6 T, then g T reduces to the traditional recall measure of information retrieval. If N > T, then g T computes the precision measure of information retrieval.

4 950 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) Fig. 1. Three texture images and their corresponding histograms of gradient directions as well as CSG vectors with k = 10. To allow us to appreciate some of the results of image retrieval by using CSG vectors, we show four instances of retrieval results with T = 5 in Fig. 2, where the retrieved images are ranked by their Euclidean distances to their respective query images. We selected 150 different types of textures from the Brodatz Album [8] and randomly cropped four subparts from each texture. Three subparts were stored in the database and one subpart was used as a query image for similarity retrieval. Thus there were 450 images in the database in which each type of texture has three different images. This example shows that the efficacy of similarity retrieval based on CSG vectors is very high. To evaluate the efficacy of image retrieval using CSG vectors, a large database was created and many queries were submitted in the experiment as follows. First of all, a set of 150 images of size pixels with different textures were selected from the Brodatz Album. Each texture image was then partitioned into 16 nonoverlapping images of pixels. Thus the image database in our prototype system contains 2400 texture images. We used every database image as a query image and evaluated the efficacy of image retrieval based on the average performance of these 2400 queries Retrieval results of using CSG vectors vs gradient vectors Table 1 lists the average image retrieval efficacy based on CSG vectors and gradient vectors for the five cases with k = 10, 20, 30, 40, and 60 by choosing T = 10. Notice that the number of bins (k) used in the histogram of gradient directions implies the length of both gradient vectors and CSG vectors. This table shows that the retrieval efficacy of using CSG vectors is always higher than that of using gradient vectors. For example, in the case of k = 10, the retrieval efficacy based on CSG vectors is 93.20% while the retrieval efficacy based on gradient vectors is 87.75%. In the case of k = 30, the retrieval efficacy based on CSG vec-

5 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) Fig. 2. Four instances of image retrieval results. The retrieved images are ranked by the Euclidean distances to their respective query images. Table 1 Retrieval efficacy with T = 10 based on CSG vectors and gradient vectors for various lengths of vectors k =10 k =20 k =30 k =40 k =60 Gradient vector 87.75% 86.31% 85.11% 84.01% 77.22% CSG vector 93.20% 92.99% 92.31% 88.94% 88.51% tors is 92.31% while the retrieval efficacy based on gradient vectors is only 85.11%. Furthermore, if we choose k = 60 for generating CSG vectors and k = 10 for generating gradient vectors, then the length of a CSG vector is 4 360/60 = 24 and the length of a gradient vector is 360/10 = 36. As a consequence, the length of a CSG vector is much shorter than that of a gradient vector (24 vs 36); however, the discriminating capability of CSG vectors is more powerful than that of gradient vectors (88.51% efficacy vs 87.75% efficacy).

6 952 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) Image retrieval by reduced-length CSG vectors The length of a CSG vector can be reduced while still preserving the capability of discriminating textures. Notice that the gradient vectors generated from the LL, LH, HL, and HH sub-images through a wavelet decomposition are not equally informative for texture discrimination. Different combinations of the gradient vectors of the sub-images may have different discriminating capabilities. Table 2 shows the retrieval results based on possible combinations of the gradient vectors extracted from the four sub-images. First of all, we may notice that LL is most informative among the four sub-images. The retrieval efficacy purely based on the gradient vectors extracted from the LL sub-image was 87.39%. The retrieval efficacy based on the gradient vectors extracted from the LH, HL, and HH subimages were 73.91, 72.99, and 73.76%, respectively. To increase the retrieval efficacy, we may compose two or more gradient vectors extracted from different sub-images to form a CSG vector. Since the LL part is most informative, it must be included as the key component of a CSG vector. Among the six cases of combining two sub-band gradient vectors, LL + LH (91.44% efficacy) and LL + HL (91.47% efficacy) have the best performance, LL + HH (88.36% efficacy) and LH + HL (87.80% efficacy) are the second best, LH + HH (77.14% efficacy) and HL + HH (75.84% efficacy) are the worst. The reason why LH + HH and HL + HH have the worst performance is because they consider low-frequencies only in one single direction. If we compare the cases of combining three sub-band gradient vectors, then LL + LH + HL is the best, LL + LH + HH and LL + HL + HH are the second best, and LH + HL + HH is the worst. The retrieval efficacy was promoted to 93.08% for the combination of LL + LH + HL. If the gradient vectors of the four sub-band images are all included in the CSG vector, an average retrieval efficacy of 93.20% can be achieved. Overall speaking, the experimental results suggest that using LL + LH + HL combination to build a CSG vector is probably the best in terms of retrieval efficacy and computational efficiency Comparisons with other methods In this subsection, we would like to compare the result of using CSG vectors with those of using two other closely related texture descriptors, the wavelet energy signature [15] and the gradient vector, in terms of the efficacy of image retrieval. The results of comparison among these three methods are shown in Table 3, where T represents the size of a short list provided by the user. In our experiment, there are 16 relevant images in the database with respect to each query (i.e., N =16in the formula of measuring the retrieval efficacy g T ). In Table 3, our approach demonstrates its superiority over Table 2 Retrieval efficacy based on possible combinations of the gradient vectors obtained from wavelet decomposition Sub-image Efficacy (%) LL LH HL HH LL + LH LL + HH LH + HL LH + HH LL + HL HL + HH LL + LH + HL LL + LH + HH LL + HL + HH LH + HL + HH LL + LH + HL + HH 93.20

7 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) Table 3 Comparison of CSG vector with wavelet energy signature (WES) and gradient vector in terms of retrieval efficacy T WES (%) Gradient vector (%) CSG vector (%) the other two methods for T = 5, 10, 15, 20, 30, 40, and 50. The first three rows, where N > T, show that the precision measure of our method is much higher than those of the other two methods. The last four rows, where N < T, show that the recall measure of our method is also much higher than those of the other two methods. 5. Segmentation using reduced-length CSG vectors Our segmentation process is divided into three stages: split, merge, and boundary refinement. In the splitstage, we will divide an image into blocks with each block having a homogeneous texture. Assume that an image I is of size m m pixels. We can imagine that there are four sub-blocks in this image, namely, I NW (the northwest sub-block), I NE (the northeast sub-block), I SW (the southwest sub-block), and I SE (the southeast sub-block). Thus each sub-block is of size m/2 m/2 pixels. Let CSGV(I) be the CSG vector extracted from image I. Let d be the Euclidean distance between the CSG vectors associated with two blocks. Furthermore, we define a block of image as having a homogeneous texture if the Euclidean distance between every two sub-blocks is less than a threshold value g (we chose g = 0.1 in our experiment). If a block of image is determined as non-homogeneous, it will be divided into four sub-blocks. The splitting process repeats for these four sub-blocks recursively until each sub-block has a homogeneous texture or becomes a block of size pixels. According to our experience, a block of image smaller than pixels can not give us enough information for determining a texture. The merge-stage will merge neighboring blocks with the same texture. The merge process starts with the blocks of size pixels. Two neighboring blocks will be merged together if the distance between them is less than a threshold f. Usually the threshold for merging is greater than that of splitting. In our experiment, we set f to 0.7 to achieve a good result. The boundary refinement stage will smooth the edges of the segmented areas. First of all, the image obtained from the merge-stage is divided into blocks of size again. We define a block as an edge block if at least one of its four neighboring blocks is labeled differently. Assume that the four neighboring blocks of an edge block are B 1, B 2, B 3, and B 4. For each pixel c in an edge block, we can find a virtual block B of size with c as its center. Then we calculate the distance between B and B i for 1 6 i 6 4. If B k has the minimum distance from B, then pixel c is re-labeled to the same as the class number of B k. Consequently, an edge block can be dissolved by merging its pixels with one of its neighboring blocks. Fig. 3 shows the results of segmentation for two synthesized images. The images in the middle row are the intermediate results after the merge-stage. The images in the third row of the same figure are the final results after the boundary refinement stage. Fig. 4 shows the results of segmentation for another two synthesized images. Segmentation for natural scene images has been an challenging problem. However, our approach of using CSG vectors as the texture descriptor for segmenting natural scene images is quite successful from the perspective of human visual perception. For example, Fig. 5 shows the segmentation results for two natural scene images. The image on the left side was segmented into regions containing mountain, sky, trees, creek, and flowers. The image on the right side was segmented into regions containing sky, hill, roof, windows, door, chimney, trees, barn, ground, and grass. As we can see that these results are very consistent with what we expected if segmentation is performed manually.

8 954 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) Fig. 3. The segmentation results of two synthesized images. 6. Summary and conclusions Image retrieval from image databases has becoming pervasive in recent years due to the advent of large storage media and network communication techniques. Image segmentation has been a fundamental and difficult problem for image processing systems. Having an appropriate feature descriptor is the key to achieve successful results in both image retrieval and image segmentation.

9 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) Fig. 4. The segmentation results of two other synthesized images. In this paper, we proposed a new texture descriptor, called CSG vector, which can be generated by composing the gradient vectors obtained from the LL, LH, and HL sub-images through a wavelet decomposition of an original image. The CSG vector can be used as a powerful texture descriptor for image retrieval and image segmentation. Our experimental results show that the capability of discriminating textures (or identifying similar textures) of using CSG vectors is better than those of using gradient vectors and wavelet energy

10 956 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) Fig. 5. The segmentation results of two natural scene images. signatures in terms of the efficacy of image retrieval. Overall speaking, our approach can achieve 93.20% efficacy in texture image retrieval. Furthermore, texture image segmentation based on CSG vectors can generate very good results for both synthesized and natural scene images. References [1] J. Beck et al., Spatial frequency channels and perceptual grouping in texture segregation, Comput. Vision Graph. Image Process. 37 (1987) [2] A.C. Bovik, Analysis of multichannel narrow band filters for image texture segmentation, IEEE Trans. Signal Process. 39 (1991) [3] A.K. Jain, Learning texture discrimination masks, IEEE Trans. Pattern Anal. Machine Intell. 18 (1996) [4] M. Unser, M. Eden, Multiresolution feature extraction and selection for texture segmentation, IEEE Trans. Pattern Anal. Machine Intell. 11 (1989) [5] I. Daubechies, Orthonormal bases of compactly supported wavelets, Commun. Pure Appl. Math. 41 (1988) [6] I. Daubechies, The wavelet transform, time-frequency localization and signal analysis, IEEE Trans. Inform. Theory 36 (1990) [7] G.M. Stephane, A theory for multiresolution signal decomposition: the wavelet representation, IEEE Trans. Pattern Anal. Machine Intell. 11 (1989) [8] P. Brodatz, Textures: A Photographic Album for Artists and Designers, Dover, New York, [9] V. Manian, R. Vasquez, Scaled and rotated texture classification using a class of basis functions, Pattern Recogn. 31 (1998) [10] S.R. Fountain, T.N. Tan, Efficient rotation invariant texture features for content-based image retrieval, Pattern Recogn. 31 (1998)

11 P.W. Huang et al. / J. Vis. Commun. Image R. 17 (2006) [11] M. Gorkani, R. Picard, Texture orientation for softing photos at a glance, Proc. IEEE Conf. Pattern Recogn. (1994) [12] R.M. Haralick, L.G. Shapiro, Computer and Robot Vision, vol. I, Addison-Wesley, Reading, MA, [13] D. Ballard, C. Brown, Computer Vision, Prentice Hall, Englewood Cliffs, [14] M. Kankanhalli et al., Cluster-based color matching for image retrieval, Pattern Recogn. 29 (1996) [15] G. Van de Wouwer et al., Wavelet correlation signatures for color texture characterization, Pattern Recogn. 32 (1999) P.W. HUANG received the BS degree in applied mathematics from National Chung-Hsing University in 1973, the MS degree in mathematics from Texas Tech University in 1978, and the Ph.D. degree in Computer Science and Engineering from Southern Methodist University in He was with Texas Instruments in Dallas as a member of technical staff, supervisor, lead engineer, section manager, and software development manager from 1978 to His working experiences include bubble memory testing and application, IC design automation, Lisp machine on a chip design, automatic program generation, production planning and scheduling, and wafer factory automation. He was the department head and has been a professor in the Computer Science Department at National Chung-Hsing University. Since September of 2002, he also has been serving as the vice president of National Huwei University of Science and Technology located in Yunlin County of Taiwan. His current research interests are in the fields of multimedia database, medical imaging, visual inspection, pattern recognition, and artificial intelligence. S.K. DAI received his M.S. and Ph.D. degrees both in Applied Mathematics from National Chung-Hsing University in 1999 and 2003, respectively. He is also an engineer of Chunghwa Telecom Co. at Taichung, Taiwan. His research interests are in computer graphics, image processing, and pattern recognition. P.L. LIN received her B.S. degree in engineering science from National Cheng-Kung University in 1973, M.S. in mathematics from Texas Tech University in 1978, M.S. and Ph.D. in electrical engineering from Southern Methodist University in 1992 and 1994, respectively. She was with the Corporate Manufacturing Technology Center of Texas Instrument as a senior engineer and a project manager from 1979 to Since 1994, she has been an associate professor in the Department of Computer Science and Information Management at Providence University. Now she is a professor and the director of the Computer and Communication Center at Providence University. She holds two U.S. patents in mobile robot guiding systems. Her research interests include network security, cryptography, image processing, and factory automation. She is a senior member of SME.