APPLYING TEXTURE AND COLOR FEATURES TO NATURAL IMAGE RETRIEVAL

Transcription

1 APPLYING TEXTURE AND COLOR FEATURES TO NATURAL IMAGE RETRIEVAL Mari Partio, Esin Guldogan, Olcay Guldogan, and Moncef Gabbouj Institute of Signal Processing, Tampere University of Technology, P.O.BOX 553, FIN Tampere, ABSTRACT This paper evaluates the performance of texture and color features applied to natural image retrieval. Natural images usually contain both texture and color, and therefore those features should also be considered together in a retrieval process. Combination of texture and color features is important, since it may result in closer connection to semantic. The idea in this study is to empirically evaluate the effect of texture and color features to the query. Retrieval capability of those features is first evaluated separately and finally the combined effect is studied. Finally, the goal is to find out what kind of features, or combination of features, yield the most successful results for certain types of query images. 1. INTRODUCTION Due to rapid increase in the size of digital image and video collections, efficient browsing, searching, and retrieval methods are gaining more importance. Existing indexing and retrieval systems, such as MUVIS [1], QBIC [7], Netra [8], and Photobook [9] provide framework and techniques for indexing and retrieving visual information. Retrieval systems in general are quite efficient in color retrieval; however, the results using only color features are not necessarily semantically meaningful. For example, querying with image containing a view of a blue lake might result in retrieving image with a lady wearing a blue shirt. Several attempts have been made to achieve query results closer to semantic. The most common approach is to apply relevance feedback [11, 12], which improves the query performance by adjusting iteratively the original query based on the relevant and irrelevant image examples designated by the user. CBIR system ifind [12] provides a framework, which combines keywordbased semantics and low-level feature based relevance feedbacks to achieve higher retrieval accuracy using fewer number of feedback iterations required from the user. However, still the problem with relevance feedback approach is, that several iterations and user interaction are needed. Therefore, efficient feature combinations for semantically meaningful queries should be studied. The idea in this paper is to empirically evaluate the effect of texture and color features for natural image retrieval. Query performance of those features is first evaluated separately and finally the combined effect is studied. Chapter 2 introduces the MUVIS system, which is used in the experiments. Chapter 3 describes the used texture and color features. Experimental results and their evaluation are provided in Chapter 4. Conclusions are drawn in Chapter THE MUVIS SYSTEM MUVIS [1] is a multimedia video indexing and retrieval system, which is developed at Tampere University of Technology. It is capable of indexing and retrieving various media types, such as images, video, and audio. Feature extraction is performed mainly offline while indexing, but features for the query image are extracted online at the retrieval stage. MUVIS basically consists of four applications: AVDatabase, IDatabase, DbsEditor, and MBrowser. AVDatabase is a real-time video/audio indexing application and the main video database creator. It has two modes: Create a database or Append into an existing database. Similarly, IDatabase application is developed to handle all image indexing capabilities. Once the database is created, several features can be added or removed using DbsEditor. It can also be used to merging two databases. The main media retrieval and browser terminal is called MBrowser. It has a built-in search and query engine, which is capable of finding media primitives in any database and for any media type that is similar to the queried media source. Retrieval results for the queried source are produced by comparing its feature vector with the feature vectors of the media primitives available in the database. Euclidean distance is used as a similarity measure between two feature vectors and minimum Euclidean distance yields the best similarity. MBrowser application is used as a terminal for all retrieval experiments. Sample retrieval setups in MBrowser are shown in Figure 1 and Figure 3. When performing the query, user can select which combination of the available features to use. For advanced queries also the different weights for the features may be applied. Adjustability of the features is very important, since the most suitable features for each query are application dependent.

2 3. TEXTURE AND COLOR FEATURES 3.1. Color features Colors can be described using different color spaces, such as RGB, HSV, and YUV. RGB is hardwareoriented, and therefore is very common in image processing research. However, HSV and YUV color models correspond better to human visual perception [2]. Also according to [4] HSV and YUV produce perceptually better results than RGB. Therefore, the experimental results shown in this paper are produced using both HSV and YUV color spaces. Color histograms are the most commonly used in color retrieval. The histogram reflects the statistical distribution of the intensities of the three color channels. The color histogram is computed by discretizing the colors within the image into bins and counting the number of pixels falling into each bin. Similarity between histograms can be evaluated by computing the L 1, L 2 or other distances [5, 10] Texture features Textural features can be roughly divided into two categories: spatial and frequency-based methods. Spatial methods such as co-occurrence matrices [13] operate on actual pixel values, while in frequency-based methods an image is first transformed into the frequency domain and some features are then extracted. In order to estimate the similarity between different gray level co-occurrence matrices, Haralick [13] proposed 14 statistical features extracted from them. To reduce the computational complexity, only some of the features should be selected for the experiments. Here we apply the 4 most relevant and widely used features, entropy, energy, contrast, and inverse difference moment. Detailed description of these features is given in [15]. As indicated by the psychological results [6], the human visual system analyzes the textural images by decomposing the image into a number of filtered images, each of which containing intensity variations over a narrow range of frequencies and orientations. Therefore, the multi-channel filtering approach is intuitively appealing by allowing us to exploit differences in dominant sizes and orientations in texture. More detailed description of using Gabor features in image retrieval is given in [3]. Since cooccurrence matrices and Gabor wavelet features concentrate on different aspects of texture, it is useful to apply them simultaneously to natural image retrieval. 4. EXPERIMENTAL RESULTS 4.1. Description of the test database The database used in the experiments consists of 996 unconstrained natural color images with various sizes. Images of the database can be roughly divided into the following categories according to their contents: buildings, faces, gymnastics, outdoor, teamsports, and miscellaneous images. Each of the main categories contains around 100 images and the rest are miscellaneous. Example query images are selected from different categories. The experiments will later be extended to include colored texture databases, where more improvement in the retrieval performance is expected Experimental setup 10 example images from each of the main categories are taken as query images and the best matches for them are retrieved using only color, only texture or combination of those features. The color feature vector used in the experiments consists of 10 subfeatures: 5 of them are HSV histograms with different number of bins for hue, saturation, and value. The other 5 are the corresponding YUV histograms for luminance and chrominance components. The texture feature vector for each image is obtained from Gabor wavelet function using 4 orientations and 3 scales and from cooccurrence matrices using 4 different distances. Weights for each feature are adjusted to be uniform, so each of the features has equal effect on the query. MUVIS MBrowser application was used as the terminal for the retrieval experiments. A sample retrieval setup is shown in Figure 1. A group of 9 people evaluated the retrieval performances for each query as follows [14]. The image in the first rank is not evaluated at all, since it is the same with the queried image. Each person in the evaluation group gave a subjective grade for each of the 11 retrieved images. The given grades G i are integer numbers within the range [0-5], which have corresponding subjective meanings as presented in Table 1. Grade Subjective Meaning 0 Not related 1 Slightly related 2 Related 3 Similar 4 Fairly similar 5 Almost identical Table 1 The evaluation grades Each of the 11 graded images has an associated weight W i = (13 i) related to its rank i. The Performance Value (P) is calculated using the following formula: 12 P = G i W i= 2 Grades for sample retrieval setup shown in Figure 1 are presented in Table 2. The P value in this case is calculated using grades as follows: P = (5*11) + (3*10) + (4*9) + (0*8) + (3*7) + (2*6) +(1*5) + (1*4) + (1*3) + (1*2) + (3*1) = 171. i (1)

3 Figure 1 A sample retrieval setup and associated rankings [14] Rank Grade Table 2 Grades given by a person for the sample retrieval setup in Figure 1 Average Performance Value (AP) for each category is first evaluated for each person in the evaluation group is AP = (2) N where N is the number of considered query images from the class in question. Query Performance Value (QP) for each category is the average of AP values designated by each person in the evaluation group for that particular class. QP values for each class are shown in Figure 2. N j = 1 QP = (3) M M is the number people in the evaluation group. Theoretical maximum value for P, AP, and QP is Evaluation of the results Query example is shown in Figure 3. The query image is in the upper left corner and the best matches are listed row-wise to the right of it. The query (a) is M j= 1 P j AP performed using color features. The query (b) utilizes only texture features and the query (c) uses both color and texture features. Combined query produced semantically more meaningful results than the one using only color feature, as can be seen from the Figure 3. It also shows that when using the features simultaneously some of the false matches appearing in the query (a), can be removed from the closest best matches. Figure 2 shows the overall retrieval results for color and texture features in all of the categories. The combined query produces the best results for almost all of the categories. Especially clearly the improvement can be observed in gymnastics and teamsports categories. For face images the retrieval accuracy increases significantly when using texture features. The only category that seems to suffer slightly from the use of texture features is the building category. Mean value for color QP value in all the categories is 152, for texture 128 and for both of them 163. Using texture and color features simultaneously achieve the best overall retrieval performance for natural images.

4 5. CONCLUSIONS Retrieval performance for most of the categories improves when color and texture features are employed for retrieval simultaneously. Additionally, most of the semantically irrelevant images that are retrieved using single feature disappear from the list of best matches. Therefore, the retrieved images are generally closer to semantic. The use of color features generally results in more accurate retrieval results than the use of texture features. This might result from the fact that natural images do not generally contain certain (or dominant) texture information, while color information is more precise. It may be expected that extracting texture features from segmented images would improve the retrieval results. Considering the complexity of texture feature extraction, it may be more efficient to use color features in limited applications. In certain categories, such as gymnastics and faces, more clear texture contents yield relatively higher performance using texture features. In the case of queries with unknown image categories, it is beneficial to perform the query using color and texture features simultaneously. Texture features mostly represent additional information about the general structure of the image. Consequently together with color features, texture features may improve the retrieval performance by removing semantically irrelevant images from the list of best matches. 6. REFERENCES [1] M. Gabbouj, S. Kiranyaz, K. Caglar, B. Cramariuc, F. Alaya Cheikh, O.Guldogan, and E. Karaoglu, MUVIS: A Multimedia Browsing, Indexing and Retrieval System, in Proc. of the IWDC 2002 Conference on Advanced Methods for Multimedia Signal Processing, Capri, Italy, Sept [2] R.C. Gonzalez, and R.E. Woods, Digital Image Processing, Second Edition, Prentice-Hall, 2002, p [3] B.S. Manjunath, and W.Y. Ma, Texture Features for Browsing and Retrieval of Image Data, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp , August [4] G. Paschos, Perceptually uniform color spaces for color texture analysis: an empirical analysis, IEEE Trans. on Image Processing, vol. 10, no. 6, pp , June [5] A. Del Bimbo, Visual Information Retrieval, Morgan Kaufmann Publishers, San Francisco, California, 1999, p [6] T.S. Lee, Image Representation Using 2D Gabor Wavelets, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 18, no. 10, pp , October [7] C. Faloutsos, W. Equitz, M. Flickner, W. Niblack, D. Petkovic, and R. Barber, Efficient and Effective Querying by Image Content, Journal of Intelligent Information Systems, vol.3, pp , [8] W.Y. Ma, and B.S. Manjunath, NeTra: A Toolbox for Navigating Large Image Databases, IEEE Int. Conf. On Image Processing, ICIP 97, Santa Barbara, October [9] A. Pentland, R.W. Picard, and S. Sclaroff, Photobook: Content-Based Manipulation of Image Databases, International Journal of Computer Vision, 18(3), pp , [10] Y. Rui, T.S. Huang, and S.-F. Chang, Image Retrieval: Current Techniques, Promising Directions and Open Issues, Journal of Visual Communication and Image Representation, vol. 10, no. 1, pp , March [11] J. Laaksonen, M. Koskela, S. Laakso, and E. Oja, PicSOM content-based image retrieval with selforganizing maps, Pattern Recognition Letters, 21(2000), pp , [12] Y. Lu, C. Hu, X. Zhu, H. Zhang, and Q. Yang, A Unified Framework for Semantics and Feature Based Relevance Feedback in Image Retrieval Systems, ACM Multimedia, pp , [13] R. M. Haralick, K. Shanmugam, and I. Dinstein, Textural Features for Image Classification, IEEE Trans. on Systems, Man, and Cybernetics, Vol. SMC-3, No.6, pp , November [14] E. Guldogan, and O. Guldogan, Compression Effects on Content-Based Multimedia Indexing and Retrieval Using Color and Texture Attributes, M. Sc. Thesis, Tampere University of Technology, January [15] M. Partio, B. Cramariuc, M. Gabbouj, and A. Visa, Rock Texture Retrieval using Gray Level Co-occurrence Matrix, NORSIG-2002, 5 th Nordic Signal Processing Symposium, On Board Hurtigruten M/S Trollfjord, Norway, October 4-7, 2002.

5 Overall Color and Texture Scores for Natural Images Color 150 QP Texture Color & Texture sp or ts r Te am oo ut d O es st ic G ym na Bu ild Fa c in gs 0 Image Classes Figure 2 Overall Color and Texture Scores for Natural Image (a) (b) Figure 3 Sample query setup using (a) only color, (b) only texture, (c) both color and texture (c)