SIEVE Search Images Effectively through Visual Elimination

Transcription

1 SIEVE Search Images Effectively through Visual Elimination Ying Liu, Dengsheng Zhang and Guojun Lu Gippsland School of Info Tech, Monash University, Churchill, Victoria, 3842 {dengsheng.zhang, Abstract. Existing Web image search engines index images by textual descriptions including filename, image caption, surrounding text, etc. However, the textual description available on the Web could be ambiguous or inaccurate in describing the actual image content and some images irrelevant to user s query are also returned by text-based search engines. In this paper, we propose to integrate the existing text-based image search engine with visual features, in order to improve the performance of pure text-based Web image search. The proposed algorithm is named SIEVE. Practical fusion methods are proposed to integrate SIEVE with contemporary text-based search engines. In our approach, text-based image search results for a given query are obtained first. Then, SIEVE is used to filter out those images which are semantically irrelevant to the query. Experimental results show that the image retrieval performance using SIEVE improves over Google image search significantly. Keywords: CBIR, RBIR, SIEVE, web image searching, decision tree. 1. Introduction Due to the explosive growth of the World Wide Web, nowadays mammoth amount of images are available on the Web. However, unlike text documents, images on the Web are not categorized properly according to their contents. This makes the Web a large unstructured image database. In order to make use of this vast database of Web images, it is desirable to have an effective Web image search engine that meets users need [1]. As Web images usually come with HTML source code including textual descriptions, existing Web image search engines such as Google, Yahoo! and AltaVista index images by textual descriptions including filename, image caption, surrounding text, etc [1], [2], [3], [4]. Google, for example, has indexed over 425 million images and allows users to search this index using textual queries. However, the textual description available on the Web can be ambiguous or inaccurate in describing the actual image content, irrelevant images are also returned by text-based search engines. For instance, Web image filenames may be misleading, surrounding text might not describe the content of an image, or a word may have multiple meanings resulting in irrelevant retrievals by matching the right word but with the wrong meaning. Research has been done to refine the search results returned by existing text-based image search engines and deliver the images which are more relevant to user queries. A hierarchical clustering technique using visual, textual and link information is proposed in [5]. This algorithm can organize the results returned by text-based image

2 search engines into different semantic categories to help users in quickly finding the desired images. Tested on top 500 images returned for a few queries by text-based image search, it is found that the re-organized image list better facilitates users in finding their desired images. This algorithm requires that the number of clusters in which images are to be categorized should be known beforehand, which is still an open research challenge. The co-ranking framework proposed in [6] aims to re-rank the Web images returned by text-based search engines, such that the images irrelevant to user query are moved to the tail of the list. The framework segregates the low-level image features into disjoint subsets (views). One-Class SVM (support vector machine) is used as the learning algorithm in each view and the output includes the ranking score for all the images considered. Then, the results from multiple views are combined to improve the final ranking of images according to their relevance to the user query. Experimental results on the top 10 images returned by Google image search shows that the framework improves the precision of Web image search. However, the scalability of the framework is questionable and further work is needed to analyse its re-ranking performance for more images returned by Google. In previous work, we have developed content-based techniques for image database retrieval, including salient low-level image feature extraction techniques and a methodology to derive high-level concepts from such low-level image features [7], [8], [9], [10], [11]. Our experimental results on natural scenery image database confirm the effectiveness of our methods on stand alone image database where no textual information is available. However, in order to apply it to Web image search, further work is necessary to combine visual image content with the textual description available on the Web. Existing text-based search engines, such as Google, are among the most widely used web tools and have done a good job in Web image search. It is more practical and advantageous to improve the retrieval performance by further refining their image search results than developing a completely new image search engine In this paper, we propose to integrate the existing text-based image search engine with the region based image retrieval (RBIR) technique we have developed (SIEVE), in order to improve the performance of text-based Web image search. Practical fusion methods are proposed to integrate SIEVE into contemporary text-based search engines. The rest of the paper is organized as following. In Section 2, the SIEVE algorithm is described in details. Section 3 shows experimental results and compares SIEVE with Google image search. In Section 4, we propose three methods of integrating SIEVE with existing text based image search engines and discuss the scalability issue. Finally, Section 5 concludes the paper. 2. The SIEVE Algorithm In this section, we present the details of our Web image search algorithm SIEVE, which integrates image semantic learning into Web image search. The idea of using SIEVE is very similar to object classification done by a human being. First, objects of interest are roughly distinguished from other very different objects either manually or through certain hand tools. Then, the collected objects are subject to visual inspection to confirm each object of interest from unwanted objects.

3 Fig. 1 shows the block diagrams of the proposed retrieval system and the SIEVE module. The input to the system is a keyword query submitted by the user, the system first comes out a ranked list of retrieved images given by the text based image search engine. The list of retrieved images by text based image search engine is then input to SIEVE for further analysis. For each image in the list, SIEVE first segments it into different regions. Then, color and texture features of each region are extracted using the methods presented in [7], [8] and [9], [10] respectively. The region color feature is the dominant color in HSV space and the region texture feature is the Gabor feature obtained using a novel padding algorithm [10]. In [11], we have developed a semantic template based decision tree reasoning algorithm and derived a set of decision rules to learn a set of concepts in natural scenery images. Using these decision rules, the lowlevel features of a region are mapped to semantic concepts. Fig. 1. Block diagrams of the retrieval system and the SIEVE module. Each query in our experiments corresponds to a high-level concept. It is assumed that each image has at least one dominant region representing its semantic meaning. Given a user query, for each image in the returned list, if any region it contains is relevant to the query concept, then the image is considered relevant to the query. If none of the region it contains is relevant to the query, the image is considered irrelevant to the query and is eliminated from the list. Refer to the human classification analogy, the retrieval list given by the text based image search engine is

4 the rough selection step. The SIEVE analysis is the visual inspection because it refines the list by using visual features which are also what human being perceives the images. 3. Experiments and Results Analysis This section presents the experimental results of SIEVE for Web image search and compares its performance with Google image search results. The process simulates the integration of the SIEVE module with the Web search engine. At this moment, the system works with natural scenery images because the underlying decision tree is trained using natural images. Therefore, the queries in our experiments are limited to this class of images currently. 3.1 Web Image Collection and Performance Evaluation To test the retrieval performance of SIEVE, 10 queries are selected, including mountain, beach, building, firework, flower, forest, snow, sunset, tiger and sea. Google image search can return up to thousands of images for a query, however, users are usually only interested in the first few pages and unlikely to take time to view the remaining pages. Therefore, for each query, the top 100 images are downloaded from the first 5 pages. For a given query, each image in the returned list is segmented into different regions using JSEG [7], and the regions with size over 5% of the entire image are selected. Then, low-level features of these regions are extracted. Next, the semantic based decision tree method is used to learn the concept of each region in an image and decide whether the image is relevant to the query or not. In Web image search scenario, it is difficult to know how many relevant images there are in the database for a given query. Hence, the conventional precision and recall measurement is not used, instead, the bull s eye measurement is used. The bull s eye measures the retrieval precision among the top K retrieved images. 3.2 Experimental Results We observed that among the top images returned by Google, most of them are relevant to the query. However, many images in the rest of the list are irrelevant due to semantic misunderstanding. Using SIEVE, the performance of Web image retrieval is significantly improved by filtering out many of these irrelevant images. Fig. 2 compares the average performance of Google image search and SIEVE for the 10 different queries we selected. Overall, the retrieval precision of SIEVE is significantly higher than that of Google. For instance, Fig. 3 and Fig. 4 list the top 25 and 20 images using query tiger and snow respectively. It is shown that the retrieval results of SIEVE are more relevant to the queries than those of Google. In the results given by Google image search, there are some irrelevant images whose surrounding text contains the keyword tiger and are hence selected as relevant images. For example, the seventh image in Fig. 3 is described as tiger pear

5 in tyre, and the description for the twelfth image for query snow in Fig. 4 is: Snow Game publishers are starting to push the envelope further and further SIEVE removes many of such images from the list, whose visual content are obviously irrelevant to the query Precision SIEVE Google K Fig. 2. Average Performance of SIEVE and Google Image Search for the 10 Queries It is observed that in some circumstances, SIEVE removes some relevant images from the returned list. There are mainly two reasons for this. The first comes from the limitation in the decision tree as the training samples do not cover every case in the image database. Fig. 5 shows a few such images. For query tiger, the eighth image (a black-and-white cartoon tiger as shown in Fig. 5(a)) in the list is removed by SIEVE, as its color is very different from the training samples. Another sample is the thirty-seventh image for query snow, as shown in Fig. 5(b), whose color is different from most of the snow images due to the dark light. Therefore it is also considered irrelevant by the decision tree. In some cases, the decision tree may classify irrelevant images as relevant. For instance, the eleventh image for query snow is a snow leopard, as shown in Fig. 5(c). Although this image is irrelevant to the query snow, it has been classified as snow because a certain region in the image (the chest of the leopard) is visually similar to snow. Another example is given in Fig. 5(d), which is a movie ad, not a sunset image. SIEVE did not remove it from the list for query sunset, as it contains regions visually very similar to sunset and the decision tree cannot tell the difference. For queries with clear semantic meaning, like sunset and firework, Google can have high retrieval performance. However, the results are still improved by using SIEVE, especially when K (total number of returned images) is large. As examples, Fig. 6 and Fig. 7 list the top 20 images given by Google image search and SIEVE using query sunset and firework, respectively. For the query sunset, the fifth and twentieth images in the list are relevant to the query, they are removed by SIEVE. This is because sunset regions are very small in these images, and are ignored by the system. In most cases, very small regions are not meaningful and ignoring such regions helps to find salient regions which are more important. However, in some cases like the sunset images mentioned above, small

6 regions may be semantically important. Other factors than size should be considered in future to deal with this type of situations. The seventeenth image in the list is about NASA - Sunset Planets. Venus, Jupiter and the Moon are gathering for a beautiful sunset sky show, and it is irrelevant to the query. SIEVE successfully removes it from the list. The eleventh image in Fig. 7 returned by Google does not contain a normal firework texture pattern. Thus, the decision tree does not recognize it and mistakenly removes it from the list. A few other irrelevant images such as the fifteenth and eighteenth images are successfully removed from the list by SIEVE. To summarize, the experimental results demonstrate that SIEVE effective improves the performance of Web image search by filtering out many irrelevant images from the image list returned by text-based search engine, although some relevant images may also be removed due to the limited dataset used to train the decision tree. Fig. 3. Top 25 Images Returned by Google (left) and by SIEVE (right) using Query Tiger. Fig. 4. Top 20 Images Returned by Google (left) and by SIEVE (right) using Query Snow. Fig. 5. Examples of Classification Errors.

7 Fig. 6. Tope 20 Images Returned by Google (left) and SIEVE (right) using Query Sunset. Fig. 7. Top 20 Images Returned by Google (left) and SIEVE (right) using Query Firework. 4. Discussions In this section, we first suggest three approaches for integrating SIEVE with existing text-based Web image search engines. Next, we discuss how to reduce SIEVE response time for real-time Web image search by making use of both text-based index and semantic-based index. Finally, the scalability of SIEVE is discussed in the context of extending it for a wide range of user queries. 4.1 Integration of SIEVE with Text Based Image Search Engine In the following, we propose different integration scenarios and take Google as an example of text based image search engine. In all these scenarios, Google and SIEVE are integrated in a hierarchy, with Google at the bottom layer to net the roughly similar images based on textual analysis. The results from Google are passed on to SIEVE for visual analysis. Scenario 1 SIEVE is integrated with Google on the server side. In this scenario, a user sends an image search query to Google using a standard Web browser. Google searches its image database using its own text-based image search algorithms. However, the resultant image list is not returned to the user immediately. Instead, the returned image list is input to SIEVE for visual analysis. SIEVE eliminates those

8 semantically irrelevant images from the list and returns the final image list to user. In this case, SIEVE is opaque to users, and the client side needs no modification. Users feel no difference from existing image searching experience, only to receive more accurate results due to the integration of SIEVE with Google. Scenario 2 SIEVE is integrated with the Web browser as a plug-in. A user query is directed by the SIEVE to Google. The results from Google are received by SIEVE but not shown to user immediately. SIEVE then performs semantic level filtering on the received image list and returns the filtered results to the user. While the server side remains unaltered in this case, the client side needs to install the plugin in order to make use of SIEVE for semantic level filtering. Scenario 3 SIEVE is used as an application software. This SIEVE software directs user queries to various Web image search engines. The user has the option of selecting more than one search engines to search for images. SIEVE software takes user queries and uses Application Programmable Interfaces (APIs) provided by various search engines, such as Google API, to send user queries and obtain image search results. SIEVE receives text-based image search results from one or more search engines, filters out these images on semantic level and presents the results to the user. This approach has the advantage that users can search multiple search engines simultaneously. This minimizes the delay caused by searching through many search engines one after the other by the users themselves. Moreover, the chances of quickly finding the most relevant images from the Web are maximized. Another advantage is that the users are not dependent on Google or other Web search engines for integration. However, in this case, the users need to install the SIEVE on their machines, and they do not have the freedom to use it everywhere, contrary to the Web browser based integration solutions discussed in scenarios 1 and SIEVE Response Time Refer to Fig. 1(b), to determine whether an image is irrelevant to a query or not, we need to first apply image segmentation to obtain image regions, and then extract lowlevel region features. With region features available, SIEVE then derives the highlevel concept of each region according to the decision rules obtained from decision tree. As the decision rules are obtained off-line, deriving region concepts on-line can be done easily by testing the region against the decision rules. Hence, the main computation cost of SIEVE comes from image segmentation and low-level region feature extraction. Our experimental data show that, on an average, image segmentation and region feature extraction together takes about three seconds for an image, on a Pentium IV 3G MHz processor. Considering that significant time is spent on learning the semantics of each image, a semantic index has to be built off-line for real-time Web image search. As proposed in Section 4.1, SIEVE and text-based search engine can be integrated in a hierarchy. In the long run, Google or any other text-based search engine has to make provision of storing this semantic index along with the text-based index in the database. 4.3 Scalability

9 The scalability of SIEVE highly depends on the performance of the image semantic learning method employed. In the current version, SIEVE is built on top of the decision tree which is limited to natural scenery images. Although a limited concept set is used to test its performance, the decision tree can accommodate more semantic concepts, provided their corresponding distinct feature templates are available for inclusion in the training dataset. For concepts in different domains, we can generate different decision trees by using different types of attributes (image features) and defining different templates accordingly. For example, shape is not used in learning concepts for natural scenery images, but it may be important in other domains such as tools database including screwdrivers, pliers, wrench, etc. In addition, dominant color is used to describe region colors, as most natural scenery image regions have single dominant color. However, this may not be true for images in other domains such as fabric cloth for which color template including more than one color has to be defined. To use SIEVE for general applications, one possible way is to classify user queries into different categories (domains) and learn decision rules for the concepts in each category. With a large number of user queries and Web images collected, queries can be classified by first clustering all Web images into different semantic categories (such as natural scenery, architectures and musical instruments), and then study the relationship between query keywords and image categories to map the keywords to each category. 5. Conclusions In this paper, we have proposed to use SIEVE to improve text based Web image search. Specifically, SIEVE makes use of visual features which represent image semantic content, to eliminate irrelevant images from the image list returned by a text-based image search engine. The idea behind the technique is intuitive and simple, it is easy to be integrated with existing Web image search engines. Our experimental results show it is very promising. It has significant improvement over Google image search in terms of retrieval accuracy. Different methods of integrating SIEVE with existing text-based search engines in real time have also been proposed. Compared with similar integration approaches in literature, the proposed SIEVE is linear to the size of the text-based result set, whereas existing approaches such as the re-ranking and grouping steps increase their complexity by a higher order. Compared with conventional content based image retrieval systems, the proposed SIEVE is also much more efficient. It does not need to access the entire huge image database, instead, it lets the more efficient text based image search engine to net the relevant images from the image database and it then works on the small returned list of images by eliminating the irrelevant images from the netted list. Currently, we have used only a limited concept set related to natural scenery images to test the performance of SIEVE. In the future, we will expand our concept sets to include a much larger number of categories, so as to further improve the retrieval accuracy. For concepts in different domains, we plan to use different types of features and define different semantic templates to represent the concepts. Multikeywords query will also be considered in our future implementation.

10 References [1] C. Thao and E. V. Munson, A Relevance Model for Web Image Search, In Proc. of International Workshop on Web Document Analysis (WDA2003), UK, Aug. 3, (2003) [2] W. -H. Lin, R. Jin and A. Hauptmann, Web Image Retrieval Re-Ranking with Relevance Model, In Proc. of IEEE/WIC International Conference on Web Intelligence (WI'03), (2003) [3] Google image search: accessed in December, [4] Yahoo image search: accessed in December, [5] D. Cai, X. He, Z. Li, W. -Y. Ma and J. -R. Wen, Hierarchical Clustering of WWW Image Search Results using Visual, Textual and Link Information, In Proc. of ACM Inter. Conf. on Multimedia, (2004) [6] J. He, C. Zhang, N. Zhao and H. Tong, Boosting Web Image Search by Co- Ranking, In Proc. of Int. Conf. on Acoustics, Speech, and Signal Processing, (2005) [7] Y. Liu, D. S. Zhang, G. Lu, and W.-Y. Ma, Region-Based Image Retrieval with High-Level Semantic Color Names, In Proc. of IEEE 11th International Multi- Media Modelling Conference (MMM05), Melbourne, Australia, January, 12-14, (2005) [8] Y. Liu, D. S. Zhang, G. Lu and W. -Y. Ma, Region-based Image Retrieval with Perceptual Colors, In Proc. of Pacific-Rim Multimedia Conference (PCM2004), Dec. (2004) [9] Y. Liu, D. S. Zhang, G. Lu and W. -Y. Ma, Study on Texture Feature Extraction in Region-Based Image Retrieval System, In Proc. of Inter. Multimedia Modelling Conf. (MMM2006), Beijing, (2006) [10] Y. Liu, W. Ma, D. S. Zhang, and G. Lu, An Efficient Texture Feature Extraction Algorithm for Arbitrary-Shaped Regions, In Proc. of IEEE 7th International Conference on Signal Processing (ICSP2004), Vol. 2, Beijing, China, Aug. 31 Sept. 4, (2004) [11] Y. Liu, D. S. Zhang, and G. Lu, Deriving High-Level Concepts Using Fuzzy-ID3 Decision Tree for Image Retrieval, In Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP05), Philadelphia, PA, USA, March 18-23, (2005)