ImgSeek: Capturing User s Intent For Internet Image Search

Transcription

1 ImgSeek: Capturing User s Intent For Internet Image Search Abstract - Internet image search engines (e.g. Bing Image Search) frequently lean on adjacent text features. It is difficult for them to illustrate users search objective only by query keywords and this cause to equivocal and noisy search results which are far from satisfactory. It is important to use visual information in order to solve the uncertainty in text-based image retrieval. Here we propose an innovative internet image search approach. It only needs the user to click on one query image with the minimal effort and images from a pool fetched by textbased search are re-ranked based on both visual and textual content. Our motive is to capture the users' search objective from this one-click query image in four steps. (1) The query image is classified into one of the predefined adaptive weight class, which follow users' search intention at a coarse level. Inside each category, a definite weight schema is used to merge visual features flexible to this kind of images to better re-rank the keyword-based search I. INTRODUCTION An internet image search engine is an essential aspect for web searchers to find desired images. With the emergence of the Internet, billions of images are now widely available. The expressive expansion in the amount of such images can make the task of finding and accessing image of interest, overwhelming for users. Therefore, some supplementary processing is required in order to make such collections searchable in a useful manner. The ongoing internet image retrieval search engines, including Google image search, Bing use text (i.e., surrounding words or Keyword) to look for images, without reviewing image content. However, when the neighboring words are ambiguous or even irrelevant to the image, the search relies on text only will result in many undesired result images. Many commercial Internet image search engines still rely on keywords as queries i.e. fetch and rank images depends on neighboring text or human-submitted annotations. Users submit query keywords in the concern of finding a definite type of images. The search engine forwards thousands of images ranked by the keywords derived from the neighboring text. It is notable that text-based image search undergo from the uncertainty or ambiguity of query keywords. The keywords issued by users lean to be short. They cannot illustrate the content of images precisely. The search results are noisy and consist of images with quite 1 Yogesh A. Sankpal, 2 Vidya Chitre M.E.(Computer), Bharati Vidyapeeth College of Engineering Asst. Professor, Vidyalankar Institute of Technology sankpalyogesh@gmail.com, vidya.chitre@vit.edu.in result. (2) Depends on the visual content of the query image elected by the user and through image clustering, query keywords are enlarged to capture user intention. (3) Expanded keywords are used to enlarge the image pool to include more compatible images. (4) Expanded keywords are also used to enlarge the query image to multiple positive visual examples from which new query specific visual and textual similarity metrics are studied to further enhance content-based image re-ranking. All these steps are automatic without extra activity from the user. This is essentially prime for any commercial internet image search engine, where the user interface has to be intensely simple and manageable. Excluding this key addition, a set of visual features which are both effective and efficient in Internet image search are outlined. Keywords Image mining, Keyword expansion, Semantic features, visual query expansion, Visual similarities dissimilar semantic meanings. Figure 1 illustrates the top ranked images from Bing image search corresponds to "apple" as query. They associated with several categories, such as "apple fruit", "apple logo" due to the ambiguity of the word "apple". The ambiguity problem appears for several basis. First, the query keywords' context may be richer than users' assumption. For example, the context of the word apple involves apple fruit, apple computer, and apple logo. Figure 1. Illustrates the top ranked images from Bing image search corresponds to "apple" as query. Also the user may not have enough knowledge on the contextual description of target images. For example, if users do not know "yogi bear" as the name of a cartoon character ( Figure 2(a)) and they submit text "bear" as query to search images of "yogi bear". Most importantly, in many cases it is difficult for users to outline the visual content of target images using keywords precisely. 18

2 Figure 2(a). Figure 2(b). Figure 2. (a) Images of yogi bear (b) Google Related Searches of query bear. Concerning to clear up the ambiguity, auxiliary information has to be used to capture users' search intention. One way is text-based keyword expansion, making the textual explanation of the query more exhaustive. Current semantic-related methods find either equivalent or other semantic-related words from thesaurus, or find words frequently co-occurring with the query keywords. For example, Google image search presents the "Related Searches" feature that advise probable keyword expansions. Although, even with the same query keywords, the intention of users can be highly varied and cannot be precisely captured by these expansions. As shown in Figure 2(b), "yogi bear" is not among the keyword expansions advised by Google "Related Searches". Capturing the user's intent of internet image search is a difficult problem because of sparse data accessible regarding the searcher. In this paper, we study methods to capture user's search intent underlying internet image search queries. We do believe that appending visual information to image search is essential. Although, the interaction has to be as simple as possible. The outright minimum is One- Click. In this paper, we propose an innovative Internet image search approach. It needs the user to give only one click on a query image and images from a pool extracted by keyword based search are re ranked depends on their visual and textual likeness to the query image. The key problem to be solved in this paper is how to capture user intention from this oneclick query image. II. RELATED WORK Various Internet scale image search methods [5] are text-based and are restricted by the fact that query keywords cannot define image content precisely. Content based image retrieval [6] uses visual features to estimate image analogy. Numerous visual features were advanced for image search in recent years. Few were global features such as GIST and histogram of gradient (HOG). Some quantized local features, such as Scale Invariant Feature Transform (SIFT), into visual words, and represented images as bags-of-visual words (BoV). In order to retain the geometry of visual words, spatial information was concealed into the BoV model in multiple ways. One of the primary threats of content-based image retrieval is to review the visual similarities which will reflect the semantic importance of images. Image similarities can be reviewed from a large training set where the relevance of pairs of images is noted [7]. Since web images are highly transformed, describing a set of attributes with hierarchical relationships for them is challenging. Generally, reviewing a universal visual similarity metric for generic images is still an open problem in research. Some visual features may be more effective for certain query images than others. In order to make the visual similarity metrics more specific to the query, relevance feedback [9] was widely used to expand visual examples. The user was asked to select multiple relevant and irrelevant image examples from the image pool. A query-specific similarity metric was learned from the selected examples. In the weights of combining different types of features were adjusted according to users' feedback. Since the number of user-labeled images is small for supervised learning methods, Huang et al. proposed probabilistic hypergraph ranking under the semi-supervised learning framework. It utilized both labeled and unlabeled images in the learning procedure. Relevance feedback required more users' effort. For a web-scale commercial system users' feedback has to be limited to the minimum, such as one-click feedback. Concerning to decrease users' burden, pseudo relevance feedback enlarged the query image by taking the top N images visually most similar to the query image as positive examples. Although, due to the conventional semantic gap, the top N images may not be all semantically constant with the query image. This may decrease the performance of pseudo relevance feedback. Under the framework of pseudo relevance feedback we have one proposed trans-media similarities which combined both textual and 19

3 visual features and also the query-relative classifiers, which merged visual and textual information, to re-rank images extracted by an initial text-only search. However, since users were not needed to select query images, the users' intention could not be precisely captured when the semantic meanings of the query keywords had large variety. Beyond visual query expansion, some approaches used concept-based query expansions through mapping textual query keywords or visual query examples to high-level semantic concepts. They needed a pre-defined concept lexicons whose detectors were off-line reviewed from fixed training sets. These approach were applicable for closed databases but not for internet image search, since the finite number of concepts cannot cover the various images on the Internet. The idea of reviewing example specific visual similarity metric was explored in previous work. Although they needed training a specific visual similarity for every example in the image pool, which is presumed to be fixed. This is illogical in our application where the image pool extracted by textbased search invariably alters for different query keywords. Moreover, text information, which can appreciably improve visual similarity learning, was not advised in previous work. In our approach, keyword expansion is used to enlarge the extracted image pool and to expand positive examples. Keyword expansion was mainly used in document retrieval. Thesaurus-based methods enlarged query keywords with their semantically related words such as synonyms and hyponyms. Corpus-based methods, such as well-known term clustering and Latent Semantic Indexing, measured the similarity of words based on their co-occurrences in documents. Words most similar to the query keywords were chosen as textual query expansion. Some image search engines have the feature of expanded keywords suggestion. They mostly use surrounding text. III. METHODOLOGY The flow diagram of our approach is shown in Figure 3. A novel Internet based image search approach [1]. It requires the user to click on one query image with minimum effort and images from a pool retrieved by text-based search are re-ranked based on both visual and Figure 3. An example to illustrate the algorithm. textual content. To capture the users intention images from one-click query image in four steps: 1) The query image is categorized into one of the predefined adaptive weight categories which reflect users search intention at a coarse level. A specific weight schema is used to combine visual features adaptive to this kind of image to better re-rank the text based search result. 2) In the visual content of the query image selected by the user and through image clustering, query keywords are expanded to capture user intention. 3) To enlarge the image pool using expanded keywords to contain more relevant images. 4) keyword Expanded are used to expand the query image to multiple positive visual examples from which new query specific visual and textual similarity metrics are learned to further improve content-based image reranking. 1.1 Text-Based Search The user first submits query keywords q as an input. Then an image is retrieved from a pool of image by textbased search. Then the user is asked to select a query image from the image pool. 1.2 Re-Ranking Without Query Expansion Images in the pool are re-ranked based on their visual similarities to the query image and the similarities are computed using the weight schema specified by the category to combine visual features. 1.3 Adaptive weighted schemas The query image is classified as one of the predefined adaptive weight categories. Human can easily categorize images into high-level semantic classes such as scene, object, fruit, logo or people, and then observed that images inside these categories usually agree on the relative importance of features for similarity calculations. The query images into several typical categories, adaptively adjust feature weights are within each category. 20

4 1.4 Keyword Expansion In the keyword expansion words are extracted from the textual descriptions (such as image file names and surrounding texts in the html pages) of the top k images most similar to the query image. To save computational cost, only the top m words are reserved as candidates for further processing because the initial image re-ranking result is still ambiguous and noisy, the top k images may have a large diversity of semantic meanings and cannot be used as visual query expansion. In this approach, reliable keyword expansions are found through further image clustering. For each candidate word wi, it finds out all the images containing wi and group them into different clusters {ci, 1, ci, 2,..., ci, ti} based on visual content. The same candidate word with images may have a large diversity in visual content. Assign image to the same cluster have higher semantic consistency since they have high visual similarity to one another and contain the same candidate word. 1.5 Visual Query Expansion The clusters contain of different candidate words, cluster ci, j with the largest visual similarity to the query image is selected as visual query expansion and its corresponding word wi is selected to form keyword expansion. q = q + wi A query specific visual similarity metric and a query specific textual similarity metric are learned from both the query image and the visual query expansion. 1.6 Image Pool Expansion The image pool is enlarged through combining the original image pool retrieved by the query keywords q provided by the user and an additional image pool retrieved by the expanded keywords q. Images in the enlarged pool are re-ranked using the learned queryspecific visual and textual similarity metrics. 1.7 Re-Ranking Result The image cluster size is selected as visual query expansion and its similarity to the query image indicate the confidence that the expansion captures the user s search intent. Figure 4. Steps in proposed framework. 1.8 Summery The goal of the proposed framework is to capture user intention and is achieved in multiple steps. The user intention is first roughly captured by classifying the query image into one of the coarse semantic categories and choosing a proper weight schema accordingly. Then regarding to the query keywords and the query image given by the user, the user intention is moreover captured in two facets: (1) keyword expansion describing user intention more precisely; (2) visual query expansion which are both visually and semantically consistent with the query image. They keyword expansion frequently co-occurs with the query keywords and the visual expansion is visually similar to the query image. Moreover, it is needed that all the images in the cluster of visual query expansion contain the same keyword expansion. Therefore, the keyword expansion and visual expansion support each other and are obtained simultaneously. In the later, the keyword expansion is used to enlarge the image pool to include more images relevant to user intention, and the visual query expansion is used to review visual and textual similarity metrics which better reflect user intention. IV. CONCLUSION In this paper, we propose a innovative Internet image search approach which only needs one-click user feedback. Intention specific weight schema is proposed to merge visual features and to estimate visual similarity adaptive to query images. Textual and visual expansions are integrated to capture user intention. Expanded 21

5 keywords are used to extend positive example images and also enlarge the image pool to include more relevant images. This framework can makes it possible for industrial scale image search by both text and visual content. The proposed new image re-ranking framework consists of multiple steps, which can be improved separately or replaced by other techniques equivalently effective. In the future work, this framework can be further improved by making use of the query log data, which provides valuable co-occurrence information of keywords, for keyword expansion. One shortcoming of the current system is that sometimes duplicate images show up as similar images to the query. This can be improved by including duplicate detection in the future work. REFERENCES [1] Xiaoou Tang, Ke Liu, Jingyu Cui, Fang Wen, and Xiaogang Wang, IntentSearch: Capturing User Intention for One-Click Internet Image Search, IEEE Transactions on Pattern Analysis and Machine Intelligence vol.34 no.7, [2] F. Jing, C. Wang, Y. Yao, K. Deng, L. Zhang, and W. Ma, Igroup: Web image search results clustering, in Proc. ACM Multimedia, [3] J. Cui, F. Wen, and X. Tang, Real time google and live image search re-ranking, in Proc. ACM Multimedia, [4] Jingyu Cui, Fang Wen and Xiaoou Tang, Intentsearch: Interactive on-line image search re-ranking, in Proc. ACM Multimedia, [5] N. Ben-Haim, B. Babenko, and S. Belongie, Improving web-based image search via content based clustering, in Proc. Int l Workshop on Semantic Learning Applications in Multimedia, [6] R. Datta, D. Joshi, and J. Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys, vol. 40, pp. 1 60, [7] G. Chechik, V. Sharma, U. Shalit, and S. Bengio, Large scale online learning of image similarity through ranking, Journal of Machine Learning Research, vol. 11, pp , [8] J. Deng, A. C. Berg, and L. Fei-Fei, Hierarchical semantic indexing for large scale image retrieval, in Proc. IEEE Int l Conf. Computer Vision and Pattern Recognition, [9] K. Tieu and P. Viola, Boosting image retrieval, International Journal of Computer Vision, vol. 56, no. 1, pp , [10] B. Luo, X. Wang, and X. Tang, A world wide web based image search engine using text and image content features, in Proc. IS&T/SPIE Electronic Imaging, Internet Imaging IV., [11] Z. Zha, L. Yang, T. Mei, M. Wang, and Z. Wang, Visual query expansion, in Proc. ACM Multimedia,