Visual Query Suggestion Zheng-Jun Zha, Linjun Yang, Tao Mei, Meng Wang, Zengfu Wang University of Science and Technology of China Textual Visual Query Suggestion Microsoft Research Asia Motivation Framework Keyword suggestion mining Image Search with VQS 3 4 Motivation Towards bridging intention gap Intention Gap Intention Gap Query Query Search Engine Search Result Search Engine Search Result s can imagine what they want, but cannot express it in precise words Intention prediction: search engine guesses s search intention Query suggestion: search engine provides a list of suggested queries 5 6
Textual query suggestion Textual query suggestion Textual query suggestion: treat image search query suggestion as text search query suggestion, i.e., suggest a list of words 7 8 New query suggestion scheme : a query suggestion scheme tailored to image search provide both image and keyword suggestions refine the text-based image search using image suggestions as query examples 3 4 5 Framework Keyword suggestion mining Image Search with VQS 9 0 Visual query suggestion framework Query suggestion mining Goal: discover both image and keyword suggestions to help s express their search intention Query suggestion mining: discover image and keyword suggestions Suggestion presentation: present query suggestion responding to s operation Search with query suggestion: perform image search with image and keyword suggestions Possible data sources Query logs Click-through Search results Flickr/Photobuket/Wikipedia/
Query suggestion mining Possible solution Select keyword select image Select image select keyword Select image and keyword simultaneously Framework Keyword suggestion mining Image Search with VQS 3 4 Keyword suggestion mining Keyword suggestion mining Query Q Tags Images Keyword suggestions Query Q Tags Images Keyword suggestions Goal: Find a set of keywords with the following properties. Relevance: The selected keywords should be relevant to the query Informativeness: The selected keywords should be informative enough to reflect different aspects of the query How to measure them? 5 6 Relevance Informativeness Normalized Co-occurrence Relevance of keyword q Assumption: keyword q i and q j can reflect different aspects of Q, if the keywords co-occurring with (Q, q i ) and (Q, q j ) are different, i.e., the distribution f (q Q, q i ) and f (q Q, q j ) are different. KL divergence Relevance of keyword set Informativeness of keyword pair Informativeness of keyword set 7 8 3
Keyword suggestion mining Optimization function Solution Relevance Informativeness Framework Keyword suggestion mining Image Search with VQS 9 0 Goal: find a small number of images which can represent the corresponding keyword Affinity propagation Data graph Keyword q Images Affinity Propagation Image suggestions Full graph Sparse graph Affinity propagation Message propagation: three kinds of messages are recursively transmitted via the edges of the graph Affinity propagation Exemplar selection: the exemplar of image i is selected through the following equation. Responsibility: Availability: Self-availability: 4
Image search with VQS Framework Keyword suggestion mining Image Search with VQS Initial query Keyword suggestion Image suggestion Text-based Image Search Content-based Re-ranking 5 6 Image search with VQS Image search with VQS Content-based re-ranking Content-based re-ranking Multi-modality visual similarity Initial relevance Final relevance 7 8 Experiments Data and Setting Search performance evaluation 3 million images, 5 million tags from Flickr 30 professional s, 0 average s 5 queries covering different semantics, such as scene, object, event, etc. Professional Average Occupation researcher, graduated student marketing people, government officer, etc. Familiar with image search? Yes No 9 30 5
Data and Setting 3 million images, 5 million tags from Flickr 30 professional s, 0 average s 5 queries covering different semantics, such as scene, object, event, etc. Dose VQS perform much better, better, closely, worse, or much worse than existing engines? airshow, animal, apple, building, camping, car, disaster, flag, flight, flower, fruit, game, insect, panorama, Paris, plant, portrait, road, scenic, season, sky, sports, sunset, travel, weather. 3 3 Professional Average Is the visual query suggestion very useful, somewhat useful, or unuseful for eliciting s search intention? Professional Average 33 34 Experiments Search performance evaluation Experimental setting Search performance evaluation Comparing methods o Text-based image search with initial query (Baseline) o Text-based image search with textual query suggestion (TQS) Evaluation Metric: NDCG@k 35 36 6
NDCG @ k Search performance evaluation Conclusions Visual Query Suggestion provides both keyword and image suggestions helps s specify and deliver the search intention more precisely refines the text-based search results by using visual information k 37 38 Demo Link Thanks 40 7