PRISM: Concept-preserving Social Image Search Results Summarization Boon-Siew Seah Sourav S Bhowmick Aixin Sun Nanyang Technological University Singapore
Outline 1 Introduction 2 Related studies 3 Search results summarization: definition and model 4 The Prism algorithm 5 Evaluation 6 Conclusion B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 2 / 25
Social image search results summarization Social images Images shared through Flickr, Istagram or other platforms Images are annotated with tags by users Tag-based image retrieval (TAGIR) Queries are often short and ambiguous Search results diversification for matching user search intent Search results are not semantically or visually coherent Image search results Often presented as a ranked list of image thumbnails B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 3 / 25
Sample query results for tag queries: fruit and fly Strawberries, apples, oranges, and even market and fruit juice Aeroplanes, insects, birds, and even the act of jumping Concepts: visually and semantically distinct objects and scenes apple, orange, bird, and act of jumping are all example concepts B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 4 / 25
Key limitations of presenting results in a ranked list Fails to provide a view of common visual objects or scenes collectively Fails to provide a bird eye view of different concepts present in a query results More appealing way of presenting search results? Organize image search results in a set of image clusters Images in each cluster are semantically and visually coherent The clusters maximally cover the entire result set. Exemplar images from clusters to give a bird s-eye view of the search results. B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 5 / 25
Results summarization for queries: fruit and fly Search results summary: a set of image clusters, associated with tags concept-preserving, visually coherent, high coverage, distinctive cherry (100%) helicopter (100%) lemon (100%) jump (100%) splash (100%) insect (100%) kiwi (100%) f16 (100%), usaf (100%) citrus (100%) aeroplane (100%) pears (100%) birdofprey (100%) B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 6 / 25
Related work Exemplar-based Summarization: find a set of exemplars that summarize the image set. Image selection from clusters by image descriptors and tag topic vectors independently and then intersection [12]. Exemplar selection using a sparse Affinity Propagation (AP) [7]. Do not ensure the exemplars maximally cover the image results. Clustering-based Summarization: find blocks of similar images Clustering purely by tags or solely by visual similarity [8, 17, 19]. Multi-modal clustering by both visual and textual features: early fusion [1, 2, 9] and late fusion [10]. Do not associate each cluster with a tag concept for easy user interpretation; do not seek to find a concise set of images that maximally covers the entire result set. B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 7 / 25
Example outputs of exiting methods Shared Nearest Neighbor (Concept + Visual) [10] 1 Shared Nearest Neighbor [10] 2 Homogeneous and heterogeneous message propagation (H 2 MP) [20] 3 Canonical View [15] 4 Affinity Propagation [6] orange (78%), yellow (48%), lemon (44%), red (25%) frutas (91%), market (83%), vegetales (83%), mercado (83%) 2 H MP (Concept + Visual) [20] orange (45%), macro (36%), stilllife (27%), black (18%) red (66%), day (22%), stems (22%), snack (9%) Canonical View (Visual) [15] red (27%), food (27%), macro (25%), strawberry (16%) strawberry (42%), sky (42%), blue (28%), garden (28%) Affinity Propagation (Visual) [7] red (100%), vegetables (40%), overtheexcellence (40%), food (40%) food (33%), red (27%), black (22%), strawberry (22%) B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 8 / 25
Notations and definitions Notations: A search query Q = {q 1, q 2,..., q c }, a query keyword q is a tag A list of result images D = {i 1, i 2,..., i n } satisfying Q and D = n An image i D comprises of (a) a d-dimensional visual feature vector, and (b) a set of tags T i = {t 1, t 2,..., t Ti } associated with i, and Q T i Visual similarity graph G = (V, E, w) V : the set of images in D E: a set of undirected edges between visually similar images. w : visual similarity between images. Concept-preserving subgraph C T = (V T, E T, T ): A subgraph of G Concept subgraph is a set of images that preserves the concepts in T. Induced by V T V and images in C T share the tags T. B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 9 / 25
Concept-preserving image search summarization An decomposition of a visual similarity graph G into a set of concept subgraphs: S = {C T 1, C T 2,... C T k } and a remainder subgraph R v2 surf sand v8 beach, nikon 0.3 v10 beach sand v1 surf 0.2 boat nikon 0.3 0.2 v3 surf 0.3 0.1 0.1 0.4 0.1 v9 beach sea 0.3 0.3 0.1 v4 sea 0.1 0.5 0.5 0.5 0.4 v6 sea, bird v5 sea sand v7 sea beach surf sea surf beach sea v11 sun nikon v13 sun beach 0.4 0.4 0.4 0.4 v14 sun boat 0.1 v12 sun bird 0.4 v15 nikon v16 bird i) ii) iii) Summary of Exemplars sun R sun B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 10 / 25
High quality summary construction? Many candidate summaries There are numerous ways of decomposing G into S and R. Summary objectives Visual coherence: The average weight of visually similar images in each C T S. Distinctiveness: Clean separation of concept subgraphs, measured by concept subgraph redundancies. Coverage: Summary well represents G, measured by the ratio of images in S against G Concept-preserving: Each subgraph contains images with common concept(s). B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 11 / 25
Summary objectives: illustration high each group contains images that are visually similar low images are visually dissimilar surf high each group contains images with a common concept low no common concepts?? beach sun?? visual coherence concept preserving beach high each group of image is semantically and visually distinctive low high degree of overlap and redundancy high most images represented low many images unrepresented d50 sea sun nikon sun distinctiveness coverage B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 12 / 25
A weighted minimum k-set cover optimization model The problem Given the visual similarity graph, the goal of the social image search results summarization problem is to find an optimal set of concept subgraphs S s.t. coherence(s), coverage(s) and distinctiveness(s) are maximized. Exemplar images selected from each concept graph. A weighted minimum k-set cover optimization model Adding a concept subgraph incurs a visual incoherence cost for maximizing coherence(s) Adding a remainder subgraph incurs a remainder penalty cost for maximizing coverage(s) Find the minimum cost of subgraphs to cover all images, penalizing redundant subgraphs for controlling distinctiveness(s) B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 13 / 25
Prism algorithm Two key issues A structure to allow efficient enumeration of concept subgraphs A method to efficiently find an optimal subset of subgraphs that maximizes the summarization objectives. The Prism algorithm in 5 phrases 1 Visual similarity graph construction 2 Concept graph construction 3 Graph decomposition selection of subgraphs 4 Summary compression merge subgraphs to reduce details 5 Exemplar summary generation. B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 14 / 25
Visual similarity graph and concept subgraph 1: Visual similarity graph construction Query-dependent. Top-n results (n=1000 in our experiments) Image similarity: cosine similarity on visual features Edge between two images if similarity greater than δ 2: Concept subgraph construction A directed acyclic graph (DAG) exploration model Concept refinement: {sea} {sea, beach} {sea, beach, surf } example on next slide B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 15 / 25
Concept subgraph construction: illustration depth 0 {} depth 1 {food} {sea} {rock} depth 2 {food, cheese} {sea, beach} {sea, rock} {rock, music} depth 3 {sea, beach,surf} {sea, beach, sail} {sea, rock, cliff} Each node represents a concept graph. B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 16 / 25
Graph decomposition and summary compression 3: Graph decomposition (selection of subgraphs) Adopt a H k -approximation greedy algorithm [5]. Each iteration, add in a subgraph with minimum cost per new node covered. 4: Summary compression The concept subgraphs selected may be too fine-grained Adjusting parameter k directly may significantly affect summary coverage and distinctiveness Solution: Multi-summaries at varying granularity by aggregating subgraphs, e.g., {boat, sail, rock} and {rock, cliff } {rock} B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 17 / 25
Summary compression: illustration {sea, surf, nikon} {nikon, boat} {boat, sail, rock} {sea, surf} {nikon, boat} {boat, sail, rock} 0.8 1.0 1.0 5.0 0.8 3.0 3.5 0.8 3.5 3.0 4.0 4.0 a) S {sea, surf, hawaii} {nikon, rock, cliff} {rock, cliff} b) S 1 {nikon, rock, cliff} {rock, cliff} {sea, surf} {nikon, boat} {boat, sail, rock} {sea, surf} {nikon, boat} {rock} 1.0 3.0 c) S 2 {rock, cliff} Weight between two concept subgraphs: (a) concept relevance to the search query, and (b) number of shared concepts B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 18 / 25 d) S 3
A summary of the summarization algorithm T'' T' T concept refine T'' = {fruit,apple,red} T' = {fruit,apple} T = {fruit} select {fruit,apple,red} {fruit,apple,green} {fruit,banana} + summary compress apple banana kiwi B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 19 / 25
Experimental setup Dataset: NUS-WIDE with 269K Flickr images. Queries: 30 queries selected mainly by tag frequency Single-tag: asia, party, wedding, animals, art, city, rock, food, sun, sea, sky, nature, church, street, macro, bird Multi-tag: [sun, sea], [sun, silhouette], [blue, sea], [street, art], [sea, rock], [blue, sky], [rock, music], [macro, insect], [city, lights], [flower, macro], [cute, animals], [red, food], [graffiti, art], [birthday, party] Search results: top-ranked 1000 images for each query Visual similarity: 6 types of low-level visual features, e.g., color histogram, edge direction histogram, wavelet texture, SIFT Evaluation User study Four measures: coverage, distinctiveness, visual cohesiveness score, concept preservation score B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 20 / 25
Evaluation: user study Methods in comparison Canonical View Summarization (CV) [15] Affinity Propagation (AP) [6] H 2 MP (HY) [20] Google images (image categories) Bing images (related topics) Ratings based on four questions from 1 to 5 1 Visual appeal: Is the summary visually appealing? 2 Relevance: Are the exemplar summaries relevant to the query? 3 Comprehensiveness: Is the summary comprehensive? 4 Organization: Is the summary well organized? easy to understand at a glance? B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 21 / 25
Evaluation: user study 5 4 VisualAppeal Relevance Comprehensiveness Organization 5 4 VisualAppeal Relevance Comprehensiveness Organization 3 3 rating 2 rating 2 1 1 0 Bing Google PR AP CV HY (a) Single-tag queries 0 Bing Google PR AP CV HY (b) Multi-tag queries Google, Bing and Prism summaries better organized than others. AP has low relevance rating for prioritizing visually similar images. Hybrid methods Prism and HY benefit from exploiting both visual and conceptual features in summarization process B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 22 / 25
Evaluation score 1.4 1.2 1 0.8 0.6 0.4 0.2 0 VisualCoherence Coverage Distinctiveness ConceptPreservation PRISM AP CV HY The four measures 1 Visual cohesiveness score 2 Coverage 3 Distinctiveness 4 Concept preservation score AP and CV purely on image visual similarities; construct a partition on G, perfect coverage and distinctiveness scores. Prism has better concept preservation and better visual coherence than HY. Prism achieves the best balance of maintaining concept preservation and visual coherence of a summary. B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 23 / 25
Summary of the summarization 1 Key limitations of presenting social image search results in a ranked list 2 Image search results summarization and the desired features/objectives 3 Prism algorithm: concept-preserving summarization considering both visual and concept features Construct visual graph Refine and select concept-preserving subgraphs Compress the select subgraphs Generate exemplar images 4 Evaluation by user study and four measures B.-S. Seah, S. S. Bhowmick, A. Sun PRISM: Image Search Summarization SIGIR 14 Gold Coast 24 / 25
Dr. Aixin SUN axsun@ntu.edu.sg http://www.ntu.edu.sg/home/axsun/