Image Tag Clarity: In Search of Visual- Representative Tags for Social Images Aixin Sun, Sourav S. Bhowmick Nanyang Technological University Singapore 1
Outline Web search & query clarity Image tag search and clarity Experiments and evaluation Conclusion and discussion 2
Web search examples: bank vs. 2008 3
Summarized by Wordle.net: bank vs. 2008 4
Query performance prediction in web search Query is effective: the retrieved documents contain unusually large probabilities of words specific to the topic Query is not effective: the retrieved documents is similar to a set of randomly sampled documents the word probability distribution is similar to that of the collection Query clarity score [Cronen-Townsend 02]: 5
Tag is visually representative? The images associated with the tag are visually similar to each other It is relatively easy to find a small set of images representing the tagged images (or the tag). Tag: Sunset Tag: Zebra Tag: Asia? Tag: 2008? 6
Sunset, Zebra, Asia, 2008 from Flickr.com Sunset Asia Zebra 2008 7
Finding visual-representative tag Query: a tag t Retrieved documents: all images annotated with the tag T Image representation: bag of visual-words (as documents) Image tag clarity score: 8
Tag language model All images (documents) are equally important Images closer to the centroid are more important [Elsas 08] 9
Tag language model vs. collection language model 10
And we have a little problem here 11
Expected tag clarity score: derived from randomly assigned dummy tags 12
Normalized image tag clarity score Given a tag t with frequency freq(t) Compute the expected tag clarity score and standard derivation with multiple dummy tags of the same frequency Normalized tag clarity through zero-mean normalization Approximation: bin frequency 13
Experiments on NUS-WIDE Dataset NUS-WIDE http://lms.comp.nus.edu.sg/research/nus-wide.htm Features 500D bag of visual-words Images 269,648 Tags 5981 (with frequency >=100) Categories (or concepts) All 81 category labels appear as tags in the dataset 14
Nitc distribution 15
Top-50 Most Visual-representative Tags Tag Nitc Pfreq Tag Nitc Pfreq Tag Nitc Pfreq sunset 319.6 99.67 minimal 104.6 77.08 airplane 78.7 96.72 silhouette 211.2 97.64 beach 104.3 99.41 sand 78.4 98.08 fog 207.5 96.71 dunes 100.9 82.59 cloud 77.5 97.61 sky 197.6 99.97 dawn 100.5 91.09 foggy 77.1 68.18 sunrise 179.2 97.76 ocean 100.2 99.03 weather 76.5 95.47 charts 158.1 78.36 moon 100 94.87 morning 75.7 96.64 sun 151.9 99.1 lake 98.9 98.5 pattern 74.2 92.63 mist 138.6 94.75 night 94.1 99.5 atardecer 74.1 71.96 clouds 133.9 99.85 graphs 94 6.89 jet 74.1 93.56 lightning 129.5 73.72 graph 91.3 1.97 lines 73.7 94.9 blue 118.1 99.95 longexposure 91 97.71 dusk 73.4 95.13 sea 116.3 99.52 zebra 89.8 82.46 moleskine 72.8 70.76 minimalism 114.9 77.66 chart 89.6 20.7 southcascades 71.5 6.02 landscape 110.2 99.77 sketches 87.9 81.52 water 70.4 99.93 windmills 106.7 75.76 plane 83.8 95.79 unbuilding 70 67.31 storm 106 96.22 aircraft 82.4 96.2 craterlakenationalpark 69.4 10.58 horizon 105.5 92.44 seascape 80.6 91.72 16
Top-50 Least Visual-representative Tags Tag Nitc Pfreq Tag Nitc Pfreq Tag Nitc Pfreq people -2.9 99.55 august -1.3 80.12 june -1 86.52 asia -2.5 98.26 photographers -1.3 85.74 pics -1 58.08 brown -2.4 96.87 finepix -1.3 65.36 bottle -1 55.63 japan -2.3 98.11 religion -1.2 94.67 april -1 84.53 washington -2.2 97.06 photos -1.2 94.22 september -1 75.66 2008-2.1 98.51 smorgasbord -1.2 61.33 hungary -1 62.82 france -2 98.39 panasonic -1.2 85.32 caribou -1 80.77 picture -1.7 92.49 global -1.2 65.74 cannon -1 58.23 photograph -1.6 88.86 may -1.1 83.51 or -1 24.08 july -1.6 86.42 israel -1.1 86.94 exotic -1 62.18 china -1.6 96.99 outside -1.1 92.81 lumix -1 86.57 virginia -1.5 86.99 cool -1.1 95.7 republic -1 37.07 india -1.5 97.44 culture -1.1 93.13 canadian -0.9 64.62 ohio -1.3 87.33 royal -1.1 72.71 this -0.9 41.87 maryland -1.3 84.17 world -1.1 95.34 prayer -0.9 85.72 colorful -1.3 97.53 2005-1.1 96.05 persian -0.9 64.04 pic -1.3 58.7 iranian -1 57.33 17
Sunset vs People 18
Distribution of the 81 category labels 46 are highly representative with nitc >=10 26 are representative with 2<=nitc<10; 9 (or 11%) are non-representative with nitc<2. Events and activities: dancing, running, soccer, sports, earthquake, Scene and location: castle, town, house, temple. 19
Tag Frequency vs Tag Clarity Tag frequency percentile (bin) 20
Conclusion and discussion The concept of clarity Web search and query clarity Tag search and tag clarity The computation of tag visual-representativeness Expected tag clarity through dummy tags Normalized tag clarity score Evaluation of on NUS-Wide dataset Future work Applications: tag recommendation? image classification? Computation: sampling of tagged images? Global features? Evaluation: is water more visually representative than river? 21
Acknowledgement Lab for Media Search, for sharing the NUS-WIDE dataset MSRA grant for partially supporting this trip References: S. Cronen-Townsend, Y. Zhou, W. B. Croft: Predicting query performance. SIGIR 2002: 299-306 J. L. Elsas, J. Arguello, J. Callan, J. G. Carbonell: Retrieval and feedback models for blog feed search. SIGIR 2008: 347-354 22