Image Tag Clarity: In Search of Visual- Representative Tags for Social Images

Transcription

1 Image Tag Clarity: In Search of Visual- Representative Tags for Social Images Aixin Sun, Sourav S. Bhowmick Nanyang Technological University Singapore 1

2 Outline Web search & query clarity Image tag search and clarity Experiments and evaluation Conclusion and discussion 2

3 Web search examples: bank vs

4 Summarized by Wordle.net: bank vs

5 Query performance prediction in web search Query is effective: the retrieved documents contain unusually large probabilities of words specific to the topic Query is not effective: the retrieved documents is similar to a set of randomly sampled documents the word probability distribution is similar to that of the collection Query clarity score [Cronen-Townsend 02]: 5

6 Tag is visually representative? The images associated with the tag are visually similar to each other It is relatively easy to find a small set of images representing the tagged images (or the tag). Tag: Sunset Tag: Zebra Tag: Asia? Tag: 2008? 6

7 Sunset, Zebra, Asia, 2008 from Flickr.com Sunset Asia Zebra

8 Finding visual-representative tag Query: a tag t Retrieved documents: all images annotated with the tag T Image representation: bag of visual-words (as documents) Image tag clarity score: 8

9 Tag language model All images (documents) are equally important Images closer to the centroid are more important [Elsas 08] 9

10 Tag language model vs. collection language model 10

11 And we have a little problem here 11

12 Expected tag clarity score: derived from randomly assigned dummy tags 12

13 Normalized image tag clarity score Given a tag t with frequency freq(t) Compute the expected tag clarity score and standard derivation with multiple dummy tags of the same frequency Normalized tag clarity through zero-mean normalization Approximation: bin frequency 13

14 Experiments on NUS-WIDE Dataset NUS-WIDE Features 500D bag of visual-words Images 269,648 Tags 5981 (with frequency >=100) Categories (or concepts) All 81 category labels appear as tags in the dataset 14

15 Nitc distribution 15

16 Top-50 Most Visual-representative Tags Tag Nitc Pfreq Tag Nitc Pfreq Tag Nitc Pfreq sunset minimal airplane silhouette beach sand fog dunes cloud sky dawn foggy sunrise ocean weather charts moon morning sun lake pattern mist night atardecer clouds graphs jet lightning graph lines blue longexposure dusk sea zebra moleskine minimalism chart southcascades landscape sketches water windmills plane unbuilding storm aircraft craterlakenationalpark horizon seascape

17 Top-50 Least Visual-representative Tags Tag Nitc Pfreq Tag Nitc Pfreq Tag Nitc Pfreq people august june asia photographers pics brown finepix bottle japan religion april washington photos september smorgasbord hungary france panasonic caribou picture global cannon photograph may or july israel exotic china outside lumix virginia cool republic india culture canadian ohio royal this maryland world prayer colorful persian pic iranian

18 Sunset vs People 18

19 Distribution of the 81 category labels 46 are highly representative with nitc >=10 26 are representative with 2<=nitc<10; 9 (or 11%) are non-representative with nitc<2. Events and activities: dancing, running, soccer, sports, earthquake, Scene and location: castle, town, house, temple. 19

20 Tag Frequency vs Tag Clarity Tag frequency percentile (bin) 20

21 Conclusion and discussion The concept of clarity Web search and query clarity Tag search and tag clarity The computation of tag visual-representativeness Expected tag clarity through dummy tags Normalized tag clarity score Evaluation of on NUS-Wide dataset Future work Applications: tag recommendation? image classification? Computation: sampling of tagged images? Global features? Evaluation: is water more visually representative than river? 21

22 Acknowledgement Lab for Media Search, for sharing the NUS-WIDE dataset MSRA grant for partially supporting this trip References: S. Cronen-Townsend, Y. Zhou, W. B. Croft: Predicting query performance. SIGIR 2002: J. L. Elsas, J. Arguello, J. Callan, J. G. Carbonell: Retrieval and feedback models for blog feed search. SIGIR 2008: