Exploring Search Log Data. Theodora Tsikrika University of Applied Sciences Western Switzerland (HES-SO) Switzerland

Transcription

1 Exploring Search Log Data Theodora Tsikrika University of Applied Sciences Western Switzerland (HES-SO) Switzerland University of Copenhagen, February 22, 2012 CLEF 2011, Sept 21,

2 Sierre, Switzerland 2 2

3 HES-SO Sierre 1,500 students Institutes: Business Information Systems, Economy, Tourism Research in focussed domains: Internet of things, RFID Mobile applications Energy, Green ICT SAP centre ehealth Information Retrieval 3 3

4 MedGIFT research group Henning HenningMüller Müller Professor Professor Theodora Tsikrika Antonio Foncubierta Postdoc Ph.D. student Adrien Depeursinge Dimitriοs Markonis Postdoc Ph.D. student Manfredo Atzori Alba Garcia Postdoc Ph.D. student Alexandre Cotting Ivan Eggel Project manager Developer Alejandro Vargas (Geneva) Roger Schaer Medical Doctor Developer 4 4

5 MedGIFT research & projects Medical (multidimensional) image analysis and retrieval Multimedia information retrieval Information retrieval evaluation Test collection creation (including images and signals) User testing and task analysis Infrastructures for computation 5 5

6 Exploring search log data : researcher at CWI, Amsterdam, The Netherlands : Database Architectures and Information Access group : Interactive Information Access group VITALAS: Video & image Indexing and Retrieval in the Large Scale (FP6 IP) use-case driven project that built a prototype system dedicated to intelligent access services to multimedia professional archives advanced solutions for indexing, searching and accessing large scale of non previously (or partly) annotated multimedia content novel contributions in cross-media (audio/speech, video, image, text) indexing, content enrichment, and interactive retrieval methods 6 6

7 7 7

8 8 8

9 9 9

10 10 10

11 11 11

12 12 12

13 Search log data Examine users' information searching behaviour Unobtrusive / Naturalistic settings Broad range of user-system interactions / Significant time periods Large amounts of data / Sizable number of users No qualitative user aspects Context, situation, decision process, user satisfaction remain implicit Benefits Understand system usage Improve user experience and system effectiveness Exploitation Core ranking / Automatic query expansion / Ad matching / User modelling Web caching Query assistance Dynamic query suggestions as you type Query recommendations Media Search cluster meeting 13 13

14 Overview Motivation Search log analysis Semantic search log analysis method Study on professional image search log data Query recommendation Exploitation of clickthrough data Image annotation Image search Conclusions 14 14

15 Search log analysis Logged data timestamp, sessionid, userid, query, clicks Level of analysis Term Query Session Analysis of user behavioural patterns Query submission (formulation) Query modification Syntactic level (term-based) Semantic level Media Search cluster meeting 15 15

16 Term-based query modification analysis Addition = specification University of Copenhagen University of Copenhagen ranking Elimination = generalisation University of Copenhagen ranking University of Copenhagen Substitution = reformulation University of Copenhagen University of Aarhus Lexical variation Media Search cluster meeting 16Copenhagen University of Copenhagen Universities of 16

17 Term-based query modification analysis: limitations Reformulations Posthuma tour france Posthuma tour 2008 Vanessa Williams Serena Williams Undetermined University of Copenhagen Royal School of Library Information Science University of Copenhagen Denmark No semantics! Can we add a semantic dimension? How? Can we exploit the Linked Open Data? 17 17

18 Linked Open Data Entity = URI RDF triple = encode what is predicated about specific entities Subject: Beckham Predicate: Object: 'David Beckham' 18

19 Linked Open Data Entity = URI RDF triple = encode what is predicated about specific entities Subject: Beckham Subject: Predicate: Predicate: Object: David Beckham Object: `soccer player 19

20 Linked Open Data Entity = URI RDF triple = encode what is predicated about specific entities Subject: Beckham Subject: Predicate: Subject: Predicate: Object: David Beckham Predicate: Object: `soccer player Object: 20

21 Linked Open Data Entity = URI RDF triple = encode what is predicated about specific entities Subject: Beckham Subject: Predicate: Subject: Predicate: Object: David Beckham Predicate: Subject: Object: `soccer player Object: Predicate: Object: 21

22 Linked Open Data cloud 22 > > sources sources billion billiontriples triples 22

23 Semantic search log analysis method Input: List of search sessions (queries, query pairs) RDF triples from Linked Open Data sources Output: Query types + relative frequencies Query modification patterns + support & confidence values 1. Map queries from logs to entities in RDF triples (rdfs:label) 2. Determine types of entities and count occurrence frequencies 3. Determine semantic relations between entities of query pairs 4. Abstract semantic relations semantic patterns 5. Count occurrence frequencies 6. Rank semantic patterns based on their support and confidence 23 23

24 Semantic search log analysis method 24 24

25 Semantic search log analysis method 25 25

26 Semantic search log analysis method 26 26

27 Abstract semantic relations semantic patterns David Beckham Joe Cole DBPedia:David_Beckham -DBPedia:Nationalteam DBPedia:England_national_football_team DBPedia:Nationalteam- DBPedia:Joe_Cole Q1 -DBPedia:Nationalteam X DBPedia:Nationalteam- Q2 Nicolas Sarkozy Carla Bruni DBPedia:Nicolas_Sarkozy -DBPedia:spouse DBPedia:Carla_Bruni Q1 -DBPedia:spouse Q

28 Rank semantic patterns Which patterns are the most important? The ones that occur with higher frequency? What if these patterns are not informative and simply occur too often in the linked data? Compute expected frequencies of patterns Compute frequency of patterns between random queries Support = relative frequency Support_session Confidence = Support_session + Support_random Media Search cluster meeting 28 28

29 Overview Motivation Search log analysis Semantic search log analysis method Study on professional image search log data Query recommendation Exploitation of clickthrough data Image annotation Image search Conclusions 29 29

30 Professional image search logs analysis European news agency Commercial picture portal Millions of photographic images Professional users Search log data 10 months ~ 1 million queries / 0.5 million sessions Linked Open Data sources (22 million RDF triples) DBpedia WordNet Cornetto Getty geographical names Media Search cluster Getty Art and Architecture thesaurus meeting 30 30

31 Search log statistics (October 2008 July 2009) 31 31

32 Query frequency distribution 32 32

33 Query types Found matching URI for 79% of all queries Identified type for 68% of matched queries (about half of all queries) 33 33

34 Query types conceptual queries specific queries 34 34

35 Query types DBpedia:Person 35 35

36 Query modification patterns 24% query pairs classified using the semantic analysis 36 36

37 Query modification patterns identity relation 37 37

38 Query modification patterns partner of a person 38 38

39 Query modification patterns common property 39 39

40 Query modification patterns same type e.g., tennis players, townships,

41 Query modification patterns close relation e.g., prince and princess 41 41

42 Query modification classes Sibling relations: 19% Q1 -R X R- Q2 e.g., common property, WordNet hyponyms Direct few-to-few relations: 10% e.g., spouse Other relations: 71% 42 42

43 Term-based query modification analysis 43 43

44 Term-based vs. semantic query modification analysis 25% query pairs classified using the term-based analysis 24% query pairs classified using the semantic analysis complementary approaches 44 44

45 Accuracy of the method Semantic search log analysis method: 1. Match the query to linked data entities 2. Determine query types 3. Identify query modification patterns Accuracy of query modification pattern identification 100 query pairs randomly selected 4 judges identified the most prominent relation for 25 query pairs each (ground truth) 3 raters assessed the patterns identified by the system against the ground truth System choice classified as incorrect, approximately correct, correct Agreement among raters: 0.69 Fleiss kappa Agreement between system and ground truth: 0.61 (lenient mapping) System moderatelymedia successful Search cluster meeting 45 45

46 Overview Motivation Search log analysis Semantic search log analysis method Study on professional image search log data Query recommendation Exploitation of clickthrough data Image annotation Image search Conclusions 46 46

47 Query recommendation: approaches Existing approaches Document-based methods Search log-based methods: co-occurring queries Not previously submitted queries? Infrequent queries? Ontology-based methods Which links to select? Combinations of the above Based on semantic patterns Given: A query mapped to concept(s) Semantic patterns ranked by their support Apply patterns to concepts Suggestions ranked by their support Media Search clustervalue meeting 47 Ties broken by occurrence frequency in logs 47

48 Query recommendation: experiments Applied approach Baseline: search log-based method Top-10 co-occurring queries in the same session If suggestions < 10, then add suggestions based on semantic patterns Datasets 1,105, 766 queries 332,809 sessions 80% of sessions used for training (417,633 query pairs) 20% of sessions used for testing (64,767 query pairs) 44 semantic patterns 48 48

49 Query recommendation: results Log-based statistics Log-based statistics + Semantic patterns All queries Success rate % Coverage % * Queries that occur 5 times or less (36% of queries) Success rate % * Coverage % * Coverage: # times at least one suggestion is found Success rate: # times that suggestions include ground truth 49 Ground truth = the query immediately following the user s query in a session 49

50 Semantic search log analysis V. Hollink, T. Tsikrika, and A. P. de Vries. Semantic Search Log Analysis: a Method and a Study on Professional Image Search. JASIST, 62(4): , V. Hollink, T. Tsikrika, and A. P. de Vries. The semantics of query modification. In Proceedings of the 9th International Conference on Adaptivity, Personalization and Fusion of Heterogeneous Information (RIAO 2010), April 28-30, Paris, France,

51 Overview Motivation Search log analysis Semantic search log analysis method Study on professional image search log data Query recommendation Exploitation of clickthrough data Image annotation Image search Conclusions 51 51

52 Concept-based Image Annotation Aim: unambiguously describe the visual content of images Bridge Canal Red houses... Caption : Pretty Copenhagen

53 Concept-based Image Annotation Challenges when using supervised machine learning techniques: require labelled samples as training data laborious and expensive task when performed manually large number of semantic concepts poor generalisation of concept classifiers in other domains How can we automatically supplement/replace the manually annotated training samples? 53 53

54 Approach Automatically generate annotated training samples user-defined tags (e.g., Flickr) keywords extracted from Web pages where images are embedded clickthrough data collected in search logs traffic advantages: large quantities, no user intervention, available to all content owners, collective annotations (assessments) disadvantages: sparse, noisy, user queries not based on Media Search cluster meeting 54 strict visual criteria 54

55 Research questions 1) How can we build classifiers for annotating images with concepts using clickthrough data? methods for searchlog-based positive sample selection random negative sample selection 2) What is the effectiveness of these concept classifiers? experiments using data provided by BELGA news agency ~100k photographic images (with their text metadata) clickthrough data 55 55

56 Concept definition A concept is a clearly defined, non ambiguous entity represented by a short name keywords free-text short description Name traffic Concept Keywords traffic, traffic jam, cars, road, highway Description Image showing a high density of vehicles when on a road or highway

57 Positive sample selection using search logs Method exact select images clicked for queries exactly matching the concept name Methods textual similarity (based on IR language models) annotate each image with all queries for which it has been clicked apply stemming (yes/no) select images retrieved for query: (i) concept name (ii) concept keywords using retrieval model: (i) language model (LM) (ii) smoothed LM (LMS) Method clickgraph images clicked for the same query are likely to be relevant to each other 57 57

58 Reliability of clickthrough-based annotations methods varies greatly across concepts around 20% of the total number of concepts for each method reach Media 58 agreement of at least 0.8 Search cluster meeting 58

59 Building concept classifiers Positive samples for concept c: Nc,m images selected using one of the methods m exact, textual similarity (6 language modelling variants), clickgraph Negative samples for concept c: Nc',m = max( Nc,m,, Nc,m) images randomly selected Low-level features visual features FW (120-d vector) based on integrated Weibull distribution of edges (texture descriptor) compare region distributions to distributions of a set of reference images J. C. van Gemert et al. Robust Scene Categorization by Learning Image Statistics in Context. In International Workshop on Semantic Learning Applications in Multimedia, text features FT SVM classifier with RBFMedia kernelsearch cluster meeting 59 59

60 Experiments: datasets (provided by Belga news agency) Image collection 97,628 photographic images ~1,000 images manually annotated for each VITALAS concept Search logs 101 days (June October 2007) professional users 9,605 unique ('lightly' normalised) queries conversion to lower case removal of punctuation, quotes, and methods removal of names of major photo agencies 35,894 clicked images (out of the 97,628) Evaluation datasets 25 concepts Training: (manual annotations) (positive samples) (negative samples) 60 Test: 56,605 images 60

61 Experiments: results methods For visual features : combination of manual and searchlog-based training samples performs best consistently over all methods For text features : searchlog-based training samples produced by less noisy methods perform best 61 Text features outperform visual features 61

62 Experiments: results visual features manual+searchlog-based visual features: manual visual features searchlog-based methods For visual features : combination of manual and searchlog-based training samples performs best consistently over all methods For text features : searchlog-based training samples produced by less noisy methods perform best 62 Text features outperform visual features 62

63 Experiments: results text features searchlog-based text features: manual text features manual+searchlog-based visual features manual+searchlog-based visual features: manual visual features searchlog-based methods For visual features : combination of manual and searchlog-based training samples performs best consistently over all methods For text features : searchlog-based training samples produced by less noisy methods perform best 63 Text features outperform visual features 63

64 Concept: soccer manually annotated positive samples search log based annotated positive samples test set results visual features search log based training 64 View all results at: 64

65 Image annotation using clickthrough data: main findings Contribution of search-log training data in image annotation when using supervised machine learning is positive Scales to a large number of concepts Can take into account emerging concepts Available to all content owners avoid the generalisation problem 65 65

66 Image annotation using clickthrough data T. Tsikrika, C. Diou, A. P. de Vries, and A. Delopoulos. Image Annotation Using Clickthrough Data. In Proceedings of CIVR Τ. Tsikrika, C. Diou, A.P. de Vries, and A. Delopoulos. Reliability and Effectiveness of Clickthrough Data for Automatic Image Annotation. Multimedia Tools & Applications, 55(1),

67 Overview Motivation Search log analysis Semantic search log analysis method Study on professional image search log data Query recommendation Exploitation of clickthrough data Image annotation Image search Conclusions 67 67

68 Topic modelling of clickthrough data D. Morrison, T. Tsikrika, V. Hollink, A. P. de Vries, É. Bruno, S. Marchand-Maillet. Topic modelling of clickthrough data in image search. Multimedia Tools & Applications, Springer (to appear)

69 Conclusions Semantic search log analysis Implications for system design, search support, content management Query recommendation Beneficial for infrequent queries or queries entered for first time (long tail) Suggestions not occurring in logs (serendipitous discoveries) Explain relations between query and suggestions Combination of search logs with linked data Image annotation using clickthough data clickthrough data alone can lead to satisfactory effectiveness combination with manual annotations improves the effectiveness scalability in the number of concept detectors possibility to dynamically adapt the detector set Optimal sample size? Noise reduction? 69 69

70 Thank you!