The Use of Domain Modelling to Improve Performance Over a Query Session

Transcription

1 The Use of Domain Modelling to Improve Performance Over a Query Session Deirdre Lungley Dyaa Albakour Udo Kruschwitz Apr 18, 2011

2 Table of contents 1 The AutoAdapt Project

3 The AutoAdapt Project Automatic Adaptation of Knowledge Structures for Assisted Information Seeking 3-year EPSRC project (November November 2011) Collaboration between: University of Essex Robert Gordon University Open University

4 - Aim Aim The Task Evaluation Metrics Evaluate the effectiveness of search engines in interpreting query reformulations [4] A good search engine should be able to utilise the previous queries in the sequence of a session to provide better results that reflect the user needs throughout the session Example: Britney Spears Paris Hilton France Hotels Paris Hilton The session track provides a framework to assess this particular issue in Information Retrieval systems

5 Aim The Task Evaluation Metrics - The Task Only sessions with two queries were considered in 2010 Participants were given a set of 150 query pairs, each query pair (original query, query reformulation) represented a user session. Roughly split between: 1 Generalisation: low carb high fat diet types of diets. 2 Specification: us map us map states and capitals 3 Drifting/Parallel Reformulation: music man performances music man script. The participants were asked to submit three ranked lists of documents from the ClueWeb09 dataset: One for the original query (RL1) One for the query reformulation ignoring the original query (RL2) One for the query reformulation taking the original query into consideration (RL3)

6 Aim The Task Evaluation Metrics - Evaluation Metrics 1 Can search engines improve their performance for a given query using previous queries (Task G1)? Compare nsdcg@10 for RL1 RL2 and RL1 RL3 2 How do they perform over an entire session (Task G2)? nsdcg@10 for RL1 RL3

7 General Methodology - essex3 FCA Lattice Approach - General Methodology RL1 - Indri - original query (e.g. hoboken ) RL2 - Indri - reformulated query (e.g. hoboken nightlife ) RL3 - Indri - supports query expansion - weighted belief operators. Allow us to combine original query, reformulated query and derived expansion terms, e.g., #weight( 0.7 #combine( hoboken nightlife ) 0.3 #combine( hoboken #1( hoboken bars ) #1( hoboken nightclubs ) #1( hoboken wine bar ) ) ) All lists were filtered for spam - Waterloo Spam Filter - 70% cutoff 1 1

8 General Methodology - essex3 FCA Lattice Approach - AutoAdapt@TREC10 - essex3 Anchor log as a simulated query log has been shown to be effective in query reformulation Dang and Croft, WSDM10[1] Anchor log for ClueWeb09 available from the University of Twente[3] Use Fonseca s Association rules [2] to extract suggestions for both constituents of the session pair Use the intersection of the suggestions extracted for both constituents as useful expansions Session gps devices garmin computer worms malware us geographic map us political map Expansion terms or phrases gps devices, wikipedia, usb, gps device, gps products, garmin nuvi880, garmin gps device, visit garmin computer worms, computer security, category, worm us political map, article

9 General Methodology - essex3 FCA Lattice Approach - FCA Lattice Approach Anchor logs effective expensive Explore feasibility of generating expansion terms using Formal Concept Analysis (FCA) lattices Could we exploit the hierarchy of concepts of an FCA lattice created from documents and their index terms?

10 General Methodology - essex3 FCA Lattice Approach - FCA Lattice Approach Figure: Sample Hasse Diagram for query volvo

11 General Methodology - essex3 FCA Lattice Approach - FCA Lattice Approach

12 General Methodology - essex3 FCA Lattice Approach - FCA Lattice Approach Build lattices for both queries in the session to generate expansion terms for RL3 Initial hypotheses: 1 Lattice concepts generated from documents returned by a search engine over ClueWeb09 would be more discriminate than those generated by documents returned by a WWW search engine 2 Using a combination of common and distinct lattice concepts depending on the query type would be more discriminate than solely distinct terms

13 General Methodology - essex3 FCA Lattice Approach - FCA Lattice Approach Methods explored in creating document descriptors: Snippets returned by Microsoft s LiveSearch API Snippets returned by Indri over ClueWeb09 Noun phrases containing query from full Indri documents ClueWeb09 anchor logs for Indri result documents Methods to extract expansion terms Disjunction use concepts which appear in lattice generated by query 2 and not in lattice generated by query 1 Conjunction concepts common to both lattices Combination based on query type determined by term distribution, e.g., query 1 subsumed by query 2 (volvo : volvo semi trucks) specialisation

14 General Methodology - essex3 FCA Lattice Approach - FCA Lattice Approach Top Lattice Concepts Disjunction hoboken hoboken nightlife Derived Concepts new jersey hoboken bars hoboken bars free encyclopedia hoboken nightclubs hoboken nightclubs new york new jersey hoboken wine bar hudson county hoboken wine bar united states hoboken events Table: Example of expansion terms generated by disjunction method

15 % Average Increase % Average Increase TREC Metrics Exploratory Findings System All topics Spec. Gen. Drift. Essex Essex Lattice Table: % average increase from nsdcg@10.rl12 to nsdcg@10.rl13

16 TREC Metrics % Average Increase TREC Metrics Exploratory Findings Run nsdcg@10.rl13 (G2) all sessions Specification Generalisation Drifting Essex Essex Lattice CengageS10R Submitted nsdcg@10 (G1) ndcg@10 (G1) Run RL12 RL13 RL1 RL2 RL3 Essex Essex Lattice

17 Exploratory Findings % Average Increase TREC Metrics Exploratory Findings Search Descriptors Expansion Terms nsdcg@10.rl13 (G2) Engine Derived From Selection Method all sessions Spec. MSN Snippets Disjunction MSN Snippets Combination Indri Title, anchors Disjunction Indri Title, anchors Conjunction Indri Full Doc 1 Disjunction noun phrases containing a query term

18 The AutoAdapt Project TREC Session Track 2010 results demonstrated it is very hard to get any measureable improvement when utilising prior history FCA lattice - we believe it to be a promising structure not ready to dismiss Indexing the entire ClueWeb09 collection in house Category B Tier 1 web crawl Helps to explain success of the expansion term wikipedia Opportunity then to explore query expansion methods Effect of spam filtering

19 V. Dang and B. W. Croft. Query reformulation using anchor text. In WSDM 10: Proceedings of the third ACM international conference on Web search and data mining, pages 41 50, New York, NY, USA, ACM. B. M. Fonseca, P. B. Golgher, E. S. de Moura, and N. Ziviani. Using association rules to discover search engines related queries. In Proceedings of the First Latin American Web Congress, pages 66 71, D. Hiemstra and C. Hauff. Mirex: Mapreduce information retrieval experiments. Technical Report TR-CTIT-10-15, Centre for Telematics and Information Technology University of Twente, Enschede, April E. Kanoulas, B. Carterette, P. Clough, and M. Sanderson. Session track overview. In Proceedings of The Nineteenth Text REtrieval Conference Proceedings (TREC 2010), National Institute of Standards and Technology, To Appear.