An adaptable search system for collection of partially structured documents

Transcription

1 Samantha Riccadonna An adaptable search system for collection of partially structured documents by Udo Kruschwitz Web Information Retrieval Course A.Y

2 Outline Search system overview Few concepts and observations: Adaptive (W)IR Searching Web Documents The domain model The general framework Conclusions References 2

3 Search system overview Generic framework: online search system accessing standard search engine automatically constructed domain model built using implicit knowledge (markup) dialogue system: interacting with user by offering options to refine or relax the query 3

4 Adaptive (W)IR adaptive techniques improve their performance over time, in response to feedback they receive on prior performance [Belew,, 2000] Relevance Feedback Techniques [Gudivada et al.,1997] 4

5 Searching Web documents queries are normally very short (average 2.3 words) search engine will retrieve numerous docs most of users do not perform any query modification [Silverstein et al.,1998] OBS: system that applies a domain model to propose query refinements must perform very well for the user to accept it 5

6 Searching Web documents interactive query expansion process improves search process inexperienced user do not make good term selections from potentially relevant term list significant improvement for experienced user [Margennis and van Rijsbergen,1997] advice to help users reformulate queries improves relevance of retrieved documents [Bruza et al.,2000] 6

7 The Domain Model (DM) Domain model is automatically acquired by exploiting the markup of documents Main issues: 1. extracting the concepts 2. constructing the domain model (organize concepts as a set of simple hierarchies) 3. applying the model as part of the search system 7

8 DM: extracting the concepts CONCEPT DOMAIN-INDEPENDENT An index term c is a concept term of type n for document d, if c occurs in at least n different markup contexts Partially structured documents documents can be divided into indifferent markup context (explicit or implicit structure) [HTML docs] doc titles, bold text, underlined text, [articles] article headings, captions, summaries, 8

9 DM: extracting the concepts CONCEPT An index term c is a concept term of type n for document d, if c occurs in at least n different markup contexts document type a concept of type n is a term found in at least n markup context OBS: - type 2 and type 3 concepts - higher type concepts are more reliable, but more sparse 9

10 DM: extracting the concepts Partially structured documents documents can be divided into indifferent markup context (explicit or implicit structure) use the structure to extract significant terms not rely on a single type of information: might be used infrequently could be used for spamming (HTML metatags) no assumptions about semantic interpretation of particular type of information 10

11 DM: concept hierarchies world model set of simple concept hierarchies AIM: RELATED CONCEPT establish that there is some relation (will be used to guide search process), no interest in semantic inter-concept relations Two concepts c 1,c 2 are related concepts of type n for document d, if c 1 and c 2 are concepts of type n for document d. 11

12 DM: concept hierarchies world model set of simple concept hierarchies AIM: establish that there is some relation (will be used to guide search process), no interest in semantic inter-concept relations relation construction process: - automatic - offline 12

13 DM: concept hierarchies concepts identified in the indexing process are likely to be among user queries keywords/terms (log analysis) model construction: iterative process each concept is a potential query ( appropriate concept hierarchy) (sequence of) user request simulation, using the concepts identified in the documents (no live user queries) 13

14 DM: concept hierarchies model construction: query modification step: exploration of all related concepts following relations detected in the source data (extended by single concept) new node added if there are documents matching the refined query limit to the number of branches originating in a node ( usable model) arc weights=number of matches that a node represent 14

15 DM: applying the model application of the model as part of the search system to help the user navigate the documents retrieved assist user in ad hoc search task support browse process (vs search process): user can get a perception of actually available data 15

16 DM: applying the model Dialogue manager: central part of the search system controls access to the domain model and search engine 16

17 The Domain Model (DM) usually not available can be constructed by hand-coding (using existing domain-independent knowledge sources) by processing available data into a concept model offline (by analyzing the entire document collection) online (by investigating docs retrieved for query) 17

18 The general framework Domain Model: automatically acquired by exploiting the markup of documents built offline by analyzing the entire document collection no assumptions about the domain type or semantic content 18

19 The general framework combines UKsearch system and std search engine: query submitted to both merge of the result 19

20 The general framework ADVANTAGES: initial response time depends on search engine technology conceptual model needed only if std search is unsatisfactory 20

21 Conclusions automatically created domain model relations not present in general knowledge sources (world knowledge vs. linguistic information) changing domain suited for document collection limited in size or domain-specific portable system (could be run on completely different collections) simple domain-independent dialogue manager assisting users (interactive search task) 21

22 References [Kruschwitz,, 2003] U. Kruschwitz, An Adaptable Search System for Collections of Partially Structured Documents, Intelligent Systems, IEEE, 18 (4), Jul-Aug 2003, pp [Belew,, 2000] R. K. Belew, Finding Out About, Cambridge University Press, Cambridge, [Gudivada et al, 1998] V. N. Gudivada, V.V. Raghavan, W. I. Grosky, R. Kasanagottu, Information Retrieval on the World Wide Web, Internet Computing, IEEE, 1 (5), Sep/Oct 1997, pp [Silverstein et al, 1998] [Silverstein et al, 1998] C. Silverstein, M. Henzinger, and H. Marais, Analysis of a Very Large alta Vista Query Log, Digital SRC Technical Note ,

23 References [Margennis and van Rijsbergen,1997] M. Margennis and C.J. van Rijsbergen, The Potential and Actual Effectiveness of Interactive Query Expansion, Proc. 20th Ann. Int l ACM SIGIR Conf. Research and Development in Information Retrieval,ACM Press,1997, pp [Bruza et al., 2000] P. Bruza, R. McArthur, and S. Dennis, Interactive Internet Search: Keyword, Directory, and Query Reformulation Mechanisms Compared, Proc. 23rd Ann. Int l ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2000, pp

24 24