Intranet Search. Exploiting Databases for Document Retrieval. Christoph Mangold Universität Stuttgart

Transcription

1 Intranet Search Exploiting Databases for Document Retrieval Christoph Mangold Universität Stuttgart

2 2 /6 The Big Picture: Assume. there is a glueing problem with product P7 Has this happened before? Is there any document about the problem? Search for: P7 glue P7 P23 Error Report Middle nozzle clogs repeatedly which leads to unstable glue joint. M8 file:///z:/docs/ Doc Aim: Rank Doc as highly relevant

3 Overview 3 /6 The ContextGraph () Ranking (4) Computing the context () Implementation & Performance (2) Related Work (3) Future Work & Summary (2)

4 ContextGraph & Semantic Distance 4 /6 ErrorReport DocID URL Abstract ProductID (FK) MachineID (FK) Doc file:// Middle noz P23 M8 Product ProductID P7 P23 Production ProductID (FK) P7 P23 Name Tiger Cobra MachineID (FK) M8 M8 Machine MachineID M8 P7 Tiger Location Cobra P23 M8 Type Error Report 4 Middle nozzle clogs repeatedly which leads to unstable glue joint. file:///z:/docs/ Doc

5 Ranking 5 /6 Idea: Transfer well-proven ranking measures to the context-based scenario What s Related : Exploit the web structure Query independent: Google s PageRank / ObjectRank Query specific: Vector space model & tf.idf coming up next

6 Ranking: Vector Space Model & tf.idf 6 /6 Documents and queries are vectors in a T -dimensional vector-space where T is the set of all terms. Similar vectors denote similar documents/queries Term d d2 d3 clog glue L L nozzle q Vector entries are calculated by means of tf idf tf (term frequency): How often does the term appear in the document/query idf (inverse document frequency): How rare is the term in the document collection

7 Ranking: tf ctf 7 /6 Consider the text only tf: How often does the term appear in a document? Consider context and semantic distances ctf: How often does the term appear in the context of a document? tf( t,d) = freq( t, d) max (freq(, d)) l d l ctf( t,d) = max H l Context( d ) t k = sim( d, h H l k = t k ) sim( d, h l k ) There is a glueing problem on M8 When the glue gets.

8 Ranking: idf icf 8 /6 Consider the text only idf: How rare is the term in the document collection? Consider context and semantic distances icf: How rare is the term in the contexts of all documents? idf( t) = { δ D t δ} icf( d, t) = { δ D n V t : sim( δ, n) sim( d, n)} Term d d2 d3 clog 2 2 glue L nozzle 2 idf( clog) idf( glue) idf( nozzle) = 4 = = 5

9 Computing the Context All Pairs Shortest Path (APSP) Optimizations: Neighborhood only Documents only Implementation FloydWarshall Neighborhood Dijkstra Document Neighborhood Dijkstra Neighborhood HiddenPath [Karger 93] HiddenPath Dijkstra doc Dijkstra all FloydWarshall 5 Time [sec] Graph Size [#edges] 9 /6

10 Implementation: Architecture & Technologies /6 Indexer Index Creation Text Analysis Admin Context Analysis APSP ObjectRank ContextGraph User Search What s Related Query Processing Index Reader Search Engine DB-Mapper Indexes Documents Database Java Lucene (Apache s search engine) D2RQ (DB-ontology mapping tool, FU Berlin) Jena (hp s semantic web framework) OWL / RDF (W3C s ontology description language)

11 Performance: Query Time /6 Text only -step 2-step WhatsRelated Time [ms] Graph Size [#edges]

12 Related Work: Semantic Search Surveyed 2 approaches Semantic Web Contextual knowledge is modeled in (handcrafted) ontologies User interaction Different ontology structures require / enable a large variety of search engines Knowledge Engineer Mount Database Ontologyhill Context information Mount Document Document information 2/6

13 Related Work: Keyword Search in Databases 3/6 [Goldman, VLDB 98] Lorel DB FIND... NEAR Shortest Path [Bhalotia, ICDE 2] BANKS Relational DB as a graph Search for subgraphs [S. Agrawal, ICDE 2] DBXplorer, [Hristidis, VLDB 2, VLDB 3] DISCOVER Join tables to retrieve tuples that contain all search terms

14 Related Work: Combining Structured & Unstructured Data 4/6 Using SQL queries [Dessloch, VLDB 97] [Goldman, SIGMOD ] WSQ Unstructered data as virtual tables Computes e.g. number of appearances of search terms Using OLAP techniques [Cody, IBM Sys. Journal 4(4), 22] BIKM Information Extraction Data Warehouse

15 Future Work 5/6 Assess semantic correctness Integration of ontologies / semantic search External memory shortest path algorithm

16 Summary 6/6 Exploit DB-Information to support Document Retrieval ContextGraph Semantic-distance based ranking à la tf.idf Architecture incorporates text- and context-search Performance evaluations promise little overheads only Related work: Semantic Search & DB Keyword Search