Structured Retrieval Models and Query Languages. Query Language

Size: px

Start display at page:

Download "Structured Retrieval Models and Query Languages. Query Language"

Maryann Underwood
5 years ago
Views:

1 Structured Retrieval Models and Query Languages CISC489/ , Lecture #14 Wednesday, April 8 th Ben CartereHe Query Language To this point we have assumed queries are just a few keywords in no parpcular order and with no structure ExcepPons: Phrases Boolean retrieval A query language is a way to give the engine more informapon about the relaponships between terms, importance of terms, etc 1

2 Examples Boolean queries t1 AND (t2 OR t3) AND NOT t4 Google queries t1 t2 +t3 t4 site: allinptle: t1 * t2 Structured language #combine( biography #syn( #1(president lincoln) #1(abraham lincoln) ) ) Query Languages and Indexes The query language is only as good as the index Your query language can support phrases, It can support document structure, It can support date ranges, But none of that mahers if there s no support in the index! Monday we talked about storing informapon to support phrases and structure 2

3 Query Languages and Retrieval Models Query language is only as good as the retrieval model behind it If your retrieval model does not model phrases, If it does not model document structure, If it does not model date ranges, Then a query language supporpng those is no use. Most of the models we ve discussed so far only model terms independently One exceppon: bigram language models Query Process Query parsed according to language turned into abstract representapon Q according to retrieval model Query language parser Ranking f(q,d) Server(s) Document parsed at index Pme, features needed for retrieval model stored on disk turned into abstract representapon D according to retrieval model 3

4 A Simple Example Suppose we have an inverted index that can support phrases We have an engine that uses the vector space retrieval model and cosine similarity scoring We want to add support for phrases to the query language Vector space model does not support phrases by default We need to add phrase support to the VSM Phrases in the VSM Add phrase support to the vector space model Document represented as vector of feature weights Features are terms and phrases Therefore each phrase becomes a new dimension in the vector space Term weight: Phrase weight: w ik = tf ik D k log N n i pw ik = pf ik D k log N np i 4

5 Phrasal VSM IllustraPon Two terms, one phrase: 3 D Two terms, no phrases: 2 D white white house house white house A More Complex Example Suppose we have an inverted index that supports field informapon We have an engine that uses the vector space retrieval model and cosine similarity scoring We want to add support for field informapon to the query language, e.g. inptle:white house Vector space model does not support this by default We need to add field support to the VSM 5

6 Fields in the VSM One possibility: separate term weights for each field the term appears in w title ik = w body ik = tf title ik title k log tf body ik body k log N n title i N n body i Fields in the Vector Space Model Another possibility: weigh terms according to the fields they occur in w ik = tf ik D k log N n i f:i f fw f f k a field weight factor length of field in k If <Ptle> weight is high, terms that appear in Ptle will count for more in cosine similarity If <script> weight is low, terms that appear in script will count for less in cosine similarity 6

Fields in the VSM: IllustraPon Q = t 1 D aoer field weighpng t 1 Document: <Ptle>t1 t1</ptle> <body>t1 t2 t2 t2 t2 t2 t2 t2 t1 t2 t2 t1 t2 t2 </body> D before field weighpng t 2 Formal MoPvaPon As

7 Fields in the VSM: IllustraPon Q = t 1 D aoer field weighpng t 1 Document: <Ptle>t1 t1</ptle> <body>t1 t2 t2 t2 t2 t2 t2 t2 t1 t2 t2 t1 t2 t2 </body> D before field weighpng t 2 Formal MoPvaPon As before, VSM provides no formal mopvapon for sepng weights How do we set field weights? How do we combine field weights? How do we weight terms within fields? etc. ProbabilisPc model can offer formalism, but it is not straightorward to incorporate these features into probabilispc models 7

Combining Evidence Take each term, field, phrase, etc, as pieces of evidence about a document s relevance Combine pieces of evidence into a final score for the document There are formal

8 Combining Evidence Take each term, field, phrase, etc, as pieces of evidence about a document s relevance Combine pieces of evidence into a final score for the document There are formal ways to mopvate the combinapon Inference network model is one approach to combining evidence uses Bayesian network formalism with language model probabilipes Inference Network: Non IR Example 8

9 Inference Network in IR Inference Network Document node (D) corresponds to the event that a document is observed Representa7on nodes (r i ) are document features (evidence) ProbabiliPes associated with those features are based on language models θ espmated using the parameters μ one language model for each significant document structure r i nodes can represent proximity features, or other types of evidence (e.g. date) 9

10 Inference Network Query nodes (q i ) are used to combine evidence from representapon nodes and other query nodes represent the occurrence of more complex evidence and document features a number of combinapon operators are available Informa7on need node (I) is a special query node that combines all of the evidence from the other query nodes network computes P(I D, μ) Example: AND CombinaPon a and b are parent nodes for q 10

11 Example: AND CombinaPon CombinaPon must consider all possible states of parents Some combinapons can be computed efficiently Inference Network Operators 11

12 Inference Network Example Query: #combine( #1( abraham lincoln ) gehysburg ) µ Inference Network Example Query: #weight( 2.0 #or( #1(north korea) iraq ) 1.0 policy ) µ 12

13 Inference Network Example Query: #combine( #uw8( hurricane wind ).(Ptle) damage ) µ Ptle µ A document is viewed as a sequence of text that may contain arbitrary tags A single context is generated for each unique tag name An extent is a sequence of text that appears within a single begin/end tag pair of the same type as the context 13

14 TexPoint Display 14

15 15

16 16

17 17

Search Engine Architecture. Search Engine Architecture

Search Engine Architecture. Search Engine Architecture Search Engine Architecture CISC489/689 010, Lecture #2 Wednesday, Feb. 11 Ben CartereGe Search Engine Architecture A soiware architecture consists of soiware components, the interfaces provided by those