Web Information Retrieval Exercises Boolean query answering. Prof. Luca Becchetti

Transcription

1 Web Information Retrieval Exercises Boolean query answering Prof. Luca Becchetti

2 Material rif 3. Christopher D. Manning, Prabhakar Raghavan and Hinrich Schueze, Introduction to Information Retrieval, Cambridge University Press,

3 Bigger corpora Consider n = 1M documents, each with about 1K terms. Avg 6 bytes/term incl spaces/punctuation 6GB of data in the documents. Say there are m = 500K distinct terms among these.

4 Can t build the matrix 500K x 1M matrix has half-a-trillion 0 s and 1 s. But it has no more than one billion 1 s. matrix is extremely sparse. What s a better representation? We only record the 1 positions. Why?

5 Inverted index Brutus Calpurnia Caesar Dictionary Sorted by docid Postings

6 Inverted index/2 Consider the following documents: Doc 1 new home sales top forecasts Doc 2 home sales rise in july Doc 3 increase in home sales in july Doc 4 july new home sales rise Build the corresponding inverted index

7 The merge Walk through the two postings simultaneously, in time linear in the total number of postings entries Brutus Caesar If the list lengths are m and n, the merge takes O(m+n) operations. Crucial: postings sorted by docid.

8 Merge algorithm Ex: Term 0 AND Term 1 Index i 0 traverse Post 0 [0,,length 0-1] Index i 1 traverse Post 1 [0,,length 1-1] i 0 =i 1 =0 Do While i 0 <length 0 and i 1 <length 1 { If Post 1 (i 1 ) = Post 0 (i 0 ) } then hit!; i 0 =i 0 +1; i 1 =i 1 +1 else If Post 1 (i 1 ) < Post 0 (i 0 ) then i 1 =i 1 +1 else i 0 =i 0 +1

9 Ex. 1: More general merges What about the following queries: Brutus AND NOT Caesar Brutus OR NOT Caesar Can we still run through the merge in time O(m+n)?

10 Ex. 1: Term 0 AND NOT Term 1 How should the Boolean query x AND NOT y be handled? What is the cost of naive evaluation of this query? Write out a postings merge algorithm that evaluates this query efficiently

11 Ex. 1: Term 0 AND NOT Term 1 How should the Boolean query x AND NOT y be handled? What is the cost of naive evaluation of this query? Write out a postings merge algorithm that evaluates this query efficiently Cost of naïve evaluation: O(N), with N = max #docids

12 Ex. 1: Term 0 AND NOT Term 1 Index i 0 traverse Post 0 [0,,length 0-1] Index i 1 traverse Post 1 [0,,length 1-1] i 0 =i 1 =0 Do While i 0 <length 0 and i 1 <length 1 If Post 1 (i 1 ) > Post 0 (i 0 ) then hit Post 0 (i 0 )! ; i 0 =i 0 +1 else If Post 1 (i 1 ) = Post 0 (i 0 ) then i 0 =i 0 +1; i 1 =i 1 +1 else i 1 =i 1 +1 } Do While i 0 <length 0 hit Post 0 (i 0 )! ; i 0 =i 0 +1

13 Digression: Term 0 OR Term 1

14 Digression: Term 0 OR Term 1 Index i 0 traverse Post 0 [0,,length 0-1] Index i 1 traverse Post 1 [0,,length 1-1] i 0 =i 1 =0 Do While i 0 <length 0 and i 1 <length 1 If Post 1 (i 1 ) >Post 0 (i 0 ) then hit Post 0 (i 0 ); i 0 =i 0 +1; else if Post 1 (i 1 ) = Post 0 (i 0 ) then hit Post 1 (i 1 )]; i 0 =i 0 +1; i 1 =i 1 +1 else hit Post 1 (i 1 ); i 1 =i 1 +1 Do While i 0 <length 0 hit Post 0 (i 0 ); i 0 =i 0 +1; Do While i 1 <length 1 hit Post 1 (i 1 ); i 1 =i 1 +1; At most one of the while cycles is executed

15 Ex. 1: Term 0 OR NOT Term 1 Index i 0 traverse Post 0 [0,,length 0-1] Index i 1 traverse Post 1 [0,,length 1-1] i 0 =i 1 =0 Do While i 0 <length 0 and i 1 <length 1 If Post 1 (i 1 ) >Post 0 (i 0 ) then i 0 =i 0 +1; else if Post 1 (i 1 ) =Post 0 (i 0 ) then hit (Post 1 (i 1-1), Post 1 (i 1 )]! i 0 =i 0 +1; i 1 =i 1 +1 else hit (Post 1 (i 1-1), Post 1 (i 1 ))! ; i 1 =i 1 +1 } Do While i 1 <length 1 hit (Post 1 (i 1-1), Post 1 (i 1 ))! ; i 1 =i 1 +1 hit(post 1 (length 1-1), maxdocid)!;

16 Merging What about an arbitrary Boolean formula? (Brutus OR Caesar) AND NOT (Antony OR Cleopatra) Rewrite the above query in disjunctive normal form Could disjunctive normal form be better in this case? Provide arguments in favour or against Can we always merge in linear time? Can we do better?

17 Merging Disjunctive normal form: (Brutus AND NOT Antony AND NOT Cleopatra) OR (Caesar AND NOT Antony AND NOT Cleopatra ) Disjunctive normal form might be more efficient in this case Reason: we perform intersections first and only one union at the end, on two postings list that (hopefully) are small In general: we can always merge in O(qN) time, where q is the number of query terms and N is the number of documents Θ(N) is a worst case lower (recall OR NOT)

18 Query optimization What is the best order for query processing? Consider a query that is an AND of t terms. For each of the t terms, get its postings, then AND together. Brutus Calpurnia Caesar Query: Brutus AND Calpurnia AND Caesar

19 Query optimization example Process in order of increasing freq: start with smallest set, then keep cutting further. This is why we kept freq in dictionary Brutus Calpurnia Caesar Execute the query as (Caesar AND Brutus) AND Calpurn

20 More general optimization e.g., (madding OR crowd) AND (ignoble OR strife) Get freq s for all terms. Estimate the size of each OR by the sum of its freq s (conservative). Process in increasing order of OR sizes.

21 Exercise Recommend a query processing order for (tangerine OR trees) AND (marmalade OR skies) AND (kaleidoscope OR eyes) Term Freq eyes kaleidoscope marmalade skies tangerine trees

22 Exercise For a conjunctive query, is processing postings lists in order of size guaranteed to be optimal? Explain why it is, or give an example where it isn t

23 Exercise For a conjunctive query, is processing postings lists in order of size guaranteed to be optimal? Explain why it is, or give an example where it isn t A.: this order is not necessarily optimal. Students should provide a counterexample

24 Query processing exercises If the query is friends AND romans AND (NOT countrymen), how could we use the freq of countrymen?

25 Query processing exercises If the query is friends AND romans AND (NOT countrymen), how could we use the freq of countrymen? If countrymen were a very frequent terms (say, it appears in at least 1/2 of the documents) then it might be worth computing the list corresponding to NOT countrymen and then processing in increasing postings list size order

26 Query processing Excercise Can you process the query with only one traversal if all posting lists are in main memory? Ex: Term 0 AND Term 1. AND Term n-1 Index i k traverse Post k [0,,length k -1] i k =0, k=1,..,n k=1 Do While i k-1mod n <length k-1mod n Do While (Post(i k ) <Post(i k-1 mod n ) AND i k < length k ) i k =i k +1 If Post k (i k ) = Post k-1 (i k-1 mod n ) = =Post k-n+1mod n (i k-n+1 mod n ) then hit! i k =i k +1, k=1,..,n else k=k+1 mod n

27 Query processing exercises Process in linear time a CNF formula: (C 11 OR C OR C 1k1 ) AND..AND (C n1 OR C n2 OR C nkn ) Algorithm: If C ij = NOT Term then use the Doc id intervals not containing Term while traversing the posting list of Term For each (C i1 OR C i2... OR C iki ) implicitely consider the posting interval list I i union of the intervals for every Term C ij while traversing the posting lists Find Doc ids contained in all intervals I 1,.,I n Need all posting lists in main memory at the same time.

28 Resources for today s lecture Introduction to Information retrieval, Chapter 1 Shakespeare: Try the neat browse by keyword sequence feature!