Boolean Retrieval. Manning, Raghavan and Schütze, Chapter 1. Daniël de Kok

Transcription

1 Boolean Retrieval Manning, Raghavan and Schütze, Chapter 1 Daniël de Kok

2 Boolean query model Pose a query as a boolean query: Terms Operations: AND, OR, NOT Example: Brutus AND Caesar AND NOT Calpuria Return documents in which the terms (do not) appear. Each document viewed as a set of words.

3 Terminology Document: basic unit of information that we want to return as a result of a query. For example: Newspaper article Wikipedia page A Shakespeare book Collection: the group of documents that we will perform retrieval on. For example: Newspaper archive Wikipedia Shakespeare s collected works Term: the smallest unit of information in a query. For example: Token Lemma Compound (e.g. New York)

4 Relation to set theory In information retrieval, these operators are strongly related to set theory. When: T : the set of terms C: the document collection D : T P, where P C t T, t 1 T, t 2 T Then: t 1 AND t 2 : D(t 1 ) D(t 2 ) t 1 OR t 2 : D(t 1 ) D(t 2 ) NOT t: C \ D(t) t 1 AND NOT t 2 : D(t 1 ) \ D(t 2 )

5 Example: search

6 Example: library search

7 Example: library search

8 Why not use grep? Why don t we just use a tool such as grep? $ egrep -r '(Brutus Caesar)' docs egrep -v 'Calpuria'

9 Why not use grep? Why don t we just use a tool such as grep? $ egrep -r '(Brutus Caesar)' docs egrep -v 'Calpuria' Disadvantages for information retrieval: Slow: linear in the number of lines in the document collection. Inflexible matching: does not support more flexible matching operations. Inflexible ranking: does not support ranking of documents.

10 Incidence matrices

11 Incidence matrix Anthony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth... Anthony Brutus Caesar Calpurnia Cleopatra mercy worser

12 Incidence matrix Anthony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth... Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Brutus AND Caesar AND NOT Calpurnia:

13 Incidence matrix Anthony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth... Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Brutus AND Caesar AND NOT Calpurnia: & & ~010000

14 Incidence matrix Anthony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth... Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Brutus AND Caesar AND NOT Calpurnia: & & ~ & & =

15 Incidence matrix Anthony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth... Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Brutus AND Caesar AND NOT Calpurnia: & & ~ & & = Result: {Anthony and Cleopatra, Hamlet}

16 Incidence matrix The use of a small incidence matrix is fast: Look up the bit vector of the relevant terms. Apply bitwise operators that correspond with the search operators.

17 Incidence matrix The use of a small incidence matrix is fast: Look up the bit vector of the relevant terms. Apply bitwise operators that correspond with the search operators. Not practical for large document collections. Consider for instance the following collection: N = 1, 000, 000 documents M = 500, 000 distinct terms Requires an incidence matrix {0, 1} M N

18 Incidence matrix The use of a small incidence matrix is fast: Look up the bit vector of the relevant terms. Apply bitwise operators that correspond with the search operators. Not practical for large document collections. Consider for instance the following collection: N = 1, 000, 000 documents M = 500, 000 distinct terms Requires an incidence matrix {0, 1} M N Too large: half a trillion cells! Linear time processing of boolean operators: O(N)

19 Observation Suppose that the average document size is 1000 terms. We expect the number of unique terms per document to be (much) smaller than The incidence matrix is very sparse: mostly zeroes. Suggests that a better representation is possible.

20 Inverted indices

21 The inverted index Brutus Caesar Calpurnia Dictionary Postings

22 The inverted index Brutus Caesar Calpurnia Dictionary Postings Every term in a dictionary has a pointer to a postings list. A postings list is a sorted list of document identifiers without duplicates. A document identifier is an integer that we assign to a document while creating the inverted index. The set of all postings lists is sometimes referred to as postings.

23 Advantages of an inverted index Brutus Caesar Calpurnia Dictionary Postings

24 Advantages of an inverted index Brutus Caesar Calpurnia Dictionary Postings Smaller size: the dictionary is of size M. However, the size of a postings list for a term is the number of documents in which the term occurs. Faster: evaluation of boolean expressions is not O(N) for most interesting query terms.

25 Data structure for postings list Candidate data structures for postings lists: Sorted linked list Sorted dynamic-length array Balanced binary search tree Hash set

26 Sorted linked list Advantages: Easily extended with skip lists Disadvantages: O(n) search O(n) insertion Pointer-heavy Non-trivial memory disk mapping Not fit for compression

27 Sorted dynamic-length array Advantages: O(log(n)) search No pointers Non-trivial memory disk mapping Easy compression Disadvantages: O(n) average/worst-case insertion

28 Balanced binary search tree Advantages: O(log(n)) search O(log(n)) insertion Disadvantages: Pointer-heavy Not easily extended with skip lists Non-trivial memory disk mapping Not fit for compression

29 Hash set Advantages: O(1) search O(1) amortized insertion Disadvantages: Pointer-heavy (separate chaining). Not easily extended with skip lists Non-trivial memory disk mapping Not fit for compression

30 Query processing

31 Boolean model recap Pose a query as a boolean query: Terms Operators: AND, OR, NOT Example: Brutus AND Caesar AND NOT Calpuria

32 Looking up a term Query: Brutus

33 Looking up a term Query: Brutus Look up Brutus in the inverted index. Return the associated postings list.

34 term1 AND term2 Query: Brutus AND Caesar

35 term1 AND term2 Query: Brutus AND Caesar Retrieve the postings list for Brutus. Retrieve the postings list for Caesar. Compute the intersection of the posting lists. (How?)

36 AND of two posting lists (naive) The AND operator should produce the intersection of two postings lists. This is easy to do using a naive algorithm: let mut inter = Vec::new(); for doc1 in &self.docs { for doc2 in &other.docs { if doc1 == doc2 { inter.push(*doc1); break; } } }

37 AND of two posting lists (naive) The AND operator should produce the intersection of two postings lists. This is easy to do using a naive algorithm: let mut inter = Vec::new(); for doc1 in &self.docs { for doc2 in &other.docs { if doc1 == doc2 { inter.push(*doc1); break; } } } Complexity: O(nm), where n and m are the postings lists sizes of the operands.

38 AND of two postings lists (better) Given that the postings lists are sorted, we can easily do better. Traverse both lists synchronously.

39 AND of two postings lists let mut inter = Vec::new(); let mut p1i = 0; let mut p2i = 0; while p1i!= self.docs.len() && p2i!= other.docs.len() { let doc1 = self.docs[p1i]; let doc2 = other.docs[p2i]; } if doc1 == doc2 { inter.push(doc1); p1i += 1; p2i += 1; } else if doc1 < doc2 { p1i += 1; } else { p2i += 1; }

40 AND of two postings lists let mut inter = Vec::new(); let mut p1i = 0; let mut p2i = 0; while p1i!= self.docs.len() && p2i!= other.docs.len() { let doc1 = self.docs[p1i]; let doc2 = other.docs[p2i]; } if doc1 == doc2 { inter.push(doc1); p1i += 1; p2i += 1; } else if doc1 < doc2 { p1i += 1; } else { p2i += 1; } Complexity: O(n + m)

41 AND of two postings lists (D(t 1 ) D(t 2 )) What if D(t 1 ) D(t 2 )? Consider the following lists: Postings list 1: 8 Postings list 2:

42 AND of two postings lists (D(t 1 ) D(t 2 )) What if D(t 1 ) D(t 2 )? Consider the following lists: Postings list 1: 8 Postings list 2: Not senseful to traverse the second list: use binary search. Complexity: O(n + n log(m))

43 AND of two postings lists (D(t 1 ) D(t 2 )) let mut inter = Vec::new(); let (smaller, larger) = min_max_posting(self, other); let mut offset = 0; for doc in &smaller.docs { offset = match larger.docs[offset..].binary_search(doc) { Ok(idx) => { inter.push(*doc); idx } Err(idx) => idx, } } Posting { docs: inter }

44 term1 OR term2 Query: Brutus OR Caesar

45 term1 OR term2 Query: Brutus OR Caesar Retrieve the postings list for Brutus. Retrieve the postings list for Caesar. Compute the union of the postings lists.

46 term1 OR term2 Query: Brutus OR Caesar Retrieve the postings list for Brutus. Retrieve the postings list for Caesar. Compute the union of the postings lists. Note: The intersection should be ordered again. consider: (Brutus OR Caesar) AND Calpuria

47 OR of two postings lists let mut result = Vec::new(); let mut p1i = 0; let mut p2i = 0; while p1i!= self.docs.len() && p2i!= other.docs.len() { let doc1 = self.docs[p1i]; let doc2 = other.docs[p2i]; } if doc1 == doc2 { result.push(doc1); p1i += 1; p2i += 1; } else if doc1 < doc2 { result.push(doc1); p1i += 1; } else { result.push(doc2); p2i += 1; }

48 OR of two postings lists (continued) if p1i!= self.docs.len() { result.extend_from_slice(&self.docs[p1i..]); } if p2i!= other.docs.len() { result.extend_from_slice(&other.docs[p2i..]); }

49 NOT term Query: NOT Caesar

50 NOT term Query: NOT Caesar Retrieve the postings list for Caesar. Construct a list with all documents. Remove the postings for Caesar from the list of all documents.

51 NOT term Query: NOT Caesar Retrieve the postings list for Caesar. Construct a list with all documents. Remove the postings for Caesar from the list of all documents. Inefficient! Not frequently used in isolation.

52 NOT term Query: Brutus AND NOT Caesar

53 NOT term Query: Brutus AND NOT Caesar Evaluation: Retrieve the postings list for Brutus. Retrieve the postings list for Caesar. Compute the difference between the postings lists for Brutus and Caesar

54 Difference of two postings lists let mut result = Vec::new(); let mut p1i = 0; let mut p2i = 0; while p1i!= self.docs.len() && p2i!= other.docs.len() { let doc1 = self.docs[p1i]; let doc2 = other.docs[p2i]; } if doc1 == doc2 { p1i += 1; p2i += 1; } else if doc1 < doc2 { result.push(doc1); p1i += 1; } else { p2i += 1; }

55 Difference of two postings lists (continued) if p1i!= self.docs.len() { result.extend_from_slice(&self.docs[p1i..]); }

56 Difference of two postings lists (binary search) let mut inter = Vec::new(); let mut offset = 0; for doc in &self.docs { offset = match other.docs[offset..].binary_search(doc) { Ok(idx) => idx, Err(idx) => { inter.push(*doc); idx } } } Posting { docs: inter }

57 Query optimization

58 Introduction A query can often be processed more quickly by: Changing the processing order; and/or rewriting the query.

59 Processing order Consider the query: Brutus AND Caesar AND Calpuria The AND operator is associative and commutative

60 Processing order Consider the query: Brutus AND Caesar AND Calpuria The AND operator is associative and commutative Different evaluation orders possible: (Brutus AND Caesar) AND Calpuria Brutus AND (Caesar AND Calpuria) (Brutus AND Calpuria) AND Caesar

61 Processing order Consider the query: Brutus AND Caesar AND Calpuria The AND operator is associative and commutative Different evaluation orders possible: (Brutus AND Caesar) AND Calpuria Brutus AND (Caesar AND Calpuria) (Brutus AND Calpuria) AND Caesar The different orders evaluate to the same result, but could not be equally efficient.

62 Processing order Consider the query: Brutus AND Caesar AND Calpurnia Should we process this as: (Brutus AND Caesar) AND Calpurnia; or (Calpurnia AND Brutus) AND Caesar?

63 (Brutus AND Caesar) AND Calpurnia Brutus Caesar Calpurnia Dictionary Postings Let s assume that there are 20 documents containing Caesar. Documents processed: 1. (Brutus AND Caesar): steps = 28 steps 2. 1 AND Calpurnia: up to 8 steps + 4 steps = up to 12 steps Total: up to 40 steps

64 (Calpurnia AND Brutus) AND Caesar Brutus Caesar Calpurnia Dictionary Postings Documents processed: 1. (Calpurnia AND Brutus): steps = 12 steps 2. 1 AND Caesar: up to 4 steps + 20 steps = up to 24 steps Total: up to 36 steps

65 General rule for intersection Process postings lists from small to large. Leads to the smallest intermediate postings lists and therefore likely the least amount of work. If the intermediate postings lists are small, intersection with a binary search pays off.

66 Estimating processing with ORs Consider the query: (madding OR crowd) AND (ignoble OR strife) AND (killed OR slain). In what order can we process the conjuncts? Approximate the disjunction sizes by summing the size of the disjuncts.

67 Query rewriting Consider the query (Exercise 1.5): (Brutus OR Caesar) AND NOT (Antony OR Cleopatra)

68 Query rewriting Consider the query (Exercise 1.5): (Brutus OR Caesar) AND NOT (Antony OR Cleopatra) Not ideal: we first form the larger sets for: Brutus OR Caesar and Antony OR Cleopatra before computing set differences

69 Query rewriting General heuristic: Disjunction high in the expresion tree. Conjunction low in the expression tree. Use De Morgan and distributive laws: (Brutus OR Caesar) AND NOT (Antony OR Cleopatra)

70 Query rewriting General heuristic: Disjunction high in the expresion tree. Conjunction low in the expression tree. Use De Morgan and distributive laws: (Brutus OR Caesar) AND NOT (Antony OR Cleopatra) (Brutus OR Caesar) AND NOT Antony AND NOT Cleopatra

71 Query rewriting General heuristic: Disjunction high in the expresion tree. Conjunction low in the expression tree. Use De Morgan and distributive laws: (Brutus OR Caesar) AND NOT (Antony OR Cleopatra) (Brutus OR Caesar) AND NOT Antony AND NOT Cleopatra (Brutus AND NOT Antony AND NOT Cleopatra) OR (Caesar AND NOT Antony AND NOT Cleopatra)