Search Engines. Gertjan van Noord. September 17, 2018

Size: px

Start display at page:

Download "Search Engines. Gertjan van Noord. September 17, 2018"

Felicia Green
5 years ago
Views:

1 Search Engines Gertjan van Noord September 17, 2018

2 About the course Information about the course is available from:

3 Last week Normalization (case, diacritics, stemming, decompounding,... ) Posting List intersection with Skip Pointers Phrase Queries Posting List with positions

4 Promised: decompounding is harder than you think omroepers paperassen rotspartij zonnestroom plantenteelt uitslover kredietverstrekkers

5 Promised: decompounding is harder than you think omroepers paperassen rotspartij zonnestroom plantenteelt uitslover kredietverstrekkers om roe pers paper assen rot spar tij zon nest room plan tent eelt uit s lover krediet verstrek kers

6 Book Exercises

7 This week: Tolerant Retrieval Wildcard queries Spell correction Alternative indexes Finding the most similar terms

8 Wildcard queries: * mon*: find documents with words that starts with mon. *mon: find documents with words that end with mon. Harder. mo*n: find documents with words that start with mo and end with n. harder. Even m*o*n: Yet harder.

9 Wildcard queries Step 1: find all terms that fall within the wildcard definition Step 2: find all documents containing any of these terms

10 Wildcard queries Step 1: find all terms that fall within the wildcard definition B-trees Permuterm index K-gram index Step 2: find all documents containing any of these terms

11 Dictionary structures Hash. Very efficient lookup and construction, but a hash cannot be used to find terms that are close to the key. Python dictionaries are implemented by hashes. Binary tree, B-tree, Tries. Data-structures in which data is kept sorted (and balanced). Fairly efficient search, but more costly to construct. Words with same suffix are close together in the result, and therefore these structures can potentially be used for tolerant retrieval.

12 Hash

13 Binary tree

14 B-tree Extension of binary tree in which the tree remains balanced

15 Trie

16 Wildcard queries: * mon*: Easy with B-tree, easy with trie. *mon: Maintain additional B-tree or trie for all words in reverse mo*n: Intersect mo* and *n. Use reverse tree for *n. m*o*n:??

17 Wildcard queries: * mon*: Easy with B-tree, easy with trie. *mon: Maintain additional B-tree or trie for all words in reverse mo*n: Intersect mo* and *n. Use reverse tree for *n. m*o*n:?? Use permuterm index or K-gram index

18 K-gram index K-gram: group of K consecutive items. Here: characters. For example, if K=3, the K-gram index has keys of three consecutive characters. The key points to all terms which contain that sequence of three characters. Index for dictionary lookup, not for document retrieval. In a k-gram index, a key points to all relevant search terms.

19 Split words in K-grams, K=3 kitchen \/ $kitchen$ \/ $ki kit itc tch che hen en$

20 K-gram index, K=3 In a k-gram index, a key points to all relevant terms. $ki en$ che ink itt kit ==> {kinkiten kitchen kitten} ==> {kinkiten kitchen kitten kzen} ==> {bitch kitchen witch} ==> {kinkiten kinky} ==> {bitter kitten} ==> {kinkiten kitchen kitten}

21 K-gram index, K=3 In a k-gram index, a key points to all relevant terms. $ki en$ che ink itt kit ==> {kinkiten kitchen kitten} ==> {kinkiten kitchen kitten kzen} ==> {bitch kitchen witch} ==> {kinkiten kinky} ==> {bitter kitten} ==> {kinkiten kitchen kitten} The terms are sorted (why?)

22 K-gram index for wildcard queries Initial Query: kit*en Mapped to: $kit*en$ Search in K-gram index: $ki AND kit AND en$

23 K-gram index for wildcard queries Initial Query: kit*en Mapped to: $kit*en$ Search in K-gram index: $ki AND kit AND en$ Result: kinkiten kitchen kitten Postprocessing required: kinkiten

24 K-gram index for wildcard queries Initial Query: kit*en Mapped to: $kit*en$ Search in K-gram index: $ki AND kit AND en$ Result: kinkiten kitchen kitten Postprocessing required: kinkiten The remaining terms are used in OR query: kitchen OR kitten

25 Query processing What to do for this query: se*ate AND fil*er Expand se*ate to OR-query, e.g., selfhate OR seagate Expand fil*er to OR-query, e.g., filter OR filler Combine into ((selfhate OR seagate) AND (filter OR filler))

26 Spell Correction for Query Terms If a query term is not present in the term index (or if it is very rare)... Find similar terms Calculate similarity to the query term Use most similar one(s) Use most frequent one(s)

27 Spell Correction for Query Terms Find similar terms: K-gram index Calculate similarity to the query term: Jaccard, Levenshtein Use most similar one(s) Use most frequent one(s)

28 Spell Correction for Query Terms Find similar terms: K-gram index For instance: a term t 1 is similar to t 2 if one of the 3-grams of t 1 and t 2 are identical. For unknown term t 1, collect all of the terms in the 3-gram index of all 3-grams. Lots of candidates, only use good ones?

29 Spell Correction for Query Terms Query: brook $bro broek, brok, brommen, brons roo roomijs,vuurrood,brood,rook ook wierookstaafjes,stookolie,brood,rook ok$ werknemersblok,varkenshok,rook

30 Spell Correction for Query Terms Query: brook $bro broek, brok, brommen, brons roo roomijs,vuurrood,brood,rook ook wierookstaafjes,stookolie,brood,rook ok$ werknemersblok,varkenshok,rook From all those, only select the ones that are close to the original term

31 Spell Correction for Query Terms Calculate similarity to the query term Jaccard Jaccard coefficient: A B A B A: trigrams in term t 1 B: trigrams in term t 2

32 Jacard A B A B A: brook: $br,bro,roo,ook,ok$ B: rook: $ro,roo,ook,ok$

33 Jacard A B A B A: brook: $br,bro,roo,ook,ok$ B: rook: $ro,roo,ook,ok$ A B: roo,ook,ok$ A B: $br,bro,roo,ook,ok$,$ro

34 Jacard A B A B A: brook: $br,bro,roo,ook,ok$ B: rook: $ro,roo,ook,ok$ A B: roo,ook,ok$ A B: $br,bro,roo,ook,ok$,$ro Jaccard: 3/6 = 0.5

35 More precise Minimum Edit Distance Levenshtein Distance

36 Levenshtein Distance Distance between A and B: Minimum number of insertions, deletions or substitutions to map A to B

37 Levenshtein Distance Distance between A and B: Minimum number of insertions, deletions or substitutions to map A to B A: brook B: rook

38 Levenshtein Distance Distance between A and B: Minimum number of insertions, deletions or substitutions to map A to B A: brook B: rook Distance: 1

39 Levenshtein distance bakker brak otter boter bloed bode ondersteboven binnenstebuiten

40 Efficient Algorithm try out all possibilities? No. First compute Levenshtein distance for all prefixes Dynamic programming Suppose we need to compute distance for: ondersteboven,binnenstebuiten and we are given the following: dist(onderstebove,binnenstebuite) = 7 dist(onderstebove,binnenstebuiten) = 8 dist(ondersteboven,binnenstebuite) = 8

41 Efficient Algorithm x and y are strings a and b are symbols Suppose we need to compute distance for dist(xa,yb) and we have dist(x,y) dist(xa,y) dist(x,yb)

42 Efficient Algorithm? dist(xa,yb) and we have dist(x,y) dist(xa,y) dist(x,yb)

43 Efficient Algorithm cost(a,b): 0 if a==b; 1 otherwise There are three ways to construct xa,yb: dist(x,y) + cost(a,b) (substitution) dist(xa,y) + 1 (insdel) dist(x,yb) + 1 (insdel)

44 Efficient Algorithm cost(a,b): 0 if a==b; 1 otherwise There are three ways to construct xa,yb: dist(x,y) + cost(a,b) (substitution) dist(xa,y) + 1 (insdel) dist(x,yb) + 1 (insdel) Take the minimum

45 # b l o e d # b o d e Efficient Algorithm: matrix

46 # b l o e d # b 1 o d e Efficient Algorithm: matrix

47 # b l o e d # b o d e Efficient Algorithm: matrix

48 Efficient algorithm: matrix Each cell in the matrix represents the distance between the corresponding prefixes The final result, therefore, can be found in... Other cost functions can be possible too E.g., substitutions for characters that are pronounced similarly could be given lower cost Sometimes, other basic edit operations can be considered (e.g. transposition)

Tolerant Retrieval. Searching the Dictionary Tolerant Retrieval. Information Retrieval & Extraction Misbhauddin 1

Tolerant Retrieval. Searching the Dictionary Tolerant Retrieval. Information Retrieval & Extraction Misbhauddin 1 Tolerant Retrieval Searching the Dictionary Tolerant Retrieval Information Retrieval & Extraction Misbhauddin 1 Query Retrieval Dictionary data structures Tolerant retrieval Wild-card queries Soundex Spelling