Search engines. Børge Svingen Chief Technology Officer, Open AdExchange

Transcription

1 Search engines Børge Svingen Chief Technology Officer, Open AdExchange

2 Information retrieval (IR) IR: Looking for information in data Much research in IR since the 60s Late 90s: The first Internet search engines: Excite FPTSearch AltaVista AllTheWeb Inktomi HotBot

3 The difference between IR and search engines Information retrieval: Small data sets Homogenous data sets High precision Expert users Search engines: Large data sets Heterogenous data sets Speed Now: IR and search engines are merging

4 Applications of search engines Search engines are now used for three main purposes: Web search E-commerce: Typically online shopping sites. Enterprise search: Searching internal data in businesses.

5 Information retrieval models Three main groups of IR methodologies: Set theoretic models: Documents represented as sets of words. Boolean and fuzzy models are most common. Algebraic models: Documents represented as algebraic constructs, typically vectors. Probabilistic models: Documents are retrieved based probabilistic estimations of relevance.

6 Relevancy Relevancy: The degree to which a document is relevant to a query. Some definitions: A query q. A set of documents D. A relevancy function R(d, q) for a document d and query q. A relevancy cut-off value R 0, R(d, q) > R 0 is good enough. A set of relevant documents D rel where R(d, q) > R 0 for all d D rel. A result set D res.

7 Recall and precision Two important performance measures: Recall tells us how many of the relevant documents that are in the result set. recall = R rel R res R rel. Precision tells us how many of the documents in the result set that are relevant. recall = R rel R res R res.

8 Expanding scope Increasing types of information are used by search engines to calculate relevancy: Traditional IR: Relevancy decided by document content. Web search: Started included other information about documents, i.e. link graphs. Mobile search: Takes into accord the device from which a search is being performed, in order to return only content that can be used by the device. Personalized search: Uses personal interests and behavioral history to give different results to different users.

9 PageRank Use link graph to estimate document quality Query independent. Looks at the graph of links between documents. Assumption: Better documents are linked to by more documents. Disadvantage: Really measures popularity, not quality.

10 Components of a search engine Crawler Connectors Data preparation Indexes Query processing Result processing

11 Crawling The purpose of the crawler is to retrieve content from the web. The problem: No centralized catalog of all web pages. The solution: Start with a number of seed URLs. Retrieve the web pages. Analyze web pages for links, giving new URLs. Repeat... Crawling is difficult...

12 Connectors Retrieves content from different sources: Applications Content management systems servers Databases Etc. Responsible for keeping content up to date.

13 Data preparation Prepare data for indexing: Normalize content Metadata enrichment Categorization Linguistic analysis Etc.

14 Query and result processing Prepares queries for searching, and results for presentation: Normalize content Metadata enrichment Categorization Linguistic analysis Etc. Same as for content...

15 Why indexes How to search terabytes of data? Linear search takes to long. Answer: Use an index. A index is a mapping from a term to the set of documents containing the term.

16 How to index How to choose the right type of index? Many types of indexes available.. Different index types have different space and time complexities. Different index types perform differently for different types of queries. The choice of index types depend on the application: What data to search. What queries will be used. Level of expertise of the users.

17 Suffix trees A suffix tree is a compact suffix trie. A suffix trie is a trie containing all suffixes of a string. Basic observation: Every substring is the prefix of a suffix. Can be built in linear time. Main advantage: Suffix trees allow substring matching. Disadvantage: A suffix can take considerably more space than the original data.

18 Suffix tree example ABC BC C $ 6 ABC$ $ ABC$ $ ABC$ $ Example string: ABCABC$

19 Suffix arrays Suffix arrays are a more efficient implementation of suffix trees. Example string: ABCABC$ ABC$ ABCABC$ BC$ BCABC$ C$ CABC$

20 Inverted files Inverted files looks at individual terms. Each term points to documents containing the term (with positions). Advantage: Creates smaller index than original data set. Disadvantage: Limited queries, no substrings Inverted files often refer to a dictionary instead of actual terms.

21 Inverted files, example Two documents: doc1: This is a test. doc2: So is this. a is so test this (doc1,2) (doc1,1), (doc2,1) (doc2,0) (doc1,3) (doc1,0), (doc2,2)

22 Scaling search engines Search engines need to handle huge scaling requirements. There are two main dimensions in which to scale: Data volume Query volume

23 Scaling linearly In the following, a linearly scalable search architecture will be described. Required hardware is O(data volume) Required hardware is O(query volume)

24 Data partitioning A data collection D is given. On this collection an equivalence relation is defined. From the equivalence classes form a partition P = {D i }. This means that D i, D j (D i P, D j P D i D j = ) and D i P = D. On the subsets of D, a function σ : P (D) N gives a measure of the actual data size.

25 Equivalence relation properties Being an equivalence relation, fulfills the following requirements: (d, d) for all d D (reflexiveness) (d 1, d 2 ) (d 2, d 1 ) (symmetry) (d 1, d 2 ) and (d 2, d 3 ) (d 1, d 3 ) (transitivity)

26 Query definitions A set of queries Q is given. Each q Q is of the form q = {q, P }, where q is a query representation (for instance, query string). P is the subset of P that is of relevance to the query, so that P D.

27 Time distribution of queries It is assumed that the set of queries Q follow a Poisson distribution characterized by the average λ. This means that the probability of k queries arriving during a time unit is equal to P(k) = e λ λ k k! The number of queries arriving in non-overlapping intervals are therefore considered independent.

28 Types of nodes We have four types of nodes: Processing nodes. Query distribution nodes. Result accumulation nodes. Data preprocessing nodes.

29 Distributed architecture

30 Processing nodes The set of nodes N proc is used to solve the set of queries Q. A function φ : N proc P specifies how the data set D is distributed to the set of nodes.

31 Query distribution nodes The set of nodes N distr distributes the query q = {q, P } to the set of processing nodes used to process the query. This set is given by the function δ : Q P (N proc ).

32 Result accumulation nodes Upon completion of the query processing, the results are accumulated by the set of nodes N acc.

33 Data preprocessing nodes In some cases the data D on which the queries will work need to be preprocessed. A set of nodes N pre will serve this task. These nodes will not be discussed further.

34 Problem solving steps To evaluate a query q = {q, P }, the following steps are performed: 1. Distribution. The query q = {q, P } is distributed to the subset δ (q) of N proc δ is chosen so that N proc δ(q) φ (N proc) P. 2. Parallel evaluation. The query q will be evaluated in parallel on the processing nodes δ (q). 3. Result accumulation and merging. Upon completion of the parallel solving process, the results from the processing nodes are accumulated and merged into the final result.

35 Performance specifications Each processing node N N proc is assumed to have the following performance specifications: An average of kproc queries can be handled in a time unit. Up to a data amount of σmax can be handled. It is assumed that is decided so that max Di P σ (D i ) σ max. Each query distribution node N N distr is assumed to be able to distribute queries to up to k distr other nodes. Each result accumulation node N N acc is assumed to be able to accumulate results from up to k acc other nodes.

36 Two-dimensional scalability Processing nodes organized in a matrix: Each column contains a full replica of all data. Each row contains the same data. The distribution and accumulation nodes are organized as trees: Queries are first distributed to columns. Each query goes to a single column. Queries are then distributed rows. Each query goes to all rows in a column.

37 Data distribution tree

38 The matrix N proc,1,1 N proc,1,2 N proc,1,c N proc,2,1 N proc,2,2 N proc,2,c N proc,r,1 N proc,r,2 N proc,r,c

39 Fault tolerance General principles: It is not acceptable that some of the data in D is not available. It is acceptable that the performance goes down until the error is corrected. Simple strategy: Don t use columns with faulty nodes. (Complex topic...)

40 Linear scalability r = c = σ (D) σ max λ k proc N proc = cr N distr = N proc 1 N acc = N proc 1 Somewhat simplified... Assumes worst case conditions, no extra fault tolerance.

41 Linear scalability, proof N = N proc + N distr + N acc = N proc + N proc 1 + N proc 1 = 3 N proc 2 σ (D) = 3 σ max λ k proc 2

42 Pattern Matching Chip (PMC)

43 PMC overview Data Data distribution Pattern Matching Result Processing Match reports

44 The Comparison Element > = not or not = MUX

45 Searching for a string = c = b = a

46 Searching for a regular expression a b a = c = b = a = c = b = a b b a b b b = c = b = a = c = b = a c b b = c = b = a

47 The Processing Element sc i res[i] sc res[i-1] ff out [i] res[i-1] ff out [i+1] res[i] ff in D res[i]

48 Binary distribution tree M[i] : MUXed distribution node : The MUX for PE[i] : Simple distribution node Data source M[0] From neighbouring tree : MUXed PE M[4] : PE shifting data M[6] M[2] Results

49 Implementation of binary distribution tree 0 M 1 M 2 M 3 M RESULTS 7 M 6 M 5 M 4

50 Binary distribution tree with sequence control sc 0 M 1 M 2 M 3 M RESULTS res[i] 7 M 6 M 5 M 4 res[i] sc res[i-1] ff out [i] ff out [i+1] res[i] ff in res[i-1]

51 Larger binary distribution tree with sequence control s c res[i] res[i-1] M s c res[i] res[i] sc M res[i-1] s c R E S U L T S res[i-1] ff out [i] res[i] ff out [i+1] res[i] ff in res[i-1] M s c res[i] res[i-1] res[i-1]

52 The result selector res1 eq1 res2 eq2 sel doc res eq

53 Result selector operations COMPARE [C] Performs alphabetical/numerical comparison (L==R) [==] Compares L and R (L > R) [>] Compares L and R (L R) [ ] Compares L and R L + R [+] Adds L and R ((L+R) C) [ C] Compare (L + R) to C ((L+R) C) [ C] Compare (L + R) to C ((L+R)==C) [==C] Compare (L + R) to C

54 Implementing all boolean functions 1/2 F0 0 A [ 3] B Null F1 (A AND B) A [ 2] B F2 (A AND NOT B) A [>] B F3 (Transfer A) A [+] B B subtree not used, generates always 0 F4 (A NOR NOT B) B [>] A Subtrees swapped F5 (Transfer B) A [+] B A subtree not used, generates always 0 F6 (A XOR B) A [==1] B F7 (A OR B) A [ 1] B

55 Implementing all boolean functions 2/2 F8 (A NOR B) A [ 0] B F9 (A XNOR B) A [==] B Equivalence F10 (NOT B) A [ 0] B A subtree not used, generates always 0 F11 (A OR NOT B) A [ ] B Implication, if B then A else true F12 (NOT A) A [ 0] B B subtree not used, generates always 0 F13 (NOT A OR B) B [ ] A Implication, subtrees swapped, see F11 F14 (A NAND B) A [ 1] B F15 1 A [ 0] B Identity

56 Data distribution tree with result selectors 0 MUX 1 MUX 2 MUX 3 MUX sel sel sel sel sel sel sel 7 MUX MUX MUX 6 5 4

57 sc sc sc sc sc sc sc sc lat lat lat lat lat lat lat lat res res res res sc sc sc sc lat lat lat lat res sc lat res res sc lat Data distribution and result gathering CE0 CE1 CE2 CE3 CE4 CE5 CE6 CE7

58 The end.