CS-C Data Science Chapter 9: Searching for relevant pages on the Web: Random walks on the Web. Jaakko Hollmén, Department of Computer Science

Transcription

1 CS-C Data Science Chapter 9: Searching for relevant pages on the Web: Random walks on the Web Jaakko Hollmén, Department of Computer Science

2 Contents of this chapter Story of the World Wide Web Finding relevant nodes in a network Search Engines 2

3 Short history of the Web : Tim Berners-Lee Figure: In March 1989, Tim Berners-Lee submitted a proposal to CERN for a new information management system. Comment from his superior: "vague but exciting... " 3

4 Short history of the Web : first browsers appear Figure: In the autumn of 1992 info.cern.ch had a page about new servers and pages 4

5 Short history of the Web - Link collections Figure: Jerry and David s Guide to the World Wide Web. Later under the name Yahoo; a large hand-edited list of Web addressses. 5

6 Short history of the Web s search engine AltaVista Figure: Digital created AltaVista to advertise its server: "These can be used to index the whole Web." Altavista made it easy to look for Web pages, but the search result order was quite random. 6

7 Short history of the Web : Google Figure: Google figured out how to sort search results according to usefulness. 7

8 Web search How can we find relevant pages (about e.g. data mining) on the Web? The first methods didn t scale too well: info.cern.ch: a bulletin board about new pages Yahoo: a hand-edited index If our query is data mining how can we automatically find pages where the words appear (or appear sequentially)? How can we figure out which ones are the most relevant pages? 8

9 Web search: Word search String algorithms: many useful and interesting algorithms Preprocessing involves a search of the whole Web page network by following links For every word X, we store a list of pages where X appears When the query is X AND Y we go through the lists for both words, returning the pages which appear on both lists In practice this is trickier than it looks on paper: there is a LOT of data, the answer should be available in under a second A lot of interesting technology, see e.g. 9

10 Web search: Relevance Results 1-10 of about 50,300,000 The old way of solving the problem: the user refines the query by adding suitable words and by using the operators AND, OR, NOT, NEAR How do we pick the pages shown at first? Heuristics: Word frequency (bad idea) A page is a good one if there are many links to it A page is a good reference on X, if X is mentioned in the pages linking to it (a human has already assessed the page) 10

11 Hubs and authorities Hubs and authorities : Jon M. Kleinberg: Authoritative Sources in a Hyperlinked Environment, IBM Research Report 1997; SODA 1998 Good authority pages have many hubs in common A good hub shows good judgement by linking to good authorities 11

12 Search for the sets of best pages and links 1/2 First we look for the pages where X and Y occur Using heuristics, we choose e.g. 200 out of these: let s call this the root set S We form the set T : S + the pages linking to a page in S + the pages linked to by a page in S Figure: Hubs and authorities. Root set S expanded with the pages linking to or linked to by a page in S. 12

13 Search for the sets of best pages and links 2/2 Let s think of the set T as a directed graph: pages are nodes, links are directed edges Let E denote the set of directed edges in T : (u, v) E when page u links to page v The number of links pointing to a page is not a very good metric of relevance 13

14 Hubs and authorities: Principle A good hub points to good authorities Good authorities are pointed to by good hubs Circular definition? 14

15 Computing good hubs and authorities iteratively 1/2 We can get rid of the circular definition with an iterative algorithm (compare with c-means) Each page s in T is given a hub weight h s and an authority weight a s We need to initialize these somehow: h s = k s = frac1 n, s T Iterative update rules: a s h s h t t T,(t,s) E a t t T,(s,t) E 15

16 Computing good hubs and authorities iteratively 2/2 The authority weight of page s is the sum of hub weights h s of the pages that link to s. The hub weight of page s is similarly the sum of authority weights a s of the pages linked to by s. After each iteration the squared sums of the weights are scaled to 1: as 2 = 1, hs 2 = 1 s T s T 16

17 Computing good hubs and authorities with matrices 1/3 The iterative algorithm can easily be rewritten for matrices Let s think of the weights as vectors a = (a s ) s T, H = (h s ) s T and the graph as the matrix M M = (1if(s, t) E, otherwise0) Now the iterative update rules can be rewritten like this: a M T H, H Ma 17

18 Computing good hubs and authorities with matrices 2/3 With the scaling added, the complete iteration rule becomes a M T H M T H, H Ma Ma Let s do a few iterations with 1 for initialization: a 1 M T 1 M T 1 H 1 M1 M1 { a 1... MM T 1 H 1... M T M1 { a 1... M T MM T 1 H 1... MM T M1 After i iterations we have a = 1... (M T M) i 1 18

19 Computing good hubs and authorities with matrices 3/3 The matrix M T M is symmetric, which means its eigenvalues and eigenvectors are real and it can be diagonalized M t M = VD n V T where D = diag(λ 1,..., λ n ) With the technical assumption λ 1 > λ 2 we have (M T M) i 1 = VD i V T 1 V diag(λ i 1, 0,..., 0) V T 1 = (λ i 1v T 1 1)v 1 Now a is the eigenvector of M t M corresponding to the largest eigenvalue Likewise H is an eigenvector of MM t In Matlab the eigenvectors are found with [V,D] = eig(m *M) 19

20 Results from the Kleinberg paper Figure: Authorities. 20

21 PageRank Sergey Brin and Larry Page, 1998 (Google) The Anatomy of a Large-Scale Hypertextual Web Search Engine A page is relevant if relevant pages link to it Circular definition? (Again, an iterative algorithm is used) 21

22 PageRank: Computation 1/5 If page t links to pages s 1, s 2,..., s k, the relevance r of page t is distributed among these with the weight 1/k Figure: The four pages of a universe with their present relevances, links, and distributed relevances. 22

23 PageRank: Computation 2/5 Now we have a relevance vector ( PageRank ) r Fr, where the matrix F is defined by { 1/deg(t), iftlinks tos F s,t = 0, otherwise where deg(t) is the number of links (h) With suitable assumptions r is the eigenvector of F corresponding to the largest eigenvalue, so the iteration converges Another interpretation: A random surfer starts with a page and follows random links. The relevance of a page is the probability that the surfer will end up on the page (over a long period of time). What if... 23

24 PageRank: Computation 3/5 Figure: The last page of the Internet! 24

25 PageRank: Computation 4/5 If a page has no links, we add links to everywhere. Here n is the number of pages. 1/deg(t), if t links to s F s,t = 1/n, if t has no outbound links 0, otherwise Let s add links to everywhere anyway: the random surfer is allowed to be bored sometimes (and we avoid technical difficulties in the convergence proof) 1 ϵ + ϵ, if t links to s deg(t) n F s,t = 1/n, if t has no outbound links ϵ/n, otherwise 25

26 PageRank: Computation 5/5 Figure: AltaVista users looked at results on several pages. Google users only look at the first few results. 26

27 Summary In this chapter we looked at finding relevant Web pages by using the links between pages. The idea of the hubs and authorities algorithm is that good hubs link to good authorities and correspondingly good authorities are linked by good hubs. The PageRank algorithm computes page relevance like a random surfer. The Google search engine uses PageRank. 27