Use of LINK information

Transcription

1 Web vs. Ordinar Text Retrieval Inforation Retrieval on the Web Based on lecture aterial b Prof. Weii Meng Web pages are ver voluinous and diversified widel distributed on an servers. extreel dnaic/volatile. Web pages have ore structures (extensivel tagged). are extensivel linked. a often have other associated etadata Web users are ordinar people without special training! the tend to subit short queries. [~.7wrds/ 97->~2.4/ ] There is a ver large user counit. 2 Use of Link Inforation () Use of LINK inforation Hperlinks aong web pages provide new docuent retrieval opportunities. e.g.: Anchor text can be used as index ter for a referenced page The ranking score (siilarit) of a page with a quer can be spread to its neighboring pages. Links can be used to copute the iportance of web pages based on citation analsis. Links can be cobined with a regular quer to find authoritative pages on a given topic. 3 4

2 Initial Idea: Web as a big graph. II. PageRank [Brin&Page] A surfer/pigeon keeps randol clicking on links. The iportance of a page is the probabilit that the surfer finds herself on that page. then rank the returned pages in decreasing order of iportance Coputing PageRank (cont d) PageRank principles: A page has an iportance weight If a page is linked to b an pages, then the page is likel to be iportant. If a page is linked to b iportant pages, then the page is likel to be iportant even though there aren t too an pages linking to it. The iportance of a page is divided evenl and propagated to the pages it points to: Coputing PageRank (cont d) PageRank Definition: For each web page u, let OUT u = the set of pages u points to, IN u = the set of pages that point to u, N u = the nuber pages in OUT u. If all pages equall likel then probabilit R(u) of landing on page u would be R(u) = Σ v IN u / N v since fro each node v onl out of N v edges lead to u. But now all pages are no longer equall likel! Instead, each page gives awa soe of its iportance to its neighbors; define iterative forula R i (u) = Σ ( R i- (v) / N v ) v IN u PageRank can be coputed iterativel: Initialize all page ranks to be /N --- N = nuber of vertices in the Web graph. This gives R (u) 27 Exaple (b Ullan) Equations:! = /3 + a /2! /3 +! = /3 + a /2 Can be rephrased using atrices /3 /2 /3 a /3 /2 Call it atrix M 28

3 Coputing PageRank (cont d) Matrix representation Let M be an N N atrix and M[u,v] be the entr at the u-th row and v-th colun. M[u,v] = /N v if page v has a link to page u M[u,v] = if there is no link fro v to u Let R i be the N rank vector for i-th iteration and R be the initial rank vector (sa for each node) Then R i = M R i- or we can just tr to solve equation R = M R Exaple 2 (b Ullan) Equation R = M R :! = /2 + a /2! /2 +! = a /2 3/2 /2 5/4 9/8 /8 /2 6/5 6/5 3/5 a /2 /2 a /2 / Ullan Ullan PageRank: proble a /2 /2 a /2 /2 Page Rank: proble 2 a /2 /2 a /2 /2 Equation R = M R: = /2 + a /2 /2 = a /2 Equations R = M R: = /2 + a /2 /2 = a /2 + /2 /2 /2 /4 5/8 3/8 /4 /2 3/2 /2 7/4 5/8 3/8 2 3 is a dead-end 3 is a spider trap: once ou enter it, ou never leave, so all other pages end up looking useless - rank! 32

4 Page Rank (called Pigeon Rank in text) A solution to spider trap and other probles Conceptuall, at an point the surfer a randol choose to jup to soe totall different page, rather than follow links fro current page. Let probabilit of staing be d, and hence of juping be -d. This leads to the equation R = d (M R) + ( d) (Actuall, ore coplex forula but don t worr) Ullan Exaple 4 Equations R =.8(M R ) +.2: =.8( /2 + a/2) +.2.8( /2) +.2 =.8(a /2 + ) /2 /2 a /2 /2 a 7/ 5/ 2/ PageRank SUMMARY for ever node v, the probabilit of following one of the links to a neighbor is /(nuber of outgoing links fro v) construct atrix M, where row j entries show for ever other node k the probabilit of following link fro k into j Solve equation like R = d (M R) + ( d) to get the ranks of each node PageRank Coputation Googling expensive but done once for all docs (web pages) Bad aspect: PageRank is quer independent! To ake up for this, cobine IR-stle siilarit easure with global page rank: ranking_score(quer q, page p) = w*si(q, p) + (-w) * R(p), if si(q, p) > where < w < soe constant (epirical) = if si(q,p)= 35 38

5 Use of Link Inforation agerank defines the global iportance of web pages but the iportance is doain/topic independent. e often need to find iportant/authoritative pages which are relevant to a given quer. hat are iportant web browser pages? Which pages are iportant gae pages?! Idea: Use a notion of topic-specific page rank Involves using a non-unifor probabilit 43