Web consists of web pages and hyperlinks between pages. A page receiving many links from other pages may be a hint of the authority of the page

Transcription

1 Link Analysis

2 Links Web consists of web pages and hyperlinks between pages A page receiving many links from other pages may be a hint of the authority of the page Links are also popular in some other information retrieval applications Citation analysis a document cites references Customer relationship management a customer refers other customers to services Phone call networks a user calls another user J. Pei: Information Retrieval and Web Search -- Link Analysis 2

3 Hyperlinks Hyperlinks are popular on the Web <a href= >Dr. Jian Pei s Web page</a> The hyperlink from A to B heuristically represents an endorsement of page B by the creator of A The anchor text pointing to B is a good description of B J. Pei: Information Retrieval and Web Search -- Link Analysis 3

4 Using Anchor Text Finding summary of web pages Many page authors do not provide a good summary of the page Anchor text often describes the purpose of the links which provide good summary of the target pages Retrieval: the frequent terms/patterns in the anchor text of links to a target page Finding aspects/facets of web pages A page may contain information in many aspects Anchor text of a link is often specific Retrieval: clustering anchor text each significant cluster is a candidate aspect (Abusing) anchor text for spamming Many commercial search engines use anchor text in computing the relevance of a page to a query Creating many links with specific anchor text may boost ranking of a target page in some target queries J. Pei: Information Retrieval and Web Search -- Link Analysis 4

5 Finding Frequently Visited Pages J. Pei: Information Retrieval and Web Search -- Link Analysis 5

6 Random Walks If a page has k out-links, a user has the same probability to visit the next page by following a link What if a page does not have out-links? Teleport: a user jumps to any nodes in the Web graph Even if a page has out-links, a user still may jump (conduct teleport operation) with a probability If we conduct a sufficient number of random walks in a large web graph, the scores on nodes may largely converge after a sufficiently large number of steps the scores reflect the expectations of frequencies that the pages are visited J. Pei: Information Retrieval and Web Search -- Link Analysis

7 Markov Chains A discrete-time stochastic process N states each page corresponds to a state An N x N transition probability matrix P Each entry is in interval [0, ] The entries in each row of P add up to At any give time-step, the Markov chain can be in one of the N states The transition probability of the next state depends on only the current state J. Pei: Information Retrieval and Web Search -- Link Analysis 7

8 Markov Property i, j, P ij [0,] i, N j= P ij = A matrix with non-negative entries that satisfies the Markov Property is called a stochastic matrix A stochastic matrix has a principal left eigenvector corresponding to its largest eigenvalue Transition probability matrix P J. Pei: Information Retrieval and Web Search -- Link Analysis 8

9 Random Walks and Markov Chains Let α be the probability of teleport operations Adjacency matrix A if there is a hyperlink from page i to page j, then A ij = ; otherwise A ij = 0 Transition probability matrix P If a row of A has no s, set each element to /N For each of the other rows Divide each in the row by the number of s in the row Multiply the resulting row by ( α) Add α/n to every entry of the resulting matrix J. Pei: Information Retrieval and Web Search -- Link Analysis 9

10 Measuring Expected Visit Probability A Markov chain is ergodic if there exists a positive integer T 0 such that for all pairs of states i, j in the Markov chain, if it is started at time 0 in state i then for all t > T 0, the probability of being in state j at time t is greater than 0 Irreducibility: there is a sequence of transitions of non-zero probability from any state to any other Aperiodicity: the states are not partitioned into sets such that all state transitions occur cyclically from one set to another J. Pei: Information Retrieval and Web Search -- Link Analysis 0

11 Measuring Expected Visit Probability For any ergodic Markov chain, there is a unique steady-state probability vector π that is the principal left eigenvector of P such that if η(i, j) is the number of visits to state i in t steps, then η( i, t) lim = π ( i) t t π(i) is the steady-state probability for state i The random walk with teleporting results in a unique distribution of steady-state probabilities over the states of the induced Markov chain The steady-state probability of a state is the PageRank of the corresponding Web page J. Pei: Information Retrieval and Web Search -- Link Analysis

12 PageRank Computation Power iteration: let x be a random initial t probability vector, compute x P until it converges (t =, 2, ) Rationale x P When t is large, is very similar to Stimulating random walks t t+ x P J. Pei: Information Retrieval and Web Search -- Link Analysis 2

13 J. Pei: Information Retrieval and Web Search -- Link Analysis 3 Example 2 3 Let α = 0.5 = P ) ( ) ( ) ( (,0,0) x P x x P x x = = = = = =

14 Topic-Specific PageRank Biased random walks: on a page containing k out-links, the probability that a user follows a link is proportional to the probability that the anchor text is relevant to the topic Biased teleporting: jumping to a random Web page chosen non-uniformly: the teleporting probability of a page is proportional to the probability that the page is relevant to the topic J. Pei: Information Retrieval and Web Search -- Link Analysis 4

15 Examples J. Pei: Information Retrieval and Web Search -- Link Analysis 5

16 Personalized PageRank A user may be interested in multiple topics Model the user s interest as a linear combination of a small number of topic page distributions Consequently, the personalized PageRank of a page is the linear combination of the unerlying topic-specific PageRanks J. Pei: Information Retrieval and Web Search -- Link Analysis

17 Example User interest = (0. sports, 0.4 politics) PageRank = 0.π + 0.4π sports politics J. Pei: Information Retrieval and Web Search -- Link Analysis 7

18 Hubs and Authorities What are the useful pages about good restaurants? Authoritative pages: some pages which provide details and well accepted reviews about some restaurants Hub pages: some pages which provide carefully compiled lists of restaurants, possible in category Hubs and authorities A hub points to many authorities An authority is pointed to by many hubs Hyperlink-induced topic search (HITS): computing a hub score and an authority score for each page J. Pei: Information Retrieval and Web Search -- Link Analysis 8

19 Choosing Good Web Pages How can we get a set of Web pages containing good hubs and good authorities on topic good restaurants? Some of those pages may not contain the word good restaurants A two step approach Root set: use a text index to get all pages containing good restaurants Base set: include the root set and the pages that either link to or are linked to by a page in the root set Rationale Including pages linked to by any pages in the root set captures good authorities if the root set contains good hub pages Including pages linking to any pages in the root set captures good hubs if the root set contains good authority pages The method can be used in cross-language retrieval J. Pei: Information Retrieval and Web Search -- Link Analysis 9

20 J. Pei: Information Retrieval and Web Search -- Link Analysis 20 Hub/Authority Score Computation For page v, let h(v) and a(v) be the hub score and the authority score, respectively Use h(v) = a(v) = as the initial values Let A be the adjacency matrix v y y v y h v a y a v h ) ( ) ( ) ( ) ( Aa A a h AA h Aa A a h AA h h A a Aa h T a T h T T T λ λ = = λ h and λ a are the eigenvalues of AA T and A T A, respectively

21 Example J. Pei: Information Retrieval and Web Search -- Link Analysis 2

22 Algorithm J. Pei: Information Retrieval and Web Search -- Link Analysis 22

23 Example Adjacency matrix A h = ( ) a = ( ) J. Pei: Information Retrieval and Web Search -- Link Analysis 23

24 Summary and To-Do List Links are important in Web search Carrying important information that may not be obtained from text Using anchor text Link analysis methods PageRank HITS Read Section 4.5 J. Pei: Information Retrieval and Web Search -- Link Analysis 24