CS535 Big Data Fall 2017 Colorado State University 9/5/2017. Week 3 - A. FAQs. This material is built based on,

Transcription

1 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week 3--1 S535 IG T FQs Programming ssignment 1 We will discuss link analysis in week3 Installation/configuration guidelines for Hadoop and Spark are uploaded Port assignment has been posted PRT 1. TH OMPUTING MOL FOR IGTNLYTIS 2. W SL LINK NLYSIS TP teams ny change? Please let me know omputer Science, olorado State University 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week 3--3 Today s topics Link analysis PageRank Link nalysis 1. PageRank 2. Hyperlink-Induced Topic Search (HITS) 3. entrality nalysis 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week 3--5 This material is built based on, nand Rajaraman, Jure Leskovec, and Jeffrey Ullman, Mining of Massive atasets, ambridge University Press, 212 hapter 5 Link nalysis 1. PageRank lgorithm 1

2 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week 3--6 efinition of PageRank Link analysis ata-analysis technique used to evaluate relationships (connections) between nodes function that assigns a real number to each page in the Web The higher the PageRank of a page, the more important it is There is NOT one fixed algorithm for assignment of PageRank 9/5/217 S535 ig ata - Fall 217 Week 3--7 xample [1/5] Page has links to, and Page has links to and Page has a link to Page has links to and 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week 3--9 xample [2/5] Suppose that a random surfer starts at page Page, and will be the next with probability probability of being at xample [3/5] Now suppose the random surfer at has probability of ½ of being at, ½ of being at and of being at or 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample [4/5] Transition matrix M What happens to random surfers after one step M has n rows and columns What is the transition matrix for this example? xample [5/5] 1 M= The first column a surfer at has a probability of next being at each of the other pages The second column a surfer at has a ½ probability of being next at and the same for being at 2

3 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week What does this matrix mean? [1/6] The probability distribution for the location of a random surfer column vector whose jth component is the probability that the surfer is at page j P(j i) Right stochastic matrix Real square matrix with each row summing to /5/217 S535 ig ata - Fall 217 Week What does this matrix mean? [2/6] If we surf at any of the n pages of the Web with equal probability The initial vector v o (row vector) will have 1/n for each component If M is the transition matrix of the Web fter the first one step, the distribution of the surfer will be Mv o fter two steps, M(Mv o ) =M 2 v and so on The product of two right stochastic matrices is also right stochastic Goal: Finding a row eigenvector of the transition matrix M Mv = M 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week What does this matrix mean? [3/6] Multiplying the initial vector v by M a total of i times The distribution of the surfer after i steps The probability for the next step from the current location The probability for being in the current location What does this matrix mean? [4/6] The probability xi that a random surfer will be at node i at the next step x i = m ij v j mij is the probability that a surfer j at node j will move to node i at the next step vj is the probability that the surfer was at node j at the previous step 1 1/ /4 The distribution of the surfer after the first step 1/4 = - Probability for being in the next possible location 1/4 9/5/217 S535 ig ata - Fall 217 Week What does this matrix mean? [5/6] The limit is reached when multiplying the distribution by M another time does not change the distribution The limiting v is a row eigenvector of M Since M is right stochastic (its columns each add up to 1), v is the principle eigenvector Its associated eigenvalue is the largest of all eigenvalues The principle eigenvector of M Where the surfer is most likely to be after a long time 9/5/217 S535 ig ata - Fall 217 Week What does this matrix mean? [6/6] The distribution of the surfer approaches a limiting distribution v that satisfies v = Mv provided two conditions are met: 1. The graph is strongly connected It is possible to get from any node to any other node 2. There are no dead ends Nodes that have no arcs out For the Web, 5-75 iterations are sufficient to converge to within the error limits of double-precision arithmetic 3

4 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample 1 M= xample 1 M= Suppose we apply this process to the matrix M The initial vector v and v1 after multiplying M 1 ¼ ¼ v 1=Mv = ¼ =? ¼ Use your worksheet Suppose we apply this process to the matrix M The initial vector v and v1 after multiplying M 1 ¼ 9/24 ¼ 5/24 v 1 =Mv = ¼ = 5/24 ¼ 5/24 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week What is the v 2? 1 M= Suppose we apply this process to the matrix M The initial vector v and v1 after multiplying M 1 ¼ 9/24 ¼ 5/24 v 1=Mv = ¼ = 5/24 ¼ 5/24 xample continued The sequence of approximations to the limit We get by multiplying at each step by M is ¼ 9/24 15/ /9 ¼ 5/24 11/48 7/32 2/9 ¼ 5/24 11/48 7/32 2/9 ¼ 5/24 11/48 7/32 2/9 This difference in probability is not noticeable In the real Web, there are billions of nodes of greatly varying importance The probability of being at a node like is orders of magnitude greater than others 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week Problems we need to avoid Link nalysis 1. PageRank lgorithm hallenges in PageRank lgorithm for the real Web ead end page that has no links out Surfers reaching such a page will disappear In the limit, no page that can reach a dead end can have any PageRank at all Spider traps Groups of pages that all have outlinks but they never link to any other pages What if we allow the ead end and Spider traps in the PageRank computation? 4

5 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week voiding ead nds xample If we allow dead ends The transition matrix of the Web is no longer stochastic Some of the columns will sum to rather than 1 Remove the arc from to à becomes a If we compute M i v for increasing powers of a substochastic matrix Some of all of the components of the vector go to substochastic matrix matrix whose column sums are at most 1 Importance drains out of the Web No information about the relative importance of pages 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample xample Remove the arc from to à becomes a dead end àif a random surfer reaches, they disappear at the next round Repeatedly multiplying the vector by M: ¼ 3/24 5/ ¼ 5/24 7/ ¼ 5/24 7/ ¼ 5/24 7/ M= The probability of a surfer being anywhere goes to as the number of steps increase 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week pproaches to dealing with dead ends 1. Recursive deletion Step 1. rop the dead ends from the graph Step 2. rop their incoming arcs as well Step 3. Repeat Step 1 and 2 xample of recursive deletion (1/4) Step 1. rop the dead ends from the graph Step 2. rop their incoming arcs as well Step 3. Repeat Step 1 and 2 2. Taxation Modify the process by which random surfers are assumed to move about the Web 5

6 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample of recursive deletion (2/4) The final matrix for the graph is ½ M = ½ 1 ½ ½ 1/6 3/12 5/24 2/9 3/6 5/ /9 2/6 4/12 8/24 3/9 xample of recursive deletion (3/4) We still need to compute deleted nodes ( and ) was the last to be deleted We know all its predecessors have PageRanks ( and ) Therefore, PageRank of = x 2/9 + ½ x 3/9 = 13/54 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample of recursive deletion (3/4) Now, we can compute the PageRank for Only one predecessor, The PageRank of is the same as that of (13/54) The sums of the PageRanks exceeds 1 It cannot represent the distribution of a random surfer It provides a good estimate (2) Spider Traps 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week Spider Traps and Taxation xample 1 Spider traps set of nodes with no dead ends but no arcs out (from the set of nodes) This can appear intentionally or unintentionally on the Web There is a simple spider trap of 4 nodes Spider traps causes the PageRank calculation to place all the PageRank within the spider traps M= 1 F 6

7 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample 2 If we perform the usual iteration to compute the PageRank of the nodes, we get The arc out of changed to point to itself 1/6 1/ /6 5/ / /6 4/ /6 1/6 1/12 is a simple spider trap of one node It still has outgoing edge to itself It is not a dead end M= 1 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week If we perform the usual iteration to compute the PageRank of the nodes, we get ¼ 3/24 5/ ¼ 5/24 7/ ¼ 14 29/48 25/288 1 ¼ 5/24 7/ ll the PageRank is at Once there, a random surfer can never leave To avoid Spider traps, we modify the calculation of PageRank llowing each random surfer a small probability of teleporting to a random page Rather than following an out-link from their current page 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week The iterative step, where we compute a new vector estimate of PageRanks v from the current PageRank estimate v and the transition matrix M is Where β is a chosen constant Usually in the range.8 to.9 v' = βmv + (1 β)e / n e is a vector for all 1 s with the appropriate number of components n is the number of nodes in the Web graph v' = βmv + (1 β)e / n The term βmv represents the case where, With probability β, the random surfer decides to follow an out-link from their present page The term (1-β)e/n is a vector ach of whose components has value (1-β)/n Represents the introduction with probability 1-β of a new random surfer at a random page 7

8 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week If the graph has no dead ends The probability of introducing a new random surfer is exactly equal to the probability that the random surfer will decide not to follow a link from their current page Surfer decides either to follow a link or teleport to a random page If the graph has dead ends The surfer goes nowhere The term (1-β)e/n does not depend on the sum of the components of the vector v, there will be some fraction of a surfer operating on Web When there are dead ends, the sum of the components of v may be less than 1 ut it will never reach 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample of Taxation () M= 1 v' = βmv + (1 β)e / n β =.8 2/5 4/15 2/5 v = 4/15 4/5 2/5 v + 4/15 2/5 xample of Taxation (2/2) First few iterations: ¼ 9/ /45 15/148 ¼ 13/6 53/3 77/45 19/148 ¼ 25/6 153/3 2543/45 95/148 ¼ 13/6 53/3 77/45 19/148 y being a spider trap, gets more than ½ of the PageRank for itself ffect is limited ach of the nodes gets some of the PageRank 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample 1 ompute the PageRank of each page assuming β=.8 Link nalysis xamples M= v = β M v +(1-β)e/n ⅓ ½ ½ ⅓ =.8 ⅓ ½ ⅓ + (1-.8)e/3 ⅓ ½ ⅓ 8

9 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week xample 2 clique Set of nodes with all possible arcs from one to another Suppose the Web consists of a clique of n nodes and a single additional node that is the successor of each of the n nodes in the clique 9/5/217 S535 ig ata - Fall 217 Week xample 2 etermine the PageRank of each page assuming β=.8 Is there a dead end? Is there a spider trap? 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample 2 etermine the PageRank of each page assuming β=.8 Is there a dead end? Yes. Is there a spider trap? No. Having clique does not mean that there is a spider trap xample 2 ¼ ¼ ¼ M= ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ⅕ v= ⅕ ⅕ ⅕ ⅕ β M v +(1-β)e/n ¼ ¼ ¼ ⅕ 1 ¼ ¼ ¼ ¼ ⅕ 1 =.8 ¼ ¼ ¼ ⅕ +(1-.8) 1 /5 ¼ ¼ ¼ ⅕ 1 ¼ ¼ ¼ ¼ ⅕ 1 9/5/217 S535 ig ata - Fall 217 Week xample 3 Suppose that we recursively eliminate dead ends from the Web graph to solve the remaining graph Suppose that the graph is a chain of dead ends, headed by a node with a self-loop What would be the PageRank assigned to each of the nodes? 9/5/217 S535 ig ata - Fall 217 Week xample 3 Remove all of the dead ends recursively Z Z 9

10 S535 ig ata Fall 217 olorado State University 9/5/217 Week 3-9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample 3 Remove all of the dead ends recursively What is v and M? xample 3 What is v and M? v =[1], M = [1], PageRank of = 1 PageRank of = ½ 1 PageRank of = PageRank of = ½ PageRank of = PageRank of = ½ Z 9/5/217 S535 ig ata - Fall 217 Week /5/217 S535 ig ata - Fall 217 Week xample of Taxation () M= v' = βmv + (1 β)e / n β =.8 2/5 4/15 2/5 v = 4/15 2/5 v + 4/15 2/5 xample of Taxation (2/2) For the first few iterations: ¼ 9/ /45 15/148 ¼ 13/6 53/3 77/45 19/148 ¼ 13/6 53/3 77/45 19/148 ¼ 13/6 53/3 77/45 19/148 1