Roadmap. Roadmap. Ranking Web Pages. PageRank. Roadmap. Random Walks in Ranking Query Results in Semistructured Databases

Transcription

1 Roadmap Random Walks in Ranking Query in Vagelis Hristidis Roadmap Ranking Web Pages Rank according to Relevance of page to query Quality of page Roadmap PageRank Stanford project Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd. The PageRank Citation Ranking: Bringing Order to the Web. Started Google

2 PageRank Make use of the link structure of the web to calculate a quality ranking (PageRank) for each web page. Each page has unique PageRank, independent of keyword query PageRank does NOT express relevance of page to query PageRank is a Usage Simulation Random surfer Given a random URL Clicks randomly on links After a while gets bored and gets a new random URL The number of visits to each page is its PageRank. PageRank Calculation Intuition PageRank of page P increases when pages with large PageRanks point to P. PageRank Calculation PR(A)=(-d) + d*(pr(t)/c(t)+ + PR(Tn)/C(Tn)) d: damping factor, normally this is set to T,, Tn: pages pointing to page A PR(A): PageRank of page A. PR(Ti): PageRank of page Ti. C(Ti): the number of links going out of page Ti. Note: d is needed due to PageRank sinks Example of Calculation () Example of Calculation (2) *0.85 *0.85/2 *0.85/2 *0.85 *0.85 2

3 Each page has not passed on 0.5, so we get: : 0.85 (from ) (not transferred) = : (from ) (not transferred) = : 0.85 (from ) (from ) (from Page A) (not transferred) = : receives none, but has not transferred 0.5 = 0.5 Example of Calculation (3) : 2.275*0.85 (from ) (not transferred) = : *0.85/2 (from ) (not transferred) = : 0.5*0.85 (from ) *0.85(from Page B) + *0.85/2 (from ) +0.5 (not transferred) =.925 : receives none, but has not transferred 0.5 = Example of calculation (4) After 20 iterations, we get Example - Conclusions has the highest PageRank, and page A has the next highest: page C has a highest importance in this page graph! More iterations lead to convergence of PageRanks. Google Uses PageRank as one of the criteria to rank keyword query results. Other criteria (may) include: Term frequencies Term proximities Term position (title, top of page, etc) Term characteristics (boldface, capitalized, etc) Link analysis information Category information Popularity information 3

4 Roadmap Hubs & Authorities Jon M. Kleinberg: Authoritative Sources in a Hyperlinked Environment. JACM 46(5): (999) HITS ( Hypertext-Induced Topic Search) developed by Jon Kleinberg, while visiting IBM Almaden. IBM expanded HITS into Clever. IBM doesn't see Clever as real-time search engine. But create constantly refreshed lists of relevant pages for categories Hubs & Authorities Rank pages according to keyword query (in contrast to PageRank) Hubs & Authorities Good hub: page that points to many good authorities. Good authority: page pointed to by many good hubs. Given Keyword Query, assign a hub and an authoritative value to each page. Pages with high authority are results of query Hubs & Authorities Calculation : Root Set and Base Set Using query term to collect a root set of pages from text-based search engine (AltaVista) Hubs & Authorities Calculation : Root Set and Base Set (Cont d) Expand root set into base set by including (up to a designated size cut-off) all pages linked to by pages in root set all pages that link to a page in root set Typical base set contains roughly pages Base Set Root Set Root Set 4

5 Hubs & Authorities Calculation Iterative algorithm on Base Set: authority weights a(p), and hub weights h(p). Set authority weights a(p) =, and hub weights h(p) = for all p. Repeat following two operations (and then re-normalize a and h to have unit norm): h(v ) h(v 2 ) h(v 3 ) v v 2 v 3 a( p) = q points to p p h(q) p h( p) = p points to a(q) q v v 2 v 3 a(v ) a(v 2 ) a(v 3 ) Z H X Y Example: Mini Web h x a = h y A = a h z a H = M x z y Ai i * T i * i A = M H M = X Y Z X Y Z 0 H = M 0 M H T i * i T i * M * Ai A = M 0 Example Hubs & Authorities Calculation M = 0 Z 0 X 0 Y M T = 0 M M T = Iteration H = A = M T M = X is the best hub Z is most authoritative Theorem (Kleinberg, 998). The iterates a(p) and h(p) converge to the principal eigenvectors of M T M and MM T, where M is the adjacency matrix of the (directed) Web subgraph. PageRank v.s. Authorities Roadmap PageRank (Google) computed for all web pages stored in the database prior to the query computes authorities only Trivial and fast to compute HITS (CLEVER) performed on the set of retrieved web pages for each query computes authorities and hubs easy to compute, but real-time execution is hard 5

6 Keyword Search in Databases Result of Keyword Query Result is tree T of nodes where: each edge corresponds to an edge of the data graph every keyword contained in a node of T no node of T is redundant (minimal) The label of a node is: Type (Value) degree Query: Vagelis, Gravano Assume that SIGMOD 0 has 500 attendees and 50 papers. Each paper has 0 references and 2 authors. Example R: Vagelis PREFER SIGMOD 0 L.Gravano R2: Vagelis PREFER Fagin PODS96 Top-k ICDE2002 L.Gravano Roadmap R3: Vagelis PREFER Insignificant paper Insignificant2 paper Unknown Gravano Previous Work R: R2: R3: XKeyword, DISCOVER, DBXplorer, Goldman98: Score is inverse of path distance between nodes. BANKS: Weighted distance output: R, R2, R3 Previous Work Keyword Queries XKeyword. V. Hristidis, Y. Papakonstantinou, A. Balmin. ICDE 2003 DISCOVER. V. Hristidis, Y. Papakonstantinou. VLDB 2002 DBXplorer. S. Agrawal et al. ICDE 2002 Three step architecture Data stored in DBMS Schema use BANKS. G. Bhalotia et al. ICDE 2002 Database viewed as graph No schema info Steiner tree problem approximations Proximity searching in databases. R. Goldman et al. VLDB 998 Database viewed as graph No schema info hub nodes 6

7 Previous Work Prior work: output: R, R2, R3 R: R2: R3: Intuitively R3 shows a tighter connection than R (higher relevance between keywords) But R2 connects objects of higher importance than R3 (higher quality of result) Relevance and Quality can be contradicting factors Random Walks (RW) Score of result A~B: Probability that a random walk goes from A to B Captures Relevance, but ignores Quality of result. P(A B C) = /degree(a) * /degree(b) Random Walks (RW) RW: output: R3, R2, R But R2 connects objects of higher importance than R3 (higher quality of result) Relevance and Quality can be contradicting factors R: R2: R3: Random Walks + PageRank (RW+PR) Score of result A~B: Probability that a random walk starting from any node, goes through both A and B. Captures both Relevance and Quality of result. Score = PR(A)* P(A~>B)+ PR(B)* P(B~>A) P(A~>B) can be computed using PageRank algorithm setting the pagerank source to {A} Random Walks + PageRank (RW+PR) R: R2: R3: Example - Details The following table shows the scores of the results according to 3 ranking methods XKeyword RW PR+RW R /4 /(7E+9).2E-7 R2 /5 /(4E+9).7E-7 R3 /4 /(2E+7) 9E-8 RW+PR: output: R2, R, R3 Assuming Ranking XKeyword RW RW+PR Vagelis PR /000 R, R2, R3 R3, R2, R R2, R, R3 L.Gravano Unknown Gravano /50 /

8 Random Walk Variations Page vs Structured Ranking Criterion Definition Comments Relevance Quality Both Probability of random walk starting from a result node traverses the rest of the result nodes For each result node, calculate probability of random walk starting from any node of graph Probability that a random walk starting from any node of the graph, goes through the result nodes. Corresponds to PageRank. Also, part of the authority of a page is attributed to its quality. A combination of PageRank (computed offline) and random walks starting from a result node could be used. Web (PageRank/Authorities) All edges have same meaning (hyperlinks) Straightforward notion of direction Single type (page) of nodes Return single object (page) Structured Databases Each edge has different semantics No Multiple types Return tree of objects Open issues Efficiently calculating RW First thoughts: Two ways DISCOVER-like with CNs BANKS-like, using shorthest path progressively Edges must have different weights for PR and RW calculation. (eg: Paper cites Paper is one-way for PR but two-way for RW) How to assign PR and RW weights on schema graph? Conclusions The concept of Random Walks has proven very useful in ranking Web pages Can also be used in ranking results of queries in structured/semistructured databases. Problem is more complicated 8