Dynamic and Historical Shortest-Path Distance Queries on Large Evolving Networks by Pruned Landmark Labeling

Transcription

1 WWW 14 Dynamic and Historical Shortest-Path Distance Queries on Large Evolving Networks by Pruned Landmark Labeling Takuya Akiba (U Tokyo) Yoichi Iwata (U Tokyo) Yuichi Yoshida (NII & PFI)

2 Distance on Graphs Context-Aware Search [Ukkonen+,CIKM 08][Potamias+,CIKM 09] Socially-Sensitive Search [Vieira+,CIKM 07][Yahia+,VLDB 08][Maniu+,CIKM 13] Social Network Analysis Biological Analysis Route Navigation

3 Traditional Static Distance Querying Given a static graph G = V, E 1. construct an index to 2. answer distance d G s, t s t Goal: Good trade-off Scalability Indexing time Index size Query Performance Query time (Precision)

4 Previous Static Methods Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] Pruned Landmark Labeling [Akiba-Iwata-Yoshida, SIGMOD 13]

5 Real Networks are Dynamic and Evolving Online Social Networks: New friends Microblogs: New replies, new retweets Collaboration networks: New papers, etc.

6 Contributions Pruned Landmark Labeling [SIGMOD 13] State-of-the-art static exact method Contribution 1 Dynamic Pruned Landmark Labeling Contribution 2 Historical Pruned Landmark Labeling

7 Contribution 1: +Dynamic Update Dynamic Pruned Landmark Labeling Graph Indexing Index Incremental Update In milliseconds Change New edge Index Update

8 Distance Time Contribution 2: +Historical Queries Historical Pruned Landmark Labeling Index Query: transition of distance Time

9 Applications What will be made possible?

10 Real-time Network-aware Search B C New Edge A D A B Search Result Instantly Search Result A B C D

11 Network Evolution Analysis Ego Network of v (subgraph induced by v and neighbors) Distance to v v Real example using our method from Facebook subgraph

12 Network Evolution Analysis Ego Network of v (subgraph induced by v and neighbors) Distance to v v Cluster 1 Cluster 2 Real example using our method from Facebook subgraph

13 Network Evolution Analysis Ego Network of v (subgraph induced by v and neighbors) Distance to v v Cluster 1 Cluster 2 Cluster 1 Cluster 2 Real example using our method from Facebook subgraph

14 Network Evolution Analysis Average Distance Effective Diameter Closeness Centrality Temporal Hop Plot

15 Limitation: Graph Update Types Supported updates: Vertex additions and Edge additions Why no removals?

16 Why No Removals? Application Side 1. Additions are much more frequent than removals 2. There are networks even with no removals such as communication networks and collaboration networks 3. We can combine with periodical reconstruction to also reflect removals in, say, hours

17 Why No Removals? Technical Side Static Methods Efficiency: Fast and scalable Update: N/A Efficiency = Index size, indexing time, query time, etc Better Actually, also better! Dynamic PLL (this work) Efficiency: Comparable with static methods Update: Additions Fully Dynamic Methods Efficiency: Poor (Large index, low scalability) Update: Additions and removals Because removals are much more harder, methods would be too complex.

18 Contribution 1: Dynamic Pruned Landmark Labeling

19 Outline 1. Indexing Framework (=Data structure and query algorithm) 2-Hop Cover [Cohen+,SODA 02] 2. Offline Index Construction Pruned Labeling [SIGMOD 13] 3. Online Index Update Resumed PBFS & Prefixal Queries

20 Assumption Undirected Unweighted (We can easily obtain directed and/or weighted version)

21 Contribution 1: Dynamic Pruned Landmark Labeling Index Framework

22 2-Hop Labeling: Index Data Structure Commonly used framework (= Data Structure + Query Algo.) [Cohen+ 02], [Cheng+ 09], [Jin+ 12], [Abraham+ 12] and ours Index: label L v = l 1, δ 1, l 2, δ 2, v δ 1 δ 2 l 1 l 2 l i V, δ i = d G v, l i δ 3 l 3 Example L(1): L(2): L 3 : Vertex Distance Vertex Distance Vertex Distance d G 1,10 = 5

23 2-Hop Labeling: Query Algorithm Query: d G (s, t) = min l L s L(t) d G s, l + d G l, t Paths through common vertices L(1): Example Vertex Distance s t L 3 : Vertex Distance Distance between vertex 1 and 3: : = 7 Answer min{6, 7} = : = 6

24 2-Hop Labeling: Challenge Challenge: Computing (and maintaining) labels Correctness (Exactness) Sizes of labels (Index Size & Query Time) Efficiency (Scalability)

25 Contribution 1: Dynamic Pruned Landmark Labeling Offline Index Construction

26 Our Approach 1. Naïve Landmark Labeling Conduct a BFS from every vertex 2. Pruned Landmark Labeling Pruning during BFSs

27 Naïve Landmark Labeling (w/o pruning) 1. L 0 an empty index 2. For each vertex v 1, v 2,, v n Conduct a BFS from v i Label all the visited vertices L i u = L i 1 u v i, d G u, v i BFS from v 1 BFS from v 2 BFS from v 3 BFS from v n L 0 L 1 L 2 (n times ) L n Empty Result

28 Naïve Landmark Labeling (w/o pruning) After a BFS from 1 L 1 (1): L 1 (2): L 1 (3): Vertex 1 Distance 0 Vertex 1 Distance 2 Vertex 1 Distance After a BFS from 2 L 2 (1): L 2 (2): L 2 (3): Vertex 1 2 Distance 0 2 Vertex 1 2 Distance 2 0 Vertex 1 2 Distance 2 3

29 Naïve Landmark Labeling (w/o pruning) 1. L 0 an empty index 2. For each vertex v 1, v 2,, v n Conduct a BFS from v i Label all the visited vertices L i u = L i 1 u v i, d G u, v i Θ(nm) preprocessing time, Θ n 2 Inpractical! space

30 Pruned Landmark Labeling 1. L 0 an empty index 2. For each vertex v 1, v 2,, v n Conduct a pruned BFS from v i Label all the visited vertices L i u = L i 1 u v i, d G u, v i PBFS from v 1 PBFS from v 2 PBFS from v 3 PBFS from v n L 0 L 1 L 2 (n times ) L n Empty Result

31 Pruned BFS Pruned BFS from v i v i Distance δ u Current incomplete index L i 1 If QUERY v i, u, L i 1 We do not label u this time We do not traverse edges from u δ Prune u

32 Example First BFS from vertex 1 L 1 (1): L 1 (2): Vertex 1 Distance 0 Vertex 1 Distance 2 L 1 (6): Vertex 1 Distance 1 Second BFS from vertex 2 QUERY 2, 6, L 1 = = 3 = d(2,6) Vertex 6 is pruned.

33 Example The search space gets smaller and smaller

34 Theorems: Correctness Theorem 4.1 for any s, t and i. QUERY s, t, L i = QUERY s, t, L i Naïve: L 0 L 1 L 2 L n Equal through function QUERY Pruned: L 0 L 1 L 2 L n Empty Final

35 Theorems: Correctness Theorem 4.1 for any s, t and i. QUERY s, t, L i = QUERY s, t, L i Corollary 4.1 (Correctness) for any s, t QUERY s, t, L n = d G s, t i.e., our method is exact.

36 Performance in Real Networks We conduct BFSs from vertices with higher degree We can exploit the structures: Hubs [Theorem 4.3, SIGMOD 13] Core-fringe structure [Theorem 4.4, SIGMOD 13]

37 Contribution 1: Dynamic Pruned Landmark Labeling Online Index Update

38 Problem Setting Vertex addition: trivial Insert an isolated vertex Add edges Therefore, we concentrate on edge addition.

39 New edge: (u, v) Update of Naïve Labels For each vertex v 1, v 2,, v n Resume the BFS rooted at v i Update the labels

40 Example New edge Distance from the root on the last snapshot 2 Check and update the distance of endpoints Resume the BFS Not visited

41 New edge: (u, v) Update for Pruned Labels? For each vertex v 1, v 2,, v n Resume a pruned BFS rooted from v i Update the labels Too inefficient since it takes Ω(n) time

42 New edge: (u, v) Update for Pruned Labels? For each vertex v i? Resume a pruned BFS rooted from v i Update the labels

43 New edge: (u, v) Update for Pruned Labels For each vertex v i L v L(u) Resume a pruned BFS rooted from v i Update the labels Why? Pruned before endpoints still no need to traverse (The proof is in the paper)

44 Contribution 2: Historical Pruned Landmark Labeling

45 Problem Definition Offline Indexing Given: Graph timestamp for each vertex and edge Describing the time when each edge (or vertex) appeared

46 Problem Definition [Query 1] Snapshot Query Given: u, v and time τ Answer: d τ (u, v) [Query 2] Change-point Query Given: u, v Answer: {τ 1, τ 2, } and distances Distance at time τ Moments that distance d u, v has changed

47 Contribution 2: Historical Pruned Landmark Labeling Index Framework

48 Historical 2-Hop Labeling: Data Structure Normal 2-Hop Label L(1): Vertex Distance d G 1,10 = 5 Historical 2-Hop Label L(1): Vertex Distance Time d 1, 7 2 if time 3 Since there is no edge deletion, distance only decreases

49 Historical 2-Hop Labeling: Data Structure Normal 2-Hop Label L(1): Vertex Distance d G 1,10 = 5 Historical 2-Hop Label L(1): Vertex Distance Time d 1, 7 2 if time 3 Both queries in (almost) linear time, i.e. O L s + L t time

50 Contribution 2: Historical Pruned Landmark Labeling Offline Index Construction

51 Index Construction Algorithm Standard 2-Hop (Dynamic) 1. Naïve Labeling 2. Pruned Labeling Historical 2-Hop 1. Naïve Labeling 2. Pruned Labeling

52 Naïve Historical Labeling (w/o Pruning) 1. Start from an empty index L 0 2. For each vertex v 1, v 2,, v n Conduct a BFS from v i Add pairs to all the labels 3. L n is the final index Already non-trivial since networks are temporal

53 Dynamic Programming Let s be the source vertex. d: integer(distance), v V T[d][v] = least time that d G s, v d Initialization T 0 s = 0, T 0 v = v s Step T d + 1 v = min {max T d w, t(v, w) } w N v v t v, w edge creation time t v, v 0 Θ(Dm) time

54 BFS-like Dynamic Programming Edges from the vertices whose value did not change in the previous step No need to see in the next step To avoid vain edge check 1. Prepare a queue 2. Push a vertex when its value is changed 3. Only see edges from the vertices in the queue

55 Pruned Historical Labeling 1. Start from an empty index L 0 2. For each vertex v 1, v 2,, v n Conduct a pruned BFS-like DP from v i Add pairs to all the labels 3. L n is the final index

56 Suppose we are Pruned BFS from v i visiting u with distance δ and time t If QUERY v i, u, t, L i 1 We do not label u this time δ Prune u We do not traverse edges from u

57 Online Incremental Update Can be done similarly to the Dynamic PLL

58 Experimental Evaluation

59 Indexing Time (sec) Dynamic PLL: Indexing Time > 1 day > 1 day > 1 day Environment: Xeon X5670, 48GB Linux, g TD [EDBT 12] IS-Label [VLDB 13] DPLL 1 Epinion Enron P2P YouTube Wikipedia More scalable than other static methods

60 MB μs Dynamic PLL:Index Size and Query Time Index Size Query Time Epinions Enron Epinions Enron TD [EDBT 12] IS-Label [VLDB 13] DPLL Comparable Also Better Environment: Xeon X5670, 48GB Linux, g++

61 Time (ms) Dynamic PLL: Update Time Environment: Xeon X5670, 48GB Linux, g Index Rebuild Incremental Update 0.1 Epinion Enron P2P YouTube Wikipedia Update can be done in milliseconds

62 μs ms sec MB Historical PLL (in comparison with Dynamic PLL) Indexing Time Index Size DPLL HPLL DPLL HPLL 1 Epinions Enron 1 Epinions Enron Query Time Update Time 10 DPLL Epinions Enron HPLL (snapshot) HPLL (change-point) 0.1 Epinions Enron DPLL HPLL

63 μs ms sec MB Historical PLL (in comparison with Dynamic PLL) Indexing Time Index Size DPLL HPLL DPLL HPLL 1 Epinions Enron Epinions Enron Just slightly worse Query Timethan Dynamic PLL Update Time 1 10 DPLL Epinions Enron HPLL (snapshot) HPLL (change-point) 0.1 Epinions Enron DPLL HPLL

64 μs ms sec MB Historical PLL (in comparison with Dynamic PLL) Indexing Time Index Size DPLL HPLL DPLL HPLL 1 Epinions Enron 1 Epinions Enron Query Time Update Time 10 DPLL Epinions Enron HPLL (snapshot) HPLL (change-point) 0.1 Epinions Enron DPLL HPLL

65 Summary Distance queries on graphs Based on Pruned Landmark Labeling (PLL) Contribution 1: Dynamic PLL Applications: Real-time network-aware search Contribution 2: Historical PLL Applications: Network evolution analysis