Efficient Top-k Shortest-Path Distance Queries on Large Networks by Pruned Landmark Labeling with Application to Network Structure Prediction

Transcription

1 Efficient Top-k Shortest-Path Distance Queries on Large Networks by Pruned Landmark Labeling with Application to Network Structure Prediction Takuya Akiba U Tokyo Takanori Hayashi U Tokyo Nozomi Nori Kyoto U Yoichi Iwata U Tokyo Yuichi Yoshida NII & PFI January 29, 2015 The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15) 1

2 Proximity Measures on Networks Strength of relevance between vertex pairs B A C (A, B) > (A, C) 2

3 Distance as a proximity measure Applications of distance as the proximity measure Context-Aware Search [CIKM 08][CIKM 09] Socially-Sensitive Search [CIKM 07][VLDB 08][CIKM 13] Efficient indexing methods Landmark [CIKM 09] [WSDM 10] [CIKM 10] [ICDE 12] TD [SIGMOD 10] [EDBT 12] 2-Hop [SIGMOD 12] [ESA 12] [SIGMOD 13] 3

4 Graph Indexing Methods Given a graph G = V, E 1. construct a data structure (an index) to 2. answer queries quickly Graph queries s 1, t 1, s 2, t 2, Index answer Reachable! 4

5 Distance as a proximity measure Advantage: Scalable and fast Drawback: Poor expressiveness Unweighted graphs Small-world phenomenon Distances are small integers 5

6 Graph Distance (a) 4 (b) 4 (c) 4 6

7 Top-k Distances As a proximity measure, we propose to use Top-k distances defined as Lengths of k shortest paths k = 1 standard distance Loops are allowed 7

8 Graph Distance Top-k Distances (a) 4 [ ] (b) 4 [ ] (c) 4 [ ] 8

9 Graph Distance Top-k Distances (a) 4 [ ] (b) 4 [ ] (c) 4 [ ] 9

10 Top-k Distances as a proximity measure? Top-k distances are vectors of dimension k Not scalar direct substitution impossible 10

11 Top-k Distances as a proximity measure? Top-k distances are vectors of dimension k as feature vectors, nice with Machine Learning [ ] [ ] [ ] SVM Yes No Yes 11

12 Previous Algorithms for Top-k Distances BFS-based naïve algorithm [folklore] O n + m k time Eppstein s algorithm [Eppstein, 1998] O n + m + k time, but slow in practice for moderate k In contrast to many indexing methods for standard distance no indexing methods have been proposed We are to evaluate many pairs of vertices Proportional to graph size not scalable 12

13 Contributions at a glance Contribution 1 An indexing method for top-k distance queries Based on pruned labeling [Akiba+,SIGMOD 13] Simple, fast and scalable Enables Contribution 2 Top-k distances as a proximity measure Suited to machine learning Empirical study on link prediction 13

14 Indexing Method for Top-k Distance Queries Part 2: Algorithm 14

15 Trivial extension? No! from standard distance queries? Main new challenge Carefully avoiding double-counts The same path may come differently Naïve approach may count a path twice or more s t 15

16 2-Hop Cover for distance queries [Cohen+, 2002] Data Structure: a label for each vertex v Label L v L v = u 1, δ 1, u 2, δ 2, δ i = d v, u i L v v δ 3 u 3 δ u2 2 u 1 δ 1 16

17 2-Hop Cover for distance queries [Cohen+, 2002] Data Structure: a label for each vertex v Label L v L v = u 1, δ 1, u 2, δ 2, δ i = d v, u i L v v δ 3 u 3 δ u2 2 u 1 δ 1 Query: 2-hop paths using labels L s min d s, u + d G u, t u L s L(t) s t L t 17

18 Top-k 2-Hop Cover (this work) Data structure Distance Label L v L v = u 1, δ 1, u 2, δ 2, >v δ i = d j th v, u i L v v δ 3 u 3 δ u2 2 u 1 δ 1 Loop Label C v C v = δ 1, δ 2, δ 3, v δ i = d i th v, v C v δ 1 δ 2 v >v d j th v, u i is restricted distance 18

19 Top-k 2-Hop Cover (this work) Data structure Distance Label L v + Loop Label C v Query: 3-hop paths L s C u L t s t 19

20 Indexing Algorithm Challenge on computing labels Correctness (Exactness) Sizes of labels (Index Size & Query Time) Efficiency (Scalability) Algorithm based on pruned labeling [Akiba-Iwata-Yoshida, SIGMOD 13] + Performance improving techniques 20

21 Experimental Evaluation Part 3: Experiments 21

22 Indexing time [s] Indexing Time Experimental Results Social networks Web graphs 1, ,000 10,000,000 Graph size E < 1 hour on graphs with ten million edges k = 8, Intel Xeon X5670 (2.93 GHz), 48 GB Memory, C++ 22

23 Index size [MB] Index Size Experimental Results 10,000 1, Social networks Web graphs 0 1, ,000 10,000,000 Graph size E < 10 GB on graphs with ten million edges k = 8, Intel Xeon X5670 (2.93 GHz), 48 GB Memory, C++ 23

24 Query time [ms] Query Time Experimental Results Proposed Naive Eppstein X faster , ,000 10,000,000 Graph size E Consistently <0.1ms k = 8, Intel Xeon X5670 (2.93 GHz), 48 GB Memory, C++ 24

25 Application to Link Prediction Part 3: Experiments 25

26 Application to Link Prediction Link Prediction Problem [Liben-Nowell+, 2003]? Online Social Network Protein Interaction Methods 7 Baseline Methods Top-k Distances + SVM 26

27 Application to Link Prediction This work Dataset CN Jaccard Adamic Pref. Comb. SVD RWR Top-k Top-1 Facebook Facebook Last.fm GrQc HepTh CondMat Precision: AUC (Area under the ROC curve) Highlighted: statistically significant winners (by paired t-test with p < 0:05) Setting: training (60% edges) evaluation (40% edges), 10 times 27

28 Conclusions Contribution 1 An indexing method for top-k distance queries Contribution 2 Top-k distances as a proximity measure Software available! Hope to see further exploration of top-k distances in various applications! 28