Sampling Large Graphs: Algorithms and Applications

Transcription

1 Sampling Large Graphs: Algorithms and Applications Don Towsley Umass - Amherst Joint work with P.H. Wang, J.Z. Zhou, J.C.S. Lui, X. Guan

2 Measuring, Analyzing Large Networks - large networks can be represented by graphs - Facebook - WWW - Twitter - Ebay 300 million 1 Billion 50 Billion 233 Million Curse of data dimensionality!!!

3 Challenges in measurement: Information Distortion World Map in 1459 incomplete (Columbus et al. 1492) (Australia 17 th century) wrong proportions (Africa & Asia)

4 Why do we want to understand these networks? High school friendship Want to understand or find out network how did these networks evolve? who are the influential users? how does influence propagate? distribution of preference of users? communities in these networks?.etc.

5 Goals and Challenges Goals generate statistically valid characterization of network structure node pairs in this work Challenges large network sizes correcting for biases

6 How to measure: sampling algorithms Random sampling (uniform & independent) Vertex sampling Crawling Breadth First sampling (BFS) Edge sampling Random walk sampling

7 How to measure: sampling algorithms Random sampling (uniform & independent) Vertex sampling Crawling Breadth First sampling (BFS) Edge sampling Random walk sampling

8 How to measure: sampling algorithms Random sampling (uniform & independent) Vertex sampling Crawling Breadth First sampling (BFS) Edge sampling Random walk sampling

9 How to measure: sampling algorithms Random sampling (uniform & independent) Vertex sampling Crawling Breadth First sampling (BFS) Edge sampling Random walk sampling

10 How to measure: sampling algorithms Random sampling (uniform & independent) Vertex sampling Crawling Breadth First sampling (BFS) Edge sampling Random walk sampling

11 Focus of this talk Measure node pair statistics: important for many applications! 12

12 Classification of Node Pairs Classify node pair [u, v] using shortest path 1-hop node pair class if distance(u,v) = 1 2-hop node pair class if distance(u,v) = 2 13

13 Similarity Function: Homophily Homophily: tendency of users to connect to others with common interests. P. Singla and M. Richardson. Yes, there is a correlation: from social networks to personal behavior on the web. In WWW 2008 (MSN) Can infer characteristics and make recommendations Compare homophily(u,v) between different node pair classes 14

14 Similarity Function: Proximity Proximity(u,v): closeness of u and v; defined as functions such as number of common neighbors of u and v u v knowing proximity distribution of node pairs important for friendship prediction interest recommendation 15

15 Similarity Function: Distance Distance(u,v): length of shortest path between u and v in graph measure distance distribution of all node pairs to calculate average distance Twitter: 4.1 MSN: 6.6 effective diameter (the 90th percentile of all distances) small world 16

16 Problem Formulation undirected graph G = (V, E) measure characteristics of node pairs in following sets: all pairs - S = { u, v : u, v V, u v} one-hop pairs - pairs of connected nodes S (1) = u, v : u, v E two-hop pairs - pairs of nodes with at least one common neighbor S (2) = { u, v : u, v V, u v; x V st x, u, x, v E} 17

17 Problem Formulation F(u, v) - value of node pair property under study, e.g., # of common neighbors of u, v {a 1,, a K } - range of F u, v distribution of F u, v S: (ω 1,, ω K ) S (1) : (ω 1 (1),, ωk (1) ) S (2) : (ω 1 2,, ω K 2 ) ω k, ω k (1), ωk (2) - fractions of node pairs in S, S 1, S (2) with property F u, v = a k 18

18 Challenges OSNs large Facebook, Google+, Twitter, Renren, Sina Weibo,, V > 500 million users huge number of node pairs, V 2 > topology not available sampling required UVS (Uniform Vertex Sampling): sampling bias fors (1), S (2). Sometimes UVS not allowed crawling methods, such as breadth first search, and random walk (RW): sampling bias need to construct unbiased estimator 19

19 Node Pair Sampling Based on UVS Basic sampling techniques UVS: sample nodes from V uniformly weighted vertex sampling (WVS): sample nodes from V with desired probability distribution (π x : x V) independent WVS (IWVS (if we know topology) Metropolis-Hastings WVS (MHWVS): At each step, MHWVS selects a node v using UVS and then accepts the move with probability min(π v /π u, 1), otherwise remains at u. 20

20 Node Pair Sampling Based on UVS All pairs S Sampling method: use UVS to select two different nodes u and v uniformly at random Estimator: given sampled node pairs u i, v i, i = 1,, n n ω k = 1 1(F u n i, v i i=1 = a k ), k = 1,, K Accuracy (unbiased) E ω k = ω k and Var ω k = ω k ω k 2 n, k = 1,, K 21

21 Node Pair Sampling Based on UVS One hop pairs S (1) Sampling node pair [u, v] 1) sample node u according to probability distribution (π u 1 : u V), where d u π u 1 - degree of node u = d u 2 E 2) select neighbor v at random Each [u, v] sampled uniformly from S (1) 22

22 Node Pair Sampling Based on UVS One hop pairs S (1) Estimator: given sampled node pairs u i, v i, i = 1,, n ω k (1) = 1 n n i=1 Accuracy (unbiased) 1(F u i, v i = a k ), k = 1,, K E ω k (1) = ω k (1) and Var ω k (1) = ω (1) 1 2 k ωk n, k = 1,, K 23

23 Node Pair Sampling Based on UVS Two hop pairs S (2) Sampling node pair u, v 1) sample a node x according to probability distribution (π x 2 : x V) π x 2 = d x d x 1 M M - normalization constant 2) then select two neighbors u and v of node x at random 24

24 Node Pair Sampling Based on UVS Two hop pairs S (2) Sample (u, v) in S (2) with probability (2) π [u,v] = m(u, v)/m m(u, v) - number of neighbors common to u, v Estimator: given node pairs u i, v i, i = 1,, K ω k (2 ) = 1 H n 1(F u i, v i = a k ) i=1 m(u i, v i ), k = 1,, K H normalization constant 25

25 Node Pair Sampling Based on UVS Two hop pairs S (2) Accuracy: ω k (2) - asymptotically unbiased estimator of ω k (2), k = 1,, K. Meanwhile, P ω k 2 ω k 2 2εω 2 k 1 ε 1 1 nε 2 m ω k 2 + m 2 where 0 < ε < 1 and m = M/ S 2 26

26 Node Pair Sampling Based on RW Why? UVS not available, too costly. API not provided user IDs sparsely distributed Only crawling techniques can be used Breadth first search: unknown sampling bias Random walk: Walker moves to random neighbor, samples its information. For connected and non-bipartite graph, probability of RW being at node v converges to π v = d v 2 E, v V 27

27 Node Pair Sampling Based on RW All pairs S Sample node pair u i, v i by two independent RWs, where u i, v i are nodes sampled by two RWs at step i Node pair [u,v] sampled according to stationary distribution π [u,v] = d ud v, u, v V 4 E 2 28

28 Node Pair Sampling Based on RW All pairs S Estimator: given sampled node pairs u i, v i, i = 1,, n ω k = 1 J n 1(F u i, v i = a k )1(u i v i ) i=1 d ui d vi, k = 1,, K J normalization constant Accuracy: ω k, k = 1,, K is asymptotically unbiased estimate of ω k 29

29 Node Pair Sampling Based on RW One hop pairs S (1) Sampling method: random node pair [u i, v i ] sampled by RW u i, v i are nodes sampled at steps i and i + 1 Probability of RW sampling node pairs in S (1) are equal when RW reaches the steady state. 30

30 Node Pair Sampling Based on RW One hop pairs S (1) Estimator: given sampled node pairs u i, v i, i = 1,, K ω k (1 ) = 1 n n i=1 1(F u i, v i = a k ), k = 1,, K Accuracy: ω k (1 ) asymptotically unbiased estimate of ω k (1), 1 k K 31

31 Node Pair Sampling Based on RW Two hop pairs S (2) Neighborhood RW (NRW) current edge (u, v) next edge: select a random edge from edges connected to u or v, except edge (u, v) 32

32 Node Pair Sampling Based on RW Two hop pairs S (2) Select node pair by excluding same node in two edges consequently sampled by NRW. For example, output node pair [u, v] for two consequently sampled edges (u, w) and (w, v) Probability NRW samples node pair [u,v] in S (2) converges to (2) π [u,v] = m(u, v)/m m(u, v) - number neighbors common to u, v 33

33 Node Pair Sampling Based on RW Two hop pairs S (2) Estimator: given sampled node pairs u i, v i, i = 1,, n ω k (2 ) = 1 H n 1(F u i, v i = a k ) i=1 m(u i, v i ) where H = n i=1 1/m(u i, v i ) Accuracy: ω k (2 ) asymptotically unbiased estimate of ω k (2), 1 k K 34

34 Simulations: Distance Distribution Estimation B - number of sampled node pairs S - total number of node pairs error metric - NMSE ω k = E ω k ω k 2 ω k, k = 1,, K Gnutella - V 6300 B >.005 S, NMSE < 1 NMSE proportional to inverse of sqr() 35

35 Simulations: Mutual Neighbor Count Distribution in S (1) B = 0.01 S 1 node pairs sampled from S 1 36

36 Simulations: Mutual Neighbor Count Distribution in S (2) B = 0.01 S 2 node pairs sampled from S 2 37

37 Simulations: Interest Distribution Scheme Distribute interests over graph: 10 5 distinct interests: number nodes per interest ~ truncated Pareto distribution over {1,,10 3 } To distribute an interest possessed by k different nodes 1. select a random node v that can reach at least k- 1 different nodes 2. distribute interest to node v and closest k-1 nodes connected to v 38

38 Simulations: Common Interest Count Distribution for Generated Content CCDF of common interest count distribution for S, S (1), and S (2). Wiki-vote P2P-Gnutella 39

39 Simulations: Common Interest Count Distribution in S B = 0.01 S node pairs sampled from S Wiki-vote P2P-Gnutella 40

40 Simulations: Common Interest Count Distribution in S (1) B = 0.01 S (1) node pairs sampled from S (1) Wiki-vote P2P-Gnutella RW ~ IWVS > MHWVS => no topology 41

41 Simulations: Common Interest Count Distribution in S (2) B = 0.01 S 2 node pairs sampled from S 2 Wiki-vote P2P-Gnutella 42

42 Conclusions use sampling to estimate pair characteristics in sets S, S (1), and S (2). sampling methods based on independent vertex sampling and random walk produce asymptotically unbiased estimates validated approaches on wide range of graphs 43

43 Conclusions Markov Chain Mixing Times Other more powerful & elegant sampling methods: Frontier Sampling Efficiently Estimating Motif Statistics of Large Networks in the Dark. TKDD 2014 Design of Efficient Sampling Methods on Hybrid Social- Affiliation Networks. Accepted in IEEE ICDE 15 measuring, maximizing group closeness centrality over diskresident graphs. WWW 14 44

44 Thanks! Slides (will be) at r-sampling.pdf