Graph Model Selection using Maximum Likelihood

Size: px

Start display at page:

Download "Graph Model Selection using Maximum Likelihood"

Robyn Dawson
5 years ago
Views:

1 Graph Model Selection using Maximum Likelihood Ivona Bezáková Adam Tauman Kalai Rahul Santhanam Theory Canal, Rochester, April 7 th 2008 [ICML 2006 (International Conference on Machine Learning)]

2 Overview Real-world network: (Internet Graph) Random graph models: Which model to choose? Model A Model B Model C Model D Model E Picture downloaded from: projects_software/topfish.html

3 Overview Real-world network: (Internet Graph) Random graph models: 3 Model A Model B Model C Model D Model E Picture downloaded from: projects_software/topfish.html

4 Overview Real-world network: (Internet Graph) Random graph models: 3 Model A WINNER: Model B Model C Model D Model E Picture downloaded from: projects_software/topfish.html

5 Overview Real-world network: (Internet Graph) Random graph models: 3 Model A WINNER: Model B Model C Model D Model E How to choose the score?

6 Real-world networks Internet topology Picture downloaded from: visualization/projects_software/topfish.html

7 Real-world networks Internet topology WWW Picture downloaded from: blog-map-gallery.html

Real-world networks Internet topology WWW Collaboration jinetworks Co-author network of the most productive and best connected authors with the strongest co-authorship relations.

8 Real-world networks Internet topology WWW Collaboration jinetworks Co-author network of the most productive and best connected authors with the strongest co-authorship relations. Circles denote author nodes, and are labeled by the author s last name and initials. Legend: Node author; Node area size # of publications; Node area color # of unique co-authors Picture downloaded from:

9 Real-world networks Internet topology WWW Collaboration jinetworks Protein jiinteractions etc. Picture downloaded from:

10 Properties of Complex Networks small-world phenomenon Picture downloaded from:

11 Properties of Complex Networks small-world phenomenon - six degrees of jiiiseparation jiii[milgram 67] Picture downloaded from:

12 Properties of Complex Networks small-world phenomenon - six degrees of jiiiseparation jiii[milgram 67] Picture downloaded from:

13 Properties of Complex Networks small-world phenomenon - six degrees of jiiiseparation jiii[milgram 67] - high clustering jiii[watts & jiiijstrogatz 98] Picture downloaded from:

14 Properties of Complex Networks small-world phenomenon - six degrees of jiiiseparation jiii[milgram 67] - high clustering log( nodes ) jiii[watts & jiiijstrogatz 98] power-law jidegree distribution ji[siganos & 3x Faloutsos 03] log( degree ) Parameters: β (exponent), c (cutoff) i {1,,c} proportional to i -β

15 Modeling Complex Networks Erdıs-Rényi (the basic random graph model) Parameters: n # vertices p probability of an edge? For every pair of vertices: include edge with probability p

16 Modeling Complex Networks Powerlaw Random Graph [Bollobás 85; Aiello, Chung, Lu 00] Parameters: n # vertices β in, c in, β out, c out powerlaw distribution 1. For every vertex: generate indeg, outdeg 2. Randomly match red/blue half-edges

17 Modeling Complex Networks Powerlaw Random Graph [Bollobás 85; Aiello, Chung, Lu 00] Parameters: n # vertices β in, c in, β out, c out powerlaw distribution 1. For every vertex: generate indeg, outdeg 2. Randomly match red/blue half-edges

18 Modeling Complex Networks Powerlaw Random Graph [Bollobás 85; Aiello, Chung, Lu 00] Parameters: n # vertices β in, c in, β out, c out powerlaw distribution 1. For every vertex: generate indeg, outdeg 2. Randomly match red/blue half-edges

19 Modeling Complex Networks Powerlaw Random Graph [Bollobás 85; Aiello, Chung, Lu 00] Parameters: n # vertices β in, c in, β out, c out powerlaw distribution 1. For every vertex: generate indeg, outdeg 2. Randomly match red/blue half-edges

20 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex 2. In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

21 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex 1 2. In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

22 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

23 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex 1 2 p 2. In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

24 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

25 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

26 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex 1 2 q 3 2. In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

27 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex 1 2 q 3 2. In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

28 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex 1 2 p 3 2. In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

29 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

30 Modeling Complex Networks Preferential Attachment [Mitzenmacher 01] Parameters: n # vertices γ self-loop p, q (p+q<1) probability of out/in edge 1. Start with a single vertex In iteration i=2,,n create edges between vtx i and <i. Repeat until c): a) with prob. p outedge b) with prob. q inedge 4 c) with prob. 1-p-q next iter. jiiiin 2. other end-point chosen prop. to in/outdegree+γ.

31 Modeling Complex Networks Small World [Watts-Strogatz 98, Kleinberg 00] Parameters: s side of the grid α,β determine distribution on edges 1. Arrange vertices in s x s grid. 2. Add an edge from u to v with probability s α dist(u,v) -β jiiiwhere dist(u,v) is the Manhattan distance from u to v. 3. Omit isolated vertices.

32 Modeling Complex Networks Small World [Watts-Strogatz 98, Kleinberg 00] Parameters: s side of the grid α,β determine distribution on edges 1. Arrange vertices in s x s grid. 2. Add an edge from u to v with probability s α dist(u,v) -β jiiiwhere dist(u,v) is the Manhattan distance from u to v. 3. Omit isolated vertices.

33 Modeling Complex Networks Small World [Watts-Strogatz 98, Kleinberg 00] Parameters: s side of the grid α,β determine distribution on edges 1. Arrange vertices in s x s grid. 2. Add an edge from u to v with probability s α dist(u,v) -β jiiiwhere dist(u,v) is the Manhattan distance from u to v. 3. Omit isolated vertices.

34 Modeling Complex Networks Small World [Watts-Strogatz 98, Kleinberg 00] Parameters: s side of the grid α,β determine distribution on edges 1. Arrange vertices in s x s grid. 2. Add an edge from u to v with probability s α dist(u,v) -β jiiiwhere dist(u,v) is the Manhattan distance from u to v. 3. Omit isolated vertices.

35 Modeling Complex Networks Small World [Watts-Strogatz 98, Kleinberg 00] Parameters: s side of the grid α,β determine distribution on edges 1. Arrange vertices in s x s grid. 2. Add an edge from u to v with probability s α dist(u,v) -β jiiiwhere dist(u,v) is the Manhattan distance from u to v. 3. Omit isolated vertices.

36 Previous Work: Scoring Graph Models Score by model s capability to reproduce certain properties. [Medina - Matta - Byers 00] [Bu Towsley 02] [Barabási Albert Jeong 00] Closing-the-loop approach [Willinger Govindan Jamin Paxson Shenker 02] [Chen Chang Govindan Jamin Shenker Willinger 02] Performance and Likelihood metrics [Li Alderson - Willinger Doyle 04]

37 Previous Work: Scoring Graph Models Score by model s capability to reproduce certain properties. [Medina - Matta - Byers 00] [Bu Towsley 02] [Barabási Albert Jeong 00] Closing-the-loop approach [Willinger Govindan Jamin Paxson Shenker 02] [Chen Chang Govindan Jamin Shenker Willinger 02] Performance and Likelihood metrics [Li Alderson - Willinger Doyle 04] probability of a graph being generated by a specific powerlaw random graph model.

38 Our Approach: Maximum Likelihood (ML) Score ( model ) = -log ( prob. model generates G ) What: scoring random graph models wrt a given graph G Why ML? - Kolmogorov complexity inspired (Kolmogorov, Solomonoff, Chaitin, Levin 1960s) - Minimum Description Length Principle (MDL)

39 Our Approach: Maximum Likelihood (ML) Score ( model ) = -log ( prob. model generates G ) + model description length and parameter setting Related to Minimum Description Length Principle (MDL) - consider a fixed order of all possible graphs G: G 1, G 2, Prob(G 1 ) Prob(G 2 ) 0 1 Suppose this interval is of length 1/16. Then it contains one of the numbers 0/16, 1/16, 2/16,, 16/16. Thus, we can encode the interval by that number, using ~4 bits = -log (1/16).

40 Our Approach: Maximum Likelihood (ML) Score ( model ) = -log ( prob. model generates G ) Technical issues: - a model must be able to generate every graph. - node ordering - how to compute?

41 Our Approach: Maximum Likelihood (ML) Score ( model ) = -log ( prob. model generates G ) Technical issues: - a model must be able to generate every graph. - node ordering - how to compute? Node labels: a random permutatation of {1,,n}. Score ( model ) = 1 -log π n! ( prob. model generates G w. labels π )

42 Algorithms: Warm-up? Erdös-Rényi Prob ER ( G ) = p m (1-p) n(n-1)-m where m is the number of edges and n is the number of vertices of G. Prob ER ( G ) is maximized for p = m/(n(n-1)).

43 Algorithms: Warm-up cont Powerlaw Random Graph (PRG) Prob PRG ( G ) = Π v Prob(in(v)).Prob(out(v)).in(v)!.out(v)! m! where in(v) and out(v) are the indegree and outdegree of a vertex v and Prob(in(v)) = in(v) -β in / Z in Z in = Σ k=1..cin k -β in (similarly Prob(out(v))) Note: the formula assumes a simple graph G (can be easily modified for non-simple graphs).

44 Algorithms: MCMC approach Preferential Attachment (PA) Given a vertex labeling π (vertices labeled 1..n) : Prob PA ( G π ) = Π i=2...n (in(i)+out(i))! (1-p-q). Π j<i,(j,i) E p (in i (j)+γ)/(m i ). Π j<i,(i,j) E q (out i (j)+γ)/(m i ) where in i (j) is the number of in-neighbors of vertex j labeled <i (similarly out i (j)), and m i is the normalizing factor m i = Σ k=1..i-1 (in i (k)+γ) = Σ k=1..i-1 (out i (k)+γ)

45 Algorithms: MCMC approach Preferential Attachment (PA) Bottom line: Prob PA (G π) (relatively) easy to compute The problem: average over all π Score ( model ) = 1 -log π n! ( prob. model generates G w. labels π )

46 Algorithms: MCMC approach Preferential Attachment (PA) Bottom line: Prob PA (G π) (relatively) easy to compute Goal: compute -log ( π Prob PA (G π) ) Idea: design a Markov chain on all labelings where a labeling ω is sampled with probability proportional to Prob PA (G ω) σ(ω) = Prob PA (G ω) / π Prob PA (G π) How does the MC help?

47 Algorithms: MCMC approach Preferential Attachment (PA) Assume that we can sample labelings with probability: σ(ω) = Prob PA (G ω) / π Prob PA (G π) A standard trick to compute the partition function : Prob PA (G {1,2,3,,n}) / π Prob PA (G π) = π starting with 1 Prob PA (G π) π starting with 1,2 Prob PA (G π). π Prob PA (G π) π starting with 1 Prob PA (G π)

48 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? (+ Metropolis filter to get the stationary distribution)

49 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? (+ Metropolis filter to get the stationary distribution)

50 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? (+ Metropolis filter to get the stationary distribution)

51 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? Hit-and-run. (+ Metropolis filter to get the stationary distribution)

52 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? Hit-and-run. (+ Metropolis filter to get the stationary distribution)

53 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? Hit-and-run. 1 (+ Metropolis filter to get the stationary distribution)

54 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? Hit-and-run. 1 (+ Metropolis filter to get the stationary distribution)

55 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? Hit-and-run. 1 (+ Metropolis filter to get the stationary distribution)

56 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? Hit-and-run. (+ Metropolis filter to get the stationary distribution)

57 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = A swapping chain? Hit-and-run. (+ Metropolis filter to get the stationary distribution)

58 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = Metropolis filter to get the stationary distribution σ: the uniform chain goes from ω 1 to ω 2, we make the move with probability min{ 1, σ(ω 1 )/σ(ω 2 ) } (in our case = min{ 1, Prob PA (G ω 1 ) / Prob PA (G ω 2 ) } )

59 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = Metropolis filter & hit-and-run: in ω 1, choose a position (n choices), then move with probability min{ 1, σ(ω 1 )/σ(ω 2 ) } (overall probability of move is min{ 1, σ(ω 1 )/σ(ω 2 ) } /n)

60 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = Metropolis filter & hit-and-run (improved/faster): in ω 1, choose a position (n choices) proportionally to σ(ω 2 ) (does not waste any steps of the MC)

61 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = Hit-and-run summary: - have to be careful, depends on the distribution - still quite slow: massive parallelism

62 Algorithms: MCMC approach Preferential Attachment (PA) The main point: design an efficient MC with stationary distribution proportional to Prob PA (G ω) MC on permutations: ω = Hit-and-run summary: - have to be careful, depends on the distribution - still quite slow: massive parallelism e.g. does not work for the small world model we use simulated annealing (heuristically) to determine bounds on Prob SW (G)

63 Experiments AS-level Internet topology: three snapshots 97, 99, 01 n m 97 3,117 6, ,266 13, ,080 25,485

64 Experiments AS-level Internet topology: three snapshots 97, 99, n m 97 3,117 6, ,266 13, ,080 25, PA PRG SW ER PA PRG SW ER PA PRG SW ER negative log-likelihood per edge

65 Experiments PA PRG SW ER p = 0.58 β in = 1.55, c in = 610 α = p = 6.2e-4 q = 0.08 β out = 2.39, c out = 69 β = 1.9 γ = p = 0.61 β in = 1.57, c in = 1410 α = p = 3.5e-4 q = 0.08 β out = 2.44, c out = 172 β = 1.8 γ = p = 0.63 β in = 1.57, c in = 2421 α = p = 2.1e-4 q = 0.07 β out = 2.50, c out = 214 β = 1.8 γ = 0.3

algorithms experiments and implications in other contexts 3

66 Summary objective ranking mechanism algorithms experiments Open Directions design new models more general / faster algorithms experiments and implications in other contexts 3 Model A Model B Model C Model D Model E Negative log-likelihood

Graph Model Selection using Maximum Likelihood

Graph Model Selection using Maximum Likelihood Ivona Bezáková ivona@cs.uchicago.edu Department of Computer Science, University of Chicago, Chicago, Illinois 60637 Adam Kalai kalai@tti-c.org Toyota Technological Institute at Chicago, Chicago, Illinois