Extracting Communities from Networks

Transcription

1 Extracting Communities from Networks Ji Zhu Department of Statistics, University of Michigan Joint work with Yunpeng Zhao and Elizaveta Levina

2 Outline Review of community detection Community extraction Asymptotic consistency Simulation study Real data analysis Future work

3 Network data Network analysis has been a focus of attention in different fields. Social science: friendship networks Internet: computer networks, hyper-links Biology: food webs, gene regulatory networks

4 The mathematical representation of a network Given a network N = (V,E), where V is the set of nodes and E is the set of edges, we can represent this network N by an adjacency matrix A as follows: { 1 if there is an edge from node i to node j, A ij = 0 otherwise. where A can be either symmetric (for an undirected network) or asymmetric (for a directed network).

5 Community detection Communities: Networks consist of communities, or clusters, with many connections within a community and few connections between communities. Community detection problem: For an undirected network N = (V,E), the community detection problem is typically formulated as finding a partition V = V 1 V K which gives tight communities in some suitable sense.

6 Example: a school friendship network (colors represent grades)

7 Community detection problem Existing community detection methods: minimizing links between communities while maximizing links within communities (Newman, 2004). For simplicity, we consider the case of partitioning the network into two communities V 1 and V 2.

8 Min-cut To minimize R = A ij. i V 1,j V 2 However, min-cut always yields a trivial solution of V 1 = V or V 2 = V.

9 Ratio-cut (Wei and Cheng, 1989) To minimize R V 1 V 2, where V 1 and V 2 are the sizes of the two communities. Ratio-cut can avoid trivial solutions because the maximizer of V 1 V 2 is achieved at V 1 = V 2 = V /2.

10 Normalized-cut (Shi and Malik, 2000) To minimize R assoc(v 1,V) + R assoc(v 2,V), where assoc(v k,v) = i Vk,j V A ij for k = 1,2. Normalized-cut can avoid trivial solutions because an extremely small group V k may have a large ratio R/assoc(V k,v).

11 Modularity (Newman and Girvan, 2004) To maximize Q = 2 k=1 [ O kk D2 k L where O kk = i Vk,j V k A ij,d k = i Vk,j V A ij,l = 2 k=1 D k. ], Q represents the number of edges that fall within communities, minus the average value of the same quantity if edges fall at random given the degree of each node.

12 Limitation of traditional community detection methods There exists background in many real-world networks, which may not belong to any community. Traditional graph partitioning methods have difficulty in this situation.

13 Example: a school friendship network

14 Outline Review of community detection Community extraction Asymptotic consistency Simulation study Real data analysis Future work

15 Community extraction Most networks consist of a number (not known a priori) of communities, with relatively tight links within each community and sparse links to the outside, and background nodes that only have sparse links to other nodes. We propose a method that extracts communities sequentially: at each step, the tightest is extracted from the network until no more meaningful communities exist.

16 Criterion Extract one community at a time by looking for a set of nodes with a large number of links within itself and a small number of links to the rest of the network. The links within the complement of this set do not matter. To maximize where W(S) = I(S) m 2 B(S) m(n m), I(S) = A ij, B(S) = A ij, m = S. i,j S i S,j S c

17 Adjusted criterion Empirically, the previous criterion performs well for dense networks. However, it tends to find very small communities for sparse networks. To avoid small communities, we also propose To maximize W a (S) = m(n m) ( I(S) m 2 B(S) ). m(n m) The factor m(n m) penalizes communities with m close to 1 or n and encourages more balanced solutions.

18 Eigen-decomposition approximation Let D = diag( j A ij ), H = na md, then for fixed community size m, the adjusted criterion is equivalent to where max s s=m, s i {0,1} s Hs, { 1 if node i belongs to S, s i = 0 otherwise. Perform eigen-decomposition by relaxing s to a real vector.

19 Algorithm Tabu Search (Glover, 1986; Glover and Laguna, 1997): a local optimization technique based on label switching Use the eigen-decomposition result as an initial value Run the algorithm for many randomly ordered nodes

20 Outline Review of community detection Community extraction Asymptotic consistency Simulation study Real data analysis Future work

21 Block models Asymptotic consistency can be established under the assumption of block models. General block models 1 Each node is assigned to a block independently of other nodes, with probability π k for block k, 1 k K, K k=1 π k = 1. 2 Given that node i belongs to block a and node j belongs to block b, P[A ij = 1] = p ab, and all edges are independent. Block models for networks with background We can define the last block as background, by assuming p ak < p bb for all a = 1,...,K, and all b = 1,...,K 1.

22 Asymptotic consistency For simplicity, assume there is only one community and background in the network (K = 2 with parameters p 11,p 12,p 22,π and 1 π). Let c denote the true community labels, ĉ (n) denote the estimated labels, we proved Theorem For any 0 < π < 1, if p 11 > p 12, p 11 > p 22 and p 11 + p 22 > 2p 12, the maximizer ĉ (n) of both unadjusted and adjusted criteria satisfies P[ĉ (n) = c] 1 as n.

23 Key component of the proof Based on the framework established by Bickel and Chen (2009) Given a proposed label assignment s, Let R be the confusion matrix with R ab (s,c) = 1 n n i=1 I(s i = a,c i = b). The population version of the criterion can be written as a function of the confusion matrix. Key condition: The population version of the criterion is maximized by the correct confusion matrix diag(π, 1 π).

24 Outline Review of community detection Community extraction Asymptotic consistency Simulation study Real data analysis Future work

25 Simulation I Two pure communities (no backgroud) n = 1000 n 1 = 100,200,300 p 11 = 0.5,p 22 = 0.4,p 12 = 0.05

26 Evaluation Let S be the extracted set and C S be the true community that matches with S the best. PPV and NPV PPV = C S S S NPV =1 C S S c S c Purity Completeness

27 Results for simulation I Method n 1 = 100 n 1 = 200 n 1 = 300 PPV NPV PPV NPV PPV NPV Modularity (0.032) (0) (0.000) (0) (0) (0) Unadjusted (0.000) (0) (0.000) (0) (0) (0) Adjusted (0.058) (0) (0.003) (0) (0) (0)

28 Simulation II One community with background n = 1000 n 1 = 100,200,300 p 12 = 0.05, p 22 = 0.05 p 11 = 0.1,0.15,0.2

29 Results of simulation II p11=0.1 p11=0.15 p11=0.2 n1= PPV NPV PPV NPV PPV NPV n1= PPV NPV PPV NPV PPV NPV

30 Simulation III Two communities with background n = 1000 n 1 = 100,300,n 2 = 100,300 p 12 = p 23 = p 13 = p 33 = 0.05 p 11 = 0.1,0.15,0.2 p 22 = 0.08,0.12,0.16

31 Results for simulation III p11=0.1 p22=0.08 p11=0.15 p22=0.12 p11=0.2 p22=0.16 n1=100 n2= PPV NPV PPV NPV PPV NPV n1=300 n2= PPV NPV PPV NPV PPV NPV

32 Outline Review of community detection Community extraction Asymptotic consistency Simulation study Real data analysis Future work

33 Karate club network Friendships between 34 members of a karate club (Zachary, 1977). This club has subsequently split into two parts following a disagreement between an instructor (node 0) and an administrator (node 33).

34 Karate club network Community extraction Modularity

35 Political books network Links in the political books network (Newman, 2006) represent pairs of books frequently bought together on amazon.com. Blue: liberal Red: conservative

36 Political books network Community extraction Modularity

37 School friendship network The school friendship network is complied from the National Longitudinal Study of Adolescent Health (AddHealth) ( Grade 7: red Grade 8: blue Grade 9: green Grade 10: yellow Grade 11: purple Grade 12: orange

38 School friendship network Grades Modularity with 6 communities

39 School friendship network Extracting 6 communities Extracting 7 communities

40 Future work Stopping criterion Adjusted criterion ( I(S) W a (S) = [m(n m)] α m 2 B(S) ) m(n m)