A Novel Parallel Hierarchical Community Detection Method for Large Networks

Transcription

1 A Novel Parallel Hierarchical Community Detection Method for Large Networks Ping Lu Shengmei Luo Lei Hu Yunlong Lin Junyang Zou Qiwei Zhong Kuangyan Zhu Jian Lu Qiao Wang Southeast University, School of Information Science and Engineering, Nanjing 2196, China. ZTE Corporation, Nanjing 2112, China. {lu.ping, luo.shengmei, {linyl, zoujyjs, qwzhong1988,zhukuangyan, lujian198, Abstract Detection of community structure in big social network is a challenging task. In this paper, a novel hierarchical clustering method taking advantages of traditional methods such as CNM proposed by Clauset et al and Spectral Clustering is presented as well as its parallel implementation. Computer-generated networks and a real world network of about two million vertices and two billion edges we crawled from Sina Weibo are used for testing the implementation. Our parallel method achieves relatively high modularity within much shorter time than other fast algorithms (CNM method, Spectral Clustering, etc) announced be suitable for massive data. And it works well on the real world network dataset, which is difficult to be handled by traditional algorithms because of limitations in both CPU and memories. 1 Introduction How to find community structure in very large networks has been a hot-spot problem in data mining [1, 2, 3]. Community detection is a process of dividing nodes into groups such that there are more connections within groups than between them. In this paper, a novel hierarchical clustering method is presented as well as its parallel implementation based on MapReduce framework. To assist readers, Table 1 defines terms and notations used throughout the paper. Table 2 shows the time complexity of existing community detection methods and our hierarchical method, illustrating that our parallel implementation achieves relatively high modularity within much shorter time than other fast (Girvan-Newman method (GN)[4], Clauset- Newman-Moore method (CNM)[5], etc) algorithms announced be suitable for big data. corresponding author 1

2 Table 1: Notations used in the paper Symbol Quantity m number of edges n number of nodes d depth of the dendrogram p number of processing nodes t number of clusters degree of the nodei k i Table 2: Time complexity of different methods Algorithm Time Complexity GN method O(m 2 n) CNM method O(md log n) Spectral Clustering [6] O(nt 2 ) Our method O( md p log n 3 p ) 2 Clauset-Newman-Moore (CNM) Method Considering a simple network with no repeated edges and its graph is undirected and unweighted, CNM method is a hierarchical clustering method based on the greedy optimization of the quantity known as Modularity [4] [5], which indicates the quality of a partition. The definition is as follows: Q = i (e ii a 2 i ), (1) where, e ij = 1 2m A vw δ(c v,i)δ(c w,j), a i = vw j e ij. (2) In (2), A vw is 1 when vertices v and w is connected, otherwise, c v denotes the community that contains the vertex v, δ(i,j) is 1 if i = j and otherwise, and e ij is the fraction of edges in the network connecting vertices in community i and those in community j. While the global optimization of Q is a NP-competed problem, CNM method trys to greedily optimize Modularity Q by merging two communities that will have the maximum growth of Q each time. It starts with the assumption that every single vertex is a community, C (n) = {c 1,...,c n } where c i = {V i }. It will merge two communities once a time for the maximum growth of Q, when finally theqdescends, the algorithm give the result ofc (s) = {c 1,...,c s }. The process can be seen as searching the agglomerative path in the space of partition. To make the searching efficiently, CNM method maintains a maxheap of Q, the elements Q ij which means the variation of Modularity Q when we merge the community i into j. According to (1), let Q i denote the contribution of the ith community to Q, Q ij the contribution of the newly merged community, then we initially set Q ij = Q ij (Q i +Q j ) = { 1/2m ki k j /(2m) 2, if i,j are connected,, otherwise. (3) Each step merge two communities and update the Q of the communities which connected to the original two. For convenient we keep the newly merged community tag as j, and delete the ith community. The update rules are as follows. If community k is connected to both i and j, then Q jk = Q ik + Q jk. (4a) Ifk is connected toibut not toj, then Q jk = Q ik 2a j a k. (4b) If k is connected toj but not toi, then Q jk = Q jk 2a i a k. (4c) Equation (4) shows us a fact that whenever the largest Q is a negative value, the merging process will not ever make any growth ofqsince all the relevant quantity of Q in (4) will still keep negative. The variation of the ModularityQwith respect to the merging process is a concave function. 2

3 Network Traditional Method Subgraph C1l Processing Node 1 Processing Node 2 subgraph C1l Improved CNM Improved CNM Subgraph C2l Subgraph C2l Subgraph C2l Subgraph C2l Subgraph C2l Subgraph C2l Community C3l Community C3l Community C3l Community C3l Figure 1: A framwork of parallel CNM method 3 Our Method Original CNM method involves finding the largest change in Q, so parallelization is not easy. The key of parallelization of CNM lies on performing the merging work on every computing nodes while keep the relatively credible searching path. Assume that we have divided the network into c subcommunities, called 1-level subgraphs, randomly or using other algorithms. The Modularity of the clustering result is Q 1l. All these communities are distributed to computing nodes. According to the rules of merging, we would divide them into several subgraph which is called 2-level subgraphs, the intermediate result. Taking the properties of concave function into consideration, the Modularity of the result of 2-level subgraphs Q 2l will be not less than that of 1-level subgraphsq 1l. Again, to get the final result, we adopt CNM on the 2-level subgraphs. After this step we are supposed to get a higher Modularity because of the same reason mentioned above. The process is shown in Fig.1. In conclusion we parallelize the intermediate merging process of CNM in several computing nodes and finally amalgamate the 2-level subgraphs in a master node. And our parallel method will get a higher Modularity for the fist-step raw result whether it is random or result using other algorithms. There are two points worth emphasizing: 1. For the 1-level subgraph, merging process starts by initializing according to (3), where k i is not the degree in the subgraph but in the original graph, which will keep optimizing the global Modularity and make the algorithm perform better than the former. 2. For the 2-level subgraph, (3) should be modified slightly as follows for the initial community being no longer a single vertex, wheree ij anda i is defined in (2), respectively. Q ij = e ij 2a i a j. (5) While the process of 1-level subgraph is independent besides some global variable like k i. Parallelization is easy to realize. Pseudocode is shown in Appendix A. Based on the algorithm described above, two community detection algorithms are proposed as follows. 3.1 Iterative parallel CNM method The most straightforward implementation of parallel CNM method divides the network into several groups randomly and takes them as the input of our iterative method. The algorithm mentioned above is a single step in the iterations. Then each iteration step can get a new result of community detection which has a higher Modularity. Our method will be more efficient because the complexity 3

4 of one iteration step is approximately 1 p 3 times that of the original one. We called the method ICNM for short. 3.2 Parallel CNM method collaborating with Spectral Clustering The complexity of basic Lanczos method is linear related to the number of groups in Spectral Clustering method. And Spectral Clustering can t ensure that the subcommunity be a fully connected graph. Spectral Clustering method thus is not an efficient method for refined clustering with many groups, but is suitable to be the traditional method in our algorithm as shown in Fig.1, because p, which is also the number of 1-level communities, is relatively small. In conclusion, we can use Spectral Clustering method to get p subgraphs, while our method mentioned above is applied to get the final community detection result. It will be calculated only once rather than iteratively because spectral clustering results is relatively convincible. We called the method SCNM for short. 4 Empirical Evaluation In this section, we evaluate the proposed method using some benchmark networks and real-world networks. Our parallel algorithm is implemented in Hadoop Mapreduce framework which is a Google parallel computing framework on our blade server of IBM HS21*9, running RedHat 5.5 with linux kernel , hadoop.2.2 and java Tests on computer-generated networks Benchmark networks[7] is used to test our algorithm. We generate the benchmark graphs with these parameters: k = 12,k max = 1,γ = 2,β = 1,µ =.2,s min = 5,s max = 2. For the benchmark network with 1, nodes, cumulative time and the variation of Modularity for each step in our ICNM and are tested with the increasing of the iterations. The result is shown in Fig.2. From Fig.2, we can see that our method runs in much shorter times and get little higher Modularity than original CNM method. Fig.2 also shows us when to stop the iteration. For, it stop when the Modularity is much close to that of the previous iteration. In the test given below, we stop iteration when(q now Q pre )/Q now >.1 intuitively. For, as Sect(3.2) said, it will iterate once because of the high Modularity after only one iteration. Test for performance in networks with different size as shown in Fig.3 illustrates that our method is suitable for very large network rather than the small one because the iterative and parallel process need more inter-node communication overhead and the start time in our Mapreduce framework Run time of original CNM 1.9 Modularity of the result by original CNM.8 Cumulative time/s 2 15 Clustering by Spectral Clustering method 1 Modularity Clustering by Spectral Clustering method 5 Clustering randomly.2.1 Clustering randomly Iterations Iterations Figure 2: Cumulative time and the Modularity with the increasing of the iterations in our methods 4.2 Tests on real world networks In this subsection, we test our method in a real world network of 1,628,853 vertices and 159,285,54 edges we crawled from Sina Weibo which is a twitter-like micro blog in China. The community 4

5 3 25 Original CNM method Original CNM method 2.76 Run time/s 15 Modularity Network size x Network size x 1 4 Figure 3: The runtime and performance of our method in networks with different size. detection result is shown in Appendix B and the Modularity score and runtime of different methods in Table 3. The modularity is relatively high proving the credibility of the result. Table 3: The Modularity score and runtime of different methods in our Sina Weibo dataset Method Modularity Runtime/s CNM ,338 ICNM ,431 SCNM ,899 5 Significance and Impact In this paper, we proposed a parallel algorithm which is based on CNM method but change the way of optimizing the Modularity. It doesn t join communities together in pairs with the greatest increase in Modularity but find the relatively great Modularity s increase and join communities in each processing node. It can improve the performance of traditional method within a relatively short time. Based on the ideas above we also proposed two parallel hierarchical community detection methods. Our parallel implementation achieves relatively high modularity within much shorter time than original CNM method. The much improved speed of our algorithm makes it possible to find community structure in very large networks, which is not difficult to be handled by traditional algorithms because of limitations in both cpu and memories. References [1] Santo Fortunato. Community detection in graphs. Physics Reports, 486:75-174, 21. [2] Andrea Lancichinetti, Santo Fortunato. Community detection algorithms: A comparative analysis. Phisics Review E 9, 56117, 29. [3] Jure Leskovec, Kevin J. Lang and Michael Mahoney. Empirical comparison of algorithms for network community detection. Proceedings of the 19th international conference on World wide web, , 21. [4] M.E.J.Newman, M.Girvan. Finding and Evaluating Community Structure in Networks. Physical Review E 69, 26133, 24. [5] A.Clauset, M.E.J Newman and C.Moore. Finding Community Structure in Very Large Networks. Physical Review E 7, 66111, 24. [6] Ulrike von Luxburg. A tutorial on spectral clustering. Statics and Computing, 17: , 27. [7] Andrea Lancichinetti, Santo Fortunato, Filippo Radicchi. Benchmark Graph for Testing Community Detection Algorithm Phisical Review E 78, 4611, 28. 5