Label propagation with dams on large graphs using Apache Hadoop and Apache Spark

Size: px

Start display at page:

Download "Label propagation with dams on large graphs using Apache Hadoop and Apache Spark"

Gilbert Morgan
5 years ago
Views:

1 Label propagation with dams on large graphs using Apache Hadoop and Apache Spark ATTAL Jean-Philippe (1) MALEK Maria (2) (1) (2) October 19, 2015

2 Contents 1 Real-World Graphs 2 Community Detection Problem Community detection algorithms Supervised and unsupervised measures 3 Benchmarks to test our community detection algorithms 4 Proposed community detection algorithms 5 Perspectives : a Hadoop and Spark implementation, first experiments on Amazon 6 Conclusion

3 What is a graph? A simple definition A graph is a set of vertices and edges, or set of nodes that are connected by a certain degree of relationship (a certain level of similarity). A mathematical definition A graph is an ordered pair G = (V, E) where V is the set of vertices (or nodes) and E is the set of edges of the graph. We note: V = n is the number of vertices of G E = m is the number edges of G Figure 1: weighted graph Figure 2: unweighted graph

4 Contents 1 Real-World Graphs 2 Community Detection Problem Community detection algorithms Supervised and unsupervised measures 3 Benchmarks to test our community detection algorithms 4 Proposed community detection algorithms 5 Perspectives : a Hadoop and Spark implementation, first experiments on Amazon 6 Conclusion

The community detection problem The community detection problem The community detection problem is to find a partition of nodes in a way that there is a high density of edges within a group and a low

5 The community detection problem The community detection problem The community detection problem is to find a partition of nodes in a way that there is a high density of edges within a group and a low density of edges between groups. It is the detection of natural groups of vertices in networks. There is no absolute definition in the literature Figure 3: A toy example with 3 communities of different scales

6 Analysis on the community detections It exists three big families in community detection : Local method based on the node and its neighbourhood propagates an information or an operation not deterministic and very unstable weak complexity O(m) Global method based on the whole topology of the graph high complexity (often in O(n 3 )) propagate an information or an operation deterministic, subject to error propagation Hybrid method (Glocal) based on a local method which is lead by one or several global metrics complexity depends on the global metrics used better quality results than local methods not deterministic

7 Example of the three methodologies Figure 4: Agglomerative, spectral and Leader driven methods

8 Unsupervised measures To evaluate the quality of our community detection algorithms: Non supervised measures : Modularity and conductance Supervised Measures : Adjusted rand index, normalised mutual information and purity. Let f (S) the metric that captures the notion of the quality of the cluster. Optimized value of f (S) signifies a more community-like set of nodes: C The conductance : f (S) = S N S (2m S +c S ) measures the fraction of total edge volume that points outside the cluster. The modularity :f (S) = 1 4 (m S E(m S )) is the difference between m S, the number of edges between nodes in S and E(m S ), the expected number of such edges in a random graph with identical degree sequence. where N S is the number of nodes in S, and C S the number of outgoing links.

9 Supervised measures : Adjusted Rand Index (1971) Adjusted Rand Index Let S be a set of N data items, and U = {U 1, U 2,..., U R } and V = {V 1, V 2,..., V C } two partitions of S, information on the overlap between U and V can be summarized in form of a R C contingency table M = [nij] j=1...c i=1...r, where n ij denotes the number of objects that are common to clusters U i and V j. By counting ( a b) possibilities, we have N 11 : the number of pairs that are in the same cluster in both U and V ; N 00 : the number of pairs that are in different clusters in both U and V N 01 : the number of pairs that are in the same cluster in U but in different clusters in V N 10 : the number of pairs that are in different clusters in U but in the same cluster in V 2(N 00 N 11 N 01 N 10 ) ARI = (N 00 + N 01 )(N 01 + N 11 ) + (N 00 + N 10 )(N 10 + N 11 ) ATTAL Jean-Philippe Figureand 5: MALEK College Maria football Label propagation club network with dams on large graphs using Apache H

10 Supervides measures : The Normalised mutual information (1987) H(U) = R a i i=1 N log( a i N ) H(U, V ) = R C n ij i=1 j=1 N log( n ij N ) MI(U, V ) = H(U) + H(V ) H(U, V ) Values of the NMI The MI as the NMI : measures the information that partitions U and V share tells how much knowing one of these clusterings reduces our uncertainty about the other. The NMI can be considered as a function : f NMI (U, V ) [0, 1]

11 Contents 1 Real-World Graphs 2 Community Detection Problem Community detection algorithms Supervised and unsupervised measures 3 Benchmarks to test our community detection algorithms 4 Proposed community detection algorithms 5 Perspectives : a Hadoop and Spark implementation, first experiments on Amazon 6 Conclusion

12 Networks characteristics Figure 6: Network Characteristics

13 Contents 1 Real-World Graphs 2 Community Detection Problem Community detection algorithms Supervised and unsupervised measures 3 Benchmarks to test our community detection algorithms 4 Proposed community detection algorithms 5 Perspectives : a Hadoop and Spark implementation, first experiments on Amazon 6 Conclusion

14 Proposed community detection algorithm The label propagation is a non deterministic algorithm based on the propagation label. 1: Initialize nodes with unique labels, as n N, c n = l n 2: update each node s label to the label shared by most of its neighborhood, ie n N, C n = arg max l L l (n) until the convergence. 3: If every node has a label the maximum number of their neighbors have, then stop the algorithm, go to the previous step Figure 7: The label propagation This algorithm is a local method. low complexity O(m) non deterministic method very unstable (without order or preferential order)

15 Community detection algorithms : problem of Label propagation Label propagation has two major problems: It can produce a bad propagation It can produce "monster" communities It is highly unstable (case where a given order is not considered) Figure 8: The label propagation

16 Community detection algorithms : other label propagation algorithms in the literature Adding a score to each label which decreases when the geodesic distance from the label source node is too high (Leung et al. 2009) Multistep greedy agglomerative label propagation algorithm using modularity (Liu and Murata 2009) Offensive, defensive and hop attenuation label propagation (node propagation strength), (Lovro et al. 2013) Copra, Finding overlapping communities (Steve Gregory 2009) "Community Detection Using A Neighborhood Strength Driven Label Propagation Algorithm" (Xie and B.K. Szymanski 2011) Maximum overlap label core detection using MapReduce (Ovelgonne 2013) "Robust network community detection using balanced propagation"(subelj and Bajec 2013) "Controlled Label Propagation: Preventing Over Propagation through Gradual Expansion" (Rezaei and Soleymani 2015) The list is not exhaustive.

17 Community detection algorithms : proposed Method, a stabilized label propagation with dams Objective: Find community structures without the problem of bad propagation. Method which can be easily parallelizable and produces a good quality of community detection. Figure 9: The label propagation with dams We note β the percentage of edges with dams.

18 Community detection algorithms : proposed Method, an hybrid method, a stabilized label propagation with dams The edge betweenness centrality Let G = (V, E) a graph, where V is the set of nodes and E the set of edges of G. Let w the function of the weight on edges. For an unweighted graph, we have w(e) = 1. Let a path between two vertices starting at s V and finishing at t V. Note σ st the total number of shortest paths between vertices s and t. The notion of edge betweenness is based on the number of shortest paths that pass through a certain edge. The edge betweenness BC(e) for an edge e E is given by: BC(e) = s,t V,s t σ st (e) σ st where σ st (e) represents the number of shortest paths from nodes s to t and passing though the edge e E.

19 Community detection algorithms : proposed Method, the LPWD Figure 10: The LPWD Complexity: O(n 2 ) + k O(m βm)

20 Community detection algorithms : proposed Method, the LPWD Figure 11: The LPWD on Football Club

21 Community detection algorithms : a core label propagation with dams Figure 12: The core detection of Seifi et al. (2011)

22 Community detection algorithms : LPWDUS Figure 13: The LPWDUS Complexity: O(n 2 ) + O(N k (m βm))

23 Community detection algorithms : LPWDUSWS Figure 14: The LPWDUSWS The modularity can be used as score metric. Complexity : O(n 2 ) + O( 1 N k (m βm)).

24 Community detection algorithms : ECDLPWD Figure 15: The ECDLPWD Complexity: O(n 2 ) + O( 1 N k (m βm)).

25 Comparative analysis Experiences with some community detection algorithms Algorithms Q Φ NMI ARI Purity # Zachary #2 Louvain Seifi GN Spin Spectral WalkTrap Leung LPA * DPA Infomap LICOD ECDLPWD LPWDUS LPWDUSWM LPWODUS Table 1: Experiences with some community detection algorithms

26 Comparative analysis Experiences with some community detection algorithms Algorithms Q Φ NMI ARI Purity # Football #11 Louvain Seifi GN Spin Spectral WalkTrap Leung LPA * DPA Infomap LICOD ECDLPWD LPWDUS LPWDUSWM LPWODUS Table 2: Experiences with some community detection algorithms

27 Comparative analysis Experiences with some community detection algorithms Algorithms Q Φ NMI ARI Purity # Dolphins #2 Louvain Seifi GN Spin Spectral WalkTrap Leung LPA * DPA Infomap LICOD ECDLPWD LPWDUS LPWDUSWM LPWODUS Table 3: Experiences with some community detection algorithms

28 Comparative analysis Experiences with some community detection algorithms Algorithms Q Φ NMI ARI Purity # Political #3 Louvain Seifi GN Spin Spectral WalkTrap Leung LPA * DPA Infomap LICOD ECDLPWD LPWDUS LPWDUSWM LPWODUS Table 4: Experiences with some community detection algorithms

29 Contents 1 Real-World Graphs 2 Community Detection Problem Community detection algorithms Supervised and unsupervised measures 3 Benchmarks to test our community detection algorithms 4 Proposed community detection algorithms 5 Perspectives : a Hadoop and Spark implementation, first experiments on Amazon 6 Conclusion

30 Perspectives and current work: Parallel graph processing systems Figure 16: PGPS

31 Perspectives : graph partitioning with Mizan Figure 17: Dynamic graph partitioning

32 Perspectives and current work: HADOOP and MapReduce Figure 18: Hadoop architecture

33 Perspectives and current work: Spark and RDD Figure 19: Spark architecture

34 Perspectives and current work: label propagation on large graphs Current work: Work on large graphs with billions of edges. A Hadoop version for large scale graphs How to compute the edge betweenness on large graphs How to develop an in memory solution using Spark Study the parametrization the LPWDUS with α and β. Proposed an improved version of the label propagation with core detection DBLP, You Tube, Live Journal

35 Perspectives: Amazon graph Amazon graph It represents a network of products, where each vertex is a product and an edge exists between two products if they have been co purchased frequently. Figure 20: Amazon characteristics network

36 Perspectives: a label propagation on Hadoop Figure 21: Simple Label propagation on Hadoop

37 Perspectives: a core label propagation on Hadoop Figure 22: Community size distribution

38 Contents 1 Real-World Graphs 2 Community Detection Problem Community detection algorithms Supervised and unsupervised measures 3 Benchmarks to test our community detection algorithms 4 Proposed community detection algorithms 5 Perspectives : a Hadoop and Spark implementation, first experiments on Amazon 6 Conclusion

39 Conclusion Putting dams allows to increase the quality of the community detection with a local method. ECDLPWD gives better results than the LPWODUS LPWDUSWM seems in specific case to give better results than the ECDLPWD. For scale free graphs, 15% to 20% of dams with the highest edge betweenness gives better quality results Putting dams associated to core detection allows to find cores, but produces a bigger number of communities rather than the standard LPA The number of community in LPWODUS depends on α

40 Do you have any questions?

41 Please cite the following articles: Jean-Philippe Attal, Maria Malek, A new label propagation with dams, IEEE/ACM international conference on advances in social networks analysis and Mining (ASONAM), Paris, août Jean-Philippe Attal, Maria Malek, Un nouvel algorithme de propagation de labels avec barrages, Journée Réseaux Sociaux et Inteligence Artificielle (Atelier PFIA), Rennes, 29 juin and Jean-Philippe Attal, Maria Malek, Propagation de labels avec barrages sur de grands graphes en utilisant Apache Hadoop et Apache Spark (GraphX), Journée thématique : Fouille de grands graphes (JFGG15), Nîmes, octobre 2015

Community detection using boundary nodes in complex networks

Community detection using boundary nodes in complex networks Mursel Tasgin and Haluk O. Bingol Department of Computer Engineering Bogazici University, Istanbul In this paper, we propose a new community