Robustness of Centrality Measures for Small-World Networks Containing Systematic Error

Transcription

1 Robustness of Centrality Measures for Small-World Networks Containing Systematic Error Amanda Lannie Analytical Systems Branch, Air Force Research Laboratory, NY, USA Abstract Social network analysis is a necessary aspect of learning terrorist/criminal networks, or of how people interact through social networking sites like Facebook and Twitter. The metrics used to evaluate these networks are equally as important, centrality measures being the most common. Being the most routinely used metrics, centrality measures have been tested under various conditions. However, there is a lacking focus on systematic error applied to computer generated connected undirected smallworld networks. The goal of this paper is to test the centrality measures (betweenness, closeness, and degree) in such networks and test their robustness after the introduction of systematic error. The experiment found that the metrics performed poorly and their use would have to be determined based on the situation and consequences. Keywords: networks centrality; systematic error; small-world 1 INTRODUCTION The centrality measures betweenness, closeness, and degree are common metrics for measuring aspects of a social network. Betweenness is the measure of the number of nodes between a pair of nodes. In [1], Freeman claims that to calculate betweenness one must compute: ( ) ( ) (1) σ st is the total number of paths from node s to node t. σ st (v) is the number of σ st paths that contain node v. Explicitly, Equation 1 is the sum of the quotients (σ st (v) divided by σ st ) for all pairs of nodes. The value computed represents a node s ability to control the flow of information. Closeness measures a node s number of contacts including both direct and indirect ones; thus, the measure is a representation of how well connected a node is. The formula for calculating closeness from [2] is: ( ) (2) where d vj is the shortest path from v to j. Degree is the number of direct contacts a node has. According to [2], the equation to compute degree can be defined as: ( ) (3) The a ij is representative of an element in an adjacency matrix representation of the graph. The more direct connections a node has the more possibilities the node has to send/receive information. Betweenness, closeness, and degree are sensitive to changes made to a social network. Consider Figure 1 and assume the network shown is complete. Now look at Figure 2; it is missing the black node in the previous figure. That is four missing connections and now the node with the most connections is missing. In addition, the betweenness and closeness values for Figure 2 would differ from Figure 1. The values would be misleading and possibly detrimental to someone using the measures to focus on important nodes (people) like someone tracking a criminal or terrorist network. Therefore, in the case of Figure 2, the most important person according to the degree measure is missing. Besides real world scenarios, it is crucial to test the robustness of centrality measures since they are the most frequently used metrics for social network analysis. Our contribution is to determine the robustness of betweenness, closeness, and degree when it comes to small-world networks with systematic error introduced which will create bias in the subgraphs. According to [3], systematic error is error that is not determined by chance but is introduced by an inaccuracy, by observation or measurement, inherent in the system. This research uses connected small-world networks for the ground truths to simulate real world data as closely as possible. A ground truth network represents a complete network, one without missing information like a person or relationship. To mimic cases of missing information and bias (introducing systematic error), we use subgraphs of the ground truths. The creation of subgraphs involves choosing one node as the starting point and like real world scenarios the further away a node is from the starting point the less likely it will be discovered. The setup is analogous to tracking a criminal/terrorist network, and the goal was to study the performance of the metrics used on the subgraphs, specifically their robustness.

2 Fig. 1. A connected undirected small-world network where the number of neighbors was originally two. The black node is the most important person in terms of degree. 2 RELATED WORK In 1988, John M. Bolland published a paper that explores the robustness of the centrality measures (betweenness, closeness, degree, and eigenvector). He used a real world network and manipulated versions of it. Bolland then inserted random systematic variation into the data from which he was able to make several conclusions. Betweenness is usually unstable when up against random error and is sensitive to systematic variation. Secondly, closeness is susceptible to both random and systematic variation on the outer edges of a network. Bolland also concluded that degree is useful in its ability to concurrently ignore random variation while reacting to systematic variation [4]. Costenbader and Valente also dealt with real data, specifically 59 networks. Their approach uses bootstrap sampling processes to establish how sampling influences the stability of 11 different network centrality measures [5]. Of the 11, among them were betweenness, closeness, and degree. From their research, Constenbader and Valente were able to conclude that betweenness and degree had lower correlations for an undirected graph than a directed one where the correlation is between actual and sampled measures. Additionally, closeness has superior performance for directed graphs when compared to undirected ones. In 2006, Borgatti, Carley, and Krackhardt tested the Fig. 2. Figure 1 with missing information, the black node and its connections. robustness of degree, betweenness, closeness, and eigenvector centralities when it came to a large sample of random graphs. The authors of [6] control the amount of error in the observed graphs by adding or removing edges or nodes. They concluded that the measures are robust if the error is less than or equal to 10%. They were of the mind that it would be difficult to tell if such accuracy (61% or better depending on the amount of error of 10% or less) would be acceptable since it would depend on the situation and consequences. Unlike [6], this research focuses on the robustness of centrality measures under systematic instead of random Node Level TABLE I. WHY THERE ARE 171 GROUND TRUTHS WHEN THERE IS ONE SET FOR EVERY NODE LEVEL. 8% Number of Neighbors to Control (#) means the actual density of the graph 16% 24% 32% 40% 10 2 (22) 4 (44) (25) 8 (33) 10 (41) 48% Fig. 3. With increasing randomness, k-circulant graphs evolve to exhibit properties of random graphs in G(n, p). Small-world networks are intermediate between k-circulant graphs and random graphs in G(n, p) [7].

3 Algorithm 1 Altered Breadth First Search 1 st pass /* origgraph is a graph and a member to the class to which these methods belong */ Input: start node Output: hops Create hashtable visited, queue current, queue nextit Set hops to 0 Add start node to visited origgraph.getneighbors(start node) for each neighbor of the start node Add neighbor to current while current and nextit aren t empty while current is not empty Remove node from current Add node to visited origgraph.getneighbors(node) for each neighbor of the node if current, nextit, and visited doesn t contain the neighbor then Add neighbor to nextit end if Increment hops Copy contents of nextit to current error. Additionally, different from [4] and [5], the networks used are not based on real data, but are created from an algorithm to generate small-world graphs. Furthermore, [4] introduces systematic variation into the ground truth network by adding randomly-selected links to four different nodes. In contrast, we introduce a bias during the creation of subgraphs that comes from a focus on one specific node and nodes further away from the start node are less likely to be added to the subgraph. 3 METHODOLOGY Nine hundred sixty-three ground truth networks (connected undirected small-world graphs) were made using Python 2.7 and NetworkX ([7]). A ground truth network has 10, 25, 50, or 100 nodes. For each of those possibilities, a graph was generated with the probability of rewiring edges of with increments of 0.1. The higher the probability the more likely edges are to change thus creating a more random graph instead of maintaining its circulant form, see Figure 3. When rewiring an edge, one end remains the same and the other receives a new node; this introduces randomness into the network. For each generated subgraph, there would be a density of approximately 8%, 16%, 24%, 32%, 40%, and 48%, constraints permitting. NetworkX s method uses a third parameter, number of neighbors, to control the density of the graph. Neighbors are the number of nodes every node has a connection to. The algorithm in [8] considers the basis for a small-world graph is a k-circulant graph. According to [8], k is the number of neighbors every node has and must be even. Choosing four neighbors would Algorithm 2 Altered Breadth First Search 2 nd pass /* origgraph is a graph and a member to the class to which these methods belong */ Input: start node, hops, thresholdincrements Output: subgraphnodes Create hashtable subgraphnodes, hashtable visited, queue current, queue nextit Set hops to number from 1 st pass Add start node to visited origgraph.getneighbors(start node) for each neighbor of the start node Add neighbor to current while current and nextit aren t empty while current is not empty Remove node from current Add node to visited origgraph.getneighbors(node) for each neighbor of the node if current, nextit, and visited doesn t contain the neighbor then Add neighbor to nextit end if Generate a random number between 0 and 1 if generated number >= (1 (thresholdincrements * (hops + 1)) then Add node to subgraphnodes end if Decrement hops Copy contents of nextit to current mean that for each node, the node has connections to the two nodes on its left and the two on its right. The number of neighbors also has the constraint that they be less than Equation 4. (4) Any additional neighbors than that and the graph will contain duplicate edges. After the generation of the k- circulant graph, the method would randomly choose edges to rewire. One full set is 171 ground truths; for a view of the setup to determine this number see Table I. However, there was concern of a bias towards the subgraphs generated from the 100 node cases. The experiment considers every node in a ground truth as a starting point from which to create a subgraph. This is why there is a need to generate more cases for ten, twentyfive, and fifty node cases. As Table II shows, the experiment has thirty-four sets of ground truths for 10 node cases, five for 25 nodes, and two for 50. Thus, the number of instances with a ground truth of 10 nodes became 20,533; 18,606 for 25 nodes; 21,595 for 50 nodes; and 21,600 for 100 nodes. The number of samples does not include cases where a subgraph had no nodes or was exactly like a ground truth network. The additional instances allows for a more even distribution among the node levels, which will remove the bias when computing overall measures. For instance,

4 without the correction, if we compute the betweenness at every percent error value, then we would get values closer to the results of the 100 nodes cases rather than ones that represent all four node levels equally. Writing a plug-in for Gephi, open source software for graphs that allows the user to visualize and manipulate them, we were able to create all of the subgraphs. The creation process of a subgraph inserts the systematic error. Each node from the ground truth is chosen as the focal point for the subgraph. As the focal point, the other nodes surrounding it are considered to have less visibility and thereby less probability of being included in the subgraph, introducing bias. If a node is not included, then neither is its connections. This is different than [6] s approach of determining the robustness of centrality measures on imperfect data where random nodes and edges were added or removed to create a new graph. The algorithm to create a subgraph uses a slightly altered breadth first search (BFS) for connected graphs; see Algorithms 1 and 2. Instead of using one queue, the modified breadth first search uses two. One queue is for the current level of nodes for the ground truth and the second is for the next level out. A subgraph generation requires two passes through the amended BFS. The first pass (Algorithm 1) determines the number of hops (steps from the start node to the outer most node(s)). Hops or steps are the maximum number of edges it takes to visit all nodes. In Algorithm 1, every time the queue for the current level is empty, the number of hops is increased by one. Figuring out the number of steps determines the possibility of keeping (visibility) of each node. If there were four hops and the initial visibility was one, then the probability of keeping a node decreases by 0.2 at each step. Dividing the initial visibility (0.25, 0.5, 0.75, and 1, each is applied to all ground truths) by number of steps plus one allows for the possibility of keeping the outer most node(s) instead of removing them all together. Additionally, the start node has the possibility of not being included in the final subgraph. During the second pass (Algorithm 2), all nodes receive a visibility based on the number of steps it is from the start node. In addition, each node receives a number that represents its actual visibility. If the actual number of steps is greater than or equal to one minus the visibility the node is included in the subgraph. After deciding for every node in the ground truth to keep it or not, the edges that were connected to the removed nodes are also removed. The leftover nodes and edges make up the resulting subgraph. For every ground truth, the Gephi plug-in generates two output files. One contains the centrality values (betweenness, closeness, and degree) for each node and the other has the absolute difference of the subgraph values and ground truth values. The centrality values come from a statistics plug-in that came with the Gephi code. Production of the betweenness values is different than Equation 1. Instead, the creators of Gephi decided to use the algorithm in [2] to improve the speed of the calculations; the authors claim to compute the exact answer produced by Equation 1 unlike other methods that reduce computation time which only produce an approximate answer. Like betweenness, Gephi s closeness computations come to the same answers, but use an alternate algorithm. Gephi uses the algorithm from [2] that determines shortest paths. For degree, Gephi uses a node s degree. In addition to the new centrality files, a delta file contains the metrics Top 1, Top 3, and Top 10% from [6]. Top 1: Proportion of times that the most central node in the true network is also the most central node in the observed network. Top 3: Proportion of times that the most central node in the observed network is among the top three most central nodes in the true network. Top 10%: Proportion of times that the most central node in the observed network is among the top ten percent of nodes in the true network. The measures were a way of determining if the top person in a subgraph was in the set of top people in the corresponding ground truth. For the cases where multiple nodes had the same value, all nodes are included in the set of top nodes. The top measures consider this case during computation. 4 RESULTS A great deal of data was generated for and from this experiment. As mentioned before, 963 ground truths were created using NetworkX. From those, Gephi was able to create 82,334 subgraphs. The breakdown based on node level is in Table II. According to Figure 4, by increasing the number of nodes in a subgraph, it reduces the amount of error within it. We use to compute the amount of error in a subgraph. Additionally, the more nodes in a subgraph, the more accurate the centrality values are. According to Figure 5, regardless of centrality measure, the more error in a subgraph the less accurate the centrality values were. Inaccuracy in Figures 5, 6, and 7 is calculated as (6) (5)

5 Fig. 4. The average percent error for each number of nodes in a subgraph for every node level. Fig. 7. The average amount of inaccuracy for each percent error shown for every node level. TABLE II. NUMBER OF SUBGRAPHS GENERATED FROM 963 GROUND TRUTHS. IT DOES NOT INCLUDE CASES WHERE THERE WERE NO NODES OR THE SUBGRAPH WAS THE SAME AS THE GROUND TRUTH. Number of Nodes in Ground Truth Number of Subgraphs Generated for Node Level Number of Sets Created to Avoid Bias 10 20, , , ,600 1 Fig. 5. The average amount of inaccuracy for percent error shown for each centrality measure. Fig. 6. A closer look at Figure 5 s closeness and degree. and is a representation of how different the subgraph s value was to the ground truth s and how that difference compares to the ground truth s value. Betweenness had the worst performance because at 60% error the node levels started to have different amounts of inaccuracy, see Figure 7. The nodes in subgraphs, generated from ground truths with 50 and 100 nodes, had vastly different values in comparison to the ground truth values. Those cases had to compensate for a great deal of missing nodes. A subgraph made from a 50 nodes ground truth, according to the data and as seen in Figure 4, could contain 1 to 38 nodes. Figure 4 also shows that for 100 node cases, the subgraphs had anywhere from 2 to 69 nodes. We suspect the gap comes from how the subgraphs are generated and how the hops played out. Total 82,334 By observing Table III, we can see that the top person in the subgraphs is hardly ever in the top portion in the ground truths. Table IV compares our results to Borgatti et al s results for the 100 nodes, 50% density, and 50% error removed nodes case. Our values are from 100 nodes, 50% error, and 48% density; it is the case closest to Borgatti et al. s. In comparison to Borgatti et al. s case, our numbers are lower, significantly so for Top 3 and Top 10%. They concluded that the measures are robust for networks that contain error of 10% or less, even then whether the levels of accuracy are sufficient for any given purpose depends on external factors such as the consequences of error [6]. Since the accuracy levels for this experiment are lower, this indicates that betweenness, closeness, and degree are not robust when used on small-world networks containing systematic error. 5 CONCLUSION Based on Table III, it appears that the information about the top nodes in the ground truths is lost when looking at subgraphs. Not only that, according to Figure 5, the centrality values for the individual nodes in the subgraphs vary from the ground truth values, particularly betweenness. The accuracies in Table III prove that the centrality measures of betweenness, closeness, and degree are not robust enough to withstand missing information (systematic error) in connected undirected small-world networks. At least, they should not be used if the situation and consequences could not handle such low accuracy. Additionally, while Borgatti et al. found the measures to be

6 robust for small measures of error, based on Figures 5 and 6 this is not the case for this experiment. 6 FUTURE WORK Despite all the research done on the robustness of betweenness, closeness, and degree, it could be furthered still in the following ways. Firstly, an experiment could be removing edges only instead of nodes. This could then be compared to Borgatti et al.'s case for removed edges. Such an experiment would test the robustness of the centrality measures for the case where all people are known but not all of the relationships between them are. Additionally, another way to expand upon this paper's research is to always keep the start node. By doing this, one could use Bolland's metrics of local and configural sensitivity and be able to compare those results with his data. This study would determine how sensitive biased graphs (social networks where their creation starts with a focus on one person) are and possibly where. Finally, investigating the robustness of centrality measures when focusing on the most and least important nodes could further this study. We could determine how well those measures perform if the network was created from the most or least important person. REFERENCES [1] L. C. Freeman, A Set of Measures of Centrality Based on Betweenness, Sociometry, vol. 40, 1977, pp [2] U. Brandes, A Faster Algorithm for Betweenness Centrality, Journal of Mathematical Sociology, vol. 25, 2001, pp [3] (2013, May 28). Systematic error [Online]. Available: [4] J. M. Bolland, Sorting Out Centrality: An Analysis of the Performance of Four Centrality Models in Real and Simulated Networks, Social Networks, vol. 10, Sept. 1988, pp , doi: / (88) [5] E. Costenbader and T. W. Valente, The Stability of Centrality Measures When Networks are Sampled, Social Networks, vol. 25, Oct. 2003, pp , doi: /S (03) [6] S. P. Borgatti, K. M. Carley, and D. Krackhardt, On the Robustness of Centrality Measures Under Conditions of Imperfect Data, Social Networks, vol. 28, May 2006, pp , doi: /j.socnet [7] (2011, Oct. 21). NetworkX [Online]. Available: [8] D. Joyner, M. Van Nguyen, and N. Cohen, Algorithmic Graph Theory TABLE III. AVERAGE ACCURACY 10 Nodes 25 Nodes 50 Nodes 100 Nodes All Node Levels Top Top Top 10% TABLE IV. ACCURACY RESULTS FOR GRAPHS OF 100 NODES AND 50% DENSITY AND 50% ERROR Betweenness Closeness Degree Borgatti et al. s Results Our Results Borgatti et al. s Results Our Results Borgatti et al. s Results Our Results Top 1 Top 3 Top 10%