Homophily-Based Network Formation Models

Transcription

1 Homophily-Based Network Formation Models Senior Honors Project Final Report Scott Linderman School of Electrical and Computer Engineering Cornell University Frank H T Rhodes Hall Ithaca, NY swl28@cornell.edu ABSTRACT Many network formation models have been proposed to explain the characteristics of real world networks which cannot be captured by random graphs. Though random graphs do have a small diameter, they do not exhibit the high clustering and heavy-tailed degree distributions that have been observed in several real world networks. The Watts-Strogatz Small World model produces graphs with high clustering and the Barabasi-Albert Preferential Attachment model can create the power-law degree distribution of some real networks, but neither exhibits both properties alone. I believe one reason why these models do not acheive both realistic properties is that they do not incorporate the notion of homophily. The goal of this project was to explore network formation and evolution when the similarity of nodes is taken into account when forming edges. I examine many of these models and attempt to build similarity proportional neighbor selection into current models. I show one model which successfully exhibits both realistic properties. Keywords Networks, Network Formation Models 1. INTRODUCTION Homophily, the tendency for similar individuals to associate, is a known characteristic of real world networks. This tendency manifests itself in networks through clusters and communities. Its influence on the structure of networks makes it a key factor in how information or contagions spread on the network. Despite the importance of homophily, it is either absent or implicitly assumed to be uniform random in most prominent network formation models. The goal of my research project has been to analyze and experiment with network formation models with heterogeneous nodes where links are formed with some regard to the similarity of the two connecting nodes. This project has undergone many mutations, but it has always revolved around the idea of creating networks that exhibit the measured properties of real world networks while requiring similar nodes to be short distances from each other. The motivating example is a network of music listeners who desire to find new music that they enjoy, and in order to do so they associate themselves with others who have similar tastes. With this example in mind I experimented with many different models beginning with very detailed models of recommendation networks and eventually culminating with an abstract variant of prominent network formation models. In the final stage of this project I attempted to create a model which incorporates both the power law degree distribution and the high clustering that are often observed in real world networks but are not achieved by the standard models. The results of this work show that the concept of homophily is an important aspect of real networks, and that by modifying current network formation models to incoporate this idea we can create networks with more realistic properties. 2. PREVIOUS WORKS There has been a great deal of recent research on network formation algorithms, especially on models which have an aspect of randomness involved in link formation. Erdos and Renyi s seminal 1960 study [3] introduced the concept of random graphs in which each edge is formed independently with given probability. Though not necessarily realistic, this model offers interesting insight into the connectedness and degree distribution of networks. It was found that even randomly generated graphs are still connected and have a small diameter when the probability of forming an edge exceeds a certain threshold. Though random graphs have some nice attributes, they do not exhibit some of the fundamental characteristics of real networks, one of the most important being a large amount of clustering. Watts and Strogatz introduced the Small World Model [6] in 1998 to explain the formation of networks with small diameter and high clustering. Their model begins with a graph with complete local connections and then rewires a small fraction of the local connections to randomly chosen nodes. This retains high clustering,independent of the number of nodes, while also significantly reducing the average distance between nodes. Another model was introduced by Barabasi and Albert (BA) to explain the observed power law degree distributions in

2 some real world networks [2]. Their process of preferential attachment forms networks over time, a process they call growth, where each new node creates the same amount of edges, but chooses the destination vertex with probability proportional to the vertex s current degree. This favors older nodes and promotes the rich get richer concept. It was emperically determined, however, that the clustering coefficient in this model still decays in power-law fashion as the number of nodes increases. Holme and Kim proposed a model in 2002 [4] which attempted to exhibit both high clustering and a power law degree distribution. Their model explicitly allows for triadic closure to occur in an attempt to raise the level of clustering in the network. Holme and Kim achieve this by following each preferential attachment step of the BA algorithm with a Triadic Formation step. In this combined process, each new node enters the network, selects its neighbors first based on current degree, and finally randomly closes the triangles with its neighbors with a given probability. In this way, Holme and Kim are able to achieve networks with power law degree distribution with tunable clustering, based on the probability of closing triads. Though their model creates networks with both desireable characteristics, their method seems somewhat forced and unrealistic. Some attempts have been made to explain the real-world mechanisms which lead to the observed preferential attachment behavior. One of the more provactive explanations is the copying mechanism from Kleinberg et. al [5]. In this model each new node chooses a random partner to copy called the prototype. The new node then chooses each of its remaining edges by copying one of the prototype s edges with probability 1 p and choosing a random vertex with probability p. This has been shown to produce a power-law distribution of degree in a method which is equivalent to linear preferential attachment. There have been many modifications of these algorithms as well, and one of the most pertinent to this study was the Bianconi-Barabasi fitness model [1]. They modified the preferential attachment model by multiplying each node s probability of being chosen by a fitness parameter. This allows even young nodes to acquire edges quickly if they have a high relative fitness. Though this model differentiates nodes with a notion of valuability, it assumes that the value of a node is the same to all others. This is not the case in my models; rather than nodes having an inherent fitness, value is derived from similarity. 3. ATTEMPTED MODELS This project has evolved over time as various attempted models provided new insights and led to subsequent revisions. The evolution followed three main stages: I first attempted to simulate a complex network of nodes exchanging recommendations and reevaluating their neighbors based on the accuracy of their recommendations; then I tried to simplify the model by using difference equations to update the weights of an adjacency matrix; finally I attempted to incorporate homophily into three existing network models. These stages follow a trend towards simpler model with fewer parameters. I found that the more complex models were too difficult to analyze, and even though the simpler models may sacrifice some realism, their predictability makes them useful tools. 3.1 Recommendation Networks My first experiments involved a network of heterogeneous agents in a world with a finite number of valuable items. Each player had a ranking of these items, modelled as a sequence of the items, and initially they only were aware of an ordered subsequence of that ranking. In each time step the agents become aware of a previously unknown item, chosen based on the recommendations of their neighbors. With each recommendation the agents recalculate their pairwise valuations and then choose which links to form or sever. As players are exposed to more items their valuations evolve as do their choices of neighbors, thereby creating a system of simultaneous network formation and information diffusion. The model I used was fairly complex and involved very specific, hand-crafted functions for estimating the value of another player. To avoid the tedious mathematical details, the valuation function v ij can be summarized as a first-order linear predictor of the accuracy of j s recommendations to i used to scale the expected value of j s future rankings. This expected value is measured by finding a least-squares mapping of ranks of items in j s ranking to ranks for i, then finding the average predicted value of items which j could recommend. In each time step players receive recommendations from all of their neighbors and choose one recommendated item to be exposed to. This choice is made by considering the value of the player who made the recommendation, as well as the number of recommendations received for the item. For a more thorough treatent of this approach refer to the midterm report. The obvious complexity of this model makes it difficult to analyze and understand. Though I did see increasing tendency to link with similar neighbors and a slight increase in the value of the received recommendations, the results were very sensitive to the choice of parameters and not quite as expected. Rather than continuing with this model, I decided to switch to a simpler method with fewer variables. 3.2 Weight Matrix Update Rules In order to alleviate some of the complexity of the recommendation network I decided to model the network as a weighted directed graph with weight matrix W. These weights w ij were updated by two functions. First, I used a linear predictor to estimate the expected value of j s next recommendation based on previous recommendations values. Second, I added the i, jth element of the squared weight matrix W 2. The second component incorporates weight diffusion through the network based on the mantra a friend of my friend is also my friend. Recommendations were accepted from neighbors whose weight exceeded a given threshold, and items were chosen from among these recommendations in the same manner as before. Using the weight matrix eliminated the least squares predictions to value players and instead relied solely on past recommendation values and the neighbors value of the player. Though this model offered some nice simplifications to the original model, it turned out to have troublesome cases where the total weight of the matrix would increase without bound.

3 Normalizing the weights for each player prevents this symptom, but I believe the fundamental problem is that the linear predictor is not bounded and does not necessarily converge. Having found the recommendation framework and its requisite design copmlexities too unpredictable cope with, I switched course and defined similarity as Euclidean distance in feature space. This definition is far easier to analyze because it is deterministic and does not require sensitive definitions of how recommendations are made and valuation functions updated. The simplest realization of this similarity is a using a onedimensional feature. We give each node a feature value f i which is a real number on the range [0, 1]. Then, the similarity of two nodes i and j is defined as s ij = 1 f i f j (1) (a) This feature can be chosen according to any distribution, and obviously different distributions will yield different network structures. Unless otherwise stated, the following models assume the feature is drawn according to the uniform distribution on [0, 1]. The first model I used with this definition of similarity continued along the lines of the weight matrix update rules. Initially all nodes began in a small world graph with all edges equally weighted. Then at each time step each node chose one neighbor to associate with, resulting in one or more connected components of nodes. These neighbors are chosen with probability proportional to their similarity. Once these groups are chosen, each player sets the weight of those in his group equal to their similarity. The value of all players not in the group is depreciated by a constant factor β. Thus { sij j C w ij = i otherwise βw ij Figure 1: Figure (a) shows an evolution of groups in a simulation with 50 nodes. These snapshots are taken every 10 steps and the various colors show various groups. As you can see, over time the groups become more sparse, and there are more groups. This is because nodes become more likely to associate with similar neighbors. Figure shows the average similarity as defined above over time. This result is the average of 10 runs and you can see that similarity does indeed increase with time as nodes begin to exhibit homophily. where C i is the set of nodes in i s group. This update rule was designed to emphasize connecting to the most similar others and I expected that by choosing the node to associate with based on their similarity eventually groups of similar nodes would begin to appear. I found that the average similarity between nodes defined as V = 1 N N W [n] S[n] n=0 is initially around 0.7 starting in a small world, and that this model it usually increased in an oscillatory fashion towards 0.8 or even higher. When groups are chosen proportional to similarity I found that the highest weighted node for a given player oscillates around approximately 0.25 probability of being chosen, and that is not necessarily the most similar player in the entire world. This seems to indicate that the update rule is not discriminating enough against worse nodes. Perhaps over time the willingness to associate

4 with less similar nodes should decrease, just as in the simulated annealing algorithm. When every node chooses to associate with the highest weighted node, however, the average similarity approaches 1 very quickly. This, however, is a product of nodes finding their most similar neighbors and continually associating with them resulting in many groups of 2. Though this model has some promise, I doubt it will be able to create the power law degree distributions that we would expect from real social networks. Rather than continue experimenting with this model I attempted to incorporate the similarity function described above into some of the prominent network formation models since those already exhibit some desirable characteristics. 3.3 Homophily in Current Models I learned from my earlier experiments that a practical network formation model should not only exhibit the properties of the system it is trying to replicate, but should also achieve this goal using as few parameters as possible without sacrificing plausibility. With this in mind I attempted to incorporate my similarity function into three well-known network formation models: Watts and Strogatz s small world model, the Barabasi-Albert preferential attachment model, and Kleinberg s version of the prefential attachment model with the copying mechanism. There is some implicit notion of homophily that could be assumed in some current models. For instance, the ring lattice used to initially connect nodes in the Watts-Strogatz model could be interpreted as a measure of similarity where like nodes are positioned near each other on the ring. The Preferential Attachment model, however, does not have an obvious analogy for similarity based connections. I would argue that it is the notion of homophily that causes dense communities to form, and that the Watts-Strogatz model s inherent inclusion of this idea is what allows it to create dense networks with high clustering. Though Preferential Attachment creates networks with slightly higher clustering than random graphs of the same size, it has been empirically shown that the clustering coefficient still decreases with the number of nodes according to a power law. By modifying Kleinberg s variant of the Preferential Attachment model to explicitly factor similarity into link formation I create a model which exhibits high clustering as well as a heavytailed degree distribution Valuation Function In the following sections I define the total value of node j to node i as v ij = α s ij d k s + (1 α) j ik k d k where d j is the degree of vertex j and α is a constant on the range of [0,1]. Thus the value of a node is based on both its relative similarity as well as its relative degree and the tradeoff is tunable with parameter α Cost Threshold based on SW (2) I interpreted the ring lattice used to connect nodes in the Watts-Strogatz model as a result of connecting nodes to neighbors less than a certain distance away in similarity space. As a logical extension to this method, I imagined positioning nodes in a metric space which incorporates both similarity and connectivity, then followed the same approach and connected nodes to nearby neighbors. I used the valuation function above to measure similarity and connected to nodes whose value exceeded a given cost threshold. I began with all the nodes unconnected. Then at each time step every node connected to the node with the highest value, as long as the value exceeded some cost threshold. I found that the cost threshold had to be kept quite low in order for any connections to form initially, and then once a node gained enough edges to exceed the cost threshold with connectivity alone, it rapidly accumulated edges from every other node. This is not a surprise since in each time step all nodes connect to the maximum valued other, and an increase in one node s degree causes all others connectivity value to decrease. Adding an element of randomness, perhaps choosing nodes to connect to with probability proportional to their value, would most likely produce a more even degree distribution, but this does not fit as nicely with the cost threshold idea Preferential Attachment In order to work around the cost threshold sensitivity I found in the small world modifications I decided to try modifying the preferential attachment model instead. In my version of the model new nodes are created every time step. Upon creation they choose a parameter m nodes to connect to. I chose the new neighbors using fitness proportional selection, so the probability of a new node i choosing j as a neighbor is P i(j) = vij k v ik I found that this model yielded the expected power law degree distribution, but it actually had a negative affect on clustering. This was because the similarity aspect of the valuation function did not place enough emphasis on the most similar nodes. As nodes accumulated more neighbors, the probability of actually choosing a node and its neighbor quickly approached 0, even when similarity was taken into consideration. Furthermore, older nodes (those which were created first) quickly accumulated degree greater than m, so even if a new node connected to an old node, it could not possibly connect to all of his neighbors as well. This is one inherent feature of preferential attachment models which prevents perfect clustering. Nevertheless, I found that Kleinberg s variant of preferential attachment, which I call the copying mechanism, allows for much higher clustering than the normal version Holme and Kim I incorporated homophily into Holme and Kim s model of preferential attachment with a triadic closure step by choosing the original neighbors with similarity proportional selection, just as was done in the version of the Barabasi-Albert

5 Figure 3: This figure shows the clustering coefficient using the modified version of the Holme and Kim preferential attachment model for various values of p, and the effect of varying α for each p. As the probability of closing triads,p, is increased the clustering increases as expected. It appears that increased reliance on similarity actually decreases the observed clustering in these experiments, though I do not have a good explanation for why this occurs. Proof of the power law degree distribution is omitted but can be found in Holme and Kim s paper. preferential attachment model. I obtain the same results as Holme and Kim - the clustering does increase with larger values of p - but the inclusion of the similarity measure had the opposite effect of what I intended. Holme and Kim show that for larger values of N the clustering approaches an equilibrium value proportional to p. Though this method successfully creates graphs with high clustering, I felt that the method of achieving these results was rather contrived. The Kleinberg model of web formation, which I have referred to as the copying mechanism, is a more realistic model which actually exhibited higher clustering as well Copying Mechanism In order to retain the realistic characteristics of preferential attachment, my model begins with Kleinberg s copying mechanism. A new node is introduced at every time step and again chooses a prototype node to copy. Rather than choosing this prototype randomly, however, I choose based on a convex combination of the vertex s degree and the similarity of the new node. Prototype vertices are chosen in proportion to their value, the same way that new neighbors were chosen in the preferential attachment-based model. As in the preferential attachment model, the algorithm starts with m 0 unconnected vertices. At each time step a new node is added and chooses a prototype to copy based on the scheme described above. Each incoming node connects to m m 0 others, including the prototype. Each of the other vertices is chosen by either copying one of the prototype s neighbors with probability 1 p or choosing a random neighbor with probability p. Varying the parameter α of the value function from 0 to 1 varies the weight on connectivity versus similarity. When α = 0 protoypes are chosen based solely on connectivity and the model is very similar to the preferential attachment model. If α = 1 incoming nodes choose prototypes irregardless of connectivity and the resulting model lacks the powerlaw degree distribution. Intermediate values of α exhibit both heavy-tailed degree distributions and high clustering. Experimental results show that these intermediate values of α produce clustering which is much higher than in the original preferential attachment models with the same number of nodes. This clustering still decays with increasing number of nodes, but it does so much more slowly than in the first model, the Preferential Attachment model. Choosing prototypes with emphasis on connectivity results in the high degree nodes being chosen, and if the prototype s degree is larger than the number of edges the incoming node creates then there will be many possible triads of nodes which cannot all connect. When emphasis is placed on connecting to similar nodes the probability of connecting to high degree nodes is lessened and more possible triads are closed, thus the clustering coefficient is greater for larger values of α (i.e. when the emphasis on homophily is stronger). If we examine the copying mechanism alone, when prototypes are chosen uniform randomly, we see that the original model also produces relatively high clustering. The clustering for the original model can be seen in plot of Figure 4 for various values of p. For p = 1, as in the simulations of the Copying Mechanism with similarity model above, we see that clustering is still much larger than in the original Preferential Attachment model and approximately the same as in the similarity enhanced version of the Copying Mechanism for α = 1. These results are very dependendent upon p, the probability of copying the prototype s neighbors. When this probability decreases and more random neighbors are chosen, the clustering decreases quickly. Copying neighbors is essential to creating high clustering because it connects triads of nodes. Randomly choosing neighbors is unlikely to produce the same effect. 4. CONCLUSIONS Although the present results seem somewhat varied and inconclusive, I feel there is still a role for homophily in network formations models. In this project I have explored many different ways of generating networks with nodal similarity. I ve examined hand-crafted models as well as logical extensions of established network formation models with homophily considered during the edge selection process. From the mixed results, I feel a few positive highlights have emerged. My experiments yielded new insights into how similarity can affect network formation. These results show that strong group formation can emerge, it just requires the right param-

6 (a) (a) Figure 2: Figure (a) shows the resulting degree distribution for various values of paramete α. As you can see, when α approaches 0 (complete emphasis on connectivity) the model is equivalent to the original preferential attachment model and the power law is maintained. As α is increased and similarity is more heavily valued, the degree distribution tends towards a more exponential form with fewer nodes accumulating large degree. Figure shows that the clustering did not increase for larger values of α as I had hoped. Connecting with similarity proportional selection did not cause higher clustering in this model. Figure 4: Figure (a) shows the affect of varying α on the clustering coefficient vs N for m = 20 and p = 1. The clustering coefficient still seems to follow a power-law type of relationship initially, but due to computational constraints issues I was unable to examine the behavior of the clustering coefficient for larger values of N. Small world models with the same m exhibit clustering coefficient equal to approximately 0.5 and independent of N. For higher values of α we see clustering coefficients which are comparable for the range of N examined. Figure shows the affect of varying α on the degree distribution for m = 20 and p = 0. The log-log plot shows the linear relationship characteristic of the power law distribution. Intermediate values of α retain the power law form but eventually sole reliance on similarity results in a distribution which decays approximately exponentially. The number of nodes with degree less than m needs further investigation.

7 eters to simultaneously see the traditional network features like power law degree distributions and high clustering. (a) Figure 5: Figure (a) shows the affect of varying p for fixed α in the Copying Mechanism model with similarity proportional prototype selection. As expected, the clustering shows a marked decrease over all numbers of nodes as the probability p of copying the prototype is decreased. The largest difference is between p = 1.0 and p = 0.75, which seems to indicate that even a slight probability of choosing random neighbors can severely decrease the clustering. Figure shows the results of the same experiment run on the original Copying Mechanism without similarity. Notice that the clustering decreases at a faster rate in the original model which seems to indicate that the similarity component can indeed increase clustering. Many aspects of my models could be studied further. In particular, the role of similarity is directly affected by the distribution of the characteristic feature value in the population of nodes. Experimenting with the homophily models under different attribute distributions should give rise to different results. For example, if the nodal attribute is drawn from a bimodal distribution, under the right parameters, I would expect to see two distinct groups form when the weight on similarity is high. Other interesting experiments would be to shift the population s average characteristic value over time to simulate a change in general opinion of the population. The range of possible extensions with this model is unlimited. Finally, the effect of homophily in other network formation models needs to be studied. I selected the models to emulate based on their performance in achieving other desireable network properties, but adding nodal similarity to other models should also produce intersting and valuable results. Although there are many more avenues through which homophily could be introduced to network formation models and much work which could be done, these results show promise and hopefully more study will reveal further insights into role of homophily in network formation. 5. REFERENCES [1] R. Albert and A. Barabási. Statistical mechanics of complex networks. [2] A. Barabasi and R. Albert. Emergence of scaling in random networks. Science, 286: , [3] P. Erdos and A. Renyi. On the evolution of random graphs. Publication of the Mathematical Institute of the Hungarian Academy of Sciences, 5:17 61, [4] P. Holme and B. J. Kim. Growing scale-free networks with tunable clustering. Phys. Rev. E, 65(2):026107, [5] J. M. Kleinberg, R. Kumar, P. Raghavan, S. Rajagopalan, and A. S. Tomkins. The Web as a graph: Measurements, models and methods. Lecture Notes in Computer Science, 1627:1 17, [6] D. Watts and S. Strogatz. Collective dynamics of small-world networks. Nature, 393(6684): , June 1998.