Overlay Partition: Iterative Detection and Proactive Recovery

Transcription

1 Overlay Partition: Iterative Detection and Proactive Recovery Tongqing Qiu, Edward Chan and Guihai Chen State Key Laboratory for Novel Software Technology, Nanjing University, China Department of Computer Science, City University of Hong Kong, China Abstract Overlay networks provide infrastructures for a large variety of Internet applications, like file sharing, online gaming, and multimedia streaming. However, these networks often face unexpected node failures and network disconnections, causing the overlay to be partitioned into several components, which may seriously affect the performance of the network. In this paper, we analyze the cause of overplay partitions and its impact on the efficiency of the system. After explaining the notion of half-life, a measurement used to describe the evolution of peers in overlay network, we propose a new construct called halfsuccess as a measure of critical partition. Furthermore, we propose an iterative method for the detection of potential partitions and a proactive strategy for the prevention of such partitions. Simulation experiments show that our detection method can efficiently find almost all of the cut vertices at a low cost. In addition, we demonstrate that the proposed proactive scheme is more effective and much faster than reactive approaches. I. INTRODUCTION In the last few years, overlay networks have rapidly evolved into a promising platform for deploying new applications and services in the Internet [1]. An overlay network is a computer network which is built on the top of another network. Nodes in the overlay can be thought as being connected by virtual or logical links, each of which corresponds to a path, perhaps through many physical links, in the underlying network. Overlay networks can be constructed in order to route messages to destinations not specified by an IP address in peerto-peer systems [2], [3], [4], [5]. However, application-level overlay usually face unpredictable failures. This is because each node/peer is viewed as an independent entity, which can join and leave the network at will. Moreover, the underlying network is not always reliable, since packets can be lost due to transient problems, network link failures and in some cases malicious attacks. In order to provide quality services, an overlay network needs to guarantee reliability and robustness. Two basic elements to achieve this are failure detection and recovery. Failure detection algorithms can be broadly classified as either active or passive [6]. In the active approach, a node periodically sends keep-live messages; while a passive one only uses data packets to convey liveness information. While both methods can discover failures in specific nodes or links, they may not be able to detect cumulative effect of failures at the system level. Regarding failure recovery, most recent studies focus on a reactive approach to the resilience of overlay i.e. how to recover after failure is detected. The major drawback of the reactive approach is that additional information needs to be collected after a failure has been detected, resulting in a significant delay in reconstructing the overlay. More importantly, it is not always possible to collect enough information for the recovery process after the failure has occurred, especially when the overlay has been partitioned due to the failures. In this paper, we analyze a serious failure condition, overlay partition, and propose suitable detection and recovery mechanisms. Our main contributions are as follows: To the best of our knowledge, this is the first comprehensive study of overlay partitions that includes both theoretical analysis and experimental evaluation. We used the notion of half-life and propose a sign of critical partition: half-success. We formulate an iterative method to detect potential partitions in the network. Our method is distributed, lightweight and is faster than similar algorithms. We also propose a proactive recovery scheme which can recover from node departures much faster than reactive methods. II. PRELIMINARIES We consider an application overlay as an undirected connected graph G =(V,E), where V is a set of nodes and E is the set of logical links. An edge xy E means that x knows a direct way to send message to y. G is called k-connected when no two vertices of G are separated by fewer than k other vertices. The greatest integer k such that G is k-connected is the connectivity κ(g) of G. Recall that cut vertex is a vertex whose removal would disconnect the connected component containing it, and bridge means a edge whose removal would lead to this kind of disconnection. A biconnected graph is a graph with no cut vertices. A biconnected component (bcc) is a maximal biconnected subgraph. In the following two sections, we will illustrate that overlay partition occurs frequently and has a major impact on the efficiency of the system, and that it is necessary to effectively detect overlay partition and quickly recover from partition. A. The Causes of Overlay Partition In order to find the causes of overlay partition, we first address the following questions: How many node failures will /07/$ IEEE

2 result in the partition of an overlay network and how long does it take? More generally, an overlay graph G with connectivity κ(g) will be disconnected when at least κ(g) nodes fail at the same time. We call these κ(g) nodes critical nodes. The remaining ones are normal. Overlay networks are usually sparse: in overlay multicast systems or unstructured P2P systems, the minimum degree δ(g) is usually a small integer; even in structured P2P systems, δ(g) is usually no larger than O(log N). Given that κ(g) δ(g) and κ(g) << N, the number of critical nodes is much smaller than the normal ones. So we focus our study on partitions among the normal nodes. Consider a model of arrivals and departure [7]: nodes arrive according to a Poisson process with rate λ and depart according to an exponential distribution with rate µ (i.e. the expected lifetime is 1/µ). Given that only removal leads to partition, the probability p that a node fails in time τ is 1 e µτ. We call the interval from the initial time when G is connected to the time when the normal nodes are partitioned partition time τ 0. If the failed nodes are just those critical nodes, the overlay will be partitioned after κ(g) nodes fail. So the lower bound of the time to be disconnected is shown in equation 1, by letting κ(g)/n =1 e µτ, τmin 0 = 1 µ ln N (1) N κ(g) Under another extreme condition where the first failed N κ(g) are normal nodes, we can get the corresponding upper bound: τmax 0 = 1 µ ln N (2) κ(g) In reality, the remaining N κ(g) nodes are not always normal nodes. Moreover, when different nodes fail continually, normal nodes may turn into critical nodes. So the upper bound of partition time is quite loose. Based on the range τ 0 1 µ [ln N N κ(g), ln N κ(g) ], we can obtain additional information. First of all, the partition time τ 0 is influenced by two parameters: the activity µ and the connectivity κ(g). More specifically, the change of time is intuitively proportional to node s life time 1/µ. Second, if only the analytical range is meaningful (τmin 0 τ max), 0 no matter how κ(g) changes, in other words, regardless of what the overlay is like, the partition time can just be τt 0 =(1/µ)ln2, when κ(g) =N/2. Figure 1 shows the existence of this value. It is worth mentioning that Liben-Nowell et al. [8] use half-life, the time required for half of nodes to depart, to measure the evolution of P2P systems. Their notion of half-life is just τt 0. In their paper, they did not explain the reason for using half-life. We give one reason here: Half-life means how long the overlay will be partitioned regardless of the topology of the overlay. Third, because κ(g) is much smaller than N, the partition time is rather short. The trace from PlanetLab [9] also illustrates that partition occurs frequently. Moreover, due to The range of partition time Half-life Upper bound Lower bound The connectivity k(g) Fig. 1. Half life. We choose µ =1and N = 100 the distributed, even random organization of overlay, cases of κ(g) =1and τ 0 0 are very common. In this condition, the failure of one node can lead to the disconnection of the whole overlay. Consequently, one possible approach to avoid potential partitions is to periodically exploit cut vertices. It is obvious that after the first partition, the connectivity of deduced graph κ(g ) may be larger, smaller or equal to that of original graph. So the time when the next partition occurs (τ 1,τ 2,...) is unexpected, although eventually τ 0. Note that the range of τ 0, is determined by both the overlay topology (connectivity) and the number of failed nodes (activity). While our partition detection method can partially describe the topology, it is impossible to know the precise number of failed nodes in a distributed system. Furthermore, different nodes have various positions and importance in the system. So this factor cannot be directly used as a sign to detect or evaluate partitions. B. The Impact of Partitions We use the successful look-up ratio (SLR) of requests as the basic metric to examine the impact of partition. It is defined as the ratio of successful query requests to the the total number of requests. If there are m different biconnected components, and the ith component C i has N i nodes, then without any recovery, SLR = m N 2 i N 2 100%, (0 <N i <N;0<m N) (3) Obviously, the number of bbc affects the efficiency of lookups. Under an extreme condition where m = N (i.e. each node is one independent component and N i =1), each node can only find contents located in its own site, and SLR =1/N 2. Moreover, if we view SLR as a function of m, considering that m N i = N, we can easily obtain the lower bound of SLR, and the upper bound SLR max (m) =1 SLR min (m) = 1 m 2(m 1) N (4) m(m 1) N 2 (5)

3 The upper bound represents a condition that m independent nodes are disconnected from the system. When m<<n, this kind of partition can be omitted. It indicates that not all partitions are serious. When it comes to lower bound, equal size of bbc leads to the worst case scenario. Specifically, assuming that the overlay is going to be partitioned into two independent components, in the worst case SLR = 50%. We call this a half-success situation. Conversely, we have every reason to suspect that: The overlay network has been partitioned into two equal size components or even more components under the condition of half-success. We will show in the simulation section that SLR=50% is in fact a reasonable indicator of a critical partition. Overlay partitions not only greatly decrease the successful lookup ratio, but also make failure detection and recovery more difficult. First of all, if the detection method is based on periodic probing within a local range (usually the neighborhood), then it may be impossible for internal nodes within an independent component to discover the failure which occurs on the edge of components. Second, once the partition occurs, the nodes do not know how to make connections between independent components. It is therefore imperative that partitions should be detected as early as possible; and if they occur at all, recovery should be prompt. We will now introduce our iterative detection method and proactive recovery scheme which satisfy these requirements. III. APPROACH FOR MANAGING PARTITIONS In this section we present the key elements in our approach for managing partitions in an overlay network. We first describe the basic detection and recovery mechanisms, and then extend them to handle group failures i.e. partition caused by the failure of a group of nodes. A. Iterative Partition Detection There are several challenges in exploiting cut vertices. First, it is hard to collect the global information in a distributed system and even if it is possible, the overhead is relatively high. Most of the distributed algorithms to find cut vertices are based on the construction of depth first search (DFS) trees [10], [11]. For each node n in the tree except the root, check the neighbors to which n s descendants are connected to. If none of the neighbors of n s descendants are n s ancestors, n is a cut vertex. These distributed methods can find all cut vertices in a static network, but the average cost of detection is about O(N 2 ), where N is the number of nodes. Second, since the overlay is dynamic, cut vertices may fail and cause partitions before they are detected. So it is important to detect potential partition quickly. Take a recently proposed method CAM [12] as an example. It uses flooding to detect cut vertices. Although its overhead only O(N), the flooding will not stop until a specific TTL (time-to-live) value has been reached. Moreover, it cannot automatically tune the flooding range and probing times according to different overlay conditions. Fig. 2. Iterative detection We use an iterative method to locate the cut vertices. In this scheme, each node C iteratively detects some nodes and determines whether C is a cut vertex. Each node has a Boolean matrix to describe the connection relation. For any items x, y, if x and y are connected, (x, y) =true in the matrix. Based on our model of overlay, the relation is reflective, transitive and symmetric 1. The matrix also records the state of each node i.e whether it is in the active or inactive state. Initially, only C s neighbors are in the matrix and they are active. At the beginning of each partition detection cycle, node C asks active nodes to feedback their neighbors information. The message contains the C s IP address, a timestamp, and identification. The receivers respond based on the timestamp and identification of the senders. If a node has already received the message for the same requester or it is not current, then the message is simply dropped. Otherwise, the node will send back its neighborhood connection information to node C. Then C makes the necessary changes in the matrix: it first sets all active nodes to the inactive state, then adds new entries for new nodes and sets them to be active. Moreover, it changes the Boolean values based on the feedback messages. After a simple circulation of transitive closure using Warshall s algorithm [13], the information in the matrix is finalized for that cycle and the number and the size of bbc are determined using this information. For instance, in Figure 2(c) we have two components N 1 =2N 2 =5, respectively. A threshold t is used to limit the number of detection cycles, and the detection process will automatically stop when the number of components m =1. Our detection method is lightweight and effective. Assume that the overlay network has N nodes, and c is the average number of connections of a node. In the worst case, the main overhead of detection is the sum of information retrieval in each detection cycle O((c +(c 2 c)+ +(c t c t 1 ))N) = O(c t N), which is much less than the traditional DFS method O(N 2 ) when c is small. Although this overhead is the same as that of CAM, based on the fact that most of nodes are not cut vertices, in most cases our detection operations will stop before t trials. This means that the detection will be completed much earlier and overhead is reduced. This is a very useful feature since it is difficult to select in advance a good value for a constant threshold to limit the probing range. For instance, 1 The symmetric property is true when we assume that overlay is an undirected graph. In some cases, especially in some structured peer-to-peer systems, it is not true. Since detection is related to transitive closure, its correctness has nothing to do with symmetry.

4 in CAM, a large value for TTL will lead to an exponential growth in overhead. In our approach, setting a larger threshold seldom leads to problems as the detection mechanism usually terminates before the threshold is hit. Note that the detection process begins only when node C is stable. We are able to determine when this stage has been reached since many researchers [8], [14] have studied and characterized the behavior of peers in a stable state. By starting the detection process when the network is in a stable state reduces the unnecessary cost of detection and recovery for those peers that enter and exit the network shortly afterward. B. Proactive Recovery The traditional reactive overlay recovery mechanisms can be broadly classified into two categories based on their different ways of collecting information: central retrieval (CR) and local retrieval (LR). In the first approach, a peer tries to contact a set of well-known nodes and retrieve the locations of other nodes, and then re-establish the connections again. This method can be used in a loosely organized system where no specific requirements on connection exist, like Gnutella [2]. The weakness of this method is that well-known nodes with centralized information are required. Failures of such nodes may cause the whole system to crash. In the local retrieval approach, a peer asks the neighbors to collect information of nodes that are far away. This method is widely used in structured peer-to-peer systems [4], [5], [15], since the peer will qualify each candidate. As we discussed above, a major problem with this approach is that the nodes do not have precise information on how to make connections between independent components when overlay partition occurs. In order to solve this problem, we propose a proactive recovery method in which a recovery plan is devised before the partition occurs. For each cut vertex C, it must leave recovery information for their neighbors. The transfer and acknowledgement of such recovery information can be piggybacked on keep-live messages. Once its neighbors detects that node C fails, they know exactly how to avoid partition by adding new connections. Note that only one single message from the neighbors is needed to complete the recovery process. This is much faster than the reactive approach. Ideally there should be as many logical connections as possible to eliminate the partitions. Given that in a dynamic or error-prone distributed environment, it may not be possible to avoid partitions all the time. As a result the choice of component union is very important. Assuming that components C x and C y are combined, the resulting improvement in SLR after the union operation is: SLR = (N x + N y ) 2 N 2 N 2 x + N 2 y N 2 = 2N xn y N 2 (6) The result shows that the union of two components with more nodes will lead to greater improvement. So we propose a larger-one-first scheme to maximize SLR. We will first sort the m components according to their size N 1 N 2... N m, then add new connections C 1 C 2,C 2 C 3,...C m 1 C m.more Fig. 3. Different recovery conditions specifically, when there is a new connection between C x C y, it means that two neighbors of node C, n i and n j, will be connected together, where n i C x and n j C y. If there are more than one candidates in the choice of neighbors, we choose the one with less degree to balance the load. However, it is easy to proof that all these new connections C 1 C 2,C 2 C 3,...C m 1 C m are bridges. So we need to connect components C m C 1 to eliminate all introduced bridges. Specifically, we will add two connections between the two components when m =2. It is very likely that node C does not have enough time to finish probing and pre-computation of partition components. But we can still use the latest partition information to add new conditions and to ensure connectivity. This feature further illustrates that iterative detection is necessary. Figure 3 shows different recovery conditions before and after the accomplishment of the first detection cycle. In (a), since the failed node believes that there are the three components, the three neighbors are reconnected together to form a ring-like structure. In (b), because the failed node thinks that there are two components, so two connections are added. Note that not all overlays allow the addition of arbitrary connections because of the requirements of overlay construction protocols. We propose that a best effort mechanism be used to maximize the successful lookup ratio, since this allows for some flexibility and variation on how the connections are made. IV. PERFORMANCE EVALUATION We use GT-ITM [16] to generate the topology of an Internetlike network. A number of nodes are then selected as peers in the overlay network. Since the aim of the simulation is to show the correctness of our analysis and the effectiveness of our mechanism under different network conditions, we use generated traffic instead of a real trace so that we can easily control the connectivity of network as well as the scale of the system. The default behavior of peers follows the Poisson arrival and exponential departure model. A. Partition Time We start from a stable overlay with 1,000 nodes. The overlay graph is manually generated with different connectivity κ(g) =1, 4, 8, 16. Then nodes depart according to the exponential distribution with rate parameter µ (i.e., expected node lifetime is 1/µ =10, 20, 30, 40). The unit is time slot.

5 Partition time k(g)=1 k(g)=4 k(g)=8 k(g)= Average life time Successful lookup ratio % 20% 30% 40% 50% 60% 70% half-success 1.6e e+006 2e e e+006 Simulation time(ms) Fig. 4. Partition time Fig. 5. Successful look-ups ratio for varying percentage of node failures Each experimental result is based on the average of 20 simulation runs. Figure 4 shows the relationship between partition time, average life time and overlay connectivity. The result is generally consistent with our analysis in Section II. However, the partition time is not strictly proportional to the average life time, especially when the connectivity is large. This is because in our analysis, we consider a discrete condition where overlay transfers from connection to disconnection with several nodes fail but without the change of connectivity. In fact, when the nodes fail continually, connectivity may decrease due to the loss of redundant paths or connections. So the time of partition is earlier than expected. B. Impact of Overlay Partitions In this section we will illustrate two consequences of overlay partition: decreased SLR and failed recovery. We take the well-known P2P system Tapestry, which uses local retrieval recovery, as an example. There are 1000 nodes, with average round-trip-time (RTT) = 2s between any pair of nodes. Each node starts to generate lookup requests at the interval exponentially distributed about the mean time of 10s. If a look-up does not reach the destination in 20s, it is regarded as a failed look-up. Stabilization routine is trigged every 100s on average. At time = 1800s, when the topology is considered stable, we let a subset of nodes (ranging from 0% to 70%) leave the network simultaneously instead of following the exponential departure model, Recall that in our analysis, SLR = 50% is a reliable indicator for the occurrence of overlay partition. Figure 5 shows than when the SLR drops to 50%, the network never recovers from the failure. Due to the paper limitation, we merely present Tapestry as an example. According to our previous evaluation [17], some other overlay management protocols cannot successfully recover from from the condition of half-success. This result is not surprising since none of these recovery schemes focus on overlay partition. C. Effectiveness of Partition Detection We use three different metrics to evaluate the correctness of partition detection: accuracy rate (AR), false negative rate CDF of accuracy rate % Iterative 10 CAM 0 DFS Detection time(ms) Fig. 6. Cumulative distribution of the detection time (FNR), and false positive rate (FPR). It is obvious that the FNR of our cut vertices detection algorithms is zero. In other words, all nodes that are truly cut vertices will be detected. However, many nodes that are normal ones will be identified as cut vertices. This false positive rate is highly dependent on the threshold t. Table I shows the accuracy rate of partition detection. When t =3, the accuracy rate is more than 95%. Detection time is another important metric to evaluate the efficiency of partition detection methods. We compare our iterative method with other two methods: CAM and DFS. Figure 6 depicts the cumulative distribution of the detection time. It is obvious that our method is faster than the other two methods. For DFS, it has to construct a DFS tree first, so most of the time is wasted on the construction, TABLE I ACCURACY RATE OF PARTITION DETECTION t=1 t=2 t=3 t=4 AR 73.3% 83.2% 97.4% 98.2% FPR 33.8% 20.6% 6.6% 3.4% FNR 0.0% 0.0% 0.0% 0.0%

6 although DFS does provide a more accurate result. According to the description in section III, for CAM, each node will flood its request to nodes TTL hops away. Our method use an iterative method to generate an intelligent response based on the result of each fault detection cycle so that the fault detection time is shorter. Given enough time, however, these two methods achieve nearly the same accuracy rate. V. RELATED WORK Researchers are aware that the topology of overlay has a significant impact on resilience. S. Saroiu et al. [18] show that the failure of a small number of high degree nodes can effectively shatter the overlay network, which makes such a network highly vulnerable in the face of well-constructed, targeted attack. Ripeanu and Foster [19] point out that if every node in an overlay network can have a degree above a constant lower bound, the resistance to malicious attacks can be greatly strengthened. However, their work based on node degrees cannot precisely describe the influence of different nodes, nor can they be used independently to optimize overlay topology. More recently, the overlay partition problem has received some attention from researchers. For instance Sit and Morris [20] propose a cross-check approach for partition detection. By asking other nodes to do random queries and comparing their results with its own, a node can verify whether its view of the network is consistent with others. This method can detect whether a node is isolated. But based on our analysis of the impact of partitions, the isolation of a specific node does not necessarily lead to the crash of the system. Regarding partition recovery, a merging algorithm in proposed in SkipNet [21] to recover from partitions. Unfortunately, this recovery scheme is based on a specific structure and organization. As mentioned earlier in the paper, our detection method is better than CAM proposed by Liu et. al. [12]. When it comes to recovery, CAM simply connects random nodes in different components after finding the cut nodes. To our knowledge, no general, well-designed method has been proposed to repair the partitioning of an overlay network. VI. CONCLUSION In this paper, we investigate an important issue that affects the resilience of overlay: overlay partitions. We analyze the cause and impact of overlay partitions and the simulation results support our analysis. Based on the characteristics of overlay partitions, we further propose an iterative detection method to exploit the cut vertices. This method is distributed and lightweight. Moreover, we suggest a proactive recovery mechanism, which is much faster than recent reactive methods. We are now planning to extend our work by using use real traces as well as different types of peer behavior in our experiments to evaluate our partition detection and recovery method. ACKNOWLEDGEMENTS The work is partly supported by China NSF grants ( , ), Jiangsu Provincial NSF grant (BK ) and China 973 projects (2006CB303004). The Conference Participation is partly supported by Nokia Bridging the World Program. REFERENCES [1] D. G. Andersen, H. Balakrishnan, M. F. Kaashoek, and R. Morris., Resilient overlay networks, in Proceedings of SOSP 2001, [2] [3] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan, Chord: A scalable peer-to-peer lookup service for internet applications, in Proceedings of the ACM SIGCOMM, [4] A. Rowstron and P. Druschel, Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems, in Middleware 2001, [5] B. Y. Zhao, L. Huang, J. Stribling, S. C. Rhea, A. D. Joseph, and J. D. Kubiatowicz, Tapestry: A resilient global-scale overlay for service deployment, IEEE Journal on Selected Areas in Communications, vol. 22, no. 1, pp , Jan [6] S. Q. Zhuang, D. Geels, I. Stoica, and R. H. Katz, On failure detection algorithms in overlay networks, in Proceedings of INFOCOM 2005, [7] G. Pandurangan, P. Raghavan, and E. Upfal, Building low-diameter p2p networks, in Proceedings of IEEE Symposium on Foundations of Computer Science 2001, [8] D. LibenNowell, H. Balakrishnan, and D. Karger, Analysis of the evolution of peer-to-peer systems, in Proceedings of Principles of Distributed Computing(PODC 2002), [9] A. Mislove, A. Post, A. Haeberlen, and P. Druschel, Experiences in building and operating epost, a reliable peer-to-peer application, in Proceedings of Eurosys 2005, [10] W. Hohberg, How to find biconnected components in distributed networks, Journal of Parrallel and Distributed Computing, vol. 9, no. 4, pp , August [11] J. Park, N. Tokura, T. Masuzawa, and K. hagiharra, Efficient distributed algorithms solving problems about the connectivity of network, in Proceedings of Systems and Computers in Japan, [12] X. Liu, L. Xiao, A. Kreling, and Y. Liu, Optimizing overlay topology by reducing cut vertices, in Proceedings of NOSSDAV 2006, [13] T. H. Cormen, C. E. Leisereson, and R. L. Rivest, Introduction to Algorithms. MIT Press, [14] F. Bustamante and Y. Qiao, Friendships that last: Peer lifespan and its role in p2p protocols, in Proceedings of International Workshop on Web Content Caching and Distribution, [15] S. S. Lam and H. Liu, Failure recovery for structured p2p networks: protocol design and performance evaluation, in ACM SIGMETRICS Performance Evaluation Review, [16] E. W. Zegura, K. L. Calvert, and S. Bhattacharjee., How to model an internetwork, in Proceedings of INFOCOM, [17] Z. Liu, R. Yuan, Z. Li, H. Li, and C. Chen, Survive under high churn in structured p2p systems: evaluation and strategy, in Proceedings of ICCS 2006, [18] S. Saroiu, P. K. Gummadi, and S. D. Gribble, Measuring and analyzing the characteristics of napseter and gnutella hosts, Multimedia Systems, vol. 9, no. 1, pp , [19] M. Ripeanu and I. Foster, Peer-to-peer architecture case study: Gnutella network, in Proceedings of IEEE P2P 2001, [20] E. Sit and R. Morris, Security considerations for peer-to-peer distributed hash tables, in Proceedings of IPTPS 02, [21] N. J. Harvey, J. Dunagan, M. B. Jones, S. Saroiu, M. Theimer, and A. Wolman., Skipnet: A scalable overlay network with practical locality properties, in Proceedings of USITS 03, 2003.