IN structured P2P overlay networks, each node and file key

Transcription

1 242 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY 2010 Elasti Routing Table with Provable Performane for Congestion Control in DHT Networks Haiying Shen, Member, IEEE, and Cheng-Zhong Xu, Senior Member, IEEE Abstrat Consistent hashing-based DHT networks have an inherent load balaning problem. The problem beomes more severe in heterogeneous networks with nonuniform and time-varying popular files. Existing DHT load balaning algorithms are mainly foused on the issues aused by node heterogeneity. To deal with skewed lookups, this paper presents an elasti routing table (ERT) mehanism for query load balaning, based on the observation that high-degree nodes tend to reeive more traffi load. The mehanism allows eah node to have a routing table of variable size orresponding to node apaities. The indegree and outdegree of the routing table an also be adjusted dynamially in response to the hange of file popularity and network hurn. Theoretial analysis proves that the routing table degree is bounded. The ERT mehanism failitates loality-aware randomized query forwarding to further improve lookup effiieny. By relating query forwarding to a supermarket ustomer servie model, we prove that a two-way randomized query forwarding poliy should lead to an exponential improvement in query proessing time over random walking. Simulation results demonstrate the effetiveness of the ERT mehanism and its related query forwarding poliy for ongestion and query load balaning. In omparison with existing virtual-server -based load balaning algorithms and other routing table ontrol approahes, the ERT-based ongestion ontrol protool yields signifiant improvement in query lookup effiieny. Index Terms Distributed hash table, peer-to-peer, load balaning, ongestion ontrol. Ç 1 INTRODUCTION IN strutured P2P overlay networks, eah node and file key is assigned a unique ID, based on a onsistent hashing funtion. The file keys are mapped on to nodes aording to their IDs and a distributed hash table (DHT) definition. The DHT maintains topologial relationships between the nodes and supports a routing protool to loate a node responsible for a required key. Beause of their lookup effiieny, robustness, salability, and deterministi data loation, DHT networks have reeived muh attention in reent years. Representatives of the systems inlude CAN [26], Chord [33], Tapestry [34], Pastry [27], and Cyloid [32]. DHT networks have an inherent load balaning problem. It is beause onsistent hashing produes a bound of O(log n) imbalane degree of keys between the network nodes. The problem beomes even more severe as the nodal heterogeneity inreases. What is more, files stored in the system often have different popularities and the aess patterns to the same file may vary with time. It is a hallenge to design a DHT protool with the apability of ongestion ontrol. The primary objetive of ongestion ontrol is to avoid bottlenek in any node (i.e., the query load exeeds its apaity). Bottlenek may our with query overflow in. H. Shen is with the Department of Eletrial and Computer Engineering, Clemson University, 313-B Riggs Hall, Clemson, SC shenh@lemson.edu.. C.-Z. Xu is with the Department of Eletrial and Computer Engineering, Wayne State University, 5050 Anthony Wayne Drive, Detroit, MI zxu@ee.eng.wayne.edu. Manusript reeived 3 May 2008; revised 25 Jan. 2009; aepted 9 Mar. 2009; published online 17 Mar Reommended for aeptane by K. Hwang. For information on obtaining reprints of this artile, please send to: tpds@omputer.org, and referene IEEECS Log Number TPDS Digital Objet Identifier no /TPDS whih too many queries reeived by the node at a time, or with data overflow in whih a too high volume of data needed to be downloaded and forwarded by the node. DHT routing algorithms may lead to the onvergene of query load targeted for an objet on a small number of nodes around the destination, leading to bottleneks. In addition, although files are often transmitted via a diret onnetion between soure and destination, data forwarding through intermediary nodes in the query routing path is often used for the provisioning of anonymity of file sharing, as in Freenet [10], Mantis [5], Mutis [1], and Hordes [16]. Reent studies of P2P file sharing systems [13], [28] demonstrate that node apaity and node query pattern are heavily skewed in the systems. Nodes are easily beome bottleneks in suh an environment. In the past, many load balaning strategies have been proposed to deal with the network heterogeneity; see [33], [12], [4] for examples. It is known that a long key ID spae interval has a high probability of being ontated than a short interval. Most of the existing approahes share a ommon idea of virtual server, in whih a physial node simulates a number of virtual overlay servers so that eah node is assigned ID spae interval of a different length aording to its apaity. The simpliity of the approahes omes at a high ost to maintain the relationship between a node s responsible interval and its apaity. Moreover, beause the approahes are based merely on key ID assignment, they do not provide any ontrol over ongestions aused by the fators of nonuniform and time-varying file popularity. There are other approahes, based on item-movement, whih take into aount the effet of file popularity on query load [3], [31], [30]. In these approahes, heavily loaded nodes probe light ones and reassign exess load between the peers by hanging the IDs of related files or nodes. Albeit /10/$26.00 ß 2010 IEEE Published by the IEEE Computer Soiety

2 SHEN AND XU: ELASTIC ROUTING TABLE WITH PROVABLE PERFORMANCE FOR CONGESTION CONTROL IN DHT NETWORKS 243 flexible, the load reassignment proess inurs high overhead for hanging IDs, espeially in networks under hurn. Notie that the existing load balaning approahes assume that eah node (or virtual node) has the same and onstant DHT degree. That is, eah node maintains the same number of neighboring relationships, irrespetive of its apaity. The priniple of power-law networks tells that the higher degree nodes tend to experiene more query loads [2]. In light of this, in this paper, we present an elasti routing table (ERT) mehanism to ope with node heterogeneity, skewed queries, and hurn in DHT networks. Unlike urrent strutured P2P routing tables with a fixed number of outlinks, eah ERT has a different number of inlinks/outlinks, and the indegree/outdegree of eah node an be adjusted dynamially aording to its experiened traffi so as to diret query flow to light nodes. Reently, Castro et al. [7] exploited heterogeneity in ongestion ontrol by stati mapping between node indegree and apaity and biasing high-apaity nodes for overlay neighbors. Stati mapping annot deal with nonuniform and time-varying file popularity and hurn. Simple apaity bias may also make high-apaity nodes beome bottleneks. The ERT-based ongestion ontrol protool goes beyond the onstrution of apaity-aware DHTs. It deals with ongestion due to time-varying file popularity by adjusting the indegrees and outdegrees of the routing tables and apaity-aware query forwarding. We summarize the ontributions of this paper as follows:. An initial indegree assignment for the onstrution of apaity-aware DHTs. The indegrees are provably bounded.. A poliy for periodi indegree adaptation to deal with the nonuniform and time-varying file popularity. It is proved that the indegree bounds remain bounded.. A topology-aware randomized query forwarding poliy on the elasti DHTs. It is proved that the ERTenabled query forwarding leads to an exponential improvement in query proessing time over random walking.. Comprehensive simulations demonstrate the superiority of the ERT protool, in omparison with the virtual-server -based load balaning poliy and other routing table ontrol approahes. The rest of this paper is strutured as follows: Setion 2 presents a onise review of representative ongestion ontrol approahes for unstrutured and strutured P2P systems. Setion 3 shows the ERT-based protool, fousing on initial indegree assignment and periodi adjustment. Setion 4.1 gives the details of topology-aware randomized query forwarding poliy. Setion 5 shows the performane of the protool with omparison of a variety of metris. Setion 6 onludes this paper with remarks on possible future work. 2 RELATED WORK There have been many load balaning algorithms to deal with node heterogeneity and network hurn [33], [12], [4]. Virtual server [33] is a popular approah, in whih eah real node runs Oðlog nþ virtual servers and the keys are mapped onto virtual servers so that eah real node is responsible for the key ID spae of different length proportional to its apaity. It is simple in onept, but the virtual server abstration inurs large maintenane overhead and ompromises lookup effiieny. Godfrey and Stoia [12] addressed the problem by arranging a real server for virtual ID spae of onseutive virtual IDs. When a node selets its virtual servers, it first piks a random starting point and then selets one random ID within eah of ðlog nþ onseutive intervals of size ð1=nþ. In [4], Bienkowski et al. proposed a distributed randomized sheme to let a linear number of nodes with short ID spae interval to divide the existing long ID spae interval, resulting in an optimal balane with high probability. To ahieve load balane, Byers et al. [6] suggested the diret appliation of the power of two hoies paradigm, whereby an item is stored at the less loaded of two (or more) random alternatives. Sine a node with a longer interval has a higher probability of being ontated, the load balaning algorithms based on ID spae interval assignment ontrol traffi ongestion due to the node heterogeneity in apaity. Initial key ID spae partitioning is insuffiient to guarantee load balane, espeially in hurn. It is often omplemented by dynami load reassignment. Godfrey and Rao et al. proposed shemes to rearrange load between heavy nodes and light ones aording to their apaities so as to avoid bottlenek [11], [25]. They assumed that query load is uniformly distributed in the ID spae. Bharambe et al. [3] proposed a load balaning algorithm to deal with the ongestion aused by biased lookups. They defined a node s load as the number of messages routed or mathed per unit time. The algorithm proeeds in a way that heavily loaded nodes probe a number of sample nodes and requests lightly loaded nodes to leave from their urrent loations and rejoin at the loation of the heavily loaded nodes. To get the load distribution information, eah node periodially samples nodes within a ertain distane and maintains approximate histograms. This requires muh ommuniation and maintenane ost, espeially in hurn. In addition, node ID hanges due to the node leave/rejoin inurs high overhead. The idea of using irregular routing tables with respet to the node apaity has been reently pursued by Hu et al. [14] and Li et al. [17]. Their foi were on a trade-off between maintenane overhead and lookup effiieny. Hu et al. proposed to deploy large routing tables in high-apaity nodes to exploit node heterogeneity and improve lookup effiieny. Li et al. designed an Aordian mehanism on Chord to vary the table size in different network sales and hurn rates without ompromising lookup effiieny. Chun et al. [9] evaluated the impat of neighbor seletion on performane and resiliene of strutured P2Ps. The neighbor seletion takes into onsideration different types of heterogeneity, suh as node apaity and network proximity. With node apaity onsideration, a node hooses its neighbors whih have the smallest proessing delay. However, this method likely makes high-apaity neighbors beome bottleneks. In fat, always biasing the query routing toward high-apaity nodes exaerbate the problem if these nodes do not have enough apaity to handle a sudden query flow aused by hurn. The query load should

3 244 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY 2010 be assigned to high-apaity nodes as well as low-apaity nodes proportional to their apaities. Castro et al. [7] proposed a neighbor seletion algorithm to onstrut routing tables based on the node s different apaities. Its basi idea of using node indegrees to exploit node heterogeneity is similar to our initial indegree assignment algorithm. Their algorithm direts most traffi to highapaity nodes beause it does not hoose low-apaity nodes as neighbors unless the indegree bounds of highapaity nodes are reahed. In ontrast, ERT mehanism would distribute the traffi between the neighbors proportional to their apaities so as to make full utilization of both high and low-apaity nodes. Moreover, ERT mehanism handles more than node heterogeneity. It handles biased lookups and hurn by adjusting table indegree and outdegree dynamially and query forwarding. Finally, we note that the problem of ongestion ontrol is not unique in strutured P2P networks. It has been a ruial performane issue in unstrutured P2P networks, as well. Many studies have been devoted to flow ontrol in unstrutured networks; see [24], [18], [2], [19], [8] for reent examples. Like ongestion ontrol in DHT networks, their solutions are based on the priniple of power-law networks that highdegree nodes play an important role in ommuniation. Osokine [24] proposed a reative flow ontrol mehanism, in whih reeivers drop pakets when beoming overloaded and senders infer the likelihood that a neighbor will drop pakets based on the responses that they reeive from the neighbor. This mehanism is aeptable when queries are flooded aross the network, beause even if a node drops a query, other opies of the query will propagate through the network. However, it is not suitable for random walks searh [18]. Lv et al. [18] revealed that in power-law networks, suh as Gnutella, the high-degree nodes often experiene high query load. They suggested that P2P systems should adopt graph building algorithms that redue the likelihood of veryhigh-degree nodes. On the ontrary, Adami et al. [2] took advantage of the feature of power-law networks that highdegree nodes play an important role in ommuniation. They proposed a message-passing algorithm, whih forwards queries to high-degree nodes so as to speed up the proess of finding targeted files. It is notied that high-degree nodes are not neessarily high-apaity nodes and that they are prone to bottlenek if they arry an extremely large share of query traffi. To overome this disadvantage and to onsider node heterogeneity, Lv et al. [19] proposed query flow ontrol and topology adaptation algorithms to let higher apaity nodes have a higher degree, and forward queries to these nodes. The algorithms gradually hange the overlay topology, based on the status of eah neighbor link, so that queries flow toward the nodes that have suffiient apaity to handle them. However, the bias to high-apaity nodes may also make them to beome hot spots. Chawathe et al. [8] proposed an ative flow ontrol sheme, whih aknowledges the existene of heterogeneity and adapts to it by assigning flowontrol tokens to nodes base on available apaity. In the sheme, a node distributes k flow-ontrol tokens means that the node is willing to aept k queries, and a sender is allowed to diret a query to a neighbor only if it has a flow-ontrol token of the neighbor. Beause of DHTs stritly ontrolled topology and preisely defined routing algorithm, their routing table neighbors are restrited and query routing path is fixed. Sine, the TABLE 1 Notations flow ontrol algorithms were designed for flooding or random walk query routing networks, the flow ontrol algorithms on unstrutured P2P networks annot be applied for load balaning and ongestion ontrol in DHT networks. 3 ELASTIC DHT ERT is designed based on the power-law priniple of networks that a higher degree node tends to reeive more query load. The ERT-based protool onstruts routing tables with different number of outlinks for different nodes so as to distribute query load among the nodes in proportion to their apaities. To deal with skewed queries, eah node dynamially adjusts its indegree aording to its atual query load experiened. 3.1 Definitions We assume an DHT network with n physial nodes, labeled as an integer from 1 to n. Node i; 1 i n, has a apaity that the node is willing to devote or able to proess queries. We assume that node i s apaity i is a quantity that represents the number of queries that node i an handle in a given time interval T. In pratie, the apaity should be determined as a funtion of a node s aess bandwidth, proessing power, disk speed, et. We define the load of node i; l i, as the number of queries it reeives and transmits to its neighbors over time T. We refer to node with traffi load l i i as a light node, otherwise a heavy or overloaded node. In order to failitate the desription, we define notions used in this paper in Table 1. The purpose of a ongestion ontrol protool is to avoid heavy nodes in query routings and distribute query load among nodes orresponding to their apaities. From the view point of an entire system, in fair load distribution, eah node s load share is proportional to its fair load share, as defined by s i ¼ l i= P i l i i = P i : i Ideally, the fair share s i should be kept lose to 1. In this ase, a node s apaity is fully utilized and it is also not overloaded. One way to ahieve this is to measure the traffi load l i of every node periodially and forward queries aording to olleted global traffi load information.

4 SHEN AND XU: ELASTIC ROUTING TABLE WITH PROVABLE PERFORMANCE FOR CONGESTION CONTROL IN DHT NETWORKS 245 Obviously, this method is too ostly to be used in any salable overlay network. In [19], Lv et al. showed that a high-degree node in Gnutella network would most likely experiene high query load. We apply the priniple to the design of ongestion ontrol in DHT networks. We define indegree, denoted by d i ; 1 i n, as the number of inlinks of node i. Under the assumption that nodes and file queries are uniformly distributed in an DHT network without hurn, l i is diretly related to d i. New poliies will be proposed to deal with the nonuniformly distributed nodes, file queries, and hurn in Setions 3.3 and 4.1. It is known that the indegree of a node is determined by the number of outlinks of other nodes. In order to have a node s indegree to be proportional to its apaity, we reverse the relationship to determine node s outdegree by setting an appropriate value of indegree. To the end, we need to address two tehnial questions: 1. How to determine a node s indegree in order to make full use of its apaity and keep it light loaded at the same time? 2. How to onstrut ERTs with different indegrees of nodes, and meanwhile retaining the original DHT routing table funtion for lookup routing? We normalize node apaity so that the average of is 1; that is, P i i ¼ n. Reall that in a uniform system without hurn, l i is related to d i diretly. It follows that s i d i= P i d P i i = P i i and d i d i i i n P i when s i ¼ 1 ideally. Taking d i n as a onstant, we define as indegree per unit apaity. It is a system parameter and is determined as a funtion of different metris in system experiene suh as inlink query forwarding rate and query initiation rate in a nonuniform system with hurn. High-load system should have small alpha while low-load system ould have large alpha. Considering that the maximum number of queries a node an proess at a time depends on its apaity, we define node i s maximum indegree d 1 i as b0:5 þ i sine is indegree per unit apaity. In order to keep the deimal fration, 0.5 is used. The initial indegree of node i is d 1 i, where is a predefined perentage for reservation purpose. It is determined aording to system atual query load. If nodes are easily overloaded due to the query load, should be a small value. Otherwise, it ould be set to a large value. There is a trade-off in determination: if is too small, high-apaity nodes annot be fully utilized beause of low indegree, while a large makes it very possible that lowapaity nodes, even high-apaity nodes, beome heavy nodes. Moreover, large leads to extra maintenane ost for overlay onnetions. Therefore, it is important to determine a suitable. In the following, we present an initial indegree assignment algorithm for the onstrution of ERT. 3.2 Initial Indegree Assignment Like bidiretional links in Gnutella, we assume that eah DHT node i maintains a bakward outlink (bakward finger) for eah of its inlink, in order to know the nodes whih forward queries to it. Consequently, a double link is maintained for eah routing table neighbor. One node i joins the system, it needs to build its routing table based on DHT protools. In Fig. 1. Examples of Chord routing table without and with loose restrition. order to ontrol eah d i below d 1 i, we set a restrition that only nodes with available apaity d 1 i d i 1 an be the joining node s neighbors. Eah neighbor in node i s routing table reates a bakward finger to node i. After building a basi routing table, node i then probes other nodes whih an take it as their neighbor to ahieve its initial indegree d 1 i.asa result, high-apaity nodes produe high indegrees while low-apaity nodes lead to low indegrees. To probe nodes for indegree expansion, the IDs of those nodes should be first deided. The ID set an be determined in the opposite way of the original DHT neighbor seletion algorithm. We will explain it later in DHT examples. After the determination of the ID set, node i sends requests targeting to some of the nodes in the set that are not in the list of its bakward fingers. On reeiving suh a request, the node whih is responsible for the ID heks if it an take node i as its routing table neighbor. If an, it adds node i into its orresponding entry in its routing table and sends bak a positive reply. One node i reeives positive reply from a node, say node j, it builds a bakward finger to node j. In the following, we take Chord, Cyloid, Pastry, and Tapestry as DHT examples to explain indegree expansion algorithm. The algorithm an be applied to other strutured overlays that have little flexibility in the seletion of neighbors by relaxing their routing table neighbor onstraints. In Chord, a node has Oðlog nþ fingers in its routing table. We represent ID of node i as i. Then, the ðm þ 1Þth finger of node i is the suessor of ði þ 2 m Þð0 m d 1Þ. Fig. 1a shows the routing table of the node with ID ( ). Atually, Chord s routing table neighbor onstraint an be loosed. That is, the ðm þ 1Þth finger is a set of suessors sueeding the suessor of ði þ 2 m Þð0 m d 1Þ. Fig. 1b illustrates the resultant routing table of node ( ). Suh that node i an ask nodes with a set of predeessor IDs of ði 2 m Þð0 m d 1Þ to point to it. For example, assume node i s ID is ( ) and it an send requests targeting to ID 2½ ; Š to take it as their 4th routing table finger. If the ID of node j whih reeives the request is 2½ ; Š, node j takes i as its 4th routing table finger. Otherwise, node j annot take i as its 4th routing table finger. Cyloid is a onstant-degree DHT where eah node has seven outdegree. We keep the Cyloid topology but remove the restrition of the onstant degree. That is, a node s outdegree an be any value. In Cyloid with dimension d, a node ðk; a d 1 a d 2...a k...a 0 Þðk 6¼ 0Þ has one ubial neighbor ðk 1;a d 1 a d 2...a k xx...xþ and two yli neighbors ðk 1;b d 1 b d 2...b 0 Þ and ðk 1; d 1 d Þ in its routing table. Fig. 2 shows an example of routing table of Cyloid node with

5 246 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY 2010 Fig. 2. An example of Cyloid routing table. ID (4, ). By the opposite way of neighbor seletion, a node ðk 1;a d 1 a d 2...a k...a 0 Þ an send requests targeting to ðk þ 1;a d 1 a d 2...a k xx...xþ to ask nodes to take it as their ubial neighbors, and also it an send requests targeting ðk þ 1;a d 1 a d 2...a k xx:::xþ to ask nodes to take it as their yli neighbors. For instane, node i (3, ) an probe ð4; xxxxþ to inrease its indegree. Let s say, node i first sends a request targeting (4, ). Assuming node j reeives the request, if j 2ð4; xxxxþ;jadds i as its ubial neighbor and node i builds a bakward finger to j.if node i needs to inrease its indegree to 10, but it is only 6 after ubial bakward finger probing, node i probes yli bakward finger for the rest 4 indegree. Algorithm 1 shows the pseudoode of indegree expansion algorithm in Cyloid. In the pseudoode, we represent the ubial ID a d 1 a d 2...a k...a 0 in node ID ðk; a d 1 a d 2...a k...a 0 Þ as a id. Algorithm 1. Pseudo-ode for indegree expansion algorithm of Cyloid node i ðk; a d 1 a d 2...a k...a 0 Þ 1: //probe bakward fingers of ubial neighbor 2: figure out a set of ubial neighbor inlinks ID ¼ðkþ1;a d 1 a d 2...a k xx...xþ 3: id ¼ðkþ1;a d 1 a d 2...a k Þ 4: while not finish probing all IDs in ID ^ððd 1 i d i Þd 1 i Þ do 5: while id is in bakward fingers do 6: id ¼ðkþ1;a id þþþ2id 7: end while 8: probe ID for ubial neighbor inlink 9: id ¼ðkþ1;a id þþþ2id 10: end while 11: //probe bakward fingers of yli neighbor 12: figure out ID of yli neighbor inlinks ID ¼ðkþ1;a d 1 a d 2...a k xx...xþ 13: id ¼ðkþ1;a d 1 a d 2...a k Þ 14: while not finish probing all IDs in ID ^ððd 1 i d i Þd 1 i Þ do 15: while id is in bakward fingers do 16: id ¼ðkþ1;a id þþþ2id 17: end while 18: probe ID for yli neighbor inlink 19: id ¼ðkþ1;a id þþþ2id 20: end while Pastry s routing table is organized into dlog b ne (b is a onfiguration parameter) rows with 2 b 1 entries eah. An entry at row m of node i s routing table refers to a node whose ID shares the node i s ID in the first m digits, but whose ðm þ 1Þth digit is not the ðm þ 1Þth digit in node i s Fig. 3. An example of Pastry routing table. ID. For example, node ð þ an have nodes with ID ð10xxxxxxþ at its row 2, as shown in Fig. 3. Tapestry s routing table neighbor seletion algorithm is similar to Pastry s. Sine eah entry has multiple hoies, node iða d 1 a d 2...a k 1 a k...a 0 Þ an send request targeting to ða d 1 a d 2...a k 1 a k x...xþ to ask nodes to take it as their kth row entry. Algorithm 2 shows the pseudoode of indegree assignment algorithm. Algorithm 2. Pseudo-ode for indegree assignment algorithm of node i join 1: while (node i s routing table has not reated) do 2: probe another node for routing table neighbors 3: if (reeive j s posirpy) then 4: set j as routing table neighbor, i.e., reate a outlink to j 5: end if 6: end while 7: while ðd 1 i d i d 1 i Þ do 8: exeute indegree expansion algorithm 9: if reeive k s posirpy then 10: reates bakward finger to k 11: d i þþ 12: end if 13: end while fnode j exeution:g 14: if (reeive node i s neighbor probing) then 15: if ðd 1 j d j 1Þ then 16: send posirpy to i 17: build a bakward finger to i 18: d j þþ 19: else 20: send negrpy to i 21: end if 22: end if fnode k exeution:g 23: if (reeives node i s bakward finger probing) then 24: if (i an be k s routing table neighbor) then 25: reate outlink to i 26: send posirpy to i 27: end if 28: end if The initial indegree assignment algorithm proeeds repeatedly. We prove that the ERT indegree resulted from the algorithm is bounded. We assume an DHT that manages

6 SHEN AND XU: ELASTIC ROUTING TABLE WITH PROVABLE PERFORMANCE FOR CONGESTION CONTROL IN DHT NETWORKS 247 a unit-size ID spae, i.e., ½0; 1Þ IR employing arithmeti modulo 1, and the DHT uses onsisting hash [15] to partition the ID spae among the nodes. Thus, the responsible ID spae imbalane is log n. We assume that eah node i an estimate its apaity i and the network sale n within a fator of and n, respetively, of the true values, with high probability 1 ; readers are referred to [20], [23] for details of suh an estimation proess. We denote ~n as estimated n and ~ as estimated. Theorem 3.1. The initial indegree assigned to a node i is between i = Oð1Þ and i þ Oð1Þ w.h.p. Proof. With and n as the maximum error fator of a node s estimated apaity and n; ~ i is within the fator of i and ~n is within n of n w.h.p. Thus, the indegree first assigned to node i is at most b0:5 þ ~ i ð~nþ ~ i ð~nþþoð1þ i ð n nþþoð1þ i ðnþþoð1þ. The indegree first assigned to node i is at least b~ i ð~nþ 0:5 ~ i ð~nþ Oð1Þ i = ðn= n Þ Oð1Þ i = ðnþ Oð1Þ. tu Algorithm 3. Pseudo-ode for periodi indegree adaptation algorithm of node i 1: if ðg i > l Þ //node i overloaded? then 2: ask 1 2 ðl i i Þ nodes pointed by bakward fingers to delete i form their routing tables 3: if (reeive k s posirpy) then 4: delete bakward finger to k 5: d i 6: d 1 i þþ 7: end if 8: else 9: if ðg i < 1= l Þ then 10: probe 1 2 ð i l i Þ nodes for bakward finger 11: if (reeive k s posirpy) then 12: reate bakward finger to k 13: d i þþ 14: d 1 i 15: end if 16: end if 17: end if 3.3 Periodi Indegree Adaptation In pratie, nodes join and leave DHT overlays ontinuously and the files in the system may have nonuniform and time-varying popularity. Considering the fat that query load often varies with time, the initial indegree assignment is not robust enough to limit a node s query load under its apaity. To ensure that queries flow toward nodes with suffiient apaity, the ongestion ontrol protool should adapt to the hange of query rate and lookup skewness aused by nonuniform and time-varying file popularity, as well as network hurn. We design a periodi indegree adaptation algorithm to help eah node adjust its indegree periodially aording to 1. An event happens with high probability (w.h.p.) when it ours with probability 1 Oðn 1 Þ. the maximum load it experiened. Speifially, every node i reords its query load l i over T periodially and heks whether it is overloaded or lightly loaded by a fator of l ; i.e., whether g i ¼ l i = i > l or < 1= l. In the former ase, it dereases ðl i i Þ indegree by asking some of its bakward fingers to delete it from their routing tables, then deletes orresponding bakward fingers, and dereases its maximum indegree d 1 i orrespondingly. is a predefined perentage. To hoose a bakward finger to remove, it hooses the one with the longest logial distane. In the ase with the same logial distanes, it hooses the one with the longest physial distane. In the latter ase, it inreases ð i l i Þ indegree by probing other nodes to take it as their neighbors using the inlink expansion algorithm disussed in Setion 3.2 and inreases its d 1 i orrespondingly. Algorithm 3 shows the pseudoode of periodi indegree adaptation algorithm. The following theorem shows that the ERT indegree in the proess of adaptation remains bounded. Theorem 3.2. With indegree assignment and periodi adaptation algorithm, a node i has an indegree between i l max and il min, where max and min represent the maximum and minimum inoming query rate per inlink in the system, respetively. And its indegree hange is bounded in eah adaptation. Proof. Node i does not need to update its indegree when ~ i l l i ~ i = l, where i ~ i i. Assume that during a ertain time period T, the average inoming query rate per inlink of node i is i ; that is, on average, there are i queries oming from eah inlink during time T. Assume that node i has degree d i at a ertain time point during T suh that l i ¼ i d i. When l i > ~ i l, node i updates its indegree to d i ðl i ~ i Þ. Sine l i > ~ i l ; i d i > ~ i l, the indegree is at most d i ~ i ð l 1Þ;d i i ð l 1Þ. Itisd i ð i d i ~ i Þð1 Þ i d i þ i. Consequently, the indegree is dereased to between ð1 Þ i d i þ i and d i i ð l 1Þ. On the other hand, when l i < i l, node iupdates its indegree to d i þ ð~ i l i Þ. Sine l i < ~i l ; i d i < ~i l, the indegree is at least d i þ ~ i ð1 1 l Þ;d i þ i ð1 1 l Þ. It is d i þ ð~ i i d i Þð1 i Þd i þ i. Therefore, the indegree is inreased to between d i þ i ð1 1 l Þ and ð1 i Þd i þ i. The indegree is hanged until it reahes a status that ~ i l i d i ~ i = l ; i l min d i i l max. tu For example, in a network of size 2,048, if a node s apaity is 50 and its average inoming query rate is 0.5, its indegree is bounded by 100 in the ase l ¼ 1. The following theorem shows that the outdegree of an ERT is bounded as well. We leave its proof to the Appendix. Theorem 3.3. A Cyloid node has an outdegree of at most Oð 2d d ÞþOð1Þ w.h.p., where d is the DHT dimension. 2 l max min 4 DYNAMIC RANDOMIZED QUERY FORWARDING Periodi indegree adaptation may not be suffiient to deal with query load imbalane. In this setion, we present a omplementary randomized query forwarding algorithms to help forward queries toward light nodes so as to further redue lookup lateny.

7 248 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY Query Forwarding Poliies With the initial indegree assignment and periodi adaptation algorithms, eah node s routing table has a variable size. With a high probability, eah ERT has a set of outlinks in eah of its routing table entries. For example, a Cyloid node i ¼ð4; Þ has ubial outlinks pointing to nodes (3, ), (3, ), and (3, ). For an inoming query destined for its ubial neighbors based on the original routing algorithm, there would be three andidates to take the query. A simple forwarding poliy is random walk, in whih one of the outlinks is seleted randomly. Another one is gradientbased walk that forwards a query to the best andidate in terms of their workload. Instead of probing all of the neighbors to find out the best andidate, we restrit the searh spae to a small set of size b. That is, one reeiving a query, node i first randomly selets b neighbors (outlinks) and then probes the nodes in the set sequentially, until a light node is found. In the ase that all andidates are overloaded, the query is forwarded to the least heavily loaded one. For example, node i reeives a query with key (2, ) and the query should be forwarded to a ubial neighbor aording to Cyloid routing algorithm. It first randomly hooses two options (3, ) and (3, ) among the three ubial neighbors if b ¼ 2. If most of a node s neighbors are heavily loaded, the forwarding poliy annot improve the performane too muh. Although suh situation may not our frequently, in this ase, the node will generate more neighbors to handle this problem based on the periodi indegree adaption algorithm. The b-way randomized query forwarding is further enhaned by taking into aount the underlying topology information in the andidate seletion. In the topologyaware forwarding poliy, a node selets the best andidate among b neighbors by two extra riteria: lose to the target ID by the logial distane (hops) in the DHT network and lose to the node by the physial distane on the Internet; readers are referred to [31], [30] for a landmarking method to measuring physial distane between two nodes on DHT networks. In the ase that the two andidates are both lightly loaded, the loser node in logial distane is seleted. Their physial distane is used to break the tie of logial distane. Probing b neighbors is a ostly proess. How to find a good andidate from b neighbors at a relatively low ost? Query forwarding in this ontext an be regarded as a supermarket ustomer servie model (see Setion 4.2 for justifiation). The supermarket model is to alloate eah inoming task (a ustomer) to a lightly loaded server with the objetive of minimizing the time eah ustomer spends in the system. Mitzenmaher [21] proved that granting a task with two server hoies and dispathing it to one of the servers with less workload leads to an exponential improvement over the single hoie in the expeted exeution time of eah task. But a poll size larger than two gains muh less substantial extra improvement. Furthermore, Mitzenmaher et al. [22] improved the performane of two-hoie method by the use of memory. In this method, eah time a task is alloated, the least loaded of that task s hoies after alloation is remembered and used as one of the possible hoies for the next task. We adapt this memory-based randomized task dispathing method with modifiations to topology-aware randomized query forwarding. We set b ¼ 2. A node first randomly selets two options, say nodes i and j. It then selets the better one, say node i, and remember the least loaded node between i and j after node i inreases by one load unit. We assume that the node is still i, whih is used for the next query forwarding. Later, when the node needs to forward a query to the same routing table entry, it only needs to randomly hoose one neighbor, instead of two. With the remembered node i, it repeats the proess again. To further redue the heavy nodes in query routings, a query flows by the use of the information of overloaded nodes enountered before, to avoid overloaded node in the sueeding routings. For example, a lookup node has three options for forwarding a query: i 1 ;i 2, and i 3. Using the above method, it forwards the query to i 2 with information of overloaded node i 1. In the seond step, node i 2 has three options: i 1 ;j 1, and j 2. Beause it knows that i 1 is overloaded, it will not selet i 1 as a option for query forwarding. Algorithm 4 shows the pseudoode of the topology-aware randomized query forwarding algorithm. Algorithm 4. Pseudo-ode for topology-aware randomized query forwarding algorithm exeuted by node i 1: reeive query Q with overloaded node information A 2: determine the set of outlinks for the query forwarding based on DHT routing algorithm. 3: hoose options J ¼fj 1 ;j 2...g from the outlink set exluding overloaded node in A 4: if memory has a node j a then 5: randomly hoose a node j b from J 6: else 7: randomly hoose two nodes j a and j b from J 8: end if 9: //hoose the better node from j a and j b 10: probe node j a and j b for load status 11: if j a and j b are heavy then 12: add j a and j b to A 13: forward Q and A to the least heavily loaded node 14: else 15: if one node is light and one node is heavy in j a and j b then 16: add the heavy node to A 17: forward Q and A to the light node 18: end if 19: else 20: hoose nodes J log logially nearest to target ID from j a and j b 21: hoose nodes J phy physially nearest to node i from J log 22: forward Q and A to a node in J phy 23: end if 4.2 Analysis of Query Forwarding Poliies The simple query forwarding model (QFM) an be rephrased as the following: After a node reeives a query,

8 SHEN AND XU: ELASTIC ROUTING TABLE WITH PROVABLE PERFORMANCE FOR CONGESTION CONTROL IN DHT NETWORKS 249 it forwards the query to one of its neighbors. If the hosen neighbor is heavily loaded by a fator l, another speifi neighbor is turned to. This proess is repeated until the node finds a light neighbor. In the ase that all neighbor options are heavy, the query is forwarded to the least heavily loaded option. We assume that the query forwarding time for a query is onstant and inoming query is Poisson distributed [33]. The forwarding model an be regarded as a variation of strong threshold supermarket model (STSM) proposed in [21] if we take l as the threshold in the latter. In the STSM, ustomers arrive at a Poisson stream of rate n ( <1) at n FIFO servers. Eah ustomer hooses a server independently and uniformly at random and only makes additional hoies if the previous hoie is beyond a predetermined threshold. If both hoies are over the threshold, the ustomer queues at the shorter of its two hoies. The servie time for a ustomer is exponentially distributed with mean 1. A key differene between the QFM and the STSM is that the servers are homogeneous in the STSM but heterogeneous in the QFM. Along the line of analytial approah in [21], we analyze the performane of the randomized query forwarding algorithms. The following theorem shows that the twoway randomized query forwarding poliies improve lookup effiieny exponentially over random walking. Readers are referred to the Appendix for the proof. Theorem 4.1. For any fixed time spot T, the time a query waits before being forwarded during the time interval ½0;TŠ is bounded and b-way ðb 2Þ forwarding yields an exponential improvement in the expeted time for a query queuing in a server. 5 PERFORMANCE EVALUATION This setion demonstrates the distinguished properties of the ERT-based ongestion ontrol protool through simulation built on an Oð1Þ-degree Cyloid network. ERT an also be applied to other DHT networks. Simulations on other Oðlog nþ-degree networks are expeted to produe better results. We assumed a bounded Pareto distribution for the apaity of nodes [11]. This distribution reflets real-world situations where mahines apaities vary by different orders of magnitude. Reall that the maximum indegree of a node, say node i, is defined in Setion 3.2 as b0:5 þ i, where i ¼ n i = P i i is the normalized apaity of the node. That is, the node an handle this amount of queries at one time. We define node i s load as the number of queries in its query proessing queue. If node i has more than b0:5 þ i queries in its queue, it is overloaded. We further assumed that the queries be generated aording to a Poisson proess at a rate of one per seond [33], with a random soure node and a random target key, unless otherwise noted. Table 2 lists the parameters of the simulation and their default values. We evaluate the effetiveness of the ongestion ontrol protool in the following metris:. Congestion rate of a node i, as defined by g i ¼ l i = i. Ideally, the rate should be kept around 1, implying the node is neither overloaded nor under utilized, TABLE 2 Simulated Environment and Algorithm Parameters and its apaity is fully utilized. We use the metri of the 99th perentile maximum ongestion to measure the network ongestion, and use the metri of the 99th perentile ongestion of minimum apaity node to reflet the node utilization.. Query distribution share s i. Reall that share s i ¼ l i= P l i i = P : i It represents the performane of fair load distribution, i.e., the total system load is distributed among nodes based on their apaity. The objetive of fair sharing is hard to ahieve in DHT networks beause of a number of reasons. First, it is hard to ollet the load and apaity of other nodes. Seond, DHT is a dynami system with ontinuous node joins and departures, as well as ontinuous query initialization. It is hard to ontrol instant share of eah node in suh a dynami situation. Third, sine query load is not uniformly distributed among the nodes, it hanges with file popularity and hurn. Although fair sharing is not the objetive of ongestion ontrol, we use the metri of the 99th perentile share to show how it an be approximated by the ontrol of indegrees.. Query proessing time. It is determined by two fators: lookup path length and the number of heavy nodes enountered in eah path. The metri of path length reflets the performane of the query forwarding algorithm, and the metri of number of heavy nodes shows how the ongestion ontrol protool avoids heavy nodes in diret traffi flow in order to redue lookup lateny. We onduted experiments on Cyloid networks without ongest ontrol (Base) and with ERT-based ongestion ontrol (ERT). For omparison, we also inlude the results due to a virtual server load balaning method [12] (VS) and a neighbor seletion algorithm for indegree ontrol (NS) [7]. The NS algorithm bears resemblane to the ERT initial indegree assignment as to selet neighbor based on node indegree bound. However, NS always selets highapaity nodes as neighbors. It may overompromise the needs of low-apaity nodes. ERT makes full use of node apaity by letting nodes reah their indegree bounds. Moreover, ERT allows dynami indegree adaptation (A) and failitates query forwarding (F) to deal with network

9 250 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY 2010 Fig. 4. Effetiveness of ongestion ontrols. (a) Maximum ongestion, (b) ongestion of minimum apaity node, and () share. hurn and skewed lookups. We represent the ongestion ontrol in different ombinations by ERT/A, ERT/F, and ERT/AF, respetively. We measured their performane as funtions of total lookup number and query proessing speed at eah node. We varied lookup number from 1,000 to 5,000, with 1,000 inrease in eah step. We also varied the proessing time of a query in a light node from 0.1 to 2.1 seond and five times of that in a heavy node. The total query load inreases in both ases and we observed similar results in simulation. 5.1 Congestion Control Effiieny We measured eah node s maximum ongestion during all test ases and alulated the 99th perentile maximum node ongestion. Fig. 4a shows the ongestion rate due to eah method inreases as more lookup queries arrive. The NS protool produes a higher 99th perentile maximum ongestion rate than Base. It implies that a heavy node in NS has muh more load orresponding to its apaity than a heavy node in Base. This is expeted beause NS strongly biases high-apaity nodes as routing table neighbors. The high-apaity nodes may turn out to be overloaded. In ontrast, VS and ERT/AF lead to muh lower ongestion rates. That is, they are more effetive in ontrolling the load of eah node based on its apaity. Also, ERT/AF outperforms VS, espeially when the system is highly loaded. The relative performane between NS, VS, and ERT/AF, as shown in Fig. 4a, an be verified by the 99th perentile ongestion rate of minimum apaity node in Fig. 4b. It is expeted to see that the low-apaity node beomes ongested as the query load inreases. Without ongestion ontrol (Base), the ongestion rate inreases sharply. The ongestion ontrol protools delay the ourrene of ongestion. In partiular, the NS protool overproteted low-apaity nodes due to its high-apaitybiased neighbor seletion poliy. In omparison, ERT/AF keeps low-apaity nodes fully utilized, without driving them into overloaded states. To further evaluate the impat of individual fators of adaptation and forwarding, we inlude the results due to ERT/A and ERT/F in Fig. 4a. The figure shows that indegree adaptation (ERT/A) redues the ongestion rate of Base signifiantly in various load onditions and performs onsistently better than VS. Forwarding (ERT/F) alone may not work as well as VS in reduing the ongestion rate. In ERT/F, the ongestion rate would grow rapidly as the lookup number inreases. It implies that forwarding is effetive in ontrolling node ongestion when query load is light, but beomes less effetive as the system load inreases. Fig. 4 shows the 99th perentile node share. We an see that NS generates a muh higher share rate, in omparison with the other protools for the same reason of the observations in Figs. 4a and 4b. That is, NS heavily relies on high-apaity nodes for query routing. Exluding lowapaity nodes in neighbor seletion may lead to a waste of system resoures beause their apaities an be used to ease the burden of high-apaity nodes in ertain situations. In ontrast, VS and ERT/AF do not have this preferene in neighbor seletion, and they ahieve good query load sharing between heterogeneous nodes. The small gain of VS is due to its fine grained ID spae partition between virtual servers. An ideal share in DHT is diffiult to ahieve beause of the DHT strit ontrolled topology, routing algorithm, nonuniform and variable file popularity, and hurn. VS approximates fair sharing by stati ID spae assignment. However, it is at the ost of more maintenane overhead and lookup ost. It annot handle skewed lookups either. From Fig. 4, we an also see that forwarding (ERT/F) alone leads to a lower share rate than ERT/AF. It is expeted beause query forwarding that forwards queries only to nodes with suffiient apaity ould reah a balaned load distribution in onept. In the next setion, we will see it is at the ost of lookup effiieny. 5.2 Lookup Effiieny Lookup lateny is determined by two fators: lookup path length and query proessing time in eah node along the path. Fig. 5a shows the total number of overloaded nodes enountered in query routings grows with the query load. It also shows that ERT/AF leads to muh high lookup effiieny in omparison with the others. Although NS and VS improve over Base to a ertain extent, there remain a large perentage of ongested nodes in the systems in omparison with ERT/AF. NS biases high-apaity nodes for query load, whih may make them more likely overloaded as the system query load inreases. Due to DHT s stritly ontrolled topology and preise lookup algorithm, the assumption of uniformly distributed load of VS does not hold. The fixed outdegree of nodes in NS and VS prevents eah node from adapting traffi load on nodes elastially. In ontrast, ERT/AF enables eah node to math its indegree to its apaity and adapt its indegree in response to the hange of its experiened query load. Furthermore, the query forwarding operation helps avoid overloaded nodes during

10 SHEN AND XU: ELASTIC ROUTING TABLE WITH PROVABLE PERFORMANCE FOR CONGESTION CONTROL IN DHT NETWORKS 251 Fig. 5. Effetiveness of ongestion ontrol protools on lookup effiieny. (a) Heavy nodes in routings, (b) lookup path length, and () lookup time. query routing, leading to higher lookup effiieny. From the figure, we an also observe that both adaptation (ERT/A) and forwarding (ERT/F) lead to a signifiant redution of overloaded nodes in omparison with Base, NS, and VS. In another word, the effetiveness of ERT/AF in avoiding overloaded nodes in routings is attributed to ombined effets of adaptation and forwarding algorithms. Fig. 5b shows the path lengths due to different ongestion protools as the network size inreases. It is expeted that VS leads to a muh longer query path length than Base beause of the additional virtual server layer in routing. This is onsistent with the observation in [12] that VS ahieves the objetive of load balaning at the ost of lookup effiieny, and the path length inreases by at most an additive onstant. In ontrast, ERT/A and ERT/F redue the path lengths of Base. It implies that both indegree adaptation and forwarding ontribute to the effetiveness of ERT/ AF in lookup path length redution. Adaptation offers eah node more neighbor andidates to forward a query. The loality-aware forwarding algorithm takes both logial and physial distane into aount in deision making. It always hooses the neighbors that are logially losest to the destination and then physially losest to the destination. Thus, the forwarding algorithm greatly redues the lookup path length. Both indegree adaptation and forwarding algorithms redue the lookup path length, leading to muh shorter lookup path length of ERT/AF. Likewise, NS onsiders node distane in neighbor seletion and redues the lookup path length of Base signifiantly. Fig. 5 shows the average, 1st, and 99th perentiles of proessing time per query as ombined effets of redued ongested nodes and lookup path length on the overall query proessing time. Although VS redues the number of ongested nodes of Base, its benefits may be outweighed by its extended path length. Without dynami ongestion ontrol, Base and NS may forward queries to ongested nodes. Stati indegree assignment by NS only results in marginal proessing time redution. On the ontrary, ERT/ AF dramatially redues the proessing time per query of VS and Base. In ERT/AF, periodi indegree adaptation tunes eah node degree to its load adaptively, and the forwarding operation tends to diret queries to light nodes that have suffiient apaity to handle them promptly. Both of the adaptation (ERT/A) and forwarding (ERT/F) fators help improve query proessing effiieny in ERT/AF by reduing ongested nodes and lookup path length. The effiieny seems attributed more to the forwarding fator. 5.3 Indegree of Nodes and Maintenane Cost A node with a higher indegree would most likely experiene higher query load and vie versa. In order to illustrate the query load imbalane due to indegree variane, we measured the number of nodes with different indegrees in Cyloid DHT with different dimensions. In a Cyloid DHT, the nodes an be divided to two groups: low-indegree nodes and highindegree nodes. The low-indegree nodes have indegree equals to 5, and the high-indegree nodes have indegree equals to 14, 16, 18, 20, and 22 when the dimension equals to 6, 7, 8, 9, and 10, respetively. Fig. 6 illustrates the results. We an observe that there always have a number of highindegree nodes. These nodes onstitute perent of the total nodes. This means that these nodes will reeive more queries and, hene, experiene higher query load than lowindegree nodes. The experiment results onfirm the existene of query load imbalane. Reall that ERT-based ongestion ontrol protool ahieves its goal using elasti routing tables to adapt eah node indegree to its load. In addition to maintain the tables, eah node needs to maintain a list of bakward fingers (the same number of its indegree). We measured the maximum indegree and outdegree of eah node and alulated the average, 1st, and 99th perentiles of these values in eah method. As we mentioned that the node degree is fixed in NS and VS, but is variable in ERT. We use maximum indegree and outdegree instead of average for the evaluation of the management overhead of ERT in the worse ase. Figs. 7a and 7b plot the results. Although inlinks in Base and VS don t need to be maintained, we inlude their degrees for omparison. As expeted, the figures show that the degree rates of Base, NS, and VS do not hange, while the rates of ERT/AF hange as total query load hanges. Beause ERT tunes node indegrees to adapt to different query load aordingly, some lightly loaded node indegrees reah a high value to request more load. Indegree hange leads to Fig. 6. Indegrees of nodes.

11 252 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY 2010 Fig. 7. Degrees of routing tables in different ongestion ontrol protools. (a) Indegree and (b) outdegree. Fig. 8. Effetiveness of ongestion ontrol protools in skewed lookups. (a) Heavy nodes in routings, (b) lookup time, and () share. outdegree hange. The indegree and outdegree of VS are muh higher than others beause that virtual node usage leads to larger overlay size. Our results turn out that the ombination of the average, 1st, and 99th perentiles of indegree and outdegree of ERT in the worse ase is muh less than the outdegree rates of VS, respetively. Thus, to ahieve ongestion ontrol, VS needs muh higher ost for maintenane, while ERT only needs a little extra maintenane ost. 5.4 Effet of Skewed Lookup Besides node heterogeneity in apaity, query load imbalane ours with nonuniform and time-varying file popularity and peer interest variation. In this setion, we onsider the effet of skewed lookups. We onsider an impulse of 100 nodes whose IDs are distributed over a ontiguous interval of the ID spae, and whose interests are in the same 50 keys randomly hosen from the ID spae. We varied the query proess rate from 0.1 to 2.2 seond per query on a light node, with 0.5 seond inrease in eah step. Figs. 8a and 8b plot the number of overloaded nodes in routings and the query proessing time of eah method, respetively. It is surprising to see that these result values of VS are muh more than Base. As laimed by the authors in [12] that a good balane of VS is guaranteed only under the uniform load assumption; this explains why VS has poor performane in skewed lookups. In VS, a real node selets IDs of its virtual nodes randomly within onseutive intervals. When query load onentrates on a ertain ID spae interval, the load is alloated to onseutive virtual servers. Sine most of the virtual servers may reside on the same real node, the node more likely beomes overloaded. In ontrast, by assigning and adjusting node indegree based on load dynamially, ombined with topology-aware randomized forwarding algorithm, ERT/ AF an handle skewed lookups aused by the hange of file popularity and node interests. NS yields a similar lookup lateny to Base on average, but exhibits a large variane. Fig. 8 plots the 99th share of eah method. By omparing it with Fig. 4, we an observe that the share rate of eah method is higher in skewed lookups. It is expeted beause the query load onentrates on ertain ID spae part, then ertain nodes. The share rate of NS is still muh higher than others in skewed lookups beause of its strong bias toward high-apaity nodes in neighbor seletion. It is a resoure waste to let low-apaity nodes idle. 5.5 Effet of Churn In DHT networks with hurn, a great number of nodes join, leave, and fail ontinually and rapidly, leading to ontinuous hange of overlay topology. This gives another hallenge to ongestion ontrol. This setion evaluates ERT/AF s adaptability to different levels of hurn. In this experiment, the lookup rate was modeled by a Poisson proess with a rate of 1; that is, there was a lookup every 1 seond. The node join/departure rate was also modeled by a Poisson proess. We ranged node interarrival/ interdeparture time from 0.1 to 0.9 seonds, with 0.1 seond inrement in eah step. Lower time orresponds to higher hurn. Our results are olleted from all node inluding the urrent nodes in the system when all lookups omplete and the nodes departed. Fig. 9a shows the 99th perentile maximum ongestion of eah method. Comparing it with Fig. 4a, we find that the rate of eah method in hurn is lower than without hurn at the point of 3,000 lookups. It is beause with ontinuous nodes join, the same query load is distributed among more nodes than in stati DHT. The rates of NS and Base grow inversely proportional to node interarrival time, and the rates of VS and ERT/AF maintain onstant. When node interarrival/interdeparture time is 0.1 seond, the rate of

12 SHEN AND XU: ELASTIC ROUTING TABLE WITH PROVABLE PERFORMANCE FOR CONGESTION CONTROL IN DHT NETWORKS 253 Fig. 9. Effetiveness of ongestion ontrol protools in networks with hurn. (a) Maximum ongestion and (b) share. Fig. 10. Effetiveness of ongestion ontrol protools on lookup effiieny in hurn. (a) Heavy nodes in routings, (b) lookup path length, and () lookup time. NS is higher than Base s, and it dereases slightly below the Base s when node interarrival time is seonds. This implies that NS has diffiulty to ope with high hurn. Reall that in NS, high-apaity nodes have denser inlinks. In high hurn, some high-apaity nodes may don t have enough apaity for a sudden query flow, whih originally should be responsible by nodes departed. In a modest hurn, the flow is not so intense for nodes to handle. In high hurn, VS has marginally less rate than Base, whih implies that VS an deal with hurn to a ertain extend. We an also see that ERT/AF keeps the rate lose to 1 in different levels of hurn, in ontrolling node ongestion in hurn. Fig. 9b shows the 99th perentile share of eah method. It demonstrates that like in stati DHT, NS performs not so well as others in fair load balane in hurn. The 99th perentile share of ERT/AF in hurn is higher than that without hurn. It is beause ontinuous node joins and departures indue more load on some nodes relative to their apaity. On the other hand, beause of hurn, NS s 99th perentile share is higher than Base, and VS has more balaned query load distribution than others. Fig. 10a shows the number of heavy nodes in routings of eah method in hurn. We an see that the number of NS is muh higher than Base in high hurn, and the number dereases as the node interarrival time inreases; both of them are larger than the result of ERT/AF. This observation is onsistent to the findings in Fig. 9a. It onfirms that ERT/ AF performs the best in reduing heavy nodes proessing query. Fig. 10b shows the lookup path length of eah method. Comparing it with Fig. 5b, we an detet that there s no big differene, exept that ERT/AF has less path length in hurn. We also reorded average timeout for eah method. A time-out ours when a node tries to ontat a departed node. The average time-out of ERT/AF is 0 and is less than 0.06 in other approahes. The reason for shorter path lengths and less time-outs of ERT/AF is that its ERT avoids time-outs by letting eah node have multiple neighbors in eah table entry. Consequently, when a entry neighbor left, others an be used as a substitute instead of making a detour routing. Fig. 10 shows the average, 1st, and 99th perentiles of query proessing time per node of eah method. They are onsistent to those without hurn in Fig. 5, exept that NS yields higher lateny than Base in high hurn. It validates the onlusion that NS is not effiient in oping with hurn. 5.6 Effet of Adaptation and Query Forwarding To evaluate the quality of the indegree adaptation and topology-aware randomized query forwarding, we ompare ERT/AF with algorithms without indegree adaptation (ERT/F) or without query forwarding algorithm (ERT/A). Fig. 4a plots the 99th perentile maximum ongestion rates of different versions. From the figure, we an observe that the topology-aware two-way randomized forwarding algorithm is effetive in reduing the ongestion rate of Base when query load is not high, but beomes not so effetive when the system is highly loaded. For this reason, it is imperative to have a omplementary method to guarantee low node ongestion. The figure shows that the indegree adaptation algorithm redues the ongestion rate signifiantly in various load onditions, whih implies the dramati ontribution of ERT/A in ontrolling node ongestion. The 99th perentile shares in Fig. 4 onfirm the superior performane of forwarding. The query forwarding algorithm ontrols query flow to light nodes, ensuring that queries are forwarded only to nodes with suffiient apaity to handle them. By adjusting node indegree adaptively, indegree adaptation algorithm

13 254 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY 2010 also helps for fair load balaning, though the improvement is not so muh as forwarding. Fig. 5a shows the heavy node number enountered in eah lookup path. From the figure, we an observe that both forwarding and indegree adaptation greatly help eliminate heavy nodes. Their ombination demonstrates an aumulated effet. Fig. 5b plots the path length of eah version. Reall that topology-aware randomized query forwarding algorithm takes node logial distane and physial distane into aount in routing. It is expeted that forwarding leads to a short lookup path. By providing multiple neighbor andidates in eah step for query routing, indegree adaptation is also effetive in reduing path length in most ases. Overall, the ombined effet on overloaded nodes redution and lookup path length shortening results in a great saving of lookup lateny, as shown in Fig. 5. The figure shows that both of ERT/A and ERT/F redue proessing time per query of Base, and their effetiveness are aggregated together, leading to muh higher query proessing effiieny of ERT/AF. From the observations, we an onlude that both of ERT/A and ERT/F play an important role in ERT/AF in ongestion ontrol and lookup lateny redution. Lak of either one will degrade the final effetiveness of ERT/AF. 6 CONCLUSIONS DHT networks have an inherent ongestion problem aused by query load due to the nature of heterogeneity and dynamism of nodes. Nonuniform and time-varying file popularity makes the problem more severe. This paper presents a ERT-based ongestion ontrol protool for DHT networks, whih onsists of three omponents: indegree assignment, periodi indegree adaptation, and topologyaware query forwarding. Theoretial analysis establishes the bounds of the indegree and outdegree, and proves the performane of the protool in general in terms of both query load balane fator and query proessing time. Simulation results show the superiority of the ongestion ontrol protool ompared with other methods in stati network, skewed lookups, and in hurn, and show the effetiveness of eah algorithm in the protool. It makes full use of eah node s apaity while ontrol eah node s load below its apaity. It improves the lookup effiieny in DHT network by reduing lookup lateny. APPENDIX Proof of Theorem 3.3. A node i with yli index k i an have at most 2 ki IDs for routing table entry seletion. We use O i to represent a set of these IDs, and it an have at most 2 ki IDs for routing table entry bakward finger seletion. We use I i to represent a set of these IDs. Beause there are totally d 0 2 d0 IDs and n nodes in a Cyloid system with dimension d 0, the probability of n node number for an ID is d 0 2d0, denoted by. Consequently, the number of nodes in 2 k i IDs is proximately 2 ki, denoted by J i. We define average ID spae responsible by a node as I, and use I i to represent the I of node i. Let X i be the indiator variable for the event that node i probes node j for indegree expansion. If j is hosen by i, then i has a bakward finger to node j, whih inrements j s outdegree with a outlink to i. P Our purpose is to find out the upper bound of i X represented by X. We assume that Cyloid nodes also probe their suessors and predeessors for indegree expansion. The probability that node j is hosen to have an outlink pointing to i is 1 J i. The probability that node j is within a=2 number of i s suessors or predeessors is ai þ 2I j. Let s assume that node i probes m i IDs. Then, the probability is expressed as P ðjid j ID i j maxf0;m i J i giþ ¼maxf0;m i J i gi þ 2I j P ðm i >J i Þ. 1 In the ase that i 2O j, when J i m i ;E½X i Š¼m i J i ; 1 when J i <m i ;E½X i Š¼J i J i þðm i J i ÞI þ 2I j. In the ase that i 62 O j, when J i m i ;E½X i Š¼0 and when J i <m i ;E½X i Š¼ðm i J i ÞI þ 2I j. Therefore, 8 >< minfj i ;m i g 1 þ maxf0;m i J i gi E½X i Š¼ J i >: þ 2I j P ðm i >J i Þ; i 2O j ; maxf0;m i J i gi þ 2I j P ðm i >J i Þ; i 62 O j ^ i 6¼ j: E½XŠ ¼ X i2r E½X i Š ¼ X i2o j minfj i ;m i g 1 J i þ maxf0;m i J i gi þ 2I j Pðm i >J i Þ þ X i62o j^i6¼j ðmaxf0;m i J i gi þ 2I j Pðm i >J i ÞÞ ¼ X i2o j minfj i ;m i g 1 J i þ X i2r^i6¼j ðmaxf0;m i J i gi þ 2I j Pðm i >J i ÞÞ ¼ 1 X X 2 k minfj j 1 i ;m i gþ ðmaxf0;m i 2 k i gi i2o j i2r^i6¼j þ 2I j P ðm i >J i ÞÞ 1 X 2 kj 1 2k j max i2oj I i þ I max mi 2 K i þ 2 2 l max min 2 l max min i2r^i6¼j 2 d 0 þ n Oð1Þ O d 0 2 d 0 O d 0 þ Oð1Þ: We define b i ðtþ as the number of servers with i spare apaities at time t; m i ðtþ as the number of servers with at most i spare apaities at time t; p i ðtþ ¼d i ðtþ=d as the fration of servers of i spare apaity; and s i ðtþ ¼m i ðtþ=d as the fration of servers with at most i spare apaities. Suh that p i ¼ s i s i 1.Inanempty system, whih orresponds to one with no ustomers, s ¼ 1, and s i ¼ 0 for i<.afixed point is a point p in whih dsi dt ¼ 0. The rate of spare apaity hanging in a node depends on whether it has more or fewer than threshold, T, spare ut

14 SHEN AND XU: ELASTIC ROUTING TABLE WITH PROVABLE PERFORMANCE FOR CONGESTION CONTROL IN DHT NETWORKS 255 apaities. In the following, we alulate dsi dt in the ase i T 1 and i<t 1, respetively. An arriving query oupies the ith apaity of a server if one of b events happen: first, its first hoie has i þ 1 spare apaities; seond, its first hoie has T 1 spare apaities and its seond hoie has i þ 1 spare apaities;..., its first b-1 hoies have T 1 spare apaities and its bth hoie has i þ 1 spare apaities. So that, there are nðp iþ1 þ s T 1 p iþ1 þ s 2 T 1 p iþ1 þþs b 1 T 1 p iþ1þ servers whose spare apaities hange from i þ 1 to i during dt. Meanwhile, dp i servers hange their spare apaities from i to i þ 1. As a result, we get ds i dt ¼ p iþ1 þ s T 1 p iþ1 þ s 2 T 1 p iþ1 þþs b 1 T 1 p iþ1 pi ; ds i i T 1; dt ¼ ðs iþ1 s i Þ sb T 1 1 s T 1 1 ðs i s i 1 Þ; i T 1 When i<t 1, the number of queries arriving over dt is bdt, and that for an query being forwarded to a server with i þ 1 spare apaity is b i dt ¼ dðs i s i 1 Þdt. Consequently, ds i dt ¼ 1 b dm i dt ¼ ðs b iþ1 sb i Þ ðs i s i 1 Þ. The differential equations for the query forwarding model (QFM) when i< T ¼ 1 onsidering a node with b speifi neighbors is ds i dt ¼ ðsb iþ1 sb i Þ ðs i s i 1 Þ;i<T 1. We get the differential equations for QFM: ds i dt ¼ 8 >< ðs iþ1 s i Þ sb T 1 1 s T 1 1 ðs i s i 1 Þ; i T 1; ð3þ >: s b iþ1 i sb ðsi s i 1 Þ; i < T 1: ð4þ Lemma P A.1. The QFM with d 2 has a unique fixed point with 1 i¼ 1 s i < 1 given by 8 s i ¼ð AÞ A i 1 A 1 þ A i ; >< A ¼ sb T 1 1 ; T 1 i ; s T 1 1 s i ¼ bt i 1 1 >: s bt i 1 1 T 1 ; i < T 1: b 1 Proof. With the ondition dsi dt ¼ 0 for all i, we derive the value of s i inluding s T 1 when i T 1 with s ¼ 1. We summer the (3) over all if i T 1, and get s 1 ¼ A þ As, assuming A ¼ sb T 1 1 s T 1 1. By indution: s 2 ¼ð AÞð1 þ AÞþA 2, we get s i ¼ð AÞ A i 1 A 1 þ A i ; A ¼ sb T 1 1 s T 1 1 ; i T 1: By summing the (4) over all i T 1 with s 1 ¼ 0, we derive that s T 2 ¼ s b T 1. By indution: s T 3 ¼ s b T 2 ¼ s b b b T 1 ¼ 2 1 b 1 s b 2 T 1...; we get s i ¼ bt i 1 1 b 1 s bt i 1 1 P T 1 ði <T 1Þ. We use 1 i¼ 1 s i < 1 to ensure that the sum onverges absolutely. tu Proof of Theorem 4.1. By (4), we an get that in the ase when i<t 1, an inoming query arriving on a node at time t lets the node has i spare apaity with probability s iþ1 ðtþ b s i ðtþ b, and this query beomes the ð iþth query in the proess waiting queue of the server. Therefore, the expeted waiting time of the query is: X 1 i¼t 2 ð iþ s iþ1 ðtþ b s i ðtþ b ¼ð T þ 2Þs b T 1 þ X 1 i¼t 2 s b i ðtþ By (3), we an get that in the ase when i T 1, the expeted waiting time of a query is X T 1 1 ð iþða=þðs iþ1 ðtþ s i ðtþþ ¼ðA=Þ X T i¼! s i ð T þ 1Þs T 1 ; A ¼ sb T 1 1 s T 1 1 : By Lemma A.1, at t!1, the QFM onverges to the fixed point. So that the expeted waiting time for a query in a server an be made: X T ða=þ ð AÞ A i 1 A 1 i¼ þð T þ 2Þs b T 1 þ X 1 i¼t 2 þ A i bt i b b 1 ð T þ 1Þs T 1 s bt i b T 1 þ oð1þ: In QFM, the time a query waits on a node when i T 1 is less than the time when i<t 1 beause in the former ase the query is proessed by a light node. The above bound an be enlarged to P 1 i¼ sb i ¼ P 1 b i b b 1 i¼ 1. Then we an apply the proved result of exponential time improvement in [21] to QFM. tu ACKNOWLEDGMENTS This researh was supported in part by US National Siene Foundation grants CNS , CNS , CCF , MCS , CNS , CNS , and CNS An early version of this work [29] was presented in the Proeedings of ICDCS 06. REFERENCES [1] Mute, [2] L.A. Adami, B.A. Huberman, R.M. Lukose, and A.R. Puniyani, Searh in Power Law Networks, Physial Rev. E, vol. 64, pp , [3] A.R. Bharambe, M. Agrawal, and S. Seshan, Merury: Supporting Salable Multi-Attribute Range Queries, Pro. ACM SIGCOMM, [4] M. Bienkowski, M. Korzeniowski, and F.M. auf der Heide, Dynami Load Balaning in Distributed Hash Tables, Pro. Int l Workshop Peer-to-Peer Systems (IPTPS), [5] S. Bono et al., Mantis: A Lightweight, Server-Anonymity Preserving, Searhable P2P Network, tehnial report, Johns Hopkins Univ., [6] J. Byers, J. Considine, and M. Mitzenmaher, Simple Load Balaning for Distributed Hash Tables, Pro. Seond Int l Workshop Peer-to-Peer Systems (IPTPS), 2003.

15 256 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 21, NO. 2, FEBRUARY 2010 [7] M. Castro, M. Costa, and A. Rowstron, Debunking Some Myths About Strutured and Unstrutured Overlays, Pro. Seond Conf. Symp. Networked Systems Design & Implementation (NSDI), [8] Y. Chawathe, S. Ratnasamy, L. Breslau, N. Lanham, and S. Shenker, Making Gnutella Like P2P Systems Salable, Pro. ACM SIGCOMM, [9] B.G. Chun, B.Y. Zhao, and J.D. Kubiatowiz, Impat of Neighbor Seletion on Performane and Resiliene of Strutured P2P Networks, Pro. Fourth Int l Workshop Peer-To-Peer Systems (IPTPS), [10] I. Clarke, O. Sandberg, B. Wiley, and T. Hong, Freenet: A Distributed Anonymous Information Storage and Retrieval System, Leture Notes in Computer Siene, vol. 2009, pp , Springer, [11] B. Godfrey, K. Lakshminarayanan, S. Surana, R. Karp, and I. Stoia, Load Balaning in Dynami Strutured P2P Systems, Performane Evaluation, vol. 63, no. 3, pp , [12] B. Godfrey and I. Stoia, Heterogeneity and Load Balane in Distributed Hash Tables, Pro. IEEE INFOCOM, [13] P. Gummadi, R. Dunn, S. Saroiu, S. Gribble, H. Levy, and J. Zahorjan, Measurement, Modeling, and Analysis of a Peer-to- Peer File-Sharing Workload, Pro. 19th ACM Symp. Operating Systems Priniples (SOSP), [14] J. Hu, M. Li, W. Zheng, D. Wang, N. Ning, and H. Dong, SmartBoa: Construting P2P Overlay Network in the Heterogeneous Internet Using Irregular Routing Tables, Pro. Third Int l Workshop Peer-to-Peer Systems (IPTPS), [15] D. Karger et al., Consistent Hashing and Random Trees: Distributed Cahing Protools for Relieving Hot Spots on the World Wide Web, Pro. 29th Ann. ACM Symp. Theory of Computing (STOC), [16] B. Levine and C. Shields, Hordes: A Multiast-Based Protool for Anonymity, J. Computer Seurity, vol. 10, no. 3, pp , [17] J. Li et al., Bandwidth Effiient Management of DHT Routing Tables, Pro. Seond Symp. Networked System Design and Implementation (NSDI 05), [18] Q. Lv, P. Cao, E. Cohen, K. Li, and S. Shenker, Searh and Repliation in Unstrutured Peer-to-Peer Networks, Pro. Ann. ACM Int l Conf. Superomputing (ICS), [19] Q. Lv et al., Can Heterogeneity Make Gnutella Salable? Pro. Int l Workshop Peer-to-Peer Systems (IPTPS), [20] G. Manku, Balaned Binary Trees for ID Management and Load Balane in Distributed Hash Tables, Pro. 23rd Ann. ACM Symp. Priniples of Distributed Computing (PODC), [21] M. Mitzenmaher, On the Analysis of Randomized Load Balaning Shemes, Pro. Ann. ACM Symp. Parallel Algorithms and Arhitetures (SPAA), [22] M. Mitzenmaher et al., Load Balaning with Memory, Pro. 43rd IEEE Symp. Foundations of Computer Siene (FOCS), [23] S. Nath, P.B. Gibbons, S. Seshan, and Z.R. Anderson, Synopsis Diffusion for Robust Aggregation in Sensor Networks, Pro. Seond ACM Conf. Embedded Networked Sensor Systems (SenSys), [24] S. Osokine, The Flow Control Algorithm for the Distributed Broadast-Route Networks with Reliable Transport Links, tehnial report, [25] A. Rao et al., Load Balaning in Strutured P2P Systems, Pro. Int l Workshop Peer-to-Peer Systems (IPTPS), [26] S. Ratnasamy, P. Franis, M. Handley, R. Karp, and S. Shenker, A Salable Content-Addressable Network, Pro. ACM SIGCOMM, pp , [27] A. Rowstron and P. Drushel, Pastry: Salable, Deentralized Objet Loation and Routing for Large-Sale Peer-to-Peer Systems, Pro. Middleware Conf., [28] S. Saroiu, P. Gummadi, and S. Gribble, A Measurement Study of Peer-to-Peer File Sharing Systems, Pro. Multimedia Computing and Networking (MMCN), [29] H. Shen and C. Xu, Elasti Routing Table with Provable Performane for Congestion Control in DHT Networks, Pro. IEEE Int l Conf. Distributed Computing Systems (ICDCS), [30] H. Shen and C. Xu, Hash-Based Proximity Clustering for Load Balaning in Heterogeneous DHT Networks, Pro. 20th Int l Parallel and Distributed Proessing Symp. (IPDPS 06), [31] H. Shen and C. Xu, Loality-Aware and Churn-Resilient Load Balaning Algorithms in Strutured Peer-to-Peer Networks, IEEE Trans. Parallel and Distributed Systems, vol. 18, no. 6, pp , June [32] H. Shen, C. Xu, and G. Chen, Cyloid: A Salable Constant- Degree P2P Overlay Network, Performane Evaluation, vol. 63, no. 3, pp , [33] I. Stoia et al., Chord: A Salable Peer-to-Peer Lookup Protool for Internet Appliations, IEEE/ACM Trans. on Networking, vol. 11, no. 1, pp , Feb [34] B.Y. Zhao et al., Tapestry: An Infrastruture for Fault-Tolerant Wide Area Loation and Routing, Tehnial Report No. UCB/ CSD , Haiying Shen reeived the BS degree in omputer siene and engineering from Tongji University, China, in 2000, and the MS and PhD degrees in omputer engineering from Wayne State University in 2004 and 2006, respetively. She is urrently an assistant professor in the Department of Eletrial and Computer Engineering, and the diretor of the Pervasive Communiations Laboratory of Clemson University. Her researh interests inlude distributed and parallel omputer systems and omputer networks, with an emphasis on peer-to-peer and ontent delivery networks, wireless networks, resoure management in luster and grid omputing, and data mining. Her researh work has been published in top journals and onferenes in these areas. She was the program ohair for a number of international onferenes and member of the Program Committees of many leading onferenes. She is a member of the IEEE, the IEEE Computer Soiety, and the ACM. Cheng-Zhong Xu reeived the BS and MS degrees from Nanjing University in 1986 and 1989, respetively, and the PhD degree from the University of Hong Kong in He is a professor in the Department of Eletrial and Computer Engineering of Wayne State University (WSU) and the diretor of the Center for Networked Computing Systems. His researh interest inludes networked omputing systems and appliations, in partiular salable and seure Internet servies and arhiteture, sheduling and resoure management in distributed, parallel, and embedded systems, and autonomi systems management for highly reliable omputing. He has published more than 140 peer-reviewed sientifi papers in arhival journals and onferenes in these areas. He is the author of Salable and Seure Internet Servies and Arhiteture (Chapman & Hall/CRC Press, 2005) and the leading oauthor of Load Balaning in Parallel Computers: Theory and Pratie (Kluwer Aademi, 1996). He serves on the editorial boards of IEEE Transations on Parallel and Distributed Systems, Journal of Parallel and Distributed Computing, Journal of Parallel, Emergent, and Distributed Systems, Journal of Computers and Appliations, and Journal of High Performane Computing and Networking. He has also guest edited speial issues for several other journals on network servies and seurity in distributed systems. He has served a number of international onferenes and workshops in various apaities as program hair, general hair, and plenary speaker. He was a reipient of President s Award for Exellene in Teahing of WSU in 2002 and Career Development Chair Award in He is a senior member of the IEEE and the IEEE Computer Soiety, and a member of the ACM.. For more information on this or any other omputing topi, please visit our Digital Library at