Degree-Optimal Routing for P2P Systems

Transcription

1 DOI /s x Degree-Optimal Routing for P2P Systems Giovanni Chiola Gennaro Cordaso Luisa Gargano Mikael Hammar Alberto Negro Vittorio Sarano Springer Siene+Business Media, LLC 2007 Abstrat We define a family of Distributed Hash Table systems whose aim is to ombine the routing effiieny of randomized networks e.g. optimal average path length O(log 2 n/ log ) with degree with the programmability and startup effiieny of a uniform overlay that is, a deterministi system in whih the overlay network is transitive and greedy routing is optimal. It is known that (log n) is a lower bound on the average path length for uniform overlays with O(logn) degree (Xu et al., IEEE J. Sel. Areas Commun. 22(1), , 2004). Our work is inspired by neighbor-of-neighbor (NoN) routing, a reently introdued variation of greedy routing that allows us to ahieve optimal average path length in randomized networks. The advantage of our proposal is that of allowing the NoN tehnique to be implemented without adding any overhead to the orresponding deterministi network. We propose a family of networks parameterized with a positive integer whih measures the amount of randomness that is used. By varying the value, the system goes from the deterministi ase ( = 1) to an almost uniform system. Inreasing to relatively low values allows for routing with asymptotially optimal average path length while retaining most of the advantages of a uniform system, suh as easy programmability and quik bootstrap of the nodes entering the system. This work was partially supported by the Italian FIRB projet WEB-MINDS (Wide-salE, Broadband MIddleware for Network Distributed Servies), G. Chiola DISI, Università di Genova, via Dodeaneso 35, Genova, Italy G. Cordaso ( ) L. Gargano A. Negro V. Sarano DIA, Università di Salerno, via Ponte don Melillo, 84084, Fisiano (SA), Italy ordaso@dia.unisa.it M. Hammar Researh & Development, Apptus Tehnologies AB, IDEON, , Lund, Sweden

2 We also provide a mathing lower bound for the average path length of the routing shemes for any. Keywords Peer-to-peer Overlay network Greedy routing 1 Introdution Peer-to-peer (P2P) is a lass of network appliations that takes advantage of existing omputing power, omputer storage, and networking onnetivity, whih are available at the edges of the Internet. Hene P2P allows users to leverage their olletive power to the benefit of all. P2P systems have quikly beome very popular in the reent years. Several appliations exploit P2P networks. Without a doubt, file sharing systems are among the most popular. Sharing other resoures, suh as file storage and CPU time are also ommon. One of the most important benefits provided by P2P systems is the salability.ina P2P system, eah onsumer of resoures also donates resoures. Nevertheless, salability has been reognized as the entral hallenge in designing suh systems [18]. To design a salable system, several P2P systems are based on the distributed hash table (DHT) shema [5, 17, 19, 21]. A DHT is a self-organizing overlay network that allows us to add, delete and look up hash table elements. In DHT shemes, data as well as nodes are assoiated with a key and eah node in the system is responsible for storing a ertain range of keys. Eah node stores data that orrespond to a ertain portion of the key spae, and eah node uses a routing shema to forward the request for data whose key does not belong to its own key spae to the appropriate next-hop. Reently, several systems have been proposed whereby hosts onfigure themselves into a strutured network in suh a way that lookups only require a small number of hops. The effiieny of routing algorithms in P2P systems represents a key fator for ensuring salability. A fast algorithm not only makes the primitive operations (suh as aessing or inserting data) effiient, but also impats positively on maintenane, i.e., by keeping the DHT onsistent and in working order in the fae of both dynami and unpreditable join/leave of peers in the network and of node failures. 1.1 Our Results: A Road-Map In this paper, we propose a novel tehnique that allows us to retain the improvements provided by the NoN routing introdued in [13] over randomized networks, while eliminating the drawbaks in system overhead that is implied by this tehnique. Moreover, in order to evaluate the effetiveness of the proposed strategy, we apply our tehnique to two well known networks: Chord [19] and Symphony [13]. In a sense, our results allow us to retain the programmability and quik bootstrap of uniform networks while leveraging randomization to ensure effiieny. In this setion we provide a road-map to our results that an be expressed as obtaining a ombination of the ease of management of uniform networks and effiieny of randomized networks.

3 A preliminary version of some results desribed in this paper were reported at the 8th International Symposium on Parallel Arhitetures, Algorithms, and Networks (I-SPAN 2005) [2] and at the 10th IEEE Symposium on Computers and Communiatiuons (ISCC 2005) [4] Greedy Routing on Uniform Networks: Programmability and Quik Bootstrap The traditional measurements of routing effiieny (degree, average path length, et.) must often be supplemented, in pratie, with the onsideration of how simple the implementation of the routing algorithms at eah node is. In this senario, the popularity of uniform networks, aswellasthegreedy routing approah, is easily explained. In fat, the first proposed DHT systems were uniform networks and exploited a greedy routing algorithm [5, 17, 19, 21]. Definition 1 (Uniform networks [20]) Consider a network U embeddedinto a irular metri spae (ring) having 2 m identifiers labeled from 0 to 2 m 1 in lokwise order. U is uniform iff there exist 1 integers (a.k.a. jumps) j 0,j 1,...,j 1 (where j i [1, 2 m ) for eah i = 0,..., 1) suh that eah node v U is onneted by an edge to the nodes 1 v + j i, for eah i = 0,..., 1 (i.e., all the onnetions are symmetri.) An extensive study of uniform systems was arried out by Xu et al. in [20]. The most important property of uniform networks is that the greedy routing is optimal. Moreover, routing ours between the portion of key spae delimited by soure and destination (loality in the key spae) and, therefore, any possible semantis present in the keys is not lost (e.g. if proximity between keys implies proximity between peers) [13]. Greedy routing is also very simple to implement and has an impliit fault tolerane apability. Indeed, at eah step the greedy routing forwards the message to a peer that is loser to the destination. Therefore, if a faulty edge (peer) is found during the routing, the work that has been arried out is not lost. Indeed, eah step that is performed independently gets the message loser to the destination. Furthermore, a greedy routing on a uniform network is peer-ongestion-free [20] and ensures a quik bootstrap to all the peers entering the network. On the down side, uniform networks produe paths of length that are larger than what would be required in a network of that given node degree. A typial example is Chord [19] that has degree O(log n) and in whih the greedy routing produes an average path length O(log n), whereas the lower bound is (log n/ log log n). Ifwe limit our attention to uniform networks, Xuetal.[20] showed that Chord, as well as the other popular DHT systems like Pastry [5], CAN [17] and Tapestry [21], are indeed optimal. The join operation in the Chord-like protool allows the DHT to support nodes that dynamially enter the system. It is rather ostly ompared with the other operations. The number of hops required by a join operation is O((finger table size) (average path length)) w.h.p., sine we need a lookup in order to fill eah finger table 1 All the arithmeti operations on the ring are done mod 2 m.

4 entry. The leave operation is ost-free, with the exeption of stabilization osts. The latter an be redued using tehniques similar to those applied to the optimization of the join. In pratie, one wants to ensure a quik bootstrap to all the nodes entering the network and, in some ases, this is a strit requirement of the appliation. For example, DHT-based distributed file systems may want to give (rather) effiient aess to all the files as soon as a node plugs itself into the system. In uniform networks, quik bootstrap of a node v an easily be obtained by using the finger table of v s predeessor (i.e., the node that preedes v on the ring) as a starting point. In fat, v and v s predeessor finger tables are quite similar, therefore, the i-th finger of v is effiiently obtained by asking v s predeessor for its i-th finger. Of ourse, uniformity of the network is ruial in this senario. Thus, on the average, the number of hops required by a join operation beomes O(finger table size), sine the lookup for eah finger is omputed on a small portion of the ring where only an average of O(1) nodes are present. It must be pointed out that the use of randomized networks makes this approah unfeasible. This is due to the total lak of relationship between the finger tables of any pair of nodes in the network NoN Routing on Randomized Networks: Optimal Average Path Length Reently, a novel approah for routing in DHTs whih improves on greedy routing was proposed by Manku et al. in [12]. This approah, alled NoN (Neighborsof-Neighbors) or 1-lookahead routing, substantially onsists in making the greedy hoie by not only looking at the neighbors of a node, but by looking at all the nodes that are at most two hops away from the node itself. By using NoN routing, lateny an be optimally redued in several well known topologies (suh as Chord) provided that randomization is used to establish the neighbors of the nodes. Hene, the use of randomization together with the NoN routing allows us to keep the advantages of greedy routing to some extent, while optimizing the lateny. For example, while it is known that Chord is not degree optimal (it uses log n degree and has an average path length (1/2) log n), [13] highlights that by inserting randomization in the hoie of the neighbors of a node and using NoN routing, one an, with log n degree, make the lateny (i.e. average path length) drop to O (log n/ log log n). It must be pointed out that the NoN approah without randomization has proved to be [7] ineffetive for Chord, sine the average path length is (log n). Unfortunately, the results in [13] are obtained by trading off programmability and feasibility for effiieny. First of all, in NoN routing a ertain amount of overhead is inherent and should not be underestimated. Randomization and NoN routing require the list of its neighbors of neighbors be transmitted to a node. While in [13] Manku et al. argued that this an be implemented without extra ost by using keep-alive TCP messages, we believe that beause of abstration requirements, the appliation protool should not tamper with the transport protools, and that performane preditability of a NoN implementation an be seriously limited by underestimating this overhead in the analysis [3, 4]. We substantiate our laim in Set. 5 by providing a lower bound to the ommuniation ost that is inurred by the NoN routing on randomized networks. Moreover, the randomization of the original network trades the

5 derement of the expeted number of hops for the loss of uniformity, whih would ensure both easy programmability and quik bootstrap. For the sake of ompleteness, it must also be said that several other proposals have been made that give up the uniformity onstraint in order to improve the average path length of uniform networks. In fat, some researh groups (independently) proposed optimal solutions (with non-greedy routing algorithms) to build non-uniform overlay networks based on the stati butterfly networks or the de Bruijn graphs [6, 8, 10, 15]. However, suh systems lak on feasibility due to their omplex routing algorithms. Besides, Aspnes et al. [1] proposed a new onept of distributed data struture, whih allows queries based on key ordering to be performed. Anyhow, the proposed systems, alled Skip-graphs, do not address the issue of load balaning the number of resoures per peer Our Proposal: The Best of Both Worlds Our proposal ombines the programmability and startup effiieny of a uniform system with the routing effiieny of randomized networks e.g. optimal average path length O(log 2 n/ log ), where n is the network size and is the degree of eah node. After preliminary definitions in Set. 2, Set. 3 introdues a family of networks H that are parameterized by the atual amount of randomness that is used. By hanging the integer parameter, the network exhibits a behavior that goes from the deterministi ase ( = 1) to an almost uniform system. In Set. 4 we show that NoN routing on H networks based on Chord (Set. 4.1) and Symphony (Set. 4.2), allows to obtain the optimal to within onstant fators average path length for = (log n/ log ). In Set. 5, we summarize the results of the previous setion in order to allow a fair omparison with the traditional NoN routing [13], whose ommuniation ost is expliitly bounded from below. We omplete our study by experimentally showing, in Set. 6, that optimal effiieny an be obtained with relatively low values of (for example between 2 and 5) for a signifiant size of the network (around 10 6 nodes). In partiular, our approah leads to networks whih both maintain the degree and the optimal average path length of the randomized versions of the network, and almost maintain the system overhead of the deterministi version. Our simulations, whih were performed both at varying sizes of the network and at varying network loads (i.e. number of nodes with respet to number of available identifiers), validated the theoretial results. Before the Conlusions setion, we also prove a mathing lower bound in Set. 7 for = log n and for any value of. 2 Preliminary Definitions We onsider a set of n nodes lying on a ring of 2 m identifiers. Identifiers are labeled from 0 to 2 m 1 in lokwise order. All the operations are on the irular metri, e.g. mod 2 m.

6 Eah node v has an m-bit identifier, and is onneted to its predeessor P(v)and its suessor S(v) on the ring. In general, one has to deal with the ase in whih only some of all 2 m possible identifiers orrespond to a node that is atually present in the network. Hene, P(v) is the first node before v on the ring, i.e., no node identifier is present in the ring interval [P(v)+ 1,v 1]; S(v) is the first node following v on the ring, i.e., no node identifier is present in the ring interval [v + 1,S(v) 1]. Moreover, throughout this paper, when we say that a node v is onneted to id, we atually mean that v is onneted to the first node with key greater than or equal to id on the ring, i.e., to the node orresponding to the identifier id following id on the ring, suh that no node identifier is present in the interval [id,id 1]. In this paper we will exploit our ideas by applying them to the two well known overlays of Chord and Symphony, whose definitions are reported below: Chord: [19] Eah node v is onneted by edges to the nodes v + 2 i, for eah i = 0,...,m 1. Notie that Chord is a uniform network (f. Definition 1) where = m and j i = 2 i for eah i = 0,..., 1. R-Chord: [13] Eah node v is onneted by edges to the nodes v + 2 i + r(i), where r(i) is an integer hosen by v at random with uniform probability in the range [0, 2 i ), for eah i = 0,...,m 1. Symphony*: [13] Letγ denote a real number satisfying log γ = log 2m, where is the node degree (Symphony* an have arbitrary degree 1). For eah i = 1,...,, let I i =[γ i 1,γ i ) and let φ i denote a probability distribution over the integers in I i, suh that the probability of a I i is proportional to 1/a. For eah i = 1,..., an edge is established from node v to v + a i, where a i is an integer drawn from φ i. We observe that Symphony* an be onsidered the randomized version of a uniform network having jumps j i = γ i for eah i = 0,..., Neighbor of Neighbor Routing The following definition of the neighbor of neighbor (NoN) routing was provided by [13]. We assume that eah node holds its neighbors routing tables as well as its own routing table. Definition 2 ([13] Neighbor of neighbor routing) Let (V, E) be a graph embedded in a metri spae and d : V 2 R + be a metri for the nodes in the network. Neighbor of neighbor routing entails the following deision (where v is the urrent node and t is the target key): 1. Let N(v)={u 1,u 2,...,u } be the neighbors of v. 2. For eah i=1,...,,letw i1,w i2,...,w i be the neighbors of u i and let N 2 (v) = {w ij 1 i, j }. 3. Among these 2 + nodes, assume that z is the one losest to the target (with respet to metri d()). 4. If z N(v) route the message from v to z else z = w ij,forsomei and j, and route the message from v via u i to z.

7 Remark We all the (standard) definition given above, whih was analyzed in [13] from a theoretial point of view, 2-phase NoN. A more effiient definition, that we all 1-phase NoN, an be obtained by replaing Step 4 as follows: 4.Ifz N(v) route the message from v to z else z = w ij,forsomei and j and route the message from v to u i. Intuition an easily support the laim of effiieny of 1-phase NoN: re-applying the whole algorithm at node u i an only extend the hoies. Furthermore, experiments have shown that 2-phase NoN is slower than 1-phase NoN. In the NoN routing algorithm, u i might not be the neighbor of v whih is losest to the target (with respet to metri d). The algorithm ould be viewed as a greedy algorithm on the square of the graph a message gets routed to the best possible node among those at hop-distane two. 3H -Networks In this setion we introdue the novel family of H -networks. We assume that nodes are partitioned into a (predefined) number of lasses so that nodes in the same lass an be onsidered as satisfying the uniformity requirement. Namely, eah node randomly hooses one of the lasses to belong to; eah node in lass i, for i = 0,..., 1, hooses its neighbors depending on the parameter i. Definition 3 (H -Network) Let U be a generi uniform overlay network (see Definition 1), having n nodes and jumps j 0,j 1,...,j 1.Let be any given positive integer in the interval [1, 2 m ] and H() be a ryptographi hash funtion (like, e.g., SHA-1 [16]), that maps an identifier id on the interval [0, 1). Consider real numbers λ 0,λ 1,...,λ 1 suh that 0 = λ 0 <λ 1 < <λ 1 < 1. The network H -U is obtained from U as follows: For any v = 0,...,2 m 1, node v is onneted by an edge to the nodes where i = 0,..., 1, j = 2 m and v = H(v). v + j i + λ v (j i+1 j i ), (1) We refer to the integers 0, 1,..., 1 as the node lasses and to the integer v as the lass of v. Easily, v satisfies v {0,..., 1}. It is lear that the following property holds. Property 1 Eah node hooses its lass independently from among the set {0, 1,..., 1} with probability 1/. We an also observe that an H -Network is fully desribed showing the generating network U and the values λ l for l = 0,..., 1. In this paper we shall analyze the H -Networks that are originated from two signifiant overlay networks: Chord and Symphony*.

8 Considering the Chord network and defining λ l = l/, for eah l = 0,..., 1, we obtain the H -Chord network: H -Chord: Eah node v is onneted by an edge to the nodes v + 2 i + v2 i, for eah i = 0,...,m 1. We reall that Symphony* an be seen as a randomized version of the deterministi uniform network having jumps j i = γ i for eah i = 0,..., 1. By defining λ l = γ l/ 1 γ 1, for eah l = 0,..., 1, we obtain the H -Symphony* network: H -Symphony*: Eah node v is onneted by an edge to the nodes v + γ i+ v, for eah i = 0,..., A Partiular Case: H-Networks By varying the value, the network H -U goes from the underlying uniform network U when = 1, i.e., all nodes are in the same lass, to a randomized network when = 2 m. In this latter ase, eah node an hoose any value in [j i,j i+1 ) as its i-th network s jump; we will denote the resulting network by H-U. In partiular, in ase of Chord and Symphony we get: H-Chord: Eah node v is onneted by an edge to the nodes v + 2 i + H(v)2 i, for eah i = 0,...,m 1. H-Symphony*: Eah node v is onneted by an edge to the nodes v + γ i+h(v), for eah i = 0,..., 1. We will show that H-Networks are haraterized by the fat that by simply using the hash-funtion defined on the node identifiers, they allow us to preserve the O (log n/log log n) average number of hops for the NoN routing (vs. the need for log n random numbers needed in the randomized overlay, fr. R-Chord and Symphony [13]). 4 Routing in H -Networks 4.1 Routing in H -Chord In this setion we present the upper bounds we obtained for routing in H -Chord networks. Lemma 1 Consider >1. The average path length is O(log n + log n log log n ) hops for the NoN routing algorithm on H -Chord with n = 2 m nodes. Proof Consider a node s that wants to send a message to a node t at distane d(s,t) = d. Letp be the unique integer suh that 2 p d<2 p+1 log n and q = min{, log log n }. There are two ases to be taken into onsideration. Case 1: p q. In this ase, O(q) hops suffie to reah the destination, sine the distane dereases at least by a fator of 3/4 for eah exeuted hop (f. [11]).

9 Case 2: p>q. Consider the interval of size d = d/q ending in the destination t. I = (t d,t]. In order to prove the lemma in this ase, we first show that with onstant probability we an reah the interval I in two hops. Let s 1,...,s q 1 denote the first q 1 neighbors of s, that is, s i = s + 2 i s 2 i +, for i = 1,...,q 1; these are the neighbors of s in the interval [s,s + 2 q ). Moreover, let s 0 = s and S ={s 0,...,s q 1 }. Finally, denote by J l (s i ) the lth neighbor of s i, J l (s i ) = s i + 2 l si 2 l +. From now on, by NoN of s, we mean the set S = S {J l (s i ) 1 i<q,0 l<m}. We are investigating the probability of S having an outgoing edge entering the interval I, that is, the probability We an prove the following P = Pr[ 0 i<qand 0 l<ms.t. J l (s i ) I]. Claim 1 For any node s i S, the probability P that it has an outgoing edge entering the interval I is at least 1 4q. Before proving the laim let us look at its onsequenes. Let S S be a maximal subset of nodes whih belong to different lasses. Easily, S q α, for some onstant α 1, with probability not less than 1 1 α q, beause q. Sine nodes in S belong to different lasses, the ases in whih one of them has a link in the interval I are independent from one node to another. Therefore, the probability that none of these nodes has an outgoing edge reahing the interval I is ( P 1 (1 P ) S ) S 1 4q 8α. Hene, the expeted number of nodes enountered before a suessful event ours is O(1). Thus, in O(log q n) hops, the distane drops to 2 p, where p <q, and we have redued ase two to ase one. Aordingly, we need O(q+log q n) hops to reah the target t. Hene, one an find the desired value of the average path length by observing that q = min{, We now omplete the proof by showing Claim 1. log n log log n }.

10 Consider a node s i at distane d i from t. We are investigating the probability of s i having an outgoing edge entering the interval I, i.e., P = Pr[ 0 l<ms.t. J l (s i ) I]. Let p i be the unique integer ( p) suh that 2 p i d i < 2 p i+1, there are two ases to be taken into onsideration. 1. d i d 2 p i. In this ase the only jump that an reah the interval I is the p i -th. Let C be the set of lasses that allows s i to reah the interval I (i.e. C,s i + 2 p i + 2 p i I ). Sine eah node independently hooses a lass with probability 1/ we have that s i reahes the interval I with probability at least. Moreover, the distane between the i-th jump of a node belonging to lass C a and the i-th jump of a node belonging to lass a + r is at most r 2i C I 2p i. If q /4 wehave Otherwise C C d 2 p i 2 d p 1 2 p i q 1 2 p 2 p q 2p 1 1 4q. 1 q 1 2 p 1 > 1 4q.. Hene, 2. d i d < 2 p i. Consider the possibility that q /4. In this ase if s i hooses the lass 0 (i.e. si = 0) then J pi (s i ) reahes the interval I (namely, s i + d i d < J pi (s i ) = s i + 2 p i s i + d i = t). Hene, sine eah node hooses a lass independently with probability 1/ we have that s i reahes the interval I with probability at least 1/ 1/4q. Otherwise, define A = (d i d, 2 p i ), B =[2 p i,d i ],C = (2d i 2d, 2 p i+1 ).The probability that one of the hops will reah the interval I is equal to Pr[J pi 1(s i ) A or J pi (s i ) B]. We an observe that J p (s i ) C implies that J p 1 (s i ) A. Observe that for any n>16, the intervals (2d i 2d, 2 p i+1 ) and [2 p i,d i ] do not overlap. Thus, we have: C B 1 2 p i + 2 p q 2 2 p 2 C 1 2 p i d 2 2 p i 1 1 q 2 2 p 2 > 1 4q.

11 We an generalize the results so that they also hold in a ring that does not ontain all nodes. Due to onsistent hashing onstraints the n nodes an be assumed to be uniformly distributed [9]. Theorem 1 Assume >1, then the average path length is O(log n + log n log log n ) hops for the NoN routing algorithm on H -Chord in a ring of size 2 m where the number of live nodes is n 2 m. Proof Consider a soure that wants to send a message at distane d. Beause of Lemma 1, it follows that diminishing the distane to size 2 m /n takes O(log n + log n log log n ) hops. What is left to prove is that the number of live nodes in an interval I of size 2 m /n is small. The expeted number of live nodes in the interval is: [ n ] n n n 2 m E[A]=E x i I = E[x i I]= Pr[x i I]= n 1 2 m = 1. i=1 i=1 Furthermore, using the Chernoff bound, the probability that more than 1.5lnn/ ln ln n nodes lie in an interval I of size 2 m /n is less than n 2 (see Example 4.4 in [14] for more details). Hene, with probability greater than 1 1/n 2, the number of nodes that lie in the same interval is O(log n/ log log n). Corollary 1 The average path length is O( log n log log n ) hops for the NoN routing algorithm on H-Chordinaringofsize2 m where the number of live nodes is n 2 m. 4.2 Routing in H -Symphony* In this setion we analyze greedy and NoN routing on H -Symphony* networks. The following preliminary result will be a tool in the analysis of the performanes of both routing strategies. Lemma 2 Let q>1 be an integer and let denote the probability that a node of an H -Symphony* network, with n = 2 m nodes (where 1 log n) with one hop is able to diminish the distane to a target node from d to, at most, d/q, then 4q log n if q log n. Proof Without loss of generality, assume that the message is at node v = 0 and the target is t = d, we want to route the message to a node w with d w d/q. in one hop. We denote the smallest integer suh that γ l +1 >d by l. Similarly, we denote the smallest integer suh that γ l + a >d by a. Let I and H respetively denote the intervals (d(1 1/q), d] and [γ l + a 1, γ l + a ]. Sine q log n we have that H I. Hene, at least one point of the form γ l + a α for some integer α>0, belongs to I. Sine eah node independently hooses a lass with probability 1/, we have that eah point of the form γ l + a α for some integer α>0isajumpofv with probability 1/. i=1 i=1

12 Let b = q log γ Theory Comput Syst (i.e. bq log γ <(b+ 1)q log γ ). Sine q log n = q log γ, we have that b 1. Define B ={γ l + a 1,γ l + a 2,...,γ l + a b }. Weareableto show that eah element of B belongs to I. By ontradition, we assume that γ l + a b does not belong to I. We know that γ l + a γ l + a b <d(1 1/q). Hene, γ b γ l + a = γ l + a b > >dand sine γ l + a b / I, we have that d d d/q = q q 1. (2) From (2) we have that < b log γ q log( q. Sine q log( q 1 ) q 1 )>1 then <bqlog γ and we have a ontradition beause bq log γ. Sine eah element of B ould be a jump of v with probability 1, then the probability that a jump of v diminishes the distane from d to at most d/q is ( = ) B b 2 > b 2(b + 1)q log γ 4q log n. Lemma 3 Consider > 2logn, then the average path length is O( log2 n ) for greedy routing on H -Symphony* having n = 2 m nodes, when 1 log n. Proof Consider node v that holds a message destined for node t at distane d. Let denote the probability that a v s outgoing link diminishes the distane by at least half, then by using Lemma 2 (with q = 2) we have 8logn. Therefore, the expeted number of nodes enountered along the route before we ome aross a distane-halving link is O( log n ). Sine the initial distane is d, the maximum number of times the remaining distane ould possibly be halved is log d. Sine d<nit follows that the average path length is O( log2 n ). Lemma 4 Assume > log n log, then the average path length is n O(log2 log ) for the NoN routing on H -Symphony*, with n = 2 m nodes, when 1 < log n. Proof Consider node v that holds a message destined for node t at distane d and let q = log. There are two ases to take into onsideration. Case 1: log d q log n, then the remaining distane an be overed using greedy routing (f. Lemma 3) ino( log2 n log ) hops. Case 2: log d> q log n.letψ denote the event that the urrent node is able to diminish the remaining distane from d to, at most, d/q in (at most) two hops. Let denote the probability that event ψ ours, we will show that 8α log n. (3)

13 In order to prove (3), assume I = (d(1 1/q), d]. Define Q as the set of nodes that are reahable from v in zero or one hop and that are at distane less than or equal to d from v. We have that Q log γ d = log d log γ > q log n log γ = q. Let denote the probability that suh a node has a link in I. Using Lemma 2 we know that 4q log n.letq Q be a maximal subset of nodes whih belongs to different lasses. Easily, Q q α, for some onstant α 1, with probability not less than 1 α 1 q, beause q. Sine nodes in Q belong to different lasses, the ases in whih one of them has a link in the interval I are independent from one node to another. Therefore, the probability that none of these nodes has an outgoing edge reahing the interval I is ( 1 (1 ) Q 8q log n )( ) q = α 8α log n. Hene (3) holds. The expeted number of nodes enountered before a suessful event ψ ours is at most O( log n ). Sine the initial distane is d, the maximum number of times the remaining distane ould possibly be diminished is log q d. Therefore sine d<nit follows that the average path length is O( log2 n log q ). Thus, in O( log2 n log q ) hops, the distane is dereased to d, where log d q log n, and we have redued ase two to ase one. Summing the two, the total number of hops is O( log2 n log q + log2 n log ). Hene one an find the desired value of the average path length by observing that q = log. Using the same argument as Theorem 1, i.e., by observing that the number of nodes that lie on the same interval of size 2 m /n is small, one an easily prove the following theorem. Theorem 2 Assume > log n log, then the average path length is n O(log2 log ) (when 1 < log n) for the NoN routing on H -Symphony* in a ring of size 2 m where the number of live nodes is n 2 m. 5 A Comparison of H -Networks with Traditional NoN Routing Here, we would like to provide a omplete omparison of our results with the traditional NoN networks proposed by Manku et al. in [13]. Our results were obtained by introduing a limited amount of randomization (with a hash funtion) into a deterministi network: effiieny (as in non-uniform networks), programmability and quik bootstrap (as in uniform networks). Corollary 2 Assume 1 < log n and = log n/ log ; then the average path length of the NoN routing algorithm on an H -Network is O(log 2 n/ log ) hops in a ring of size 2 m, where the number of live nodes is n 2 m. Moreover, the number of hops for the ompletion of the join operation is O( log log n) (w.h.p.).

14 Proof The first part of the orollary is proved beause of Theorems 1 and 2 when = log n/ log. In a uniform system, it suffies to use the finger table of v s predeessor (if v is entering the system) whih means that its fingers an be off by at most O(1) distane (on average) with respet to what v needs. In an H -Network, v may need to go bak to at most O(log ) nodes (w.h.p.) before it an find a node of the same lass as v s. This means that its fingers may be off by at most O(log) (w.h.p.) with respet to what v needs, and that O(log ( log )) = O(log log n) hops are needed to perform a searh of eah right finger for v. Therefore, the bootstrap an be performed with O((routing table size) log log n) hops (w.h.p.). The result follows from the size of the routing table of the analyzed network. Our results are obtained without the signifiant maintenane overhead that is inurred by the NoN networks. Without loss of generality, we restrit our disussion to the Chord-like systems, although the same line of arguments are also suited for Symphony* networks. First of all, note that in order to exploit the NoN greedy routing, an R-Chord network uses (log 2 n) words of memory, sine it keeps trak of all the NoN information, whereas the H -Chord network only needs a storage of O(log n). As explained by Manku et al. [13], R-Chord uses the same network update algorithms as Chord. In addition, the neighbors of neighbors information is ommuniated during basi network maintenane. We already ommented on the feasibility of this suggestion in Set. 1. Here, we try to estimate the overhead aused by this ommuniation. The areful reader will have already notied that H -Chord uses the same network maintenane protool as Chord, i.e., without any extra overhead. As a matter of fat, the hash-funtion exploited by the H networks to find the neighbors neighbors merely provides us with identifiers. The real neighbors neighbors are the nodes whih immediately follow these identifiers on the ring. This means that the exat neighbors of neighbors information is not available in an H network, sine the identifier suggested by the hash-funtion might not orrespond to a live node. However, this information suffies to perform effiient lookups: indeed eah identifier provided by the hash-funtion allows us to evaluate the distane to the neighbor in question. Hene, we an use this distane to estimate the searh progress that ould be made in eah step. It follows that by using the estimated distanes in Theorem 1 we get an upper bound on the number of hops needed to reah the target. Below we desribe the additional ommuniation osts of R-Chord. Sine Manku et al. do not provide exat algorithms, we only give lower bounds on the ommuniation osts and try to omment on the expeted overhead in pratie. Theorem 3 The ommuniation omplexity, using R-Chord together with the 1-lookahead protool for maintaining the network struture is (log n) messages for eah node for and eah round if no failures, joins, or leaves our. Proof For eah node v i we need to hek whether its neighbors have hanged and also whether its neighbors neighbors have hanged. If no hanges have taken plae

15 an aknowledgment bit is transmitted from eah of the neighbors. Hene, to maintain the ring during a period in whih there are no failures or leaves we need to send (log n) one bit messages per round. In pratie, sine the probability of neighborhood-updates is high, it is more likely that the protool diretly heks whih neighbors of the neighbor have hanged by using a log n bit word. Hene, the ommuniation omplexity will be (log 2 n) bits per node and round. Theorem 4 Every join, leave or failure inurs a message overhead of (log 2 n) messages and (log 3 n) bits. Proof We will only prove the theorem in ase of a leave. Consider a node n l leaving the system. The leaving node has, on the average, k = (log n) inoming edges. Hene, n l is in the neighborhood list of k nodes, all them n 1,...,n k. Eah node n i has reeived a new representative n i for n l by the stabilizing algorithm of the Chord system. Due to the topologial hange, n i must ommuniate its neighbors to n i.the number of these neighbors is, on the average, (log n). Hene, (log n) messages of size (log 2 n) need to be ommuniated. Furthermore, eah neighbor n i needs to ommuniate its updated neighborhood information to all of its own neighbors. Hene, additional (log 2 n) messages of size (log n) bits eah need to be ommuniated. 6 Monte Carlo Simulation Results In order to assess the effetiveness of our proposed routing algorithm from a pratial point of view as well (in addition to the asymptoti omplexity analysis), we used a Monte Carlo simulation approah to ompare the standard Chord, the H-Chord and the H -Chord routing performane. The performane of H-Chord was found to be equivalent to the original NoN version that was presented in [13]. However, sine this paper proves that for = log n/ log log n the theoretial bound is already reahed, it makes no sense to inrease the number of lasses beyond log n/ log log n. A virtual ring is onstruted by randomly generating the identifiers assoiated with the presribed number of nodes. Then the finger list for eah node is onstruted aording to the hosen distane funtion. Finally, a large number of uniform, independent, random routing requests are issued to the node with the lowest identifier in the ring and routed through the nodes simulating either the greedy (for Chord) or the NoN (for H- and H -Chord) routing protool. The number of hops is ounted for eah route and statistis are olleted. 99% onfidene level intervals are estimated in order to ensure the preision of the simulated results. The simulation experiments are repeated for many randomly generated virtual rings, until the 99% onfidene intervals drop to below 1% of the point estimates. Figure 1 reports some of the simulation results we obtained by omparing the standard Chord, the H-Chord, and the H -Chord routing performane (the latter with only = 2 lasses).

16 Fig. 1 Monte-Carlo Simulation Results. Number of hops for different routing algorithms as a funtion of the size of the ring: average values and 90th perentile Both the average number of hops and 90th perentile number of hops (i.e., the minimal number of hops having a 90% probability of never being exeeded) are reported for eah ase (the H-Chord and H -Chord with = 2 have the same 90th perentile urve). The diagrams learly onfirm the advantage of H-Chord over Chord, not only in terms of average hop ount (ranging from an 11% redution with only 100 nodes, to a 20% redution with 1,000 nodes, up to a 27% redution with 500,000 nodes) but also in terms of 90th perentile. Moreover, the diagrams show that for small/medium size rings, H -Chord with only two lasses behaves almost as well as H-Chord (with a less than 2% differene up to 5,000 nodes on average hop ount, and idential 90th perentile over the whole range of ring size we took into onsideration). As ring size inreases the differene between H-Chord and H -Chord with = 2 lasses beomes more evident (3% with 10,000 nodes, 7% with 100,000 nodes, 10% with 500,000 nodes), but still provides a substantial advantage over Chord without substantial disadvantages (in terms of effiieny of the implementation) ompared with the uniform version of Chord. A omment is needed regarding the hoie of the number of lasses when the number of peers n varies. Indeed, is dynami, but due to its slow growth it very slowly reflets the hanges in n. Therefore, standard tehniques to estimate the number of nodes, suh as onsidering the distribution of a node s suessors [8], an easily be employed to estimate the number of lasses that should be used. Moreover, our strategy does not require eah node in the network to take on the same value of

17 . Aordingly, the value of an be easily hanged on the fly with no harm to the effiieny of the system. 7 Lower Bound In this setion we present a lower bound on the diameter and the average path length of any H -Network based on the degree of the node in the network. By degree we mean the number of jumps from eah node in the network. We first prove the following preliminary lemmas. Lemma 5 Let (n) be any funtion suh that 2 (n) log n. The diameter of any uniform network having degree O((n)log n) is (log (n) n). Proof Consider any uniform network U. Sine the degree of the network is O((n)log n) then we an assume that the degree of the network (that is, the number of jumps from eah node in U) isatmosta (n)log n, where a>2 and n>n for some N>0. Then hoose n>n. For the sake of simpliity, sine n is fixed, we hide the dependeny from n of the various parameters. Hene, denote the degree of the network with n peers by and its diameter by d. Also assume a fixed number of lasses = (n). In order to prove the lemma, we will show that d 1 4 a log n. (4) Assume, on the ontrary that (4) does not hold, that is, d< 4 a 1 log n and onsider the binomial ( +d) d. By using Stirling s formula 2 ( ) + d ( + d)! = < d!d! By denoting α = a and β = 1 (+d) (+d) d d ( ) (+d) (+d) 2 2π( + d) e ( 2π e ) ( 2πd de ) d 4 a log is an inreasing funtion of both and d,wehave ( ) + d (α log n + β log n)(α log n+β log n) < d (α log n) α log n (β log n) β log n = By notiing that ( α+β 1 4logα,wehave [ (α ) + β α ( ) ] α + β β log α β α < ( + d)(+d) d d. (5), we an write α log n, d < β log n. Sine ( (α ) + β α ( ) ) α + β β log n. (6) α β )α ( α+β β )β is an inreasing funtion of β and that β = 1 4 a log ( ) 1 α log log(1 + 4α log α) 4α log α 4logα 2 2πx(x/e) x <x! < 2 2πx(x/e) x.

18 α 4 < 1 + log α + log log α + log 5 4logα < 1. Theory Comput Syst Therefore, ( ) α + β α ( ) α + β β < 2. α From (6)wehave ( ) + d < 2 log n = n. d However, the network is uniform and, by a result in [20], we know that ( + d d β ) n. (7) Informally, the authors in [20] use elementary ombinatoris to show that the number of different pathways (in the sense that the paths end up in different destinations) ( that an be obtained on a uniform network having degree and diameter d is equal to +d ) d. Sine eah node must be able to reah all other nodes, the inequality (7) holds. Hene, there is a ontradition and we an onlude that (4) holds. Lemma 6 The average path length of any uniform network having degree (n) and diameter d(n) = O((n)) is (d(n)). Proof Let U be a uniform network with n peers and let v be a node in U. Denote by r(i) the number of nodes at distane i from v. By using the same argument as [20]it is easy to show that for eah i = 0, 1,...,d(n), ( ) (n) + i 1 r(i). i d(n) Sine the average path length is a(n) = n 1 i=0 i r(i), we have that the smallest possible average path length is obtained when eah r(i) is as large as possible, that is, { ( (n)+i 1 ) i if 1 i<d(n), r(i) = n d(n) 1 i=0 r(i) if i = d(n). In suh a ase, we have that, 3 d(n) 1 n i=0 r(i) = d(n) 1 i=0 ( ) (n) + i 1 i 3 Remember that, given two positive integers a and b, b ( a + i 1 ) ( a + b ) i=0 = whih implies i b bi=0 ( a + i 1 ) = a+b ( a + b 1 ) i b = a+b b 1 ( a + i 1 ) b 1 b i=0. i

19 = (n) + d(n) 1 d(n) 1 d(n) 2 i=0 ( ) (n) + i 1 i (n) + d(n) 1 = [n (r(d(n) + r(d(n) 1))]. (8) d(n) 1 We assume that d(n) = O((n)), that is, d(n) α (n) for some α>0 and a suffiiently large n. This implies that for suffiiently large n, from (8)wehave ( )( ) (n) + d(n) 1 d(n) 1 r(d(n)+ r(d(n) 1) n n d(n) 1 (n) + d(n) 1 Hene, a(n) = = n(n) (n) + d(n) 1 > n α + 1. d(n) i=0 i r(i) d(n) i=d(n) 1 > i r(i) n n (d(n) 1)(r(d(n)) + r(d(n) 1)) > n n (d(n) 1) α+1 > = d(n) 1 n α + 1 = (d(n)). Theorem 5 Both the diameter and the average (shortest) path length of an H - Network with degree O(log n) and 2 lasses are (log n + log n log log n ). Proof We onsider a generi H -Network H with n peers, (n) 2 lasses and degree (n). It is known that the lower bound ( log n log log n ) holds [20]. We need to show the lower bound (log n). Let H be the network obtained by augmenting H in suh a way that eah node in H maintains (n) onnetions. Namely, eah node v has all the (n) onnetions v + j i + λ l (j i+1 j i ) for i = 0,..., 1, orresponding to its membership to lass l, for eah l = 0,..., 1. We denote by (n) and d(n), respetively, the degree and the diameter of H. Obviously (n) = (n). We an observe that: (a) H is a uniform network (all the nodes maintain (n) onnetions of the same length); (b) the diameter and average path length of H is smaller than H (H is in fat obtained by adding some onnetion to H). Hene, in order to bound the diameter of H (with degree (n)), we an apply the lower bound given in Lemma 5 to H (with degree (n)). Moreover, by Lemma 6, we an see that the same bound holds on the average path length of H. 8 Conlusions We propose routing shemes that optimize the average number of hops for lookup requests in Peer-to-Peer systems. Unlike other proposed systems, our sheme does not

20 add any overhead to the system. A reently introdued variation of greedy routing, alled NoN routing, allows us to get optimal average path length with respet to the degree; higher overhead is paid ompared to previous systems due to the additional network maintenane [13]. Our proposal has the advantage of limiting randomization to suh an extent that neighborhood information an be enoded within the hash-value of the node identifier. This enables us to use NoN lookup routing without any additional overhead. Our Theorems 1, 2 and 5 prove that H -Networks make up a flexible and asymptotially optimal family of routing shemes. When = 1 the network is uniform, and by inreasing the value of we an redue the expeted number of hops down to the minimum possible (i.e. O(log 2 n/ log )) that is reahed for = log n/ log. Implementation is easy and effiient at the same time and ombines the advantages of the NoN routing (that an be implemented as a greedy routing applied to a larger set of nodes) with a quik bootstrap of nodes entering the system (whih an easily be obtained thanks to the uniformity among nodes belonging to the same lass). Stohasti simulation results allowed us to show that the partition of nodes in a very small number of lasses provides almost the same performane as H-Chord in terms of the expeted number of hops in ase of small/medium size rings. Even in ase of rings with several hundred thousand nodes, = 2 lasses may provide a substantial advantage ompared with standard Chord, though with very limited overhead in finger table onstrution. Aknowledgements The authors would like to thank the anonymous referees whose very helpful omments allowed to signifiantly improve the presentation of their work. Referenes 1. Aspnes, J., Shah, G.: Skip graphs. In: Pro. of 14th Annual ACM-SIAM Symposium on Disrete Algorithms (SODA 03), pp (Jan. 2003) 2. Chiola, G., Cordaso, G., Gargano, L., Negro, A., Sarano, V.: Overlay networks with lass. In: Pro. of 8th International Symposium on Parallel Arhitetures, Algorithms, and Networks (I-SPAN 05), Las Vegas, Nevada, USA, pp IEEE Computer Soiety (De. 2005) 3. Cordaso, G., Gargano, L., Hammar, M., Sarano, V.: Brief announement: degree-optimal deterministi routing for P2P systems. In: Pro. of 23rd Annual ACM SIGACT-SIGOPS Symposium on Priniples of Distributed Computing (PODC 04) (Brief Announement), St. John s, Newfoundland, Canada, p ACM Press, New York (Jul. 2004) 4. Cordaso, G., Gargano, L., Hammar, M., Sarano, V.: Degree-optimal deterministi routing for P2P systems. In: Pro. of 10th IEEE Symposium on Computers and Communiations (ISCC 05), La Manga del Mar Menor, Cartagena, Spain, pp IEEE Computer Soiety (Jun. 2005) 5. Drushel, P., Rowstron, A.: Pastry: salable, deentralized objet loation, and routing for largesale peer-to-peer systems. In: Pro. of the 18th IFIP/ACM International Conferene on Distributed Systems Platforms (Middleware 01), Heidelberg, Germany, pp Springer, New York (Nov. 2001) 6. Fraigniaud, P., Gauron, P.: D2B: a de Bruijn based ontent-addressable network. Int. J. Theor. Comput. Si. 355(1), (2006) 7. Ganesan, P., Manku, G.S.: Optimal routing in hord. In: Pro. of 15th Annual ACM-SIAM Symposium on Disrete Algorithms (SODA 04), New Orleans, LA, USA, pp Springer, New York (Jan. 2004) 8. Kaashoek, M.F., Karger, D.R.: Koorde: a simple degree-optimal distributed hash table. In: Proeedings of 2nd International Workshop on Peer-to-Peer Systems (IPTPS 03), Berkeley, CA, USA. Leture Notes in Computer Siene, pp (Feb. 2003)

21 9. Karger, D.R., Lehman, E., Leighton, F.T., Panigrahy, R., Levine, M.S., Lewin, D.: Consistent hashing and random trees: distributed ahing protools for relieving hot spots on the world wide web. In: Pro. of the 29th Annual ACM Symposium on Theory of Computing (STOC 97), El Paso, TX, USA, pp ACM Press, New York (May 1997) 10. Kumar, A., Merugu, S., Xu, J.(J.), Yu, X.: Ulysses: a robust, low-diameter, low-lateny peer-to-peer network. In: Pro. of 11th IEEE International Conferene on Network Protools (ICNP 03), pp IEEE, New York (Nov. 2003) 11. Manku, G.S.: The power of lookahead in small-world routing networks. Tehnial Report, Computer Siene Department, Stanford University, CA, USA (Nov. 2003) 12. Manku, G.S., Bawa, M., Raghavan, P.: Symphony: distributed hashing in a small world. In: Pro. of 4th USENIX Symposium on Internet Tehnologies and Systems (USITS 03) (Mar. 2003) 13. Manku, G.S., Naor, M., Wieder, U.: Know thy neighbor s neighbor: the power of lookahead in randomized P2P networks. In: Pro. of 36th Annual ACM Symposium on Theory of Computing (STOC 04), Chiago, IL, USA, pp (Jun. 2004) 14. Motwani, R., Raghavan, P.: Randomized Algorithms. Cambridge University Press, Cambridge (1995) 15. Naor, M., Wieder, U.: Novel arhitetures for P2P appliations: the ontinuous-disrete approah. In: Pro. of 15th ACM Symposium on Parallelism in Algorithms and Arhitetures (SPAA 03), San Diego, CA, USA, pp (Jun. 2003) 16. National Institute of Standards and Tehnology, Seure Hash Standard (NIST). gov/fipspubs/fip180-1.htm 17. Ratnasamy, S., Franis, P., Handley, M., Karp, R., Shenker, S.: A salable ontent-addressable network. In: Pro. of ACM Speial Interest Group on Data Communiation (ACM SIGCOMM 01), San Diego, CA, USA, pp (Aug. 2001) 18. Ratnasamy, S., Shenker, S., Stoia, I.: Routing algorithms for DHTs: some open questions. In: Pro. of 1st International Workshop on Peer-to-Peer Systems (IPTPS 02), Cambridge, CA, USA, pp (Mar. 2002) 19. Stoia, I., Morris, R., Liben-Nowell, D., Karger, D.R., Kaashoek, M.F., Dabek, F., Balakrishnan, H.: Chord: a salable peer-to-peer lookup protool for Internet appliations. IEEE/ACM Trans. Netw. (TON) 11(1), (Feb. 2003) 20. Xu, J., Kumar, A., Yu, X.: On the fundamental tradeoffs between routing table size and network diameter in peer-to-peer networks. IEEE J. Sel. Areas Commun. 22(1), (Jan. 2004) (A preliminary version appeared in the Pro. of IEEE INFOCOM 03) 21. Zhao, B.Y., Kubiatowiz, J.D., Joseph, A.D.: Tapestry: an infrastruture for fault-tolerant wide-area loation and routing. Teh. Report No. UCB/CSD , Computer Siene Division (EECS), University of California at Berkeley, CA, USA (Apr. 2001)