Hierarchical Peer-to-peer Systems

Transcription

1 Hieachical Pee-to-pee ystems L. Gacés-Eice 1, E.W. Biesack 1, P.A. Felbe 1, K.W. Ross 2, and G. Uvoy-Kelle 1 1 Institut EURECOM ophia Antipolis, Fance {gaces ebi felbe uvoy}@euecom.f 2 Polytechnic Univesity Booklyn, NY 11201, UA oss@poly.edu Abstact. tuctued pee-to-pee (P2P) lookup sevices such as Chod, CAN, Pasty and Tapesty oganize pees into a flat ovelay netwok and offe distibuted hash table (DHT) functionality. In these systems, data is associated with keys and each pee is esponsible fo a subset of the keys. We study hieachical DHTs, in which pees ae oganized into goups, and each goup has its autonomous inta-goup ovelay netwok and lookup sevice. The goups themselves ae oganized in a top-level ovelay netwok. To find a pee that is esponsible fo a key, the top-level ovelay fist detemines the goup esponsible fo the key; the esponsible goup then uses its inta-goup ovelay to detemine the specific pee that is esponsible fo the key. Afte poviding a geneal famewok fo hieachical P2P lookup, we conside the specific case of a two-tie hieachy that uses Chod fo the top level. Ou analysis shows that by designating the most eliable pees in the goups as supepees, the hieachical design can significantly educe the expected numbe of hops in Chod. We also popose a scalable design fo managing the goups and the supepees. 1 Intoduction Pee-to-pee (P2P) systems ae gaining inceased populaity, as they make it possible to haness the computing powe and esouces of lage populations of netwoked computes in a cost-effective manne. A cental poblem of P2P system is to assign and locate esouces among pees. This task is achieved by a P2P lookup sevice. eveal impotant poposals have been ecently put foth fo implementing distibuted P2P lookup sevices, including Chod [1], CAN [2], Pasty [3] and Tapesty [4]. In these lookup sevices, each key fo a data item is assigned to the live pee whose node identifie is closest to the key (accoding to some metic). The lookup sevice essentially pefoms the basic function of detemining the pee that is esponsible fo a given key. The lookup sevice is implemented by oganizing the pees in a stuctued ovelay netwok, and outing a message though the ovelay to the esponsible pee. The efficiency of a lookup sevice is geneally measued as a function of the numbe of pee hops needed to oute a message to the esponsible pee, as well as the size of the outing table maintained by each pee. Fo example, Chod equies O(log N) pee hops and O(log N) outing table enties when thee ae N pees in the ovelay. Implementations of the distibuted lookup sevice ae often efeed to as Distibuted Hash Tables (DHTs). Distibuted DHTs ae the cental components of a wide ange of new distibuted applications, including distibuted pesistent file stoage [5, 6], Web caching [7], multicast [8, 9], o computational gids [10]. DHTs geneally povide impovement to an application s esilience to faults and attacks. Chod, CAN, Pasty and Tapesty ae all flat DHT designs without hieachical outing. Each pee is indistinguishable fom anothe in the sense that all pees use the same ules fo detemining the outes fo lookup messages. This appoach is stikingly diffeent fom outing in the Intenet, which uses hieachical outing. pecifically, in the Intenet, outes ae gouped into autonomous systems (Aes). Within an A, all outes un the same inta-a outing potocol (e.g., RIP o OPF). pecial gateway outes in the vaious Aes un an inte-a outing potocol (BGP) that detemines the path among the Aes. Hieachical outing in the Intenet offes seveal benefits ove non-hieachical outing, including scalability and administative autonomy (e.g., at the level of a univesity, a copoate campus, o even the coveage aea of a base station in a mobile netwok). In this pape we exploe hieachical DHTs. Inspied by hieachical outing in the Intenet, we examine two-tie DHTs in which (i) pees ae oganized in disjoint goups, and (ii) lookup messages ae fist outed to the destination goup using an inte-goup ovelay, and then outed to the destination pee using an inta-goup ovelay. We will ague that hieachical DHTs have a numbe of advantages, including: They significantly educe the aveage numbe of pee hops in a lookup, paticulaly when nodes have heteogeneous availabilities. They significantly educe the lookup latency when the pees in the same goup ae topologically close and coopeative caching is used within the goups. They facilitate the lage-scale deployment of a P2P lookup sevice by poviding administative autonomy to paticipating oganizations. In paticula, in the hieachical famewok that we pesent, each paticipating oganization (e.g., institutions and IPs) can choose its own lookup potocol (e.g., Chod, CAN, Pasty, Tapesty). We pesent a geneal famewok fo hieachical DHTs. In the famewok, each goup maintains its own ovelay netwok and uses its own inta-goup lookup sevice. A top-level ovelay is also defined among the goups. Within

2 each goup, a subset of pees ae labeled as supepees. upepees, which ae analogous to gateway outes in hieachical IP netwoks, ae used by the top-level ovelay to oute messages among goups. We conside designs fo which pees in the same goup ae locally close. We descibe a coopeative caching scheme that can significantly educe aveage data tansfe delays. Finally, we also povide a scalable algoithm fo assigning pees to goups, identifying supepees, and maintaining the ovelays. Afte pesenting the geneal famewok, we exploe in detail a paticula instantiation in which Chod is used fo the top-level ovelay. Thus, in this instantiation, Chod is analogous to BGP in Intenet outing, and the inta-goup lookup sevices ae analogous to inta-a outing potocols. Using a novel analytical model, we analyze the expected numbe of pee hops that ae equied fo a lookup in the hieachical Chod instantiation. Ou model explicitly captues inaccuacies in the outing tables due to pee failues. The pape is oganized as follows: We fist discuss elated wok in ection 2. We then pesent the geneal famewok fo hieachical DHT s in ection 3. We discuss the paticula case of a two-tie Chod instantiation in ection 4, and we quantify the impovement of lookup latency due to the hieachical oganization of the pees. 2 Related Wok P2P netwoks can be classified as being eithe unstuctued o stuctued. Chod [1], CAN [2], Pasty [3], Tapesty [4], and P-Gid [11], which use highly stuctued ovelays and use hashing fo tageted data placement, ae examples of stuctued P2P netwoks. These P2P netwoks ae all flat designs (P-Gid is based on a vitual distibuted seach tee, but pee nodes ae located at the leaves level and the tee is used solely fo outing puposes). Gnutella [12] and KaZaA [13], whose ovelays gow oganically and use andom data placement, ae examples of unstuctued P2P netwoks. Ratnasamy et al. [14] exploe using landmak nodes to bin pees into goups. The basic idea is fo each pee to measue its ound-tip time (RTT) to M landmaks, ode the esulting RTTs, and then assign itself to one of M! goups. The authos then apply this binning technique to CAN, to constuct a locality-awae ovelay. In this binning-can scheme, the node id space is patitioned into M! equal-size potions, one potion coesponding to each goup. When a pee wants to join the ovelay, it pings the landmaks to detemine the goup, and hence the potion of the id space, to which it belongs; the pee then gets assigned a node id, unifomly chosen fom that potion of the node id space. This implies that duing a lookup, typically shot topological hops ae taken while the lookup message tavels though a goup; and then longe topological jumps ae taken when the message eaches the bounday of a goup. Ou hieachical DHT schemes bea little esemblance to the scheme in [14]. Although in [14] the pees ae oganized in goups accoding to locality, the lookup algoithm applies only to CAN, does not use supepees, and is not a multi-level hieachical algoithm. Ou appoach has been influenced by KaZaA, an enomously successful unstuctued P2P file shaing sevice. (Today, KaZaA has typically seveal million paticipating pees at the same time.) KaZaA designates the moe available and poweful pees as supenodes. In KaZaA, when a new pee wants to join, it bins itself with the existing supenodes, and establishes an ovelay connection with the supenode that has the shotest RTT. The supenodes ae connected though a toplevel ovelay netwok, using a popietay design. A simila achitectue has been poposed in CAP [15], a two-tie unstuctued P2P netwok that focuses on scalability and stability. Ou design is a blend of the supenode/hieachy/heteogeniety of KaZaA with the lookup sevices in the stuctued DHTs. Bocade [16] poposes to oganize the pees in a twolevel ovelay. All pees fom a single ovelay O L. Geogaphically close pees ae then gouped togethe and get assigned a epesentative called supenode. upenodes ae typically well connected and situated nea netwok access points. The supenodes fom anothe ovelay O H, and each of them must somehow announce which pees ae eachable though him. Bocade is not tuly hieachical since all pees ae pat of O L, which pevents it fom eaping the benefits of hieachically oganized ovelays discussed in section 3.1. Finally, Casto et al. pesent in [17] a topology-awae vesion of Pasty [3]. At each hop Pasty pesents multiple equivalent choices to oute a equest. By choosing the closest (smallest netwok delay) pee at each hop, they ty to minimize netwok delay. Howeve, at each step the possibilities decease exponentially, so delay is mainly detemined by the last hop, usually the longest. Ou appoach is somewhat the opposite, as we popose lage hops to fist get to a goup, and then shote local hops inside the goup. Note that ou achitectue leads to a moe natual caching scheme, as shown late in section Hieachical Famewok We begin by pesenting a geneal famewok fo a hieachical DHT. Although we focus on a two-tie hieachy, the famewok can be extended to a geneal tie hieachy in a staightfowad manne. Let P denote the set of pees paticipating in the system. Each pee has a node id. Each pee also has an IP addess, which may change wheneve it e-connects to the system. The pees ae inteconnected though a netwok of links and switching equipment (outes, bidges, etc.) The pees send lookup quey messages to each othe using a hieachical ovelay netwok, as descibed below. The pees ae oganized into goups. We will discuss how goups ae ceated and managed and how pees ae assigned to goups in ection 3.3. The goups may o may not be such that the pees in the same goup ae topologically close

3 to each othe, depending on the application needs. Each goup has a unique goup id. Let I be the numbe of goups, G i the pees in goup i, and g i the id fo goup i. The goups ae oganized into a top-level ovelay netwok defined by a diected gaph (X, U), whee X = {g 1,..., g I } is the set of all the goups and U is a given set of vitual edges between the nodes (that is, goups) in X. The gaph (X, U) is equied to be connected, that is, between any two nodes g and g in X thee is a diected path fom g to g that uses the edges in U. It is impotant to note that this ovelay netwok defines diected edges among goups and not among specific pees in the goups. Each goup is equied to have one o moe supepees. upepees, as we will discuss below, have special chaacteistics and esponsibilities. Let i G i be the set of supepees in goup i. Ou achitectue allows fo i = G i fo all i = 1,..., I, in which case all pees ae supepees (and distinction between egula pees and supepees becomes supefluous). We efe to achitectues fo which all pees ae supepees as the symmetic design. Ou achitectue also allows i = 1 fo all i = 1,..., I, in which case each goup has exactly one supepee. Let R i = G i i be the set of all egula pees in goup g i. Fo non-symmetic designs ( i G i ), an attempt is made to designate the moe poweful pees as supepees. By moe poweful, we pimaily mean the pees that ae up and connected the most. But as seconday citeia, supepees will be the pees that have high CPU powe and/o netwok connection bandwidth. g 2 g 3 g 1 g Fig. 2. Communication elationships between supepees in neighboing goups. and the cuent IP addess of at least one supepee s j j. With this knowledge, s i can send quey messages to s j. On the othe hand, if p is a egula pee, then p must fist send inta-goup quey messages to a supepee in its goup, which can then fowad the quey message to anothe goup. Regula pees must thus know the name and IP addess of the supepees in thei goup. Figue 1 shows a top-level ovelay netwok. Figue 2 shows possible communication elationships between the coesponding supepees. Figue 3 shows an example fo which thee is one supepee in each goup and the top-level ovelay netwok is a ing. Within each goup thee is also an ovelay netwok that is used fo quey communication among the pees in the goup. Each of the goups opeates autonomously fom the othe goups. Fo example, some goups can use Chod, othes CAN, othes Pasty, and yet othes Tapesty. g g 4 g 1 Fig. 1. Communication elationships between goups in the ovelay netwok. The supepees ae gateways between the goups: they ae used fo inte-goup quey popagation. To this end, we equie that if s i is a supepee in G i, and (g i, g j ) is an edge in the top-level ovelay netwok (X, U), then s i knows the name g Hieachical Lookup evice Let us fist conside a two-level lookup sevice, whee the top level manages pee goups and the bottom level pee nodes. Given a key k, we say that goup g j is esponsible fo k if g j is the closest goup to k among all the goups. Hee closest is defined by the specific top-level lookup sevice (e.g., Chod, CAN, Pasty, o Tapesty). The implementation of the lookup sevice exploits the hieachical achitectue: fist, the lookup sevice finds the goup that is esponsible fo the key; then it finds the pee within that goup that is esponsible fo the key. pecifically, ou two-tie DHT opeates as follows. uppose a pee p i G i wants to detemine the pee that is esponsible fo a key k. 1. Using the ovelay netwok in goup i, pee p i sends a quey message to one of the supepees in i.

4 2. Once the quey eaches a supepee, the top-level lookup sevice outes the quey though (X, U) to the goup G j that is esponsible fo the key k. Duing this phase, the quey only passes though supepees, hopping fom one goup to the next. A supepee in one goup uses its knowledge of the IP addesses of supepees in the subsequent goup along the oute to fowad the quey message fom goup to goup. Eventually, the quey message aives at some supepee s j G j. 3. Using the ovelay netwok in goup j, the the supepee s j outes the quey to the pee p j G j that is esponsible fo the key k. 4. Pee p j sends a esponse back to queying pee p i. Depending on the design, this esponse message can follow the evese path of the path taken by the quey message, o can be sent diectly fom pee p j to pee p i (ignoing the ovelay netwoks). This appoach can be genealized to an abitay numbe of levels. A equest is fist outed though the top-most ovelay netwok to some supepee at the next level below, which in tun outes the equest though its local ovelay netwok, and so on until the equest finally eaches some pee node at the bottom-most level. In the est of this pape, we will focus on the case of a two-level lookup sevice. The hieachical achitectue has seveal impotant advantages when compaed to the flat ovelay netwoks. Exploiting heteogeneous pees: By designating as supepees the pees that ae up the most, the top-level ovelay netwok will be moe stable than the coesponding flat ovelay netwok (fo which thee is no hieachy). This inceased stability enables the lookup sevice to appoach its theoetical optimal lookup hop pefomance (fo example, on aveage 1 2 log N fo Chod, whee N is the numbe of pees in the Chod ovelay). Tanspaency: When a key is moved fom one pee to anothe within a goup, the seach fo the pee holding the key is completely tanspaent to the top-level algoithm. imilaly, if a goup changes its inta-goup lookup algoithm, the change is completely tanspaent to the othe goups and to the top-level lookup algoithm. Also, the failue of a egula pee i G i (o the appeaance of a new pee) will be local to G i ; outing tables in pees outside of G i ae not effected. Faste lookup time: Because the numbe of goups will be typically odes of magnitude smalle than the total numbe of pees, queies tavel ove fewe hops. As we shall soon see, this popety along with the enhanced stability of the top-level ovelay can significantly educe queying delays. Less messages in the wide-aea: If the most stable pees fom the top-level DHT, most ovelay econstuction messages happen inside goups, which gathe pees that ae topologically close. Less hops pe lookup means also less messages exchanged fo the same numbe of equests. Finally, as we shall see shotly, the hieachical oganization of the pee goups is pefectly adapted to content caching, which can futhe educe the numbe of messages that need to get out of the goup. 3.2 Inta-Goup Lookup The famewok we just descibed is quite flexible and accommodates any one of a numbe of ovelay stuctues and lookup sevices at any level in the hieachy. At the inta-goup level, the goups can use diffeent ovelays, which could all be diffeent fom the top-level ovelay stuctue. If a goup has a small numbe of pees (say, in the tens), each pee in the goup could tack all the othe pees in the goup (thei ids and IP addesses); the goup could then use CARP [18] o consistent hashing [19] to assign and locate keys within the goup. The numbe of steps to pefom such an inta-goup lookup in the destination goup is O(1), since each pee uns a local hash algoithm to detemine the pee in the goup esponsible fo a key (g 2 in Figue 3) Goup g 1 hash List of pees in goup s pees CARP Goup g 2 < 10 2 pees 1 s 4 1 Top-level ovelay netwok s s Chod Goup g 3 3 > 10 pees 3 2 CAN Goup g 4 3 > 10 pees Fig. 3. The case of a ing-like ovelay netwok with a single supepee pe goup. Inta-goup lookup is implemented using diffeent lookup sevices (CARP, Chod, CAN). If the goup is a little lage (say, in the hundeds), then the supepees could tack all the pees in the goup. In this case, by fowading a quey to a local supepee, a pee can do a local lookup in O(1) steps (g 1 in Figue 3). Finally, if the goup is lage (say, thousands of pees o moe) then a DHT such as Chod, CAN, Pasty, o Tapesty can be used within the goup (g 3 and g 4 in Figue 3). In this case, the numbe of steps in the local lookup is O(log M), whee M is the numbe of pees in the goup. 3.3 Hieachy and Goup Management In the two-tie hieachical DHT, pees ae oganized in goups and, in tun, goups ae divided in egula pees and supe-

5 pees. We now biefly descibe the potocols used to manage goups. Conside pee p joining the hieachical DHT. We assume that p is able to get the id g of the goup it belongs to (e.g., g may coespond to the name of p s IP o univesity campus). Fist, p contacts and asks anothe pee p aleady pat of the P2P netwok to look up p s goup using key g. Following the fist step of the hieachical lookup, p locates and etuns the IP addess of the supepee(s) of the goup esponsible fo key g. If the goup id of the etuned supepee(s) is pecisely g, then p joins the goup using the egula join mechanisms of the undelying inta-goup lookup sevice; additionally, p notifies the supepee(s) of its CPU and bandwidth esouces. If the goup id is not g, then a new goup is ceated with id g and p as only (supe)pee. In a netwok with m supepees pe goup, the fist m pees to join a goup g become the supepees of that goup. Howeve, as peviously mentioned, supepees ae expected to be the most stable and poweful goup nodes. Theefoe, we let supepees monito the pees that join a goup and pesent good chaacteistics. upepees keep an odeed list of the supepee candidates: the longe a pee emains connected and the highe its esouces, the bette a supepee candidate it becomes. This list is sent peiodically to the egula pees of the goup. When a supepee fails o disconnects, the fist egula pee in the list becomes supepee and joins the top-level ovelay. It infoms all pees in its goup of its aival, as well as the supepees of the neighboing goups. We ae thus able to povide stability to the top-level ovelay using multiple supepees, pomote the most stable pees as supepees, and apidly epai the infequent failues o depatues of supepees. 3.4 Content Caching uch hieachical setups can be natually extended to implement coopeative caching. Conside the following modification to the lookup algoithm. When a pee p G i wants to obtain the file associated with some key k, it fist uses goup G i s inta-lookup algoithm to find the pee p G i that would be esponsible fo k if G i wee the entie set of pees. If p has a local copy of the file associated with k, it etuns the file to p; othewise, p obtains the file (using the hieachical DHT), caches a copy, and fowads the file to p. In this manne, files ae cached in the goups whee they have been peviously equested (until they ae eplaced by a cache eplacement algoithm, such as LRU). If a significant faction of equests ae fo popula files, then a significant faction of file tansfes will be inta-goup tansfes. Thus, the hieachical design can be used to ceate a P2P content distibution netwok. As a function of file sizes, cache sizes, equest distibutions and access bandwidth, one can use standad analytical techniques [20] to quantify the eduction in aveage file tansfe time and load on access links. 4 Chod Instantiation Fo the emainde of this pape we focus on a specific top-level DHT, namely, Chod. In this context, we show that the two-tie design can significantly educe lookup latency and file tansfe latency. 4.1 Oveview of Chod In Chod, each pee and each key has a m-bit id. Ids ae odeed on a cicle modulo 2 m. Key k is assigned to the fist pee whose identifie is equal to o follows k in the identifie space. This pee is called the successo of key k. In many P2P applications, once a pee p detemines the pee p that is esponsible fo a key, p then asks p fo the file associated with the key. This file might be a music file (MP3), a video, a softwae package, a document, etc. If the path fom p to p taveses a congested o low-speed link, the file tansfe delay will be long. In this section, we descibe how hieachical DHTs can be used to ceate goups of coopeative caches, which can significantly educe file tansfe delays. In many hieachical DHT setups, we expect the pees in a same goup to be topologically close and to be inteconnected by high-speed links. Fo example, a goup might consist of pees in a copoate o univesity campus, with campus netwoks using 100 Mbps Ethenet. In such an envionment, it is typically faste to etieve files fom othe pees in the local goup, athe than to etieve the file fom the global Intenet. Also, by fequently confining file tansfes to inta-goup tansfes, we educe taffic loads on the access links between the goups and highe-tie IPs. n successo( n+2 i-2 ) successo( n+2 i-1 ) finge i-1 finge i i successo(n+2 ) finge i+1 Fig. 4. Finges of a pee in a Chod ing.

6 Each pee tacks its successo and pedecesso pee in the ing. In addition, each pee tacks m othe pees, called finges; specifically, a pee with id p tacks all the successos of the ids p + 2 j 1 fo each j = 1,..., m (note that p s fist finge is in fact its successo). The successo, pedecesso, and finges make up the Chod outing table (see Figue 4). Duing a lookup, a pee fowads a quey to the finge with the lagest id that pecedes the key value. The pocess is epeated fom pee to pee until the pee peceding the key is eached, which is the closest pee to the key. When thee ae P pees, the numbe of hops needed to each the destination is O(log P ) [1]. 4.2 Inte-Goup Chod Ring In the top-level ovelay netwok, each node is actually a goup of pees. This implies that the top-level lookup system must manage an ovelay of goups, each of which is epesented by a set of supepees that may fail independently fom the goup they belong to. Chod equies some adaptations fo being used in the top-level ovelay netwok and managing goups instead of nodes. We will efe to the modified vesion of Chod as top-level Chod. the new supepees eagely update the vectos of the pedecesso and successo goups. This guaantees that each goup has an up-to-date view of its neighboing goups and that the ing is neve disconnected. Finges impove the lookup pefomance, but ae not necessay fo successfully outing equests (a equest can always be outed by following the successo pointes until it eaches its destination. Howeve, this outing leads to O(P ) hops to each the destination). Theefoe, when the supepees i of a goup g i change, we do not immediately update the finges that point to g i. Instead, we lazily update the finge tables when we detect that they contain invalid efeences (similaly to the lazy update of the finges in egula Chod ings [1]). It is woth noticing that the egula Chod must pefom a lookup opeation to find a lost finge. Due to the edundancy that ou multiple supepee appoach povides, we can choose without delay anothe supepee in the finge vecto fo the same goup. To oute a equest to a goup pointed to by a vecto (successo o finge), we choose a andom IP addess fom the vecto and fowad the equest to that supepee. When goups have moe than one supepee, this load balancing stategy avoids oveloading specific nodes o communication links, and inceases the scalability of the system. g Goup Routing Table ped. finges succ. Fig. 5. Goup outing table in the top-level Chod ovelay netwok. Each node in top-level Chod has a pedecesso and successo vecto (instead of a pointe). The vectos hold the IP addesses of the supepees of the pedecesso and successo goup in the ing, espectively. Each finge is also a vecto, which holds the IP addesses of the supepees of anothe goup an the ing. The outing table of a top-level Chod goup is shown in Figue 5. The population of goups in the top-level ovelay netwok is expected to be athe stable. Howeve, individual supepees may fail and disconnect the top-level Chod ing. When the identity of the supepees i of a goup g i changes, 4.3 Lookup Latency with Hieachical Chod In this section, we quantify the impovement of lookup latency due to the hieachical oganization of the pees. To this end, we compae the lookup pefomance of two DHTs. The fist DHT is the flat Chod DHT. The second DHT is a two-tie hieachy in which Chod is used fo the top level ovelay, and abitay DHTs ae used fo the bottom level ovelays. Fo each bottom level goup, we only suppose that the pees in the goup ae topologically close so that inta-goup lookup delays ae negligible. We shall show that the hieachical DHT can significantly educe the lookup latency, owing to the heteogeneous availabilities of the pees and the eduction of nodes in the Chod ing. In ode to make a fai compaison, we suppose that both the flat and hieachical DHTs have the same numbe of pees, denoted by P. Let I be the numbe of goups in the hieachical design. Because pees ae joining and leaving the ing, the finge enties in the pees will not all be accuate. This is moe than pobable, since finges ae updated lazily. To captue the heteogeneity of the pees, we suppose that thee ae two categoies of pees: table pees, fo which each pee is down with pobability p s. Instable pees, fo which each pee is down with pobability p, with p >> p s. We suppose that the vast majoity of the pees ae instable pees. In eal P2P netwoks, like Gnutella, most pees just

7 emain connected the time of getting data fom othe pees. This fact has aleady been obseved in [21]. Fo the hieachical oganization, we select supepees fom the set of stable pees, and we suppose thee is at least one stable pee in each goup. Because thee ae many moe instable pees than stable pees, the pobability that a andomly chosen Chod node is down in the flat DHT is appoximately p. In the hieachical system, because all the supepees ae stable pees, the pobability that a Chod node is down is p s. To compae the lookup delay fo flat and hieachical DHTs, we thus only need to conside a Chod ing with N pees, with each pee having the same pobability p of being down. The flat DHT coesponds to (N, p) = (P, p ) and the hieachical DHT coesponds to (N, p) = (I, p s ). We now poceed to analyze the lookup of the Chod ing (N, p). To simplify the analysis, we assume the N pees ae equally spaced on the ing, i.e., the distance between two adjacent pees is 2m N. Ou model implies that when a pee attempts to contact a pee in its finge table, the pee in the finge table will be down with pobability p, except if this is the successo pee, fo which we suppose that the finge enty is always coect (i.e., the successo is up o the pee is able to find the new successo. This assues the coect outing of lookup queies). Given an initial pee and a andomly geneated key, let the andom vaiable H denote the numbe of Chod hops needed to each the taget pee, that is, to each the pee esponsible fo the key. Let T be the andom vaiable that is the clockwise distance in numbe of pees fom the initial pee to the taget pee. We want to compute the expectation E[H]. Clealy E[H] = N 1 n=0 N 1 = 1 N P (T = n)e[h T = n] n=0 E[H T = n] (1) Fom (1), it suffices to calculate E[H T = n] to compute E[H]. Let h(n) = E[H T = n]. Note that h(0) = 0 and h(1) = 1. Let j n = max{j : 2 j 2m n N } The value j n epesents the numbe of finge enties that pecede the taget pee, excluding finge 0, the successo. Fo each of the finge enties, the pobability that the coesponding pee is up is p (the successo is assumed always up, so Chod outing is assued to eventually succeed). tating at the initial pee, when hopping to the next pee, the quey will advance 2jn 2 m /N pees if the j nth finge pee is up; if this pee is down but the (j n 1)th finge pee is up, the quey will advance 2jn 1 2 m /N ; and so on. Let q n(i) denote the pobability that the ith finge is used. We theefoe have h(n) = 1 + j n i=0 ( 2 i ) q n (i)h n 2 m /N The pobability that the ith finge is used is given by q n (i) = p jn i (1 p) i = 1,..., j n and by q n (0) = p jn. Combining the above two equations we obtain h(n) = 1 + p jn h(n 1) + (1 p) j n i=1 ( 2 p jn i i ) h n 2 m /N Using this ecusion, we can calculate all the h(n) s beginning at h(0) = 0. We then use (1) to obtain the expected numbe of hops, E[H]. Mean numbe of hops pe look-up nodes 2 20 nodes 2 16 nodes 2 10 nodes Pobability of node failue Fig. 6. Mean numbe of hops pe lookup in Chod. In Figue 6, we plot the expected numbe of hops in a lookup as a function of the availability of the pees in a Chod system, fo diffeent values of N. We can obseve an exponential incease in the numbe of steps only fo p > 0.7. With smalle values of p, although pees in the ing ae not totally eliable, we ae still able to advance quite quickly on the ing. Indeed, while the best finge fo a taget pee is unavailable with pobability p, the pobability of the second best choice to be also down is p 2, which is fa smalle than p. Despite the good scalability of the Chod lookup algoithm in a flat configuation, the hieachical achitectue can yet significantly decease the lookup delay. Table 1 gives the expected numbe of hops fo the flat and hieachical schemes, fo diffeent values of P, I, and p (p s = 0). We suppose in all cases goups of P I pees. As an example, fo

8 Flat Hieachical p = 0.5 p = 0.8 p s = 0 P = 2 16 I = P = 2 20 I = P = 2 24 I = P = 2 24 I = Table 1. Compaing the flat and hieachical netwoks. P = 2 20 = 1, 048, 676 pees, and p = 0.8, we obseve that about 59 hops ae needed on aveage. Howeve, if we oganize these pees in I = 2 16 = goups of 16 pees each, the numbe of hops is educed to 8. ince the numbe of steps is diectly elated to the lookup delay, we can conclude that the aveage lookup delay is divided by a facto of 7 in this case. 5 Conclusion Hieachical oganizations in geneal impove oveall system scalability. An example of a hieachical oganization is the Intenet outing achitectue with inta-domain outing potocols such as RIP o OPF, and with BGP as inte-domain outing potocol. In this pape, we have poposed a geneic famewok fo the hieachical oganization of pee-to-pee ovelay netwok, and we have demonstated the vaious advantages it offes ove a flat oganization. A hieachical design offes highe stability by using moe eliable pees (supepees) at the top levels. It can use vaious inte- and inta-goup lookup algoithms simultaneously, and teats join/leave events and key migation as local events that affect a single goup. By gatheing pees into goups based on topological poximity, a hieachical oganization also geneates less messages in the wide aea and can significantly impove the lookup pefomance. Finally, as shown in ection 3, ou achitectue is ideally suited fo caching popula content in local goups. By fist queying the esponsible pee within ones own goup, popula objects ae dynamically pulled into the vaious goups. This local-goup caching can damatically educe download delays in pee-to-pee file-shaing systems. We have pesented an instantiation of ou hieachical pee oganization using Chod at the top level. The Chod lookup algoithm equied only mino adaptations to deal with goups instead of individual pees. We have analyzed and quantified the impovement in lookup pefomance of hieachical Chod. When all pees ae available, an hieachical oganization educes the length of the lookup path by a facto of log P log I, whee I is the numbe of goups and P is the total numbe of pees. When each pee is down with a cetain pobability, a hieachical oganization educes the length of the lookup path damatically fo the case whee the failue pobability of supepees is smalle than the failue pobability of egula pees. Refeences 1. I. toica, R. Mois, D. Kage, M. Kaashoek, and H. Balakishnan, Chod: A scalable pee-to-pee lookup sevice fo intenet applications, in Poceedings of IGCOMM 2001, August Ratnasamy, M. Handley, R. Kap, and. henke, A scalable content-addessable netwok, in Poceedings of IGCOMM 2001, Aug A. Rowston and P. Duschel, Pasty: calable, distibuted object location and outing fo lage-scale pee-to-pee systems, in IFIP/ACM Intenational Confeence on Distibuted ystems Platfoms (Middlewae), (Heidelbeg, Gemany), pp , Novembe B. Y. Zhao, J. Kubiatowicz, and A. D. Joseph, Tapesty: An infastuctue fo fault-toleant wide-aea location and outing, Tech. Rep. UCB/CD , Compute cience Division, Univesity of Califonia, Bekeley, Ap A. Rowston and P. Duschel, toage management and caching in past, a lage-scale, pesistent pee-to-pee stoage utility, in Poceedings of OP 2001, Oct F. Dabek, M. F. Kaashoek, D. Kage, R. Mois, and I. toica, Wide-aea coopeative stoage with cfs, in Poceedings of OP 2001, Oct Iye, A. Rowston, and P. Duschel, quiel: A decentalized, pee-to-pee web cache, in Poceedings of PDOC 2002, July A. Rowston, A.-M. Kemaec, P. Duschel, and M. Casto, cibe: The design of a lage scale event notification infastuctue, in Poceedings of NGC 2001, Nov Zhuang, I. toica, D. Adkins,. Ratnasamy,. henke, and. uana, Intenet indiection infastuctue, in Poceedings of IPTP 02, (Cambidge, MA), Ma A. Andzejak and Z. Xu, calable, efficient ange queies fo gid infomation sevices, in Poceedings of the econd IEEE Intenational Confeence on Pee-to-Pee Computing, Linkping Univesity, weden, eptembe K. Abee, P-gid: A self-oganizing access stuctue fo p2p infomation systems, in Poceedings of the ixth Intenational Confeence on Coopeative Infomation ystems (CoopI 2001), (Tento, Italy), Gnutella Kazaa Ratnasamy, M. Handley, R. Kap, and. henke, Topologically-awae ovelay constuction and seve selection, in Poceedings of Infocom 02, (New Yok City, NY), B. Kishnamuthy, J. Wang, and Y. Xie, Ealy measuements of a cluste-based achitectue fo p2p systems, in ACM IG- COMM Intenet Measuement Wokshop, (an Fancisco, CA), Novembe B. Y. Zhao, Y. Duan, L. Huang, A. D. Joseph, and J. D. Kubiatowicz, Bocade: Landmak outing on ovelay netwoks, in In Poceedings of IPTP 02, (Cambidge, MA), Mach M. Casto, P. Duschel, Y. C. Hu, and A. Rowston, Exploiting netwok poximity in pee-to-pee ovelay netwoks, Tech. Rep. MR-TR , Micosoft Reseach, One Micosoft Way, Redmond, WA 98052, K. W. Ross, Hash-outing fo collections of shaed web caches, IEEE Netwok Magazine, vol. 11, 7, pp , Nov-Dec 1997.

9 19. D. Kage, A. heman, A. Bekhemie, B. Bogstad, R. Dhanidina, K. Iwamoto, B. Kim, L. Matkins, and Y. Yeushalmi, Web caching with consistent hashing, in Eighth Intenational Wold Wide Web Confeence, May J. F. Kuose and K. W. Ross, Compute Netwoks: A Top-Down Appoach Featuing the Intenet, 2nd edition. Addison Wesley, E. Ada and B. A. Hubeman, Fee iding on gnutella, Fist Monday, vol. 5, Oct