Topology-aware routing in structured peer-to-peer overlay networks

Transcription

1 Topologywre routing in structured peertopeer overly networks Miguel Cstro Peter Druschel Y. Chrlie Hu Antony Rowstron Microsoft Reserch, 7 J J Thomson Close, Cmbridge, C3 F, UK. Rice University, Min Street, MS32, Houston, TX 77, USA. Purdue University, 28 EE uilding, West Lfyette, I 4797, USA. Technicl Report MSRTR2282 Structured peertopeer (p2p) overly networks like CA, Chord, Pstry nd Tpestry offer novel pltform for vriety of sclble nd decentrlized distributed pplictions. They provide efficient nd fulttolernt routing, object loction nd lod blncing within selforgnizing overly network. One importnt spect of these systems is how they eploit network proimity in the underlying Internet. We present study of topologywre routing pproches in p2p overlys, identify proimity neigbor selection s the most promising technique, nd present n improved design in Pstry. Results obtined vi nlysis nd vi simultion of two lrgescle topology models indicte tht it is possible to efficiently eploit network proimity in selforgnizing p2p substrtes. Proimity neighbor selection incurs only modest dditionl overhed for orgnizing nd mintining the overly network. The resulting loclity properties improve ppliction performnce nd reduce network usge in the Internet substntilly. Finlly, we show tht the impct of proimity neighbor selection on the lod blncing in the p2p overly is miniml. Microsoft Reserch Microsoft Corportion One Microsoft Wy Redmond, WA 982

2 Introduction Severl recent systems (e.g., CA, Chord, Pstry nd Tpestry [7, 3,,, ]) provide selforgnizing substrte for lrgescle peertopeer pplictions. Among other uses, these systems cn implement sclble, fulttolernt distributed hsh tble, in which ny item cn be locted within bounded number of routing hops, using smll pernode routing tble. While there re lgorithmic similrities mong ech of these systems, one importnt distinction lies in the pproch they tke to considering nd eploiting proimity in the underlying Internet. Chord in its originl design, for instnce, does not consider network proimity t ll. As result, its protocol for mintining the overly network is very lightweight, but messges my trvel rbitrrily long distnces in the Internet in ech routing hop. In version of CA, ech node mesures its network dely to set of lndmrk nodes, in n effort to determine its reltive position in the Internet nd to construct n Internet topologywre overly. Tpestry nd Pstry construct topologywre overly by choosing nerby nodes for inclusion in their routing tbles. Erly results for the resulting loclity properties re promising. However, these results come t the epense of significnly more epensive overly mintennce protocol, reltive to Chord. Also, proimity bsed routing my compromise the lod blnce in the p2p overly network. Moreover, it remins uncler to wht etent the loclity properties hold in the ctul Internet, with its comple, dynmic, nd nonuniform topology. As result, the cost nd effectiveness of proimity bsed routing in these p2p overlys remin uncler. This pper presents study of proimity bsed routing in structured p2p overly networks, nd presents results of n nlysis nd of simultions bsed on two lrgescle Internet topology models. The specific contributions of this pper include comprison of pproches to proimity bsed routing in structured p2p overly networks, which identifies proimity neighbor selection in prefibsed protocols like Tpestry nd Pstry s the most promising technique; improved node join nd overly mintennce protocols for proimity neighbor selection in Pstry, which significncly reduce the overhed of creting nd mintining topologywre overly; study of the costs nd benefits of proimity neighbor selection vi nlysis nd simultion bsed on two lrgescle Internet topology models; study of the impct of proimity neighbor selection on the lod blncing in the p2p overly bsed on simultions on lrgescle topology model. Compred to the originl Pstry pper [], this work dds comprison with other proposed pproches to topologywre routing, new node join nd overly mintennce protocols tht drmticlly reduce the cost of overly construction nd mintennce, new protocol to locte nerby contct node, results of forml nlysis of Pstry s routing properties nd etensive simultion results on two different network topology models. The rest of this pper is orgnized s follows. Previous work on structured p2p overlys is discussed in Section 2. Approches to topologywre routing in p2p overlys re presented in Section 3. Section 4 presents Pstry s implementtion of proimity neighbor selection, including new efficient protocols for node join nd overly mintennce. An nlysis of Pstry s loclity properties follow in Section. Section presents eperimentl results, nd we conclude in Section 7. 2 ckground nd prior work In this section, we present some bckground on structured p2p overly protocols like CA, Chord, Tpestry nd Pstry. (We do not consider unstructured p2p overlys like Gnutell nd Freenet in this pper [, 2]). Spce limittions prevent us from detiled discussion of ech protocol. Insted, we give more detiled description of Pstry, s n emple of structured p2p overly network, nd then point out relevnt differences with the other protocols. 2. Pstry Pstry is sclble, fult resilient, nd selforgnizing peertopeer substrte. Ech Pstry node hs unique, uniform rndomly ssigned nodeid in circulr 28bit identifier spce. Given 28bit key, Pstry routes n ssocited messge towrds the live node whose nodeid is numericlly closest to the key. Moreover, ech Pstry node keeps trck of its neighboring nodes in the nmespce nd notifies pplictions of chnges in the set.

3 b b b c c c d d d e e e f f f O 2 28 d4c d47f d47c4 d42b d423f b c d e f fc Route(d4c) d3d3 Figure : Routing tble of Pstry node with nodeid,. Digits re in bse, represents n rbitrry suffi. ode stte: For the purpose of routing, nodeids nd keys re thought of s sequence of digits in bse ( is configurtion prmeter with typicl vlue 4). A node s routing tble is orgnized into rows nd columns. The entries in row of the routing tble contin the IP ddresses of nodes whose nodeids shre the first digits with the present node s nodeid; the th nodeid digit of the node in column of row equls. The column in row tht corresponds to the vlue of the s digits of the locl node s nodeid remins empty. Figure depicts smple routing tble. A routing tble entry is left empty if no node with the pproprite nodeid prefi is known. The uniform rndom distribution of nodeids ensures n even popultion of the nodeid spce; thus, on verge only! #"%$ '&)(+* levels re populted in the routing tble. Ech node lso mintins lef set. The lef set is the set of nodes with nodeids tht re numericlly closest to the present node s nodeid, with, lrger nd, smller nodeids thn the current node s id. A typicl vlue for is pproimtely #. #"%$/ (+*. The lef set ensures relible messge delivery nd is used to store replics of ppliction objects. Messge routing: At ech routing step, node seeks to forwrd the messge to node whose nodeid shres with the key prefi tht is t lest one digit (or bits) longer thn the current node s shred prefi. If no such node cn be found in the routing tble, the messge is forwrded to node whose nodeid shres prefi with the key s long s the current node, but is numericlly closer to the key thn the present node s id. Severl such nodes cn normlly be found in the routing tble; moreover, such Figure 2: Routing messge from node with key. The dots depict live nodes in Pstry s circulr nmespce. node is gurnteed to eist in the lef set unless the messge hs lredy rrived t the node with numericlly closest nodeid or its immedite neighbor. And, unless ll, nodes in one hlf of the lef set hve filed simultneously, t lest one of those nodes must be live. The Pstry routing procedure is shown in Figure 3. Figure 2 shows the pth of n emple messge. Anlysis shows tht the epected number of forwrding hops is slightly below! #"%$ '&)(+*, with distribution tht is tight round the men. Moreover, simultion shows tht the routing is highly resilient to node filures. 2.2 CA, Chord, Tpestry et, we briefly describe CA, Chord nd Tpestry, with n emphsis on the differences of these protocols when compred to Pstry. Tpestry is very similr to Pstry but differs in its pproch to mpping keys to nodes in the sprsely populted id spce, nd in how it mnges repliction. In Tpestry, there is no lef set nd neighboring nodes in the nmespce re not wre of ech other. When node s routing tble does not hve n entry for node tht mtches key s th digit, the messge is forwrded to the node with the net higher vlue in the th digit, modulo, found in the routing tble. This procedure, clled surrogte routing, mps keys to unique live node if the node routing tbles re consistent. For fult tolernce, Tpestry inserts replics of dt items using different keys. Like Pstry, Chord uses circulr id spce. Unlike Pstry, Chord forwrds messges only in clockwise di 2

4 G C $ $ () if ( )! (2) // is within rnge of locl lef set (mod ) (3) forwrd to #", s.th. &%'("$ is miniml; (4) else () // use the routing tble () Let )+*, ). / ; (7) if (324 eists nd is live) (8) forwrd to 24 ; (9) else () // rre cse () forwrd to #798:, s.th. (2) ).; #<), (3) /&%'?$ Figure 3: Pstry routing procedure, eecuted when messge with key rrives t node with nodeid. :CD is the entry in the routing tble t column E nd row. F is the ith closest nodeid in the lef set F, where negtive/positive inde indictes counterclockwise/clockwise from the locl node in the id spce, respectively. FHG D I nd F D I re the nodes t the edges of the locl lef set. O D represents the s digit in the key. JLKM P is the length of the prefi shred mong nd, in digits. rection in the circulr id spce. Insted of the prefibsed routing tble in Pstry, Chord nodes mintin finger tble, consisting of up to pointers to other live nodes. The E th entry in the finger tble of node refers to the live node with the smllest nodeid clockwise from LC. The first entry points to s successor, nd subsequent entries refer to nodes t repetedly doubling distnces from. Ech node lso mintins pointers to its predecessor nd to its successors in the id spce (the successor list). Similr to Pstry s lef set, this successor list is used to replicte objects for fult tolernce. The epected number of routing hops in Chord is #"%$ (. CA routes messges in dimensionl spce, where ech node mintins routing tble with Q9M#?P entries nd ny node cn be reched in Q9M# ( ISR P routing hops. The entries in node s routing tble refer to its neighbors in the dimensionl spce. Unlike Pstry, Tpestry nd Chord, CA s routing tble does not grow with the network size, but the number of routing hops grows fster thn #"%$ ( in this cse. 3 Topologywre routing In this section, we describe nd compre three pproches to topologywre routing in structured overly networks tht hve been proposed, nmely topologybsed nodeid ssignment, proimity routing, nd proimity neighbor selection [9]. Proimity routing: With proimity routing, the overly is constructed without regrd for the physicl network topology. The technique eploits the fct tht when messge is routed, there re potentilly severl possible net hop neighbors tht re closer to the messge s key in the id spce. The ide is to select, mong the possible net hops, the one tht is closest in the physicl network or one tht represents good compromise between progress in the id spce nd proimity. With T lterntive hops in ech step, the pproch cn reduce the epected dely in ech hop from the verge dely between two nodes to the epected dely of the nerest mong T nodes with rndom loctions in the network. The min limittion is tht the benefits depend on the mgnitude of T ; with prcticl protocols, T is smll. Moreover, depending on the overly protocol, greedily choosing the closest hop my led to n increse in the totl number of hops tken. While proimity routing cn yield significnt improvements over system with no topologywre routing, its performnce flls short of wht cn be chieved with the following two pproches. The technique hs been used in CA nd Chord [7, 4]. Topologybsed nodeid ssignment: Topologybsed nodeid ssignment ttempts to mp the overly s logicl id spce onto the physicl network such tht neighbouring nodes in the id spce re close in the physicl network. The technique hs been successfully used in version of CA, nd hs chieved dely stretch results of two or lower [7, 8]. However, the pproch hs severl drwbcks. First, it destroys the uniform popultion of the id spce, cusing lod blncing problems in the overly. Second, the pproch does not work well in overlys tht use onedimensionl id spce (Chord, Tpestry, Pstry), becuse the mpping is overly constrined. Lstly, neighboring nodes in the id spce re more likely to suffer correlted filures, which cn hve implictions for robustness nd security in protocols like Chord nd Pstry, which replicte objects on neighbors in the id spce. Proimity neighbour selection: Like the previous technique, proimity neighbor selection constructs 3

5 topologywre overly. However, insted of bising the nodeid ssignment, the ide is to choose routing tble entries to refer to the topologiclly nerest mong ll nodes with nodeid in the desired portion of the id spce. The success of this technique depends on the degree of freedom n overly protocol hs in choosing routing tble entries without ffecting the epected number of routing hops. In prefibsed protocols like Tpestry nd Pstry, the upper levels of the routing tble llow gret freedom in this choice, with lower levels leving eponentilly less choice. As result, the epected dely of the first hop is very low, it increses eponentilly with ech hop, nd the dely of the finl hop domintes. As one cn show, this leds to low dely stretch nd other useful properties. A limittion of this technique is tht it does not work for overly protocols like CA nd Chord, which require tht routing tble entries refer to specific points in the id spce. Discussion: Proimity routing is the most lightweight technique, since it does not construct topologywre overly. ut, its performnce is limited since it cn only reduce the epected perhop dely to the epected dely of the nerest mong smll number T of nodes with rndom loctions in the network. With topologywre nodeid ssignment, the epected perhop dely cn be s low s the verge dely mong neighboring overly nodes in the network. However, the technique suffers from lod imblnce nd requires highdimensionl id spce to be effective. Proimityneighbor selection cn be viewed s compromise tht preserves the lod blnce nd robustness fforded by rndom nodeid ssignment, but still chieves smll constnt dely stretch. In the following sections, we show tht proimity neighbor selection cn be implemented in Pstry nd Tpestry with low overhed, tht it chieves comprble dely stretch to topologybsed nodeid ssignment without scrificing lod blncing or robustness, nd tht is hs dditionl route convergence properties tht fcilitte efficient cching nd multicsting in the overly. Moreover, we confirm these results vi simultions on two lrgescle Internet topology models. 4 Proimity neighbor selection: Pstry This section shows how proimity bsed neighbor selection is used in Pstry. We describe new node join nd overly mintennce protocols tht significntly reduce the overhed compred to the originl protocols described in []. Moreover, we present new protocol tht llows nodes tht wish to join the overly to locte n pproprite contct node. It is ssumed tht ech Pstry node cn mesure its distnce, in terms of sclr proimity metric, to ny node with known IP ddress. The choice of proimity metric depends on the desired qulities of the resulting overly (e.g., low dely, high bndwidth, low network utiliztion). In prctice, verge roundtrip time hs proven to be good metric. Pstry uses proimity neighbor selection s introduced in the previous section. Selecting routing tble entries to refer to the precisely nerest node with n pproprite nodeid is epensive in lrge system, becuse it requires Q9M ( P communiction. Therefore, Pstry uses heuristics tht require only Q9M# #"%$ '& ( P communiction but only ensure tht routing tble entries re close but not necessrily the closest. More precisely, Pstry ensures the following invrint for ech node s routing tble: Proimity invrint: Ech entry in node s routing tble refers to node tht is ner, ccording to the proimity metric, mong ll live Pstry nodes with the pproprite nodeid prefi. In Section 4., we show how Pstry s node joining protocol mintins the proimity invrint. et, we consider the effect of the proimity invrint on Pstry s routing. Observe tht s result of the proimity invrint, messge is normlly forwrded in ech routing step to nerby node, ccording to the proimity metric, mong ll nodes whose nodeid shres longer prefi with the key. Moreover, the epected distnce trveled in ech consecutive routing step increses eponentilly, becuse the density of nodes decreses eponentilly with the length of the prefi mtch. From this property, one cn derive three distinct properties of Pstry with respect to network loclity: Totl distnce trveled (dely stretch): The epected distnce of the lst routing step tends to dominte the totl distnce trveled by messge. As result, the verge totl distnce trveled by messge eceeds the distnce between source nd destintion node only by smll constnt vlue. Locl route convergence: The pths of two Pstry messges sent from nerby nodes with identicl keys tend to converge t node ner the source nodes, in the proimity spce. To see this, observe tht in ech consecutive 4

6 routing step, the messges trvel eponentilly lrger distnces towrds n eponentilly shrinking set of nodes. Thus, the probbility of route convergence increses in ech step, even in the cse where erlier (smller) routing steps hve moved the messges frther prt. This result hs significnce for cching pplictions lyered on Pstry. Populr objects requested by nerby node nd cched by ll nodes long the route re likely to be found when nother nerby node requests the object. Also, this property is eploited in Scribe [2] to chieve low link stress in n ppliction level multicst system. Locting the nerest replic: If replics of n object re stored on T nodes with djcent nodeids, Pstry messges requesting the object hve tendency to first rech node ner the client node. To see this, observe tht Pstry messges initilly tke smll steps in the proimity spce, but lrge steps in the nodeid spce. Applictions cn eploit this property to mke sure tht client requests for n object tend to be hndled by replic tht is ner the client. Eploiting this property is pplictionspecific, nd is discussed in []. An nlysis of these properties follows in Section. Simultion nd mesurement results tht confirm nd quntify these properties follow in Section. 4. Mintining the overly et, we present the new protocols for node join, node filure nd routing tble mintennce in Pstry nd show how these protocols mintin the proimity invrint. The new node join nd routing tble mintennce protocols supersede the second phse of the join protocol described in the originl Pstry pper, which hd much higher overhed []. When joining the Pstry overly, new node with nodeid must contct n eisting Pstry node. then routes messge using s the key, nd the new node obtins the th row of its routing tble from the node encountered long the pth from to whose nodeid mtches in the first digits. We will show tht the proimity invrint holds on s resulting routing tble, if node is ner node, ccording to the proimity metric. First, consider the top row of s routing tble, obtined from node. Assuming the tringle inequlity holds in the proimity spce, it is esy to see tht the entries in the top row of s routing tble re lso close to. et, consider the th row of s routing tble, obtined from the node encountered long the pth from to. y induction, this node is Pstry s pproimtion to the node closest to tht mtches s nodeid in the first digits. Therefore, if the tringle inequlity holds, we cn use the sme rgument to conclude tht the entries of the th row of s routing tble should be close to. At this point, we hve shown tht the proimity invrint holds in s routing tble. To show tht the node join protocol mintins the proimity invrint globlly in ll Pstry nodes, we must net show how the routing tbles of other ffected nodes re updted to reflect s rrivl. Once hs initilized its own routing tble, it sends the th row of its routing tble to ech node tht ppers s n entry in tht row. This serves both to nnounce its presence nd to propgte informtion bout nodes tht joined previously. Ech of the nodes tht receives row then inspects the entries in the row, performs probes to mesure if or one of the entries is nerer thn the corresponding entry in its own routing tble, nd updtes its routing tble s pproprite. To see tht this procedure is sufficient to restore the proimity invrint in ll ffected nodes, consider tht nd the nodes tht pper in row of s routing tble form group of nerby nodes whose nodeids mtch in the first digits. It is cler tht these nodes need to know of s rrivl, since my displce more distnt node in one of the node s routing tbles. Conversely, node with identicl prefi in the first digits tht is not member of this group is likely to be more distnt from the members of the group, nd therefore from ; thus, s rrivl is not likely to ffect its routing tble nd, with high probbility, it does not need to be informed of s rrivl. ode filure: Filed routing tbles entries re repired lzily, whenever routing tble entry is used to route messge. Pstry routes the messge to nother node with numericlly closer nodeid. If the downstrem node hs routing tble entry tht mtches the net digit of the messge s key, it utomticlly informs the upstrem node of tht entry. We need to show tht the entry supplied by this procedure stisfies the proimity invrint. If numericlly closer node cn be found in the routing tble, it must be n entry in the sme row s the filed node. If tht node supplies substitute entry for the filed node, its epected distnce from the locl node is therefore low, since ll three nodes re prt of the sme group of nerby

7 nodes with identicl nodeid prefi. On the other hnd, if no replcement node is supplied by the downstrem node, we trigger the routing tble mintennce tsk (described in the net section) to find replcement entry. In either cse, the proimity invrint is preserved. Routing tble mintennce: The routing tble entries produced by the node join protocol nd the repir mechnisms re not gurnteed to be the closest to the locl node. Severl fctors contribute to this, including the heuristic nture of the node join nd repir mechnisms with respect to loclity. Also, mny prcticl proimity metrics do not strictly stisfy the tringle inequlity nd my vry over time. However, limited imprecision is consistent with the proimity invrint, nd s we will show in Section, it does not hve significnt impct on Pstry s loclity properties. However, one concern is tht devitions could cscde, leding to slow deteriortion of the loclity properties over time. To prevent deteriortion of the overll route qulity, ech node runs periodic routing tble mintennce tsk (e.g., every 2 minutes). The tsk performs the following procedure for ech row of the locl node s routing tble. It selects rndom entry in the row, nd requests from the ssocited node copy of tht node s corresponding routing tble row. Ech entry in tht row is then compred to the corresponding entry in the locl routing tble. If they differ, the node probes the distnce to both entries nd instlls the closest entry in its own routing tble. The intuition behind this mintennce procedure is to echnge routing informtion mong groups of nerby nodes with identicl nodeid prefi. A nerby node with the pproprite prefi must be know to t lest one member of the group; the procedure ensures tht the entire group will eventully lern of the node, nd djust their routing tbles ccordingly. Whenever Pstry node replces routing tble entry becuse closer node ws found, the previous entry is kept in list of lternte entries (up to ten such entries re sved in the implementtion). When the primry entry fils, one of the lterntes is used until nd unless closer entry is found during the net periodic routing tble mintennce. 4.2 Locting nerby node Recll tht for the node join lgorithm to preserve the proimity invrint, the strting node must be close to ()discover(seed) (2) nodes getlefset(seed) (3) forll node in nodes (4) nerode closertome(node,nerode) () depth getmroutingtblelevel(nerode) () while (depth ) (7) nodes getroutingtble(nerode,depth ) (8) forll node in nodes (9) nerode closertome(node,nerode) () end while () do (2) nodes getroutingtble(nerode,) (3) currentclosest nerode (4) forll node in nodes () nerode closertome(node,nerode) () while (currentclosest! nerode) (7) return nerode Figure 4: Simplified nerby node discovery lgorithm. seed is the Pstry node initilly known to the joining node. the new node, mong ll live Pstry nodes. This begs the question of how newly joining node cn detect nerby Pstry node. One wy to chieve this is to perform n epnding ring IP multicst, but this ssumes the vilbility of IP multicst. In Figure 4, we present new, efficient lgorithm by which node my discover nerby Pstry node, given tht it hs knowledge of some Pstry node t ny loction. Thus, joining node is only required to obtin knowledge of ny Pstry node through outofbnd mens, s opposed to obtining knowledge of nerby node. The lgorithm eploits the property tht loction of the nodes in the seeds lef set should be uniformly distributed over the network. et, hving discovered the closest lef set member, the routing tble distnce properties re eploited to move eponentilly closer to the loction of the joining node. This is chieved bottom up by picking the closest node t ech level nd getting the net level from it. The lst phse repets the process for the top level until no more progress is mde. Anlysis In this section, we present nlyticl results for Pstry s routing properties. First, we nlyze the distribution of the number of routing hops tken when Pstry messge with rndomly chosen key is sent from rndomly cho

8 P H K : * A sen Pstry node. This nlysis then forms the bsis for n nlysis of Pstry s loclity properties. Throughout this nlysis, we ssume tht ech Pstry node hs perfect routing tble. Tht is, routing tble entry my be empty only if no node with n pproprite nodeid prefi eists, nd ll routing tble entries point to the nerest node, ccording to the proimity metric. In prctice, Pstry does not gurntee perfect routing tbles. Simultion results presented in Section show tht the performnce degrdtion due to this inccurcy is miniml. In the following, we present the min nlyticl results nd leve out the detils of the proofs in Appendi A.. Route probbility mtri Although the number of routing hops in Pstry is symptoticlly! #"%$ '& (+*, the ctul number of routing hops is ffected by the use of the lefset nd the probbility tht the messge key lredy shres prefi with the nodeid of the strting node nd intermedite nodes long the routing pth. In the following, we nlyze the distribution of the number of routing hops bsed on the sttisticl popultion of the nodeid spce. Since the ssignment of nodeids is ssumed to be rndomly uniform, this popultion cn be cptured by the binomil distribution (see, for emple, [3]). For instnce, the distribution of the number of nodes with given vlue of the most significnt nodeid digit, out of ( nodes, is given by MT ( P. Recll from Figure 3 tht t ech node, messge cn be forwrded using one of three brnches in the forwrding procedure. In cse, the messge is forwrded using the lef set F (line 3); in cse using the routing tble (line 8); nd in cse using node in F (lines 3). We formlly define the probbilities of tking these brnches s well s of two specil cses in the following. Definition Let " MK ( P denote the probbility of tking brnch, t the MK P th hop in routing messge with rndom key, strting from node rndomly chosen from ( nodes, with lef set of size. Furthermore, we define " MK ( P s the probbility tht the node encountered fter the K th hop is lredy the numericlly closest node to the messge, nd thus the routing termintes, nd define " MK ( P s the probbility tht the node encountered fter the K th hop lredy shres the MK. digits with the key, thus skipping the MK P th hop. We denote " MK ( P K! " s the probbility mtri of Pstry routing. The following Lemm gives the building block for deriving the full probbility mtri s function of ( nd. Lemm Assume brnch hs been tken during the first K hops in routing rndom messge #, i.e. the messge # is t n intermedite node which shres the first K digits with #. Let $ be the totl number of rndom uniformly distributed nodeids tht shre the first K digits with #. The probbilities in tking different pths t the MK P th hop is 798 +)/9 2> &&&&% +)/ <;. +)/9 # ' +)/943 DC E?GF /9IH GJ K 2<>@? >@? +)/9 A DC >@? E F / %LE? GJ>% K ( / M E OE? / %LE? %LE S+) where " MQP D P.R P,S P clcultes the five probbilities ssuming there re P D P.R P,S nodeids tht shred the first K digits with #, but whose MK. P th digits re smller thn, equl to, nd lrger thn tht of #, respectively. Since the rndomly uniformly distributed nodeids tht fll in prticulr segment of the nmespce contining fied prefi of K digits follow the binomil distribution, the K th row of the probbility mtri cn be clculted by summing over ll possible nodeid distributions in tht segment of the nmespce the probbility of ech distribution multiplied by its corresponding probbility vector given by Lemm. Figure plots the probbilities of tking brnches,, nd t ech ctul hop (i.e. fter the djustment of collpsing skipped hops) of Pstry routing for ( TTT, with VU nd.. It shows tht the #"%$/ M ( P th hop is dominted by hops while erlier hops re dominted by W hops. The bove probbility mtri cn be used to derive the distribution of the numbers of routing hops in routing rndom messge. Figure plots this distribution for ( TTT with XU nd. The probbility mtri cn lso be used to derive the epected number of routing hops in Pstry routing ccording to the following theorem. 7

9 G M ) ) Probbilities of tking brnches PA, P, nd PC , l32, b4, Epected (hops) Hop number h prob(h,l,,p) prob(h,l,,pb) prob(h,l,,pc) Figure : Probbilities ) +) 2, ) +) #, ) +) ( nd epected number of hops for *, with )O* L nd *. (From nlysis.) Probbility.8..4, l32, b4, Epected (hops) 3.7 epected totl route distnce. To mke the nlysis trctble, it is ssumed tht the loctions of the Pstry nodes re rndom uniformly distributed over the surfce of sphere, nd tht the proimity metric used by Pstry equls the geogrphic distnce between pirs of Pstry nodes on the sphere. The uniform distribution of node loctions nd the use of geogrphic distnce s the proimity metric re clerly not relistic. In Section we will present two sets of simultion results, one for conditions identicl to those ssumed in the nlysis, nd one bsed on Internet topology models. A comprison of the results indictes tht the impct of our ssumptions on the results is limited. Since Pstry nodes re uniformly distributed in the proimity spce, the verge distnce from rndom node to the nerest node tht shres the first digit, the first two digits, etc., cn be clculted bsed on the density of such nodes. The following Lemm gives the verge distnce in ech hop trveled by messge with rndom key sent from rndom strting node, s function of the hop number nd the hop type umber of routing hops Figure : Distribution of the number of routing hops per messge for *, with )+* L nd *. (From nlysis.) Theorem Let the epected number of dditionl hops fter tking for the first time, t the K th hop, be denoted s MK ( P. The epected number of routing hops in routing messge with rndom key # strting from node rndomly chosen from the ( nodes is!s J >@? % 23 % 3 +) 2 +) +) # +) +) ( +) ( K +) (.2 Epected routing distnce The bove routing hop distribution is derived solely bsed on the rndomly uniform distribution of nodeids in the nmespce. Coupled with proimity neighbor selection in mintining the entries in Pstry s routing tbles, the routing hop distribution cn be used to nlyze the Lemm 2 () In routing messge #, fter K if is not empty, the epected K "9 EJMK &! #"$! "J % P. W hops, P is (2) In routing messge #, if pth is tken t ny given hop, the hop distnce K "9 EJMK. P is &(' (3) In routing messge #, fter K hops, if pth is tken, the hop distnce K "9 EJMK P is K "9 EJMK P, which with high probbility is followed by hop tken vi, i.e. with distnce &(' The bove distnce K "9 EJMK P comes from the density rgument. Assuming nodeids re uniformly distributed over the surfce of the sphere, the verge distnce of the net hop is the rdius of circle tht contins on verge one nodeid (i.e. the nerest one) tht shre MK P digits with #.. Given the vector of the probbilities of tking brnches,, nd t the ctul K th hop (e.g. Figure ), nd the bove vector of perhop distnce for the three types of hops t the K th hop, the verge distnce of the K th ctul hop is simply the dotproduct of the two vectors, i.e. the weighted sum of the hop distnces by the probbilities tht they re tken. These results re presented in the net section long with simultion results. 8

10 P.3 Locl route convergence Theorem 2 Let nd on sphere of rdius be the two strting nodes from which messges with n identicl, rndom key re being routed. Let the distnce between nd be *. Then the epected distnce tht the two messges will trvel before their pths merge is ; " et, we nlyze Pstry s route convergence property. Specificlly, when two rndom Pstry nodes send messge with the sme rndomly chosen key, we nlyze the epected distnce the two messges trvel in the proimity spce until the point where their routes converge, s function of the distnce between the strting nodes in the proimity spce. To simplify the nlysis, we consider three scenrios. In the worstcse scenrio, it is ssumed tht t ech routing hop prior to the point where their routes converge, the messges trvel in opposite directions in the proimity spce. In the vergecse scenrio, it is ssumed tht prior to convergence, the messges trvel such tht their distnce in the proimity spce does not chnge. In the best cse scenrio, the messges trvel towrds ech other in the proimity spce prior to their convergence. For ech of the bove three scenrios, we derive the probbility tht the two routes converge fter ech hop. The probbility is estimted s the intersecting re of the two circles potentilly covered by the two routes t ech hop s percentge of the re of ech circle. Coupling this probbility vector with the distnce vector (for different hops) gives the epected distnce till route convergence. 3* % +<( ; S 3.. +<( " H E 3 >@? >@? R ) R ) where " K "9 MQP * P C R ) ),!" C P * $#&%(')* K "9 EJMQP P in the worst cse, or P * in the',+. verge cse, or P MO * K "9 EJMQP /#*%(')* ',+. PP in the best cse, respectively, M P denotes the intersecting re of two circles of rdius centered t two points on sphere of rdius tht re distnce of 2 prt, nd 3 S4,9R M denotes the surfce re of circle of rdius on sphere of rdius. Figure 7 plots the verge distnce trveled by two messges sent from two rndom Pstry nodes with the 9 sme rndom key, s function of the distnce between the two strting nodes. Results re shown for the worst cse, verge cse, nd best cse nlysis. Averge Pstry distnce to convergence point 2 2 Worst cse, k Averge cse, k est cse, k etwork distnce between source nodes Figure 7: Distnce mong source nodes routing messges with the sme key, versus the distnce trversed until the two pths converge, for, node Pstry network, with l32 nd b4. (From nlysis.) Eperimentl results Our nlysis of proimity neighbor selection in Pstry hs relied on ssumptions tht do not generlly hold in the Internet. For instnce, the tringle inequlity does not generlly hold for most prcticl proimity metrics in the Internet. Also, nodes re not uniformly distributed in the resulting proimity spce. Therefore, it is necessry to confirm the robustness of Pstry s loclity properties under more relistic conditions. In this section, we present eperimentl results quntifying the performnce of proimity neighbor selection in Pstry under relistic conditions. The results were obtined using Pstry implementtion running on top of network simultor, using Internet topology models. The Pstry prmeters were set to + nd the lefset size U. Unless otherwise stted, results where obtined with simulted Pstry overly network of, nodes.. etwork topologies Three simulted network topologies were used in the eperiments. The Sphere topology corresponds to the topology ssumed in the nlysis of Section. odes re plced t uniformly rndom loctions on the surfce of sphere with rdius. The distnce metric is

11 bsed on the topologicl distnce between two nodes on the sphere s surfce. Results produced with this topology model should correspond closely to the nlysis, nd it ws used primrily to vlidte the simultion environment. However, the sphere topology is not relistic, becuse it ssumes uniform rndom distribution of nodes on the Sphere s surfce, nd its proimity spce is very regulr nd strictly stisfies the tringle inequlity. A second topology ws generted using the Georgi Tech trnsitstub network topology model []. The roundtrip dely (RTT) between two nodes, s provided by the topology grph genertor, is used s the proimity metric with this topology. We use topology with nodes in the core, where LA with n verge of nodes is ttched to ech core node. Out of the resulting, LA nodes,, rndomly chosen nodes form Pstry overly network. As in the rel Internet, the tringle inequlity does not hold for RTTs mong nodes in the topology model. Finlly, we used the Merctor topology nd routing models [4]. The topology model contins 2,39 routers nd it ws obtined from rel mesurements of the Internet using the Merctor progrm []. The uthors of [4] used rel dt nd some simple heuristics to ssign n utonomous system to ech router. The resulting AS overly hs 2,2 nodes. Routing is performed hierrchiclly s in the Internet. A route follows the shortest pth in the AS overly between the AS of the source nd the AS of the destintion. The routes within ech AS follow the shortest pth to router in the net AS of the AS overly pth. We built Pstry overly with, nodes on this topology by picking router for ech node rndomly nd uniformly, nd ttching the node directly to the router with LA link. Since the topology is not nnotted with dely informtion, the number of routing hops in the topology ws used s the proimity metric for Pstry. We count the LA hops when reporting the length of the Pstry routes. This is conservtive becuse the cost of these hops is usully negligible nd Pstry s overhed would be lower if we did not count LA hops..2 Pstry routing hops nd distnce rtio In the first eperiment, 2, lookup messges re routed using Pstry from rndomly chosen nodes, using rndom key. Figure 8 shows the number of Pstry routing hops nd the distnce rtio for the sphere topology. Distnce rtio is defined s the rtio of the distnce trversed by Pstry messge to the distnce between its source nd destintion nodes, mesured in terms of the proimity metric. The distnce rtio cn be interpreted s the penlty, epressed in terms of the proimity metric, ssocited with routing messges through Pstry insted of sending the messge directly in the Internet. Four sets of results re shown. Epected represents the results of the nlysis in Section. orml routing tble shows the corresponding eperimentl results with Pstry. Perfect routing tble shows results of eperiments with version of Pstry tht uses perfect routing tble. Tht is, ech entry in the routing tble is gurnteed to point to the nerest node with the pproprite nodeid prefi. Finlly, o loclity shows results with version of Pstry where the loclity heuristics hve been disbled Epected Perfect Routing Tble orml Routing Tble umber of hops o Epected Loclity Perfect Routing Tble orml Routing Tble Distnce rtio 3.8 o Loclity Figure 8: umber of routing hops nd distnce rtio, sphere topology. All eperimentl results correspond well with the results of the nlysis, thus vlidting the eperimentl pprtus. As epected, the epected number of routing TT U hops is slightly below #"%$/ nd the distnce rtio is smll. The reported hop counts re virtully independent of the network topology, therefore we present them only for the sphere topology. The distnce rtio obtined with perfect routing tbles is only mrginlly better thn tht obtined with the rel Pstry protocol. This confirms tht the node join protocol produces routing tbles of high qulity, i.e., entries refer to nodes tht re nerly the closest mong nodes with the pproprite nodeid prefi. Finlly, the distnce rtio obtined with the loclity heuristics disbled is significntly worse. This speks both to the importnce of topologywre routing, nd the effectiveness of proimity neighbor selection.

12 .3 Routing distnce 8 Perhop distnce Epected Perfect Routing Tble orml Routing Tble o Loclity Hop number Perhop distnce orml Routing Tbles Perfect Routing Tbles o Loclity Hop umber Figure : Distnce trversed per hop, Merctor topology. Figure 9: Distnce trversed per hop, sphere topology. Figure 9 shows the distnce messges trvel in ech consecutive routing hop. The results confirm the eponentil increse in the epected distnce of consecutive hops up to the fourth hops, s predicted by the nlysis. ote tht the fifth hop is only tken by tiny frction (.4%) of the messges. Moreover, in the bsence of the loclity heuristics, the verge distnce trveled in ech hop is constnt nd corresponds to the verge distnce between nodes ( M P', where r is the rdius of the sphere). Perhop distnce orml Routing Tbles Perfect Routing Tbles o loclity Hop umber Figure : Distnce trversed per hop, GATech topology. Figures nd show the sme results for the GATech nd the Merctor topologies, respectively. Due to the nonuniform distribution of nodes nd the more comple proimity spce in these topologies, the epected distnce in ech consecutive routing step no longer increses eponentilly, but it still increses monotoniclly. Moreover, the node join lgorithm continues to produce routing tbles tht refer to nerby nodes, s indicted by the modest difference in hop distnce to the perfect routing tbles in the first three hops. The proimity metric used with the Merctor topology mkes proimity neighbor selection pper in n unfvorble light. Since the number of nodes within T IP routing hops increses very rpidly with T, there re very few nerby Pstry nodes. Observe tht the verge distnce trveled in the first routing hop is lmost hlf of the verge distnce between nodes (i.e., it tkes lmost hlf the verge distnce between nodes to rech bout other Pstry nodes). As result, Pstry messges trverse reltively long distnces in the first few hops, which leds to reltively high distnce rtio. evertheless, these results demonstrte tht proimity neighbor selection works well even under dverse conditions. Figures 2, 3 nd 4 show rster plots of the distnce messges trvel in Pstry, s function of the distnce between the source nd destintion nodes, for ech of the three topologies, respectively. Messges were sent from 2, rndomly chosen source nodes with rndom keys in this eperiment. The men distnce rtio is shown in ech grph s solid line. The results show tht the distribution of the distnce rtio is reltively tight round the men. ot surprisingly, the sphere topology yields the best results, due to its uniform distribution of nodes nd the geometry of its proimity spce. However, the fr more relistic GATech topology yields still very good results, with men distnce rtio of.9, miml distnce rtio of bout 8., nd distribution tht is firly tight round the men. Even the lest fvorble Merctor topology yields good results, with men distnce rtion of 2.2 nd mimum of bout..

13 7.9.8 Convergence metric Distnce trveled by Pstry messge Distnce trveled by Pstry messge To evlute how erly the pths convergence, we use R R the metric M R R P' where, R is the distnce trveled from the node where the two pths converge to Men.37 the destintion node, nd J R nd J R re the distnces trv4 eled from ech source node to the node where the pths 3 converge. The metric epresses the verge frction of 2 the length of the pths trveled by the two messges tht ws shred. ote tht the metric is zero when the pths converge in the destintion. Figures, nd 7 show Distnce between source nd destintion the verge of the convergence metrics versus the disfigure 2: Distnce trversed versus distnce between tnce between the two source nodes. As epected, when the distnce between the source nodes is smll, the pths source nd destintion, sphere topology. 2 re likely to converge quickly. This result is importnt Men.9 for pplictions tht perform cching, or rely on efficient 2 multicst trees [, 2] Distnce between source nd destintion Figure 3: Distnce trversed versus distnce between source nd destintion, GATech topology Distnce between two source nodes.4 Locl route convergence Convergence metric Figure : Convergence metric versus the distnce bethe net eperiment evlutes the locl route conver tween the source nodes, sphere topology. gence property of Pstry. In the eperiment, nodes were selected rndomly, nd then for ech of these nodes,.9.8, other nodes were chosen such tht the topologicl.7 distnce between ech pir provides good coverge of the. rnge of possible distnces. Then, rndom keys were. chosen nd messges where routed vi Pstry from ech.4.3 of the two nodes in pir, with given key..2 Distnce trveled by Pstry messge. 8 Men Distnce between two source nodes Figure : Convergence metric versus distnce between the source nodes, GATech topology Overhed of node join protocol et, we mesure the overhed incurred by the node join protocol to mintin the proimity invrint in the routdistnce between source nd destintion ing tbles. We quntify this overhed in terms of the Figure 4: Distnce trversed versus distnce between number of probes, where ech probe corresponds to the source nd destintion, Merctor topology

14 Convergence Metric Distnce between two source nodes Figure 7: Convergence metric versus distnce between the source nodes, Merctor topology. communiction required to mesure the distnce, ccording to the proimity metric, mong two nodes. Of course, in our simulted network, probe simply involves looking up the corresponding distnce ccording to the topology model. However, in rel network, probing would likely require t lest two messge echnges. The number of probes is therefore meningful mesure of the overhed required to mintin the proimity invrint. The verge number of probes performed by newly joining node ws 29, with minimum of 23 nd mimum of 34. These results were virtully independent of the overly size, which we vried from, to, nodes. In ech cse, the probes performed by the lst ten nodes tht joined the Pstry network were recorded, which re the nodes likely to perform the most probes given the size of the network t tht stge. The corresponding verge number of probes performed by other Pstry nodes during the join ws bout 7, with minimum of 2 nd mimum of 2. It is ssumed here tht once node hs probed nother node, it stores the result nd does not probe gin. The number of nodes contcted during the joining of new node is M P #"%$ '&)(, where is the number of Pstry nodes. This follows from the epected number of nodes in the routing tble, nd the size of the lef set. Although every node tht ppers in the joining node s routing tble receives informtion bout ll the entries in the sme row of the joining node s routing tble, it is very likely tht the receiving node lredy knows mny of these nodes, nd thus their distnce. As result, the number of probes performed per node is low (on verge less thn 2). This mens tht the totl number of nodes probed is low, nd the probing is distributed over lrge number of nodes. The results were virtully identicl for the GATech nd the Merctor topologies.. ode filure In the net eperiment, we evlute the node filure recovery protocol (Section 4.) nd the routing tble mintennce (Section 4.). Recll tht lef set repir is instntneous, filed routing tble entries re repired lzily upon net use, nd periodic routing tble mintennce tsk runs periodiclly (every 2 mins) to echnge informtion with rndomly selected peers. In the eperiment,, node Pstry overly is creted bsed on the GATech topology, nd 2, messges from rndom sources with rndom keys re routed. Then, 2, rndomly selected nodes re mde to fil simultneously, simulting conditions tht might occur in the event of network prtition. Prior to the net periodic routing tble mintennce, new set of 2, rndom messge re routed. After nother periodic routing tble mintennce, nother set of 2, rndom messges re routed. Figure 8 shows both the number of hops nd the distnce rtio t vrious stges in this eperiment. Shown re the verge number of routing hops nd the verge distnce rtio, for 2, messges ech before the filure, fter the filure, fter the first nd fter the second round of routing tble mintennce. The no filure result is included for comprison nd corresponds to 3, node Pstry overly with no filures. Moreover, to isolte the effects of the routing tble mintennce, we give results with nd without the routing tble mintennce enbled efore filure After filure 3.8 After round After 2 rounds o filure.. efore filure.9 After filure After round 3 3 umber of Hops Routing Tble Mintennce Enbled Routing Tble Mintennce Disbled Distnce Rtio After 2 rounds. o filure Figure 8: Routing hops nd distnce rtio for, node Pstry overly when 2, nodes simultneously fil, GATech topology. During the first 2, messge trnsmissions fter the mssive node filure, the verge number of hops nd verge distnce rtio increse only mildly (from 3.4 to 4.7 nd. to.8, respectively). This demonstrtes the 3