Low Traffic Overlay Networks with Large Routing Tables

Transcription

1 Low Traffic Overay Networks with Large Routing Tabes Chunqiang Tang, Meissa J. Buco, Rong N. Chang, Sandhya Dwarkadas, Laura Z. Luan, Edward So, and Christopher Ward ABSTRACT The routing tabes of Distributed Hash Tabes (DHTs) can vary from size O() to O(n). Currenty, what is acking is an anaytic framework to suggest the optima routing tabe size for a given workoad. This paper () compares DHTs with O() to O(n) routing tabes and identifies some good design points; and () proposes protocos to reaize the potentia of those good design points. We use tota traffic as the uniform metric to compare heterogeneous DHTs and emphasize the baance between maintenance cost and ookup cost. Assuming a node on average processes, or more ookups during its entire ifetime, our anaysis shows that arge routing tabes actuay ead to both ow traffic and ow ookup hops. These good design points transate into one-hop routing for systems of medium size and two-hop routing for arge systems. Existing one-hop or two-hop protocos are based on a hierarchy. We instead demonstrate that it is possibe to achieve competey decentraized one-hop or two-hop routing, i.e., without giving up being peer-to-peer. We propose h-caot for one-hop routing and h-caot for two-hop routing. Assuming a moderate ookup rate, compared with DHTs that use O(og n) routing tabes, h-caot and h-caot save traffic by up to 7% whie resoving ookups in one or two hops as opposed to O(og n) hops. Categories and Subject Descriptors C..4 [Computer-Communication Networks]: Distributed Systems Genera Terms Agorithms, Design, Management, Performance Keywords Peer-to-Peer System, Overay Network, Distributed Hash Tabe. INTRODUCTION In recent years, Distributed Hash Tabes (DHTs) have been proposed as the infrastructure for buiding a wide range of distributed appications such as storage [], content distribution [4], and search IBM T. J. Watson Research Center, Hawthorne, NY 5. {ctang, mbuco, rong, uan, edwardso, cw}@us.ibm.com. Computer Science Department, University of Rochester, Rochester, NY, sandhya@cs.rochester.edu. Permission to make digita or hard copies of a or part of this work for persona or cassroom use is granted without fee provided that copies are not made or distributed for profit or commercia advantage and that copies bear this notice and the fu citation on the first page. To copy otherwise, to repubish, to post on servers or to redistribute to ists, requires prior specific permission and/or a fee. SIGMETRICS 5, June 6, 5, Banff, Aberta, Canada. Copyright 5 ACM /5/6...$5.. engines []. A DHT organizes nodes into a structured overay network and can efficienty map a key to the node that is responsibe for the key through distributed routing. The designs of DHTs vary dramaticay. Eary designs [] use sma O(og n) routing tabes, due to the concern that big routing tabes are hard to maintain and cannot scae to arge systems. Later designs use O( n) [8] or even O(n) [7] routing tabes and argue that it is feasibe to do so. This paper provides an anaytic framework to suggest the optima routing tabe size for a given workoad. A workoad is parameterized by a tupe <n,, f >, where n is the number of nodes in the system, is the average node ifetime, and f is the average number of ookups that a node processes per second (i.e., the node is the destination of the ookups). We use traffic as the uniform metric to compare heterogeneous DHTs with O() to O(n) routing tabes. Our anaysis shows that the most traffic-efficient routing size is proportiona to O(f n(n)). Our anaysis does have practica use. It heps us to identify pitfas in existing DHT designs that are mainy driven by the desire to improve ookup atency, e.g., the argument [7] that it is favorabe to maintain O(n) routing tabes for systems with miions of nodes. Our anaysis shows that it is not cost-effective to do so for systems arger than a few thousand nodes. Otherwise, it coud introduce, times more traffic than traditiona DHTs []. Most existing DHTs are intended for environments simiar to those for peer-to-peer fie sharing systems such as Gnutea and KaZaA, and hence are designed to hande a high churn rate, assuming node ifetimes as short as severa minutes [7]. Consequenty, they argue for sma O(og(n)) routing tabes, which seems reasonabe as both node ifetime and ookup rate f are ow. However, DHTs are inherenty unsuitabe for environments with a high churn rate because DHTs mandate data pacement on nodes; by contrast, a node in Gnutea stores its own data ocay. In DHTs, when a node joins, some data must be copied to that node; when the node eaves, data stored on that node must be copied to another node. Even if the routing tabes can be maintained correcty under a high churn rate [7], the high traffic due to data movement woud render the system unusabe []. Not surprisingy, most depoyed DHT appications [4, ] run on reativey stabe but unreiabe nodes. Open DHT [] is one prominent exampe. After severa years of extensive research on DHTs, Open DHT is perhaps the ony depoyed DHT running at a arge scae. It runs on PanetLab and offers services to nodes outside the DHT, incuding mobie nodes. We beieve that this mode is the future of DHT. It is unnecessary and inefficient to incude every node that uses the DHT services as part of the DHT. If a node ives for ony severa minutes, the overhead caused by the join and eave of the node and reated data movement is ikey to dwarf the services, if any, provided by the node during its short ifetime. Seecting ony good quaity nodes to provide services can resut in a DHT that is smaer, faster, and more efficient. Even KaZaA [] uses just a subset of super nodes to provide ookup services.

2 When a DHT is provided as a service to nodes outside the DHT, we assume that each DHT node on average processes, or more ookups during its entire ifetime, i.e., f. For instance, assuming a.9 hour node ifetime (the average node ifetime in Gnutea [9]), each DHT node needs to process one ookup every seconds; assuming a one week node ifetime, each DHT node needs to process one ookup every 6 seconds. We beieve this assumption f is reasonabe. If the ookup rate is extremey ow, then the DHT is underutiized. The architect shoud downsize the DHT to reduce unnecessary overheads and resource wastes, resuting in increased ookups submitted to each DHT node. Under the assumption f, our anaysis shows that arge routing tabes with severa hundred to one thousand entries actuay ead to both ow traffic and ow ookup hops. This design point transates into one-hop routing (with O(n) routing tabes) for systems with up to a few thousands nodes; or two-hop routing (with O( n) routing tabes) for systems with up to a few miion nodes. One-hop and two-hop routings are efficient in both traffic and ookup hops, but their arge routing tabes are hard to maintain. Existing proposas for one-hop or two-hop routing are either hierarchica [5, 7, 8, 5, 8] in which nodes have different roes and the oad is uneveny distributed or assume a particuar query distribution that imits its generaity [6]. We wi demonstrate that it is possibe to achieve one-hop or two-hop routing without giving up being peer-to-peer. A peer-to-peer architecture has many good properties such as resiience and oad baance, which are the reasons that originay motivated DHTs []. We propose what we beieve are the first practica non-hierarchica protocos for one-hop routing (h-caot) and two-hop routing (h- Caot). Compared with traditiona DHTs that use O(og n) routing tabes, h-caot and h-caot save tota traffic by up to 7% whie resoving ookups in one or two hops as opposed to O(og n) hops. Their fast ookups are particuary attractive for interactive appications such as search engines [] and name resoution. To maintain the arge routing tabes in a scaabe fashion, h- Caot and h-caot muticast node arrivas and departures through O(n) different trees embedded in the overay. The trees in h- Caot and h-caot are conceptua and require no expicit maintenance. h-caot s randomized agorithm further expoits virtua nodes running on the same computer to route among remote nodes in a purey peer-to-peer fashion. Both h-caot and h-caot are extremey simpe: muticast maintains the routing tabes; information in the routing tabes is then used to guide muticast and routing. The remainder of the paper is organized as foows. Section compares heterogeneous DHTs in order to identify the good design points. Sections and 4 present the design and anaysis of our onehop and two-hop protocos, respectivey. Section 5 evauates our protocos through extensive simuation. Reated work is discussed in Section 6. Section 7 concudes the paper.. OPTIMAL ROUTING TABLE SIZE Previous works [6,,, ] mainy used resiience and ookup atency as the metrics to compare DHTs with O(og(n)) routing tabes. Instead, we use tota traffic (both maintenance and ookup) as the metric to compare DHTs with O() to O(n) routing tabes under a strawman mode. Our goa is to revea the fundamenta impact of routing tabe size on the traffic of DHTs. Traffic is reevant because a ow-traffic DHT aows the architect to use a smaer and faster DHT to hande a given oad. In genera, DHTs with arger routing tabes introduce higher maintenance traffic but have fewer routing hops and hence ower ookup traffic. A good design shoud strike a baance between them to minimize the tota traffic. We assume that node ifetime foows an exponentia distribution p (t) = λ e λ t, () where λ = and is the average node ifetime. We assume that node arriva is a Poisson process with rate λ e. The probabiity that k nodes join during a time period t is P (X = k) = (λet)k e λet. () k! To maintain a stabe popuation of n nodes with an average ifetime, the node arriva rate λ e = nλ = n. We assume the ookups that a node processes foow a Poisson process with rate f. Both ookup messages and messages for routing tabe maintenance are sma. The payoad typicay incudes a DHT key and the IP address of a node. Uness otherwise noted, we assume communications use UDP/IP; ookup and maintenance messages have unit size s (incuding both packet header and payoad); the messages are expicity acknowedged and the acknowedgments have size.5s. Our anaysis ignores packet oss and retransmissions at the network ayer. We assume that the targets of ookups distribute uniformy across a nodes. We assume an idea representative for each category of DHTs in order to shed ight on the fundamentas. When it comes to a specific DHT design, we aso consider other factors such as resiience and ookup hops. Sections and 4 wi address more reaistic impementation issues. Beow, we use the tota traffic metric to compare DHTs with O() to O(n) routing tabes. Degree-Diameter Optima DHTs For a network with n nodes in which each node has d neighbors (i.e., the node degree is d), the network s diameter D (maximum hops of the shortest paths between any two nodes) is bounded [] by: D og d (n(d ) + ). We refer to DHTs that approach this ower bound as degree-diameter optima DHTs [9,,, 4]. At the abstract eve, DHTs with the same node degree introduce simiar maintenance traffic but those with optima diameters introduce ower ookup traffic. Our comparison therefore focuses on degree-diameter optima DHTs with routing tabes of different sizes. We use de Bruijn graphs [] as the representative, in which a ookup on average takes r og d n hops. We cacuate the minima traffic needed to update the routing tabes of a de Bruijn graph in the face of node arrivas and departures. In an n-node system with a node ifetime, on average n nodes join and n nodes eave each second. When a node joins or eaves, at east one message is sent to notify each of its d routing neighbors, resuting in n d messages for node arrivas and n d messages for node departures. Furthermore, at east one message is needed to inform a new node of each of its d neighbors, resuting in n d messages to set up the routing tabes for new nodes. Assuming a maintenance messages have unit size s and each is acknowedged by a packet of size.5s, the maintenance traffic is B = ( +.5)s ( n d + n d + n d). () Each node processes f ookups per second, resuting in nf ookups in tota. Each ookup takes og d n hops in a de Bruijn graph. The traffic for ookups is therefore B = ( +.5)s nf og d n. (4) In most existing DHTs, nodes probe their routing neighbors periodicay. An idea design can avoid this traffic. For instance, Caot uses overay muticast to maintain routing tabes. The traffic woud be ower if the new node copies a compete routing tabe from an existing node in a singe packet. This ony affects our resuts by a very sma constant factor. We choose not to consider this optimization here because it is adopted in few DHTs.

3 Optima routing tabe size k 4k 6k 64k 56k 4k Nodes (thousands) Optima routing tabe size Node ifetime (hours) Optima routing tabe size Lookup rate (ookups/second) (a) f= ookup/second and =.9 hours. (b) n= miion nodes and f= ookup/second. (c) n= miion nodes and =.9 hours. Figure : The optima routing tabe size d that minimizes the tota traffic (from Equation 6). Reative traffic Mh / Mdht 4 5 Nodes (a) One-hop schemes. Reative traffic Mh / Mdht Nodes (miions) (b) Two-hop schemes. Figure : Traffic reative to traditiona DHTs with O(og n) routing tabes when =.9 hours and f=. ookups/second. M h, M h, and M dht are from Equations 7-9. The tota traffic (maintenance pus ookup) in a de Bruijn graph is M = B + B =.5s ( n d + nf og d n). (5) We derive the routing tabe size d that minimizes the tota traffic by setting the derivative of M with respect to d to. M d = = d f n n n d = The f component in Equation 6 indicates that the optima routing tabe size d is proportiona to the number of ookups that a node processes during its entire ifetime. Previous comparisons mainy focused on the impact of node ifetime on system resiience, and ignored ookup rate. Our anaysis instead shows that ookup rate is a critica parameter when designing DHTs for ow traffic. Equation 6 has no cosed form soution. We sove it using Newton s method for a given workoad <n,, f> and pot the resuts for some typica workoads in Figure. This figure shows that using arge routing tabes with severa hundred to one thousand entries is actuay efficient in traffic. This transates into one-hop routing (with O(n) routing tabes) for systems with up to a few thousand nodes, or two-hop routing (with O( n) routing tabes) for systems with up to a few miion nodes. These are the good design points we focus on in Sections and 4. The above anaysis makes some idea assumptions: () when a node joins or eaves, this membership change can be efficienty disseminated to about, nodes; and () a node need not probe its, or so routing neighbors to maintain the accuracy of its routing tabe. These assumptions are obviousy not met by existing soutions [] based on a de Bruijn graph. Other systems that do use arge routing tabes are based on a hierarchy [5, 7, 8, 5, 8]. In Sections and 4, we wi present our peer-to-peer soutions. (6) One-hop Schemes In one-hop schemes, nodes know each other: d = n n. Substituting this into Equation 5, we obtain the tota traffic Two-hop Schemes M h.5s ( n + nf). (7) In idea two-hop schemes, each node has d = n routing neighbors. Substituting this into Equation 5, we obtain the tota traffic Traditiona DHTs M h = s(4.5 n.5 + nf). (8) In traditiona DHTs, each node has O(og n) routing neighbors and ookups are resoved in O(og n) hops. We consider an abstract version of the Chord protoco [], in which each node has d = og n neighbors and ookups on average take og n hops. Foowing the anaysis process in Section, we know that the abstract Chord introduces traffic M c = 4.5 n s og n to update routing tabes in the face of node arrivas and departures (see Equation and note d = og n). In addition, each node sends a heartbeat message to each of its og n neighbors every T = seconds. We assume the heartbeat messages have size.5s. The traffic for heartbeats is M c. There are nf ookups in tota. Lookups on average take og n hops. The traffic for ookups is =.5sn og n T M c = (+.5)s nf og n. The coefficient.5 is because ookup messages are acknowedged. The tota traffic therefore is M dht =M c+m c+m c =s n og n( f +.5 ). (9) T Comparing Traditiona DHTs with Others In Figure, we compare the traffic of traditiona DHTs with that of one-hop schemes and two-hop schemes. Overa, the figure shows that one-hop and two-hop schemes can have ow traffic and fast routing at the same time when f is sufficienty high, for instance, in reaistic DHTs ike Open DHT []. In contrast to the argument [7] that it is favorabe to maintain compete O(n) routing tabes for systems with up to a few miion nodes, Figure (a) shows that one-hop schemes are ony efficient for systems with up to severa thousand nodes. With a few miion nodes, a one-hop scheme coud introduce, times more traffic than traditiona DHTs. Figure (b) shows that an idea two-hop scheme can be efficient for systems with up to miions of nodes. When the system has more than miion nodes, however, two-hop schemes introduce more traffic than traditiona DHTs. Based on these observations, we propose our one-hop protoco (h-caot) for systems of medium size and two-hop protoco (h-caot) for arge systems.

4 Route Caching and Reactive Maintenance A the DHTs described above proactivey maintain the accuracy of the routing tabes. Another way to keep arge routing tabes is reactive maintenance, in which nodes cache other nodes they discovered in past ookups and reuse them in future ookups. There is no expicit maintenance operation. The drawback is that nodes may encounter frequent faiures during ookups. Next, we cacuate the probabiity of correct cache hit when nodes use their routing tabes. Suppose node N puts node S into its routing tabe when N discovers S through a ookup. The ookups that N issues foow a Poisson process with rate f. Assuming ookups are uniformy distributed, the ookups that N issues to target S is a Poisson process with rate λ v = f/n since there are n nodes. The interva between ookups from N to S foows an exponentia distribution p v(t) = λ ve λvt. Node ifetime foows an exponentia distribution p (t) = λ e λt. When node N contacts node S at time x since the ast ookup, the probabiity that S is sti aive is ( R y=x p y= (y) dy). The probabiity that node N finds node S aive when N issues a new ookup to S is therefore P cache hit = = Z x= x= [ p v(x) ( Z y=x y= p (y) dy) ] dx λ v = λ v + λ + n. () f Tabe shows the cache hit rate P cache hit under typica workoads. When ookup rate f=., about 49% of ookups fai on their first hop. A faied hop incurs a high atency as the query initiator has to wait for a ong, conservative period before it timeouts. Since the cache hit rate is not sufficienty high, we consider reactive maintenance not suitabe for interactive appications. f P cache hit Tabe : Cache hit rate in Eq (n=, and =.9 hours).. H-CALOT FOR ONE-HOP ROUTING The anaysis in Section shows that it is beneficia to use arge routing tabes. When impemented propery, they ead to both ow traffic and ow ookup hops. The chaenge, however, is to efficienty maintain the arge routing tabes in the face of frequent node arrivas and departures. To this end, we propose h-caot. It uses overay muticast to efficienty disseminate notifications for node arrivas and departures to a nodes. For systems with up to a few thousand nodes, h-caot resoves ookups in one hop with high probabiity whie introducing traffic ower than traditiona DHTs []. For arger systems, we wi introduce in Section 4 our h-caot protoco that uses O( n) routing tabes. Unike hierarchica one-hop or two-hop schemes [5, 7, 8, 5, 8], h- Caot and h-caot are purey peer-to-peer. Like Chord [], h-caot organizes nodes into a circuar ring that corresponds to an identifier space [, 6 -]. Each node is assigned an identifier by appying SHA- hashing to its IP address. We refer to a node s cockwise neighboring node aong the ring as its successor and the counter-cockwise neighboring node as its predecessor. The predecessor node and the successor node of a key are defined simiary. Each object is associated with a key drawn from the identifier space, for instance, by appying SHA- hashing to the object s content. An object is stored on the node whose identifier is the cosest to the object s key in absoute distance, regardess of the direction (cockwise or counter-cockwise). (a) Muticast process in the overay. (b) Muticast process as a tree. Figure : Muticast tree for disseminating membership changes. This exampe uses a -bit identifier space. There are 8 nodes with identifiers -7. Node just joined and acts as the root of the tree for announcing its arriva. Node seects its finger nodes at exponentiay increasing distance from itsef as its chidren in the tree. Each chid of node is responsibe for covering a range of the identifier space, for instance, the range (, 4) for node. The chidren of the root further seect their finger nodes as their chidren to expand the tree, and so forth. h-caot maintains a compete O(n) routing tabe on every node. Ideay, nodes know each other and messages are deivered directy between the source and the destination. In the case that the routing tabes are inaccurate (e.g., missing ive nodes or isted dead nodes), routing may take onger. A node N aways greediy forwards a ookup to the node P that is, to N s knowedge, the cosest in absoute distance to the ookup key. If P is the right destination, the ookup is done. Otherwise, P further forwards the ookup to the node, to P s knowedge, cosest to the destination, and so forth. If P is not responsive when N tries to forward P a ookup, N wi timeout and try the second cosest node. A communications in h-caot use UDP and messages are expicity acknowedged. As in Chord, correct routing is guaranteed so ong as each node correcty maintains its predecessor and successor (therefore a ookup aways moves coser to its destination after each step). A node maintains its predecessor and successor through periodic heartbeat messages, but does not periodicay probe any other node in its routing tabe. This is crucia to keep maintenance traffic ow. Routing tabe maintenance is described in the next section.. Handing Node Joins and Leaves We assume that a new node N knows through some out-of-band method about at east one node P aready in the system. Node N copies a compete routing tabe from node P in order to have a goba view of the system. Node N generates its identifier k by appying SHA- hashing to its IP address, and takes over objects that are coser to N from its predecessor and successor. Node N informs other nodes of its arriva by muticasting a notification through a tree rooted at N. The tree is impicity embedded in the overay (see Figure ). We first provide some definitions before describing the process for constructing the muticast tree. For a node V with identifier k, the finger nodes of node V are defined as the successor nodes of keys r i = k + i (i =,..., 59). The finger nodes of node V distribute at exponentiay increasing cockwise distance from V. As noted in Chord [], with high probabiity, each node has O(og n) distinct finger nodes (note that, for exampe, the successors of keys r and r may be the same since r and r are cose). The new node N sits at the root of the muticast tree to announce its arriva. Among nodes in its routing tabe, node N seects its finger nodes as its chidren. Let S i and s i (i =,, j) denote the j 4

5 finger nodes and their identifiers, respectivey. Nodes S i are ranked in increasing cockwise distance from N. Node N sends each node S i a message consisting of N s identifier, N s IP address, and a muticast range (s i, s i+) of the identifier space. Node S i wi be responsibe for muticasting the notification to nodes whose identifiers are in the range (s i, s i+). Together, the j finger nodes of node N hep N muticast its arriva to a nodes in the system. Node S i uses a simiar process to expand the muticast tree with its own chidren. The purpose now is to cover nodes in range (s i, s i+). Among nodes in its routing tabe, node S i seects its finger nodes that are within range (s i, s i+) as its chidren in the tree. Let P j and p i denote the chidren of node S i and their identifiers, respectivey. Node S i asks node P i to cover range (p i, p i+), which in turn expands the tree by adding their finger nodes as chidren, and so forth. A node stops expanding the tree when it finds that there is no node in the muticast range it is assigned to. The muticast trees in h-caot are transient and purey conceptua. There is no message to construct the trees before use; no probing to maintain the trees; and no message to tear down the trees after use. Nodes expand the trees just in time based on oca information. This aows successfu muticast with inaccurate routing tabes. Suppose node S is responsibe for covering key range (a, b) and node P in that key range is missing from S s routing tabe. Node S wi not seect P as its chid in the muticast tree even if P shoud be seected based on our definition of finger nodes. This mistake, however, wi not prevent node P from receiving the notification. In the worst case, node P wi receive the notification from its predecessor as the key range narrows. When a node eaves, it notifies its predecessor and successor. The predecessor propagates this membership change to a nodes through a muticast tree rooted at itsef, using a process simiar to that for node arrivas. A node may fai without notice. Its predecessor detects this through ost heartbeats and then announces its departure. The average session duration in Gnutea is.9 hours [9], which is much shorter than the mean time to faiure (MTTF) of most modern systems. We consider most node departures as vountary rather than due to hardware or software faiures. We recommend that the overay software running on a node aways notifies its predecessor when the user coses the appication, which aows the predecessor to prompty muticast the node s departure thereby keeping the routing tabes up to date.. Handing Faiures Without fauts, each node receives a membership change notification through a muticast tree exacty once. Fauts, however, are unavoidabe. There are severa scenarios in which a notification may not be propagated to some nodes. Suppose a node S in a muticast tree asks its chid P to forward a notification to nodes in a key range that incudes nodes W,, W j. If P is no onger in the system, S wi timeout due to the missing acknowedgment from P. S deetes P from its routing tabe, and tries using another node to forward the notification. The retria may succeed but the notification has aready been deayed such that some nodes hod inaccurate routing tabes onger. If P dies after receiving and acknowedging the notification but before forwarding it, nodes W,, W j wi miss this notification atogether. Inaccurate routing tabes do not persistenty resut in faied ookups so ong as nodes propery maintain their predecessors and successors. However, inaccurate routing tabes degrade routing performance by taking more hops (when ive nodes are missing from the In impementation, the message ony needs to incude node N s IP address and node S i+ s IP address since their identifiers are simpy SHA- hashings of the IP addresses. routing tabes) or more frequenty encountering faied hops (when dead nodes are kept in the routing tabes). In a ong-running environment, it is important to ensure that errors in the routing tabes do not accumuate over time and eventuay ead to an unacceptabe routing performance. To this end, we propose node reannouncements to address the probem of missing ive nodes, and routing entry timeouts to address the probem of stae dead nodes. When node N joins, it muticasts a message to announce its arriva. Periodicay, every h seconds afterwards, if node N is sti aive, it muticasts a message to re-announce its existence. Nodes that missed previous announcements now have an opportunity to pick it up. Therefore, the number of missing nodes in a routing tabe does not accumuate over time. The period h is chosen such that the probabiity that a node ives onger than h seconds is. Assuming node ifetime foows an exponentia distribution p (t)=λ e λt, where λ = and is the average node ifetime, we have Z h p (t)dt = = h = n.7. () Nodes need to know in order to compute h. A node can ocay estimate by observing the ifetimes of nodes for which it received both birth and death notifications. When a node P receives a notification regarding the existence of a node N, P adds N into its routing tabe and associates an h second timer with this routing entry. If node N is aready in the routing tabe, node P resets the timer to h seconds. When the timer fires, node P deetes node N from its routing tabe. Ideay, if node N is aways aive, node P receives N s re-announcements periodicay and keeps N in the routing tabe. If node N dies and the notification fais to reach node P, P wi purge N from its routing tabe after the timer fires. Therefore, the number of dead nodes in a routing tabe does not accumuate over time. In summary, with timeouts and re-announcements, routing tabes become soft-state images of the system. When the system stabiizes and no fauts occur, the routing tabes converge to a correct goba view. The overhead, however, is the traffic for re-announcements as we as the cost for timer book-keeping. We quantify the traffic overhead beow. A ive node re-announces its existence every h seconds (see Equation ) and haf of the nodes eave before they make their first re-announcements. Hence, haf of the nodes make their first re-announcements, among which haf of them ive ong enough to make their second re-announcements, and so forth. The average number of re-announcements that a node makes during its ifetime is P i= ( )i =. During a node s ifetime, it muticasts a notification for its birth and death, respectivey. Adding reannouncements increases muticast messages by 5%. The benefit is a soft-state protoco that handes fauts ceany.. Traffic Anaysis In this section, we compare the tota traffic in h-caot with that in traditiona DHTs []. Simuation resuts in Section 5 show that h-caot maintains accurate routing tabes and resoves most ookups in one hop. Beow we assume one-hop routing for a ookups. Each second, there are nf ookups in tota. Lookup messages have size s and are acknowedged by packets of size.5s. The traffic to process nf ookups is L o ( +.5)s nf. () Next, we cacuate the traffic for maintenance. Each second, n nodes join. We estimate that the notifications for node arrivas have size s (see Footnote ). We assume that notifications are deivered to every node exacty once. The traffic to muticast notifications for node arrivas is 5

6 M o = ( +.5)s n n. () The coefficient.5 is because messages are acknowedged by packets of size.5s. On average, each node re-announces its existence once during its ifetime. The traffic for re-announcements is M o = ( +.5)s n n. (4) Each new node obtains a compete n-entry routing tabe from a node aready in the overay. A routing entry incudes a node P s IP address and some properties such as P s bandwidth. It is not necessary to transmit P s identifier since the identifier is simpy the SHA- hashing of the IP address. Copying a routing tabe is a buk transfer; it does not incur per entry packet overhead or acknowedgment. We estimate the traffic to transmit one entry is.5s bytes. The traffic for copying routing tabes is M o =.5s n n. (5) Each second, n nodes eave the system. The notification for a node departure contains ony the IP address of the eaving node and a propagation range. We assume that the notification has size s. The traffic to propagate node departures is M o4 = ( +.5)s n n. (6) Every T = seconds, a node sends two heartbeats, one to its predecessor and one to its successor. We assume that the heartbeat messages have size.5s. The traffic for heartbeats is M o5 =.5s n T. (7) The tota traffic (maintenance pus ookup) in h-caot is M o = L o + M o + M o + M o + M o4 + M o5 = s n(.5f + T n ). (8) Dividing M o by M dht in Equation 9, we get the reative traffic R o between h-caot and traditiona DHTs: R o = Mo 6. M dht f n og n. (9) The traffic in h-caot is dominated by the muticast traffic for routing tabe maintenance. For systems with up to a few thousand nodes, the maintenance traffic is we compensated for by the savings from efficient one-hop ookups. As a resut, h-caot can introduce ess tota traffic than traditiona DHTs. The reative traffic R o decreases as node ifetime or ookup rate f increases. n R o grows with the system size (the component), indicating that it is not economica to use h-caot for very arge sys- og n tems. This probem is not unique to our design; it is inherent in any one-hop scheme [5, 7]. Membership update traffic in one-hop schemes grows quadraticay with the system size, due to more frequent membership changes in a arge system and the fact that each change is notified to more nodes. Figure 4 pots the exact reative traffic R o in Equation 9. When n=,4 nodes, =.9 hours, and f=. ookups/second, h-caot saves traffic by %; when ookup rate f increases to.5, h-caot saves traffic by 7%. In addition to the benefit of ow traffic, h- Caot resoves ookups much faster than traditiona DHTs, i.e., in one hop as opposed to O(og n) hops. Traffic reative to traditiona DHTs ,84 nodes 4,96 nodes,4 nodes Node ifetime (hours) (a) f=. ookups/node/second. Traffic reative to traditiona DHTs ,84 nodes 4,96 nodes,4 nodes Lookup rate (ookups/second) (b) Node ifetime =.9 hours. Figure 4: Reative traffic between h-caot and traditiona DHTs (the exact R o in Equation 9, which grows with O( f )). 4. H-CALOT FOR TWO-HOP ROUTING When the system is very arge, efficient one-hop routing is no onger feasibe. This is because the maintenance traffic in onehop schemes grows quicky with O(n ) (see Equations 7 and 8). By contrast, the maintenance traffic in two-hop schemes that use O( n) routing tabes grows with O(n.5 ) (see Equation 8). When n is arge, the difference between O(n ) and O(n.5 ) is significant, for instance, when n= 6, n /n.5 =. Figure (b) shows that, even for very arge systems (up to miion nodes), the tota traffic of an idea two-hop scheme can sti be ower than that of traditiona DHTs []. Moreover, two-hops schemes resove ookups in two hops, much faster than traditiona DHTs. Our goa, therefore, is to design a practica two-hop protoco that approaches the performance of the idea two-hop scheme. The main chaenge is to maintain the arge O( n) routing tabes in the face of frequent node joins and eaves and to do two-hop routing with the O( n) routing tabes in a peer-to-peer fashion. To this end, we propose our h-caot protoco. Unike existing hierarchica two-hop protocos [7], h-caot is purey peer-to-peer. Beow, we first present a basic version of h-caot and then describe how to make it adaptive. 4. The Basic h-caot h-caot is a further deveopment of h-caot. It aso organizes nodes into a ring topoogy. The basic version of h-caot partitions the ring into continuous regions of equa size caed sices (see Figure 5(a)), and runs a protoco simiar to h-caot inside each sice. A membership change that happens in a sice is ony propagated to nodes in the same sice. Inside a sice, nodes know each other. h-caot resoves a ookup in two hops. The first hop routes the ookup between the source sice and the destination sice. The second hop deivers the ookup within the destination sice. Since nodes in the same sice know each other, the second hop is trivia. The chaenge is to route between two arbitrary sices in one hop. For this purpose, each computer N runs two virtua nodes, N and N, caed sister nodes. N s identifier is the SHA- hashing of N s IP address and N s identifier is the SHA- hashing of N s identifier (i.e., doube hashing of N s IP). Beow we refer to virtua nodes simpy as nodes. To route a message between two sices, h-caot tries to find a node in the source sice whose sister node sits in the destination sice to forward the message. That is, sister nodes act as gateways to connect different sices. Suppose there are a arge number of nodes S i (i =,, j) in a sice S. The sister nodes P i of nodes S i are randomy distributed a over the identifier space because the node identifiers are generated randomy. Given an arbitrary destination sice D, with high probabiity, one of these sister nodes P i may sit in sice D. In other words, with high probabiity, we can find a pair of sister nodes to connect sices S and D. 6

7 st hop n d hop (a) Iustration of h-caot. Probabiity at east pair of sisters to connect sices at east pairs of sisters to connect sices c = #nodes_in_a_sice / #sices (b) Prob. of finding sister nodes to connect two random sices. Figure 5: Highights of the basic version of h-caot. Figure 5(a) is an iustration of h-caot. Nodes at the two ends of a dashed ink are sister nodes, e.g., nodes u and u. Suppose node s in sice S wants to route a message to the node in sice D that is responsibe for key d. Node s searches its routing tabe for a node u in the oca sice S whose sister node u resides in the destination sice D. Node s sends the message to node u. Nodes u and u are two virtua nodes running on the same computer. Node u then directy forwards the message to the destination since node u knows a nodes in sice D. We next derive a proper configuration for h-caot. We want the sices to be sma such that the traffic for membership updates inside sices is ow. But we aso want the sices to be sufficienty arge such that the probabiity of finding two sister nodes to connect two random sices is high. Let k denote the number of sices, m denote the number of nodes in a sice, and n denote the number of computers. k m = n since each computer runs two virtua nodes. Let c = m be the main parameter for h-caot. We have k number of sices: k = p n/c () number of nodes in a sice: m = cn. () Let S and D denote two random sices. There are k sices in tota. The sister of a node is randomy distributed in the identifier space. For a node in sice S, the probabiity that its sister node is in sice D is p =. Among the m nodes in sice S, on average k c = m/k nodes have sister nodes in sice D. The probabiity that exacty x nodes in sice S have sister nodes in sice D foows a Binomia distribution: «m P (X =x) = p x ( p) m x e c cx () x x! P (X ) = P (X=) e c () P (X ) = P (X=) P (X=) e c ce c. (4) The approximation above expoits the fact that this Binomia distribution approaches a Poisson distribution when m is arge. P (X ) is the probabiity that there exists at east one pair of sister nodes to connect two random sices; P (X ) is the probabiity that there exist at east two pairs of sister nodes to connect two random sices. Figure 5(b) pots P (X ) and P (X ). This figure shows that, with high probabiity, we can find sister nodes to connect two random sices. Hence the two-hop routing in Figure 5(a) can be accompished. We opt for configuration c = 5. c = 5 = m = n, k =.4n, (5) P (X ).99, P (X ).96 (6) With this configuration, the probabiity of finding more than one pair of sister nodes to connect two sices is aso high (.96). This offers an opportunity to consider network proximity when routing messages among sices. If there exists more than one node to reach the destination sice, we can choose the node that has the owest atency to forward the message. Currenty, our simuator does not expoit proximity-aware routing. 4. Making h-caot Adaptive Ideay, the number of nodes in a sice (m = cn) and the number of sices (k = p n/c) shoud automaticay adapt as the system size n changes. In existing soutions for two-hop routing [7, 8], nodes need to unanimousy agree upon the number of sices, making it impossibe to do decentraized adaptations based on ony oca knowedge. Beow, we show how to make h-caot adaptive. The key observation is that, the use of sices in h-caot is competey artificia. So ong as a node knows a sufficient number of nodes randomy distributed in the identifier space, given a message to any destination, it can route the message in one hop to a pace very cose to the destination by using one of those random nodes (the first hop). Furthermore, so ong as each node knows a sufficient number of neighbors aong the ring, the message can be deivered to its destination in one hop when it is aready at a pace very cose to the destination (the second hop). Hence, h-caot is abe to accompish two-hop routing without using sices. More specificay, the routing tabe of a node N incudes its m cockwise neighbors aong the ring and m counter-cockwise neighbors aong the ring. We refer to the continuous range in the identifier space that spans over these m neighbors as node N s neighbor zone. Neighbor zones essentiay repace the roe of sices in Figure 5(a). Unike the fixed sices, each node has its own neighbor zone centered at itsef and need not know the neighbor zones of others. Beow we aways assume that, whenever a node N knows about a node P, N automaticay knows about P s sister. Hence there are actuay m nodes in a node s routing tabe: m nodes in its neighbor zone (the neighbor set ) and their m sisters (the sister set ) that are randomy scattered in the identifier space. The routing agorithm is the same as that in h-caot. Given a ookup, a node N greediy forwards the ookup to the node P that is, to N s knowedge, the cosest in absoute distance to the ookup key. Node P either returns the object or further forwards the ookup greediy. When a node N searches its routing tabe for a node P that is cosest to the destination, N does not distinguish between whether P is from its neighbor set or its sister set. Nodes have no notion of sices either. The ony rue is greedy forwarding. As in h-caot, correct routing is guaranteed so ong as each node correcty maintains its predecessor and successor. 4. Routing Tabe Maintenance The maintenance protoco for h-caot is simiar to that for h- Caot but with a major difference: when a node joins or eaves, the muticast notification is ony sent to its m cockwise neighbors and m counter-cockwise neighbors aong the ring, rather than a nodes in the system. Nodes do not know the exact number n of computers in the system. They estimate n and m from oca knowedge. The processes for announcing node arrivas and departures are simiar. Beow we use a node arriva as the exampe. When a new computer N joins, it functions as two virtua nodes N j (j=, ). N s identifier is the SHA- hashing of N s IP address and N s identifier is the SHA- hashing of N s identifier (i.e., doube hashing of N s IP). Nodes N and N execute the same protoco but function independenty as if they were rea nodes. Beow we use N j to refer to either of them. Node N j joins the ring topoogy and obtains a copy of the routing tabe from its predecessor P. Suppose the routing tabe incudes a tota of y neighbors of P, either cockwise or counter-cockwise. The neighbors of node 7

8 P are aso neighbors of node N j. Node N j adds into its routing tabe the y neighbors and node P. Suppose the size of the continuous region of the identifier space spanned over by the y + neighbors (incuding nodes P and N j) is z. N j estimates the tota number of computers in the system as estimated tota computers: n = 6 (y + ). (7) z The size of N j s neighbor zone is estimated as b = 6, where k k = p n/c. Suppose N j s identifier is d. N j s neighbor zone is estimated neighbor zone: K = [ d b, d + b ]. (8) Note that the operations are in moduo 6. Node N j purges from its routing tabe neighbors that are outside K. Different nodes may estimate the sizes of their neighbor zones differenty. Since the sices (neighbor zones) are configured to be sufficienty arge (c = m = 5), the variance of the estimation is we toerated. With k high probabiity, a node can forward a message in one hop to any region in the identifier space through the sisters of nodes in its neighbor zones. Node N j needs to muticast a notification about its arriva to a nodes in its neighbor zone K. The muticast process is simiar to that of h-caot but the notification is propagated both cockwise and counter-cockwise. In h-caot, the finger nodes of a node are defined as the successor nodes of keys r i = k+ i (i =,..., 59). In h-caot, the forward-finger nodes of a node are defined as the successor nodes of keys r i = k + i (i =,..., 58) and the backward-finger nodes are defined as the predecessor nodes of keys r i = k i (i =,..., 58). In the identifier space, the finger nodes of a node distribute at exponentiay increasing distance from the node, either cockwise or counter-cockwise. The muticast process to cover nodes in node N s neighbor zone K = [d b, d + b] works as foows. Node N spits K into a backward range K b = [d b, d] and a forward range K f = [d, d + b]. It muticasts notifications through two different trees T b and T f to cover ranges K b and K f separatey. The tree T f is constructed using the inks between nodes and their forward-finger nodes. The muticast process over tree T f is exacty the same as that in h-caot (see Figure ). The muticast process in tree T b is the same as that in tree T f except that the notification traves over inks between nodes and their backward-finger nodes. Like h-caot, h-caot aso uses timeouts and re-announcements to make the routing tabes soft-state images of the system. Before a re-announcement, a node aways re-estimates the system size n and its neighbor zone K. The re-announcement wi cover nodes in the updated neighbor zone. This heps nodes with a ong ifetime adapt as the system evoves. 4.4 h-caot vs. Other Two-hop Schemes Existing protocos for two-hop routing aso partition the overay into sices and nodes in the same sice know each other [7, 8]. There are severa major differences between h-caot and these hierarchica protocos. () h-caot is purey peer-to-peer and extremey simpe. Each node knows O( n) neighbors aong the ring That s it! Notification muticast uses these neighbors; routing aso uses these neighbors. There are neither sices nor muticast trees to maintain; both are conceptua. By contrast, existing hierarchica protoco [7] partitions the overay into units and sices, and designates nodes as sice eaders, unit eaders, ordinary nodes, and sice representatives. Nodes have different roes and run different protocos. () h-caot distributes oad eveny across nodes. Each pair of sister nodes carry some traffic between two sices. By contrast, for each sice, existing protocos seect a few nodes to act as gateways to carry a incoming traffic from other sices. () h-caot estimates neighbor zones from oca knowedge and adapts as the system evoves. By contrast, existing protocos use fixed sices and cannot adapt easiy. 4.5 Traffic Anaysis We compare the traffic in h-caot with that in traditiona DHTs [] through anaysis. The process is simiar to that for h-caot, but there are n virtua nodes for a system with n computers and each notification is sent to ony m = cn = n nodes. Simuation resuts in Section 5 show that h-caot maintains very accurate routing tabes and resoves most ookups in two hops. As an approximation, we assume two-hop routing for a ookups. Each second, there are nf ookups in tota. Lookup messages have size s and are acknowedged by packets of size.5s. The traffic to process nf ookups is L t ( +.5)s nf. (9) Next, we cacuate the traffic for maintenance. Each second, n new nodes (or n computers) join. Conceptuay, the notification for a node join incudes its IP address, its identifier, its sis- ter node s identifier, and a boundary and direction (cockwise or counter-cockwise) for the notification to be propagated. We estimate the notification message has size s. (Note that the identifiers and boundaries are simpy SHA- hashings of the IP addresses and we need not transmit them. See Footnote ). We assume that each notification is deivered to m = n nodes exacty once. The traffic to muticast notifications for node arrivas is M t = ( +.5)s n m. () The coefficient.5 is because messages are acknowedged by packets of size.5s. On average each node re-announces its existence once during its ifetime. The traffic for re-announcements is M t = ( +.5)s n m. () Each new node copies a routing tabe from its predecessor. Conceptuay, the routing tabe incudes m neighbors and the sisters of those m neighbors. We ony need to transfer the IP addresses of the m computers since the m identifiers are just SHA- hashings of the IP addresses. Copying a routing tabe is a buk transfer; it does not incur per entry packet overhead or acknowedgment. We estimate the traffic to transmit one entry of the routing is.5s. The traffic for copying routing tabes is n M t =.5s n m () Each second, nodes eave the system. The notification for a node departure contains ony the IP address of the eaving node and a propagation range. We assume that the notifications have size s. The traffic to propagate node departures is M t4 = ( +.5)s n m. () Every T = seconds, a node sends two heartbeats, one to its predecessor and one to its successor. We assume that the heartbeat messages have size.5s. There are n nodes in tota. The traffic for heartbeats is M t5 =.5s 4n T. (4) The tota traffic (maintenance pus ookup) for h-caot is M t = L t + M t + M t + M t + M t4 + M t5 = s n(f + T n ). (5) 8

9 Traffic reative to traditiona DHTs.5.5.5,48,576 nodes,7 nodes 6,84nodes Node ifetime (hours) (a) Traffic reative to traditiona DHTs.5.5.5,48,576 nodes,7 nodes 6,84nodes Lookup rate (ookups/second) (b) Average ookup hops Node ifetime (minutes) (a), nodes. Average ookup hops Nodes (b) hour node ifetime. Figure 6: Reative traffic between h-caot and traditiona DHTs (the exact R t in Equation 6, which grows with O( f )). (a) Vary node ifetime (ookup rate f=.). (b) Vary ookup rate (node ifetime =.9 hours). Comparing Equations 8 and 5, we see that h-caot is more scaabe than h-caot. h-caot s traffic grows with O(n.5 ) whie h-caot s traffic grows with O(n ). Dividing M t by M dht in Equation 9, we get the reative traffic R t between h-caot and traditiona DHTs: R t = Mt 4 M dht f n og n. (6) Like h-caot, the traffic for h-caot is dominated by membership updates. The maintenance traffic, however, is we compensated for by the savings from efficient two-hop ookups when f. Figure 6 pots the exact R t in Equation 6. When n=,7 computers and f=. ookups/second, h-caot saves traffic by %; when ookup rate f increases to.5, h-caot saves traffic by 6%. When the ookup rate f further increases to, h-caot saves traffic by 6% even for a,48,576-node system. In addition to the benefit of ow traffic, h-caot resoves ookups much faster than traditiona DHTs, i.e., in two hops as opposed to O(og n) hops. 5. EXPERIMENTAL RESULTS We buit an event-driven simuator to evauate h-caot and h- Caot. The simuator consists of 5,5 ines of C++ code. It simuates a compete system, incuding dynamic node arrivas and departures, timeouts, and network deays. We do not simuate the network-eve packet detais. Limited by the GB memory of our computers, we can simuate h-caot with up to, nodes and h-caot with up to 6, computers (i.e.,, virtua nodes). Modeing network topoogies and atencies is sti an open research topic. We foow the approach [6] that focuses on ensuring the simuated network atencies foow the distribution of rea network atencies in the Internet. In our simuator, the network atencies between nodes are randomy samped from the King dataset [], which is extracted from rea measurements of the round-trip times (RTTs) between,48 DNS servers. We divide the RTTs by two to obtain one-way atencies. Excuding the empty entries in the RTT matrix, the average one-way atency is 9ms. Uness otherwise noted, the simuation works as foows. The system starts with one node and continuousy adds more nodes unti the popuation reaches n. From then on, node arriva is a Poisson process with rate λ e = n. Node ifetime foows an exponentia distribution with a mean. The system popuation stabiizes around n as nodes join and eave. After the system undergoes n membership changes, i.e., a tota of n nodes have joined or eft the system, the simuator enters the evauation phase. It takes a snapshot of the routing tabes and uses them to evauate routing performance, during which each node on average issues, random ookups. Our simuator modes the ookups that a node issues as Faied hops per ookup Figure 7: Routing hops per ookup in h-caot Node ifetime (minutes) (a), nodes. Faied hops per ookup Nodes (b) hour node ifetime. Figure 8: Faied routing hops per ookup in h-caot. a Poisson process with rate f. However, uness reated statistics are needed, the simuator does not fuy execute the ookups issued before the evauation phase. We found this optimization important to make the simuation time manageabe when the system size is arge. This optimization makes the reported ookup performance more pessimistic because the overooked ookups can hep detect and fix some inaccurate entries in the routing tabes. Beow, we present resuts regarding various aspects of h-caot and h-caot, incuding traffic, routing performance, resiience in the face of membership changes, and the abiity to adapt as the system size evoves. 5. h-caot We first present resuts on h-caot. Figures 7(a) and 7(b) show the average routing hops per ookup when varying node ifetime and system size, respectivey. In both figures, the ookup hops are very cose to one, indicating that the routing tabes are very accurate. For instance, with a one hour ifetime and one thousand nodes, the average routing hops are ony.8. In Figure 7(a), the routing performance improves as the node ifetime increases. The absoute improvement, however, is sma because the routing hops are aready very cose to one. Figure 8 reports the average number of faied hops encountered per ookup. (Note that a faied hop does not necessariy ead to an irresovabe ookup. The system aways retries aternative routing paths.) Both missing ive nodes and isted dead nodes can ead to inaccurate routing tabes, among which the atter is particuary harmfu as it significanty increases ookup atencies. In our simuator, it takes a timeout that is 8 times of the average one-way network atency to detect a faied hop before trying an aternative. Figure 8(a) shows that the faied hops w reduce dramaticay as the node ifetime increases. With a haf an hour ifetime, w=.6; with a one hour ifetime, w=.5 ( faied hop out of, ookups). Comparing Figures 8(a) and 8(b), we see that the faied hops are much more sensitive to node ifetime than to system size. Comparing Figures 7(a) and 8(a), we find that the number of faied hops is a more reveaing metric of h-caot s performance than the number of ookup hops because of the high cost of faied hops. 9