Implementing Aggregation/Broadcast over Distributed Hash Tables

Transcription

1 1 Implementing Aggregation/Broadcast over Distributed Hash Tables Ji Li Ben Leong Karen Sollins Abstract We present an algorithm for implementing aggregation/broadcast over existing Distributed Hash Tables (DHTs). Our algorithm builds an aggregation tree in a bottom-up fashion by mapping nodes to their parents in the aggregation tree with pre-determined function and maintains it using soft-state. We believe that the aggregation/broadcast function is a basic service that should be layered over existing DHTs to provide them with the ability to disseminate/collect information efficiently on a global scale and perhaps even to search for answers to arbitrary semistructured queries. I. INTRODUCTION A Distributed Hash Table (DHT) is essentially a distributed resolution mechanism that manages the distribution of data among a changing set of nodes by mapping keys to nodes. DHTs allow member nodes to efficiently locate stored resources by name without using any centralized servers. A large number of DHTs, including Chord [1], Tapestry [2], Pastry [3], and CAN [4], have thus far been proposed and these systems are expected to eventually become the fundamental components of many large scale peer-to-peer distributed applications in the near future. Although the most commonly deployed peer-to-peer applications are file-sharing applications like Napster [5], Gnutella [6], Kazaa [7], and Freenet [8], we believe that the peer-to-peer paradigm will eventually be applied to the organization of centrally administered distributed systems. As highlighted by Bawa et al. [9], the administrators of such systems may want to collect statistical information about the network, even though they may not have direct control over the membership changes. Aggregated information on system parameters, such as network size, lifetime distribution, most popular files, etc., may be used to tune and optimize the performance of a network at runtime. In particular, the recently proposed randomized DHT topologies, like Viceroy [1] and Symphony [11], often require network size estimates to tune routing performance. In the case of a peer-to-peer storage network ([12], [13]), users may want to know about the approximate amount of available storage; in the case of a peer-to-peer computational Grid [14], nodes may want to know about the amount of computation resources available when deciding on how to schedule jobs in the network. More specifically, like Bawa et al. [9], our goal is to solve the node aggregation problem defined as: Devise a scheme to enable any node in a P2P network to issue a query that computes an aggregation function (MIN, MAX, COUNT, SUM, AVG) over data residing at nodes in the network. In our view, a good aggregation/broadcast scheme for DHTs must satisfy two criteria, namely scalability and robustness. With respect to scalability, we want to avoid flooding schemes that generate excessive redundant messages and we also want to ensure that there is good loadbalancing, in the sense that no node in the network should be liable for forwarding a disproportional amount of network traffic. In terms of robustness, we want a scheme that is resilient to the dynamicity of nodes joining, leaving and failing arbitrarily and independently. In this paper, we assume that we have at hand a robust and efficient DHT and our algorithm uses the underlying infrastructure to build efficient aggregation/broadcast trees. Our key idea is to use a family of functions to map a node to its parent node in an aggregation tree, to construct the tree in a bottom-up fashion and to maintain it using soft-state. Our proposed bottom-up approach is better than traditional broadcast based approaches because it generates a minimal amount of network traffic and we are guaranteed to be able to reach all nodes (eventually). II. RELATED WORK Data aggregation over a distributed system has been studied in different contexts such as large scale databases [15], wireless sensor network applications ([16], [17]), etc. In particular, Madden et al. [16] proposed a scheme for declarative aggregate queries to be distributed and executed over sensor networks. They focused mostly on bandwidth- and power-efficient aggregation. We attempt to solve the problem proposed by Bawa et al. [9] with a bottom-up approach and in a different con-

2 2 text. In [9], Bawa et al. assume that the peer-to-peer network is unstructured, and hence their proposed schemes are limited to top-down flooding-based techniques. From our work with EpiChord [18], we believe that it is feasible to build and maintain a underlying robust DHT infrastructure at relatively low cost and thus it is useful to attempt to solve the aggregation problem over DHTs. Our approach in using a continuous function to map nodes onto nodes is similar in spirit to the generalized scheme for building DHTs proposed by Naor and Wieder [19]. In their scheme, the functional mapping is used to build the underlying DHT, while in our scheme, we assume a robust DHT and we seek instead to build an additional structure over the underlying DHT infrastructure. Most of the recently proposed randomized DHT routing protocols ([2], [1], [11], [18]), require network size estimates to optimize lookup performance. Most of the current techniques rely on the assumption that node s are uniformly distributed within the address space, and estimates are obtained by extrapolating from observed node densities. These techniques typically require messsages and provide only probabilitistic guarantees on the accuracy, while our scheme requires only messages and have somewhat better accuracy guarantees independent of the actual distribution of node s. By solving the aggregation problem, we have also indirectly solved the broadcast problem. There is a large body of literature in this area. In general, existing schemes can be categoried either as a top-down spanning-tree approach ([21], [22]) or a flooding-based approach ([23], [24]). A major drawback of the former is that when nodes fail in the middle of the hierarchy, large sections of the original spanning tree lose contact with the root, while a major drawback of the latter tend to generate redundant traffic. There are also many related DHT-based application-level multicast proposals ([25], [26], [27]). III. AGGREGATION BY FUNCTIONAL MAPPING In this section, we describe our bottom-up tree construction algorithm. We assume that we have an efficient and robust DHT available as the basic infrastructure 1. The key idea in our aggregation tree-building algorithm is to use a many-to-one function,, to map each node uniquely to a parent node in the aggregation tree based on its. More specifically, the parent node for a node is the node which owns the, where is the minimum number of times that has to be applied In general, any robust DHT will suffice. The only restriction/requirement that we impose is that the DHT must have a circular namespace, i.e. Chord, Tapestry or Pastry. to before the node no longer owns the. If node owns for! #"%$%$%$&"(', then is the root of the aggregation tree. If we consider the nodes in a DHT as nodes in a graph and the child-parent relations determined by to be directed edges, the resulting graph is a directed tree converging at the root. More specifically,, which we also call a Parent Function, is any function that satisfies the following conditions: *)+, ) (1) -. / )"+ (2) (":)+<; (":)+(" =>"? A@CB (3) where ) is an owned by the root of the aggregation tree, is any valid node 3 and "?DE is a distance metric defined for two valid node s and D in the namespace. Theorem 1: If a function satisfies the above conditions, there is a directed path from all nodes to the (root) node that owns the id ). Proof: Theorem 1 is a direct consequence of Condition (2). Given an arbitrary node, We know that there is a path from to ("F>,@!B. Since - GH), there must therefore be a directed path from to ). A. Tree-Building Protocol A node learns about the parent function,, for a DHT when it attempts to joins the network through an existing node. Given a parent function, the tree-building protocol is straightforward: 1) When a node joins the system, it determines and performs a lookup to locate the node responsible for I, say node D. 2) Node sends a message to node D to register itself as a child and node D adds node to its list of child nodes. 3) It is henceforth the responsibility of node to contact node D periodically to refresh its status as a child node; if node D fails to hear from node after a specified expiry duration, node will be deleted from its list of child nodes. 4) As the system evolves, it is possible that node D may fail or it may no longer be responsible for. In the former, node will discover node D s failure when it tries to refresh its status with node D ; in the latter, D will inform node that it is no longer the parent when node tries to refresh its status. In both cases, node will simply perform another lookup for to determine its new parent and register with it instead.

3 B ' ) B B. A Simple Parent Function There are many functions that will satisfy Conditions (1) to (3) above. The following is a simple PSfrag replacements example:, mod for B ; mod for ; C where is a node, and is a parameter that determines address space bits the branching factor of the tree, and. As shown in Figure 1, the aggregation tree resulting from this function is rooted at the node owning the B. By direct substition, we can easily verify that this function satisfies the three conditions stated above. The expected height of a spanning tree constructed with this function is 6, where is the size of the network and the expected branching factor is approximately if the node are uniformly distributed in the namespace. PSfrag replacements increasing Fig. 1. Aggregation pattern of the simple parent function. is the root id, and the arrows show the directed edges of the aggregation tree from child to parent. C. A More General Parent Function We can generalize the function described in Section III- B to allow for an arbitrary root ) by a simple change of coordinates. The resulting parent function is as follows:, ) mod! mod (" " for B ; # ) mod $ )% & mod! mod (" for ; )+ mod $ C The resulting aggregation tree is illustrated in Figure 2. A nice property of this aggregation/broadcast tree is that if the node s are uniformly distributed over the address space (which a common assumption for many of the existing DHTs), the resulting tree is likely to be balanced. Nodes s for which )+ mod )(+* B" will converge clockwise to the root, while node s for which, ) G mod -(.* "? will converge counterclockwise. increasing / Fig. 2. Aggregation pattern of the generalized parent function with as the root of the tree. The arrows show the directed edges of the aggregation tree from child to parent. D. An Example of Aggregation - Network Size Estimation Although our goal is to compute aggregation functions (MIN, MAX, COUNT, SUM, AVG), we validate our approach with by evaluating a network size estimation function, since the key to obtaining good estimates of the various aggregation functions is to have robust directed paths from each node to the root of the aggregation tree. To estimate network size, we simply enhance the tree-building protocol described in Section III-A as follows: 1) When a node first joins the system and locates its parent node, it tells its parent node that it has a subtree size of 1. 2) When a node refreshes its status with its parent, it will report its subtree size which is the sum of the subtree sizes of all its child nodes plus one (i.e. itself) to the parent. 3) Eventually, the root of the aggregation tree will have an estimate of the size of the network. Once the root of the aggregation tree obtains an estimate of the network size, what it does with the estimate is a matter of policy and depends on the application/dht. In DHT where nodes require the network size estimate to optimize their performance, the root may wish to broadcast the estimate down the aggregation tree periodically. ) IV. PRACTICAL IMPLEMENTATION We have described an approach for building an aggregation tree rooted at a fixed 3 ). In a practical system, we will want to have the capability to set up aggregation trees rooted at arbitrary node s. To this end, our proposal is to have one permanent aggregation/broadcast tree rooted at a pre-determined well-known )1 for each DHT. If the default tree is the only avaiable broadcast tree in the system and a node, say, wants to construct a new aggregation/broadcast tree rooted at itself, it sends a message to the root of the default tree specifying the parent function with ) as its own. The message is then broadcast down 3

4 4 the default tree and all nodes will eventually receive this message and participate in the new aggregation/broadcast tree rooted at. When a new node joins the network, it will get a list of all currently existing aggregation trees from its parent node. Nodes also periodically exchange information on the existing aggregation trees. In this way, all nodes will eventually discover and participate in all existing aggregation trees. In order to avoid overloading the default broadcast tree, nodes that wish to set up new broadcast trees will randomly choose among all the other existing broadcast trees to perform its initial broadcast, if possible. When the root node of a tree wishes to tear down its aggregation tree, it simply broadcasts a message telling all nodes that the tree should be torn down. The message is guaranteed to eventually reach all nodes. If a root fails without notification, subsequent aggregation messages will eventually end up at a node which discovers that it has become the root of tree that it did not set up. It can decide to keep the tree or it can broadcast a message to tear it down. As highlighted by Bawa et al. in [9], one major drawback of using trees for aggregation is that trees inherently present single points of failure, i.e. the failure of any node in the intermediate levels of the tree will lead to a large number of nodes being cut off temporarily from the root. Although our bottom-up approach is inherently more robust and resilient to failures than existing top-down approaches, aggregates obtained by a single-tree approach will experience periodic glitches in a highly dynamic network. To improve the robustness of the estimates, multiple trees can be constructed and the same aggregates can be computed over them. By distributing the s of these trees over the 3 space uniformly and adopting some quorum system or by averaging over the estimates obtained from several trees, we can further improve the robustness of the aggregation computation. V. PERFORMANCE We evaluated our aggregation algorithm by evaluating the accuracy of the network size estimates obtained via simulation. Our simulations were performed with the Epi- Chord [18] network simulator written in Java. Our experiments were run on single-processor Intel P4-2.2GHz machines with 1GByte RAM running Linux with Java(TM) 2 Runtime Environment version A. Simulation Setup Our experiments are divided into discrete time periods. In each period, a number of nodes (determined according to a Poisson distribution) are created and join the network at a randomly chosen node. The node of a new node is uniformly distributed within the namespace. At the point when a node joins the network, we set its departure time according to an exponential distribution and nodes are removed from the network when their lifespans expire. In all our simulations, the default root ) is set to, and is set to 4. In our simulations, we have assumed that the estimates of network size are broadcast to all nodes down the aggregation tree and that the broadcast is done separately from the aggregation. It is possible to adopt another approach where estimates from a parent node to its child nodes are piggybacked on the acknowledgement messages, but we decided to evaluate the two processes separately because the periodicity of the two processes can conceivably be different depending on the application. B. Tree Construction In the first experiment, we evaluated the overhead of our algorithm in terms of network traffic by counting the number of messages sent in both the aggregation and broadcast process. In this experiment, the node failure rate is approximately 1% per time period. The communication cost in terms of messages sent as a function of network size is shown in Figure 3. The number of messages in each round of aggregation or broadcast are at least twice the network size, because in the aggregation each node sends an aggregation message to its parent and receives an acknowledgement (except for the root), and in the broadcast process each node receives a broadcast from its parent and sends back an acknowledgement (except for the root). Additional messages are needed to repair the tree when nodes fail. The children of a failed node have to perform a lookup to find their new parent, which requires extra messages 2. Figure 3 demonstrates that at a node failure rate of 1% per time period, the number of aggregation messages and broadcast messages are only a little more than twice the network size. The height of the resulting aggregate/broadcast tree affects the performance of the algorithm since it determines the time it takes for an aggregation message to propagate from a leaf node to the root and also the time it takes for a broadcast message from the root to propagate to the leaf nodes. Another important property of the aggregation tree is the distribution of the branching factor at the intermediate nodes as it directly affects load balancing. Ideally, The number of lookup messages is dependent on the underlying DHT. EpiChord is relatively efficient and provides a worst case lookup performance of messages in the worst case and a modal lookup performance of messages in most cases.

5 5 35 Aggregation messages Broadcast messages 16 Actual Network Size Average Estimated Network Size Number of messages 2 15 Network size Network size Fig. 3. Communication overhead (in terms of messages) for aggregation and broadcast Time period Fig. 5. Network size estimation. we would like all the intermediate nodes in the aggregation tree to have approximately the same branching factor for all nodes and to have a small spread in the range of branching factors. The relation between tree height, branching factor and the size of the network is shown in Figure Average height of aggregation tree Average branching factor Maximum branching factor 4 35 In order to investigate the robustness of our aggregation/broadcast algorithm, we ran experiments with networks of differing node failure rates. The maximum, minimum and average ratios of the size estimate to the actual network size are shown in Figure 6. Even at a node failure rate of 3%, i.e. 3% of the nodes in the network leave without notification in each time period, our algorithm still maintains an accuracy to within #B of the actual network size, which demonstrates that our algorihm is quite robust. Average height of aggregation tree Network size Fig. 4. Plots of tree height and branching factor against network size. Average branching factor Estimated size (as a proportion of total network size) Average Range of estimates C. Network Size Estimation In Figure 5, we show the evolution of the network size estimate at the root of the aggregation tree for a network with a node failure rate of 5% per time period. As the network is growing, the estimates tend to be smaller than the actual network size as expected. The downward spikes are likely to be due to the failure of intermediate nodes high up in the aggregation tree, leading to temporarily losses in the node count. Our results demonstrate however that our algorithm recovers rapidly from such failures. Fig Node failure rate Network size accuracy under various node failure rates. VI. CONCLUSION AND FUTURE WORK In this paper, we have presented a bottom-up approach for the building of an aggregation/broadcast tree over a DHT by mapping from a continuous function into the discrete node space. The major advantage of our bottom-

6 6 up function-based aggregation scheme compared to existing top-down and flooding-based approaches, is that it has a relatively low overhead and yet is guaranteed to eventually reach all nodes in the network. By having each node determine its parent in the aggregation tree at runtime based on the parent function, the system is robust and recovers quickly from node failures. Our scheme is also flexible in that parameters in the function can be changed to control the structure and characteristics (i.e. height and branching factor) of the resulting aggregation tree. There are many possible parent functions that can conceivably be adopted in our approach. We have only thus far explored a very simple one. It is conceivable that by adopting a different, we may be able to improve the performance of the algorithm. REFERENCES [1] Ion Stoica, Robert Morris, David Karger, Frans Kaashoek, and Hari Balakrishnan, Chord: A scalable Peer-To-Peer lookup service for internet applications, in Proceedings of the 21 ACM SIGCOMM Conference, 21, pp [2] B. Y. Zhao, J. D. Kubiatowicz, and A. D. Joseph, Tapestry: An infrastructure for fault-tolerant wide-area location and routing, Tech. Rep. UCB/CSD , UC Berkeley, April 21. [3] Antony Rowstron and Peter Druschel, Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems, Lecture Notes in Computer Science, vol. 2218, pp. 329??, 21. [4] Sylvia Ratnasamy, Paul Francis, Mark Handley, Richard Karp, and Scott Shenker, A scalable content addressable network, Tech. Rep. TR--1, Berkeley, CA, 2. [5] Napster, [6] Gnutella, [7] KaZaA, [8] Ian Clarke, Oskar Sandberg, Brandon Wiley, and Theodore W. Hong, Freenet: A distributed anonymous information storage and retrieval system, Lecture Notes in Computer Science, vol. 29, pp. 46+, 21. [9] M. Bawa, H. Garcia-Molina, A. Gionis, and R. Motwani, Estimating aggregates on a peer-to-peer network, in Submitted for publication, 23. [1] Dahlia Malkhi, Moni Naor, and David Ratajczak, Viceroy: A scalable and dynamic emulation of the butterfly, in Proceedings of 21st ACM Conf. on Principles of Distributed Computing (PODC 2), July 22. [11] Gurmeet Manku, Mayank Bawa, and Prabhakar Raghavan, Symphony: Distributed hashing in a small world, in Proceedings of 4th USENIX Symposium on Internet Technologies and Systems, March 23. [12] Frank Dabek, A cooperative file system, M.S. thesis, MIT, September 21. [13] Athicha Muthitacharoen, Robert Morris, Thomer M. Gil, and Benjie Chen, Ivy: A read/write peer-to-peer file system, in Proceedings of 5th Symposium on Operating Systems Design and Implementation, 22. [14] Ben Paechter, T. Baeck, M. Schoenauer, A.E. Eiben, and J. Merelo, A distributed resource evolutionary algorithm machine, in Proceedings of the 2 Congress on Evolutionary Computation (CEC 2), San Diego, USA, 2, IEEE, pp , IEEE Press. [15] Jim Gray, Surajit Chaudhuri, Adam Bosworth, Andrew Layman, Don Reichart, Murali Venkatrao, Frank Pellow, and Hamid Pirahesh, Data cube: A relational aggregation operator generalizing group-by, cross-tab, and sub-totals, J. Data Mining and Knowledge Discovery, vol. 1, no. 1, pp , [16] S. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong., Tag: a tiny aggregation service for ad-hoc sensor networks, in 5th Symposium on Operating Systems Design and Implementation (OSDI), 22. [17] Jerry Zhao, Ramesh Govindan, and Deborah Estrin, Computing aggregates for monitoring wireless sensor networks, in Proc. 1st IEEE Intl. Workshop on Sensor Network Protocols and Applications (SNPA), 23. [18] Ben Leong, Erik D. Demaine, and Ji Li, Epichord: Implementing parallized DHT lookup with epidemic caching., July 23. Submitted to IEEE Infocomm 24. [19] Moni Naor and Udi Wieder, Novel architectures for P2P applications: the continuous-discrete approach, 22. [2] Gurmeet Singh Manku, Routing networks for distributed hash tables, in Proceedings of the 22nd ACM Symposium on Principles of Distributed Computing (PODC 23), Boston, Massachusetts, July 23, ACM. [21] Sameh El-Ansary, Luc Onana Alima, Per Brand, and Seif Haridi, Efficient broadcast in structured P2P networks, in 2nd International Workshop on Peer-to-Peer Systems (IPTPS 3), Berkeley, CA, 23. [22] Y. Chawathe and M. Seshadri, Broadcast federation: An application-layer broadcast internetwork, in Intnl. Workshop on Network and Operating Systems Support for Digital Audio and Video, 22. [23] M.-J. Lin, K.Marzullo, and S.Masini, Gossip versus deterministic flooding: Low message overhead and high reliability for broadcasting on small networks, Tech. Rep. Technical Report CS , University of California, San Diego, [24] M. Portmann and A. Seneviratne, Cost-effective broadcast for fully decentralized peer-to-peer networks, Computer Communication, vol. Special Issue on Ubiquitous Computing, Elsevier Science, 22. [25] Sylvia Ratnasamy, Mark Handley, Richard Karp, and Scott Shenker, Application-level multicast using content-addressable networks, in 3rd International Workshop on Networked Group Communication, November 21. [26] Antony I. T. Rowstron, Anne-Marie Kermarrec, Miguel Castro, and Peter Druschel, SCRIBE: The design of a large-scale event notification infrastructure, in Networked Group Communication, 21, pp [27] Rongmei Zhang and Y. Charlie Hu, Borg: A hybrid protocol for scalable application-level multicast in peer-to-peer networks, in The 13th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSS- DAV), Monterey, California, 23.