A Survey and Comparison of. Application-level Multicast Protocols

Transcription

1 A Survey and omparison of Application-level Multicast Protocols Xing Jin Wan-hing Wong S.-. Gary han oi-lun Ngan epartment of omputer Science The ong Kong University of Science and Technology lear Water Bay, Kowloon ong Kong {csvenus, Tel: Fax: Abstract Applications such as Internet-TV and software distribution often require multicast for their delivery. As of today, IP multicast still experiences much difficulty in commercial deployment. In order to overcome it, researchers recently have shifted their focus to application-level multicast (ALM), where the multicast-related functionalities are moved to end-hosts. ALM builds an overlay topology consisting of end-to-end unicast connections between end-hosts. The major concern is how to route data along the topology efficiently. epending on the methods of building data delivery tree, the existing ALM protocols may be classified as mesh-based or tree-based. We have chosen Narada, elaunay Triangulation and Scribe as the representatives of mesh-based approach, while NIE and Overcast are chosen for tree-based approach. We describe in detail and illustrate with examples their basic mechanisms. Using Internet-like topologies, we also compare their performance in terms of network stress, stretch, and overhead incurred. Index Terms Application-level multicast, multicast, overlay network, performance evaluation This work is supported, in part, by the ompetitive Earmarked Research Grant from the Research Grant ouncil in ong Kong (KUST64/E).

2 I. INTROUTION Applications such as Internet-TV, multi-party conferencing, and software distribution require pointto-multipoint Internet communication. In IP multicast, the routers take the responsibility to replicate packets[]. In Figure (a), we show the mechanism of IP multicast, where the end-hosts are denoted by circles and the routers by squares, with the cost of a link as indicated. Source S needs to deliver data to recipients, and 3. Routers R to R5 replicate and forward data packets along the physical links formed by the spanning tree rooted at source S. espite its proposal more than a decade ago, IP multicast still has not been widely deployed, mainly due to the following reasons: Multicast management issues IP multicast standard does not provide solutions for many important multicast management issues for large-scale commercial deployment, such as group management, multicast address allocation and support for network management []. Stateful routers An IP multicast router needs to maintain per group state for packet replication. This state-keeping makes routers unscalable in terms of the number of multicast groups that can be supported. Lack of support of higher level functionalities IP multicast is based on best-effort data delivery, and hence is not reliable. This is not desirable for some applications such as multi-party games and software distribution. Although a number of reliable multicast protocols such as RMTP [3], SRM and PGM have been proposed, many challenges such as the heterogeneity of receivers and feedback issues have to be addressed before they can be extensively deployed. ue to the above problems, researchers have recently shifted their attempts from the network layer to the application level, i.e., the multicast-related functionalities are moved from routers to end-hosts. This is the so-called application-level multicast (ALM). We show an example of ALM delivery mechanism in Figure (b). As opposed to Figure (a), S establishes unicast connections with and 3, while in turn delivers data to via unicast. Multicast is hence achieved via piece-wise unicast connections. learly, end-hosts, instead of routers, are responsible for replicating and forwarding multicast packets. In this way, the spanning tree of ALM forms an overlay topology which consists of only end-hosts and unicast connections between end-hosts. In discussing ALM, we hence often can abstract the network to consist of end-hosts only. In the rest of the paper, we will focus on such overlay topology and data delivery. Implementing multicast functionalities at the application level addresses many issues associated with IP multicast, and comes with a number of strengths: Absence of multicast routers ALM is built on an overlay topology without the need of multicast routers. In other words, routers do not need to maintain any multicast group information and hence

3 3 S R 4 R S R 4 R 3 R5 3 6 R4 R3 3 R5 3 6 R4 R3 (a) Packet flows for IP multicast. (b) Packet flows for application-level multicast. Fig.. A omparison between IP multicast and application-level multicast. In IP multicast, packets are duplicated and forwarded by routers. But, in application-level multicast, this is done by the end-hosts. are stateless, Leverage of current unicast protocols Since the connections are based on unicast, the existing solutions and functionalities of unicast protocols in the transport layer (such as TP and UP) can be straightforwardly applied in ALM. When designing an ALM protocol, the following metrics are usually used to evaluate its performance: Link and node stress Because multiple overlay edges may span the same underlying physical link, a packet may traverse multiple times along a physical link. Moreover, a host may need to forward packets to many hosts. In order to quantify these, link stress and node stress are often used, defined as the number of copies of a packet transmitted over a given physical link and forwarded by an end-host, respectively. The distribution of stresses indicates whether an ALM protocol balances load fairly. If it does not, network traffic will be concentrated at several physical links or several hosts. As an example, refer to Figure (b) again. Since both overlay edges S and S 3 span the same physical link S R, the link stress of S R and the node stress of S are both. Stretch Since a packet may take multiple unicast segments before reaching its destination, it experiences higher delay as compared to the path formed by multicast-capable routers. Stretch is a general term used to refer to such increase in latency. learly, ALM protocols designed for latency-sensitive applications (such as conferencing applications) should have a low stretch. In order to quantify the stretch, the following two metrics are often used: Relay elay Penalty (RP), defined as the ratio of the overlay latency from the source to a given host to the delay along the shortest unicast path. Path Length, defined as the number of unicast segments taken before a packet reaches a given host.

4 4 For example, in Figure (b), the path of ALM packets going from S to is S R R R R3, instead of the shortest path S R R4 R3. Therefore the RP of is ( )/( ) = /5 =, while the path length is. ontrol message overhead In ALM protocols, control messages in general serve two purposes: onnectivity maintenance: Since the multicast group is generally not a static environment (due to failure of links and nodes and joining and leaving of group members), periodic message exchange among hosts is essential to maintain the connectivity of the overlay topology. Network condition measurement: Most ALM protocols continue to improve its connectivity by measuring the round-trip time and available bandwidth between hosts in order to reduce the stress and stretch. Overhead Ratio of a protocol is often used to measure the control overhead, which is defined as the amount of non-data traffic to that of data traffic. Non-data traffic consists of the control packets for connectivity maintenance and network condition measurement. learly, a high ratio indicates much overhead. Since delivering packets in ALM is less efficient than IP multicast, how to build an efficient overlay topology for data delivery becomes a major concern. Moreover, since the multicast group is not a static environment, the overlay topology formed should be able to recover partition when it occurs. Apart from these, the average control-messaging overhead of an ALM protocol should be scalable with the size of a multicast group. There are in general two approaches to multicast packets in ALM: mesh-based or tree-based. In meshbased approach, joining members first form an overlay mesh, and build the multicast tree(s) on top of it. Examples of such approach are Narada [4], T [5], Scribe [6], ALMI [7], MAN, GOSSAMER and Bayeux. On the other hand, in tree-based approach, the multicast tree is built directly without need of preconstructed mesh. Examples of such approach are NIE [8], Overcast [9], Yoid, MTP. Our goals in this paper are to educate readers the basic mechanisms of some representative protocols in each category and present detailed quantitative comparisons among them in the same setting (as opposed to some previous comparisons which are mostly qualitative in nature). II. MES-BASE PROTOOLS In a mesh-based protocol, a mesh-like topology is first built, in which a host connects to a number of other hosts termed as neighbors. There can be multiple paths connecting a pair of hosts on the mesh. Based on the mesh, single or multiple data delivery tree(s) are built, according to the specific protocols.

5 5 ALGORTIM I EVALUATEUTILITY(u, v) utility for each m in multicast group {u} 3 do current latency latency between u and m along mesh 4 new latency latency between u and m along mesh when uv were added 5 if (new latency < current latency) current latency new latency 6 then utility utility + current latency 7 Return utility ALGORITM II EVALUATEONSENSUSOST(uv) ost uv number of group members for which u uses v as the next hop for forwarding packets. ost vu number of group members for which v uses u as the next hop for forwarding packets. 3 return max(ost uv, ost vu ) Because delivery trees are embedded in the overlay mesh, delivery efficiency depends on how well overlay mesh is constructed. On the one hand, more overlay edges means more direct connections between hosts, and hence lower latency. On the other hand, fewer mesh edges means lower node stress. Therefore, in routing decision, selecting the overlay edges for low latency and node stress is important. Another issue is that mesh partition leads to tree partition; therefore partition avoidance and detection mechanisms are important. In order to illustrate and understand in a more concrete manner the basic mechanisms involved, we have chosen three representative schemes, namely, Narada, elaunay Triangulation and Scribe, and discuss their operations in detail in the following. A. Narada Narada is suitable for Internet conferencing applications, where participants can be both sources and receivers at the same time. In Narada, an overlay mesh is constructed by periodically adding and dropping connections between hosts. A host periodically exchanges control messages with its neighbors to maintain connectivity and build its own routing table with the shortest widest path algorithm, of which bandwidth and latency are considered primary and secondary metrics, respectively. A joining host first obtains a list of already-joined hosts from a rendezvous point. The joining host then randomly selects a few of them to connect to. Through periodic exchange of refresh messages among

6 6 A B E A B E 5 5 c F F (a) The utility of adding edge B is (b) The consensus cost of edge AF is. Fig.. a) By adding edge B, the delay from B to is reduced from 9 (along the path B A F ) to (along the path B ), and the delay from B to F is reduced from 7 (along the path B A F ) to 4 (along the path B F ), and the delay from B to E is reduced from 9 (along the path B A F E) to 3 (along the path B E). Therefore the utility of B is AE is one only. = b) Since only the shortest paths between A and E passing through the connection AE, the consensus cost of neighbors, the changes in membership due to joining or leaving of hosts can eventually be propagated to all hosts. In Narada, reverse path algorithm is used to multicast packets: when host u receives a multicast packet from source s, it forwards the packet to those neighbors from which u is on the shortest path to s []. Narada uses a periodic mesh refinement algorithm to add or drop connections so as to incrementally improve the overlay mesh. A host u periodically probes some other hosts to evaluate the overall delay that can be reduced if connected to them. If the reduction of a host is above a certain threshold (the adding threshold), u connects to it. Meanwhile, u also computes the consensus cost of an edge between its neighbor, say the edge uv. For all the shortest paths from u to the other nodes in the network, u computes the number of them include uv as one of their edges. v also computes likewise. The maximum of these two numbers is the cost. If it is below a certain threshold (the dropping threshold), the edge uv is disconnected. Note that the adding and dropping thresholds are not fixed values, but are functions of maximum and minimum fanout of all the nodes in the overlay mesh. Therefore Narada controls the maximum and minimum fanout to avoid a host from having too many connections and becoming a bottleneck. We show the algorithm to compute the overall delay reduction (termed as utility) for two nodes u and v if a connection were made in ALGORITM I, and show an example in Figure (a), where hosts B and are not connected at first. When B probes, it finds that by connecting to the delay from itself to can be reduced from 9 (along the path B A F ) to (along the path B ), the delay from B to F can be reduced from 7 (along the path B A F ) to 4 (along the path B F ), and the delay from B to E can be reduced from 9 (along the path B A F E) to 3 (along the path B E). Therefore the utility of B is = If the value is above the

7 7 a b a b d c d c (a) (b) Fig. 3. a) Two adjacent triangles forming a convex quadrilateral abd and bdc violating the T property. b) Restoration of T property by disconnecting a from c and connecting b and d. adding threshold, B will connect to. We also show how to evaluate the consensus cost in ALGORITM II given an edge uv and an example in Figure (b). onsider the edge AF (i.e., A connects to host F initially.) For both A and F, because there is only one shortest path which includes the edge AF (the direct path AF ), the consensus cost of AF is one. Therefore, if the value of the dropping threshold is higher than one, AF will be dropped. Narada detects mesh partition with the help of periodical exchanging refresh messages. Basically, if host u loses a sequence of refresh messages from host v, host u suspects that v is dead and probes v. If v is still alive, then u connects to v in order to recover the partition. Since this algorithm does not interact with the rendezvous point, it is robust to the failure of any host. Narada is not scalable in term of the number of hosts. This is because: As a flat routing algorithm, the shortest widest path algorithm holds the fact that the size of routing table is as large as the group size. Therefore the state maintained by a host is in the order of O(N). A control message contains the states of all hosts in the group; therefore, the total control overhead in the system is high for large groups (O(N )). Another potential problem is that Narada s convergence time can be very long. Simulation results show that it is difficult to form a stable mesh, even after a very long time. This is due to the inconsistent criteria for adding and dropping edges. B. elaunay Triangulation (T) In T, each host has a geographical coordinate and hosts first form an overlay mesh based on these coordinates. ompass routing is used to route a packet from one point to another. T protocol connects the nodes together in a triangular manner so that the mesh satisfies elaunay Triangulation property, i.e., the minimum internal angle of the adjacent triangles in the mesh are maximized []. To illustrate the triangulation process, consider that points a, b, c and d form a convex quadrilateral abcd. There are two possible ways to triangulate it as shown in Figure 3. Since the minimum internal angle of abc

8 8 n n n u t n 3 Fig. 4. When u receives a unicast packet with destination t, it forwards the packet to n. When u receives a multicast with source t, it forwards the packet to n and n 3. and acd (Figure 3(a)) is less than that of abd and bcd (Figure 3(b)), T protocol transforms the former configuration into the latter. The mesh formed this way connects those geographically close nodes together. In T protocol, the joining process is bootstrapped by a T server, which caches a list of joined host. A joining host u first queries the T server for some already-joined hosts. u then sends via unicast join requests to these hosts, which in turn sends back those joining requests along the T mesh towards u until reaching a set of hosts nearby u. These nearby hosts then connect to u. Note that the newly established connections may violate the T property. To restore it, every host periodically tests its connections against the property, and drops those failing in the test. A host also discovers nearby hosts through periodical exchange of control messages with its neighbors, and connects to any nearby hosts if this does not violate the T property. To illustrate this process, suppose that initially hosts a, b, c and d are connected as in Figure 3(a). Because the connection ac violates the T property, it is dropped when host a (or c) tests it. After that host b discovers d through control messaging, and connects to d since bd satisfies the T property. Then the resultant overlay mesh (as shown in Figure 3(b)) satisfies the T property. To route a packet from one point to another in T protocol, compass routing can be used. When host u receives a unicast packet with destination v, it first computes the slope between the two coordinate points u and v and let the slope be s. u then computes the slopes of all its N neighbors with itself. Let these be s i, i N. u forwards the packet to the neighbor which has the closest slope to s, i.e., u forwards to neighbor j where s j s is the minimum for all s i s. We illustrate the compass routing in Figure 4, where u needs to route a packet destined to t to one of its neighbors. Because the slope between u and n is the closest to the slope of u and t, host u forwards the unicast packet to host n. In T, as similar to Narada, reverse path forwarding algorithm is used to multicast packets. When host u receives a multicast packet from source s, it forwards the packet to those neighbors if u is on the path

9 9 from them to s. Refer back to Figure 4 again. When host u receives a multicast packet with source t, it forwards the packet to n and n. This is because the slope of t n is the closest to slope u n among all the slopes of u n, n n and n n, while the slope of t n is the closest to slope of u n among the slopes of u n, n n and n 3 n. In T protocol, mesh partitioning is detected with the T server. For each connected mesh, one and only one host is elected as leader (usually the leader is the one with the greatest coordinate ). The leader periodically exchanges control messages with the server. When the mesh is partitioned, there would be more than one leaders communicating with the server. The server can then recover the partition by requesting one of them to connect to the others. As compared to Narada, T protocol is much more scalable in terms of the number of hosts. This is because compass routing is a kind of local routing, which is based on the coordinates of hosts only directly connected to a node. Therefore the size of the routing table maintained by a host depends on the number of neighbors it has only, which is on average less than or equal to six. owever, T protocols come with the following two weaknesses: Inaccurate host location estimation The geographic locations of hosts in general do not correlate well with the latencies between hosts in the Internet; therefore, the end-to-end delay along a T overlay may be quite large. Single point of failure The partition detection and recovery scheme relies on a T server, which forms a single point of failure.. Scribe In Scribe, there are many hosts in the network but a multicast group only covers or spans a subset of them. Those hosts which are not group members take part in packet forwarding in the Scribe network. A possible application for Scribe is Internet chatroom, where usually a small set of users out of a possibly large pool belongs to the same multicast group. The large pool of hosts jointly takes part in forwarding packets for the group members in the system. Scribe provides multicast group management for data delivery. It builds on top of Pastry, which provides the actual host-to-host routing and content-delivery mechanisms. Scribe first connects hosts together as a Pastry overlay mesh, i.e., the larger group. Then it constructs an overlay tree for each multicast group on The coordinate of host A is greater than host B iff the y-coordinate of A is greater than that of B, or, if their y-coordinates are the same, the x-coordinate of A is greater than that of B. elaunay Triangulation is a planar graph, which has at most 3N edges. Therefore the average number of neighbors is less than or equal to 3N N = 6.

10 TABLE I AN EXAMPLE ROUTING TABLE OF OST 33 FOR PASTRY. olumn R xxx xxx xxx o 3xx 3xx 3xx w 33x 33x 333x top of the mesh such that the tree branches are embedded with the mesh edges []. learly, a host can be a tree node of multiple overlay trees. When a host receives a packet of a multicast group, it simply forwards the packet to all of its children in the corresponding multicast overlay tree. In Scribe, the non-leaf nodes are referred to as forwarders since they forward data to their children for data dissemination. In Pastry, a host is identified by a random key termed NodeId of value between and M (one may think of the key as the host address). (NodeI can be made unique with high probability by using common message digest functions.) Each key is expressed in base B. A host constructs its own routing table based on the leading prefix of the destination NodeId. The routing table of a host with NodeId u = [u u... u l ] (in base B) has l = log B (M + ) rows and B columns. The entry at the rth row and cth column ( r l and c B) is for routing to a destination with NodeI matching r prefixes of u and has a digit of value c at the rth position. More formally and specifically, the (r, c) entry is for the forwarding of a host with NodeId v = [v v... v l ] where v = u, v = u,..., v r = u r, and v m = c, while v m+... v l are don t care. The routing table shows which entry to look into based on the leading prefix of the destination NodeI. In the lookup, maximum prefix-matching is used. Upon a match, the entry would indicate the next-hop NodeI for the packet to be forwarded to. It should be noted that the routing table is constructed in such a way that the next-hop NodeI also has the same leading prefix as the entry. Each entry indicates at most one next-hop host although there may be a number of destination hosts matching the prefix. To reduce the stretch, a host periodically probes those possible next-hop hosts to select the one with the shortest round-trip time. We show in Table I an example of the routing table for a host of index 33, with M = 55, B = 4, and x can be any digit between and B (i.e., don t care digit). Note that the next-hop host for the entries (at the (, 4), (, 4), (3, 3), and (4, ) entries) is the host 33 itself. Each entry indicates the next-hop NodeI with the same prefix (not shown) to be forwarded to. When the host receives a packet of destination 33, it forwards the packet to the next-hop host as indicated by the (3, ) entry, learly, in Pastry overlay, at each hop the packet is one step closer to its destination as the length of

11 prefix matching between the current NodeI and destination NodeI is increased by. Therefore, the path length is equal to O(log d M). 3 A multicast group in Scribe is assigned with a key (the group identifier), which is in the same key space as NodeId. A joining host first sends a join request with the group identifier as the destination key along the Pastry overlay until the request reaches a host receiving data of the same multicast group. The join request turns all hosts along the path into forwarders even though they are not members of the multicast group. Therefore, the overlay tree is an aggregation of Pastry paths from the interested hosts to the host whose key is numerically closest to the group identifier. This tree is free of loops because the distance to the destination progressively reduces upon each hop. 4 Therefore, an overlay tree is topologically sorted by the distance from the destination. We show an example of multicast routing in Pastry in Figure 5, with each digit being either or (i.e., B = ) and the group identifier being. There are 8 hosts in the networks, and only the hosts (unshaded nodes),, and are group members and they join in such order. We arrange the hosts into concentric circles based on the length of their matched prefix with the group identifier, with the circles separated by a distance of. In this example, the topological order of the multicast tree is [], [], [, ], [,,, ], where hosts within the square brackets are interchangeable. When host joins the multicast group (whose key has only one matched prefix), it sends a join request to host (two matched prefixes), which in turn forwards the request to host. Similarly, the joining host sends a join request to host which has a longer matching prefix than. owever since host has already been a forwarder, it suppresses the request. learly, the tree is without any loop. To maintain the connectivity of the overlay tree, each host periodically sends refresh messages to its children. Any host fails to receive refresh messages would assume that its parent is dead and rejoins the multicast group to recover the partition. To reduce the refresh overheads, multicast packets can serve as implicit refresh messages. There is also an algorithm to remove bottleneck in the data delivery tree by limiting the children number of a host through delegation of its children to other nodes []. When a host is overloaded, it first identifies the multicast group which consumes the most resources and sends a control message appended with its children NodeIs to the farthest child within that group from itself. Upon receiving the control message, the child then chooses a new parent among the children listed in the 3 There are some special cases which need to be considered, such as the case when a host cannot find any neighbors with the same leading prefix as its table entry. This is taken care of by comparing the numerical difference, rather than the prefix, between the destination NodeI and some other set of neighbor NodeIs of the host. In this case, the routing is less efficient but the distance to the destination still monotonically decreases upon each hop. The details of routing mechanism of Pastry has been discussed in []. 4 Suppose P = {h, h,... } is the Pastry path with h being the joining host, h is the first host, etc. The distance from the destination from h i is longer than that from h i+.

12 3 prefix digits matched prefix digits matched prefix digit matched. Fig. 5. The loop-free tree built by Scribe. message. (Note that this algorithm may break the loop-free property of the Pastry tree, and hence has to rely on other mechanisms for recovery.) The size of the routing table at a host in Pastry is O(log B M), and hence Scribe is scalable in terms of group size. owever, the tree-building algorithm requires a host to serve other groups not of its interest. Moreover, the performance of Pastry routes depends on the key distribution. There may be cases where even if two hosts are very close in location together, they may be separated by many hops on the Pastry overlay due to their poor match in prefixes. As a result, a high stretch value results. III. TREE-BASE PROTOOLS In tree-based protocols, a multicast tree is built directly over all joining members, and no mesh topology is needed. The tree structure does not change with the source host. Because of that, a single node failure or a loop can destroy the tree, thereof making it more fragile than a mesh. Loop-avoidance mechanism is hence an important issue in tree-based protocols. We have chosen NIE and Overcast as two representative schemes in this category and discuss their basic operations in detail in the following. A. NIE NIE is suitable for low-bandwidth streaming applications with a large number of receivers. It organizes hosts into a multi-layer hierarchical structure, with the highest layer consists of only one host and the

13 3 L3 3 L L B F L A B E F G 3 Fig. 6. Nice organizes hosts into a multiple-layer hierarchical structure, where hosts on each layer are grouped into a number of cluster. lowest layer consists of all the hosts in the group. A host joins a number of layers in a bottom-up manner, and hosts of the same layer are grouped into a number of clusters. We show an example of NIE in Figure 6, where shadowed boxes denote clusters, and white circles denote end-hosts. (The arrows will be explained later.) All hosts (i.e., A to ) join the bottom layer (L ), where hosts are grouped into a number of clusters (namely,,, and 3). The leader of each cluster (B,, F and ) joins the layer one level up (L ). The grouping of hosts into clusters and selection of cluster leaders are then repeated. In NIE, the clusters are organized into a hierarchical tree, with the upper cluster branching out a number of child clusters in the lower layer. A host may join multiple layers, and belong to different cluster in different layer. We denote a host s cluster peers as the set of all the nodes sharing clusters with the host. A joining host u first selects a cluster from layer L to join by successive probing from the highest layer to the lowest: it first queries the rendezvous point for the host of the highest layer. The host tells u the leaders of the layer one level down. By measuring the round trip time with these leaders, u selects the closest one and queries it for the leaders being covered on the layer one level down. The process is repeated until u finds the closest leader of a cluster on the lowest layer. u then joins the cluster. We show an example in Figure 7. Joining host I first queries the rendezvous point for the host on the highest layer, i.e.,, and then it queries for hosts being covered on the lower layer, i.e., and. I selects the closest one to repeat the process. Eventually, joining host I finds the closest cluster 3 on layer L with which it joins. Unlike Scribe, the overlay tree for data delivery in NIE is not pre-constructed. When a host receives a multicast packet from a host of cluster c, it simply forwards the packet to all its cluster peers except

14 4 L3 3 RP L3 3 RP L3 3 RP L L L L B F L B F L B F L A B E F G I L A B E F G I L A B E F G I (a) (b) (c) 3 3 L3 RP L3 RP L L L B F L B F L A B E F G I L A B E F G I 3 3 (d) (e) Fig. 7. In NIE, the joining host selects a cluster on layer L to join with by successively probing from highest layer to lowest layer. those in cluster c. Referring back to Figure 6 again. When host receives a multicast packet from host B of cluster, it forwards the packet to its cluster peers in and, i.e., nodes and, respectively. Therefore the maximum path length is twice the number of layers (i.e., O(log k (N))), where k is the cluster size, and the maximum node stress is equal to the product of the cluster size and the number of layers (i.e., O(k log k (N))). To maintain the overlay topology, a host periodically sends heartbeat messages to its cluster peers. Therefore, sudden leaving of any member can be detected through the loss of heartbeat messages. NIE also limits the size of a cluster from k to 3k. A cluster will split into two clusters if its size is above the upper bound, or merges with another cluster if its size falls below the lower bound. Nice is efficient in terms of end-to-end delay, since the path length of forwarding a data packet is of order O(log k (N)). owever, NIE creates bottlenecks at the top-layer and higher-layer nodes, since all the joining members have to query one node at each layer of the hierarchy.

15 5 B. Overcast Overcast is designed for single-source applications, e.g., TV-broadcasting. It tries to maximize each host s bandwidth from the source. Latency, on the other hand, is not the major concern. A new member joins the multicast tree by contacting its potential parents. The root node is all new nodes default potential parent. The new node estimates its available bandwidth to this potential parent. It also estimates the bandwidth to the potential parent through each of this potential parent node s children. If the bandwidth through any of the children approximates to the direct bandwidth to the potential parent, the closest one (in terms of network hops) of all the qualified children becomes the new potential parent and a new round commences. If there is no qualified children, the procedure stops and the current potential parent becomes new node s parent. To estimate the bandwidth, the node measures the download time of K bytes, which includes all the service costs. If the measured bandwidths to two nodes are within % of each other, we consider the two nodes equally good and select the closer one. A node periodically reevaluate its position in the tree. It measures the bandwidth to its current siblings, parent and grandparent. It will move below its sibling if that does not decrease its bandwidth back to the root. Also, it will move one level up for higher bandwidth. An example is shown in Figure 8, where node R is the root and node is the new member. Before joins, all other nodes have formed a multicast tree. The thickness of the arrows indicates the overlay link bandwidth. Initially, contacts root R. Since its direct bandwidth to R approximates to the bandwidth through node B, it switches to B. This process is repeated and then it switches to E. Since the switching to G leads to lower end-to-end bandwidth, stays at that level thereafter. In Overcast, each member maintains its ancestor list for partition avoidance and recovery. A member rejects any connection requests initiated by its ancestor(s) to avoid looping. When a member detects that its parent has left the multicast group, it connects to its ancestors one by one, from its grandparent to the root, until a live member is found. Therefore, the loading is distributed along the path to the root, and the root is not easily overloaded. Overcast also includes an up/down protocol for information exchange. Each node, including the root, maintains a table of information about all its descendants and a log of all changes to the table. Each node periodically checks in with its parent. If a child fails to contact its parent within a given interval, the parent will assume the child and all its descendants have died. It will then modify the table. A node also modify the table if new children arrive. uring these periodical check-ins, a node reports new information that it has observed or been informed of. By this protocol, the root can maintain up-to-date information about all the other nodes.

16 6 R R R A B A B A B E F E F E F G G G (a) Initial stage. (b) Intermediate stage. (c) Final stage. Fig. 8. The joining procedure of Overcast. Node is the new member. The root node has to handle each new member s joining request and is likely to be system bottleneck. This problem remains for the up/down protocol. To overcome this single point of failure problem, Overcast proposes to use a linear structure at the top of the multicast tree. Figure 9 shows an example. The black node is the root. The root and two grey nodes are configured linearly. Each grey node has enough information to act as the new root in case of root failure. owever, this technique increases latency. Overcast concentrates on bandwidth allocation, which is different from all above protocols. One key issue in tree building is the bandwidth estimation. urrent estimation technique is not accurate enough, and testing results may not conform the cases during data transmission. This will affect the tree efficiency in terms of bandwidth. In the worst case of building a tree, every newly joining node has to contact all the existing nodes. This leads to a complexity of N O( i) = O(N ), i= where N is the number of nodes. On the average case, time complexity should not be that bad. We will introduce its simulation results in the next section. IV. OMPARISONS In this section, we compare the performance of the protocols discussed. We summarize in Table II the comparison of the protocols in terms of the mechanism of the overlay construction and maintenance, partition detection and the recovery scheme. We also show in Table III the performance of these protocols in terms of path length, node stress, and the size of routing table with respect to group size N, the

17 7 Fig. 9. Linear structure at the top of an Overcast tree, which helps overcome the single point of failure problem. number of possible keys in the network M (for Scribe), and the cluster size K (for NIE). From it, we see that the maximum path length for Scribe and Nice increases only logarithmically with group size, while Narada and Overcast in the worse case may have very long path. Although the path length of T in the worst case is O(N), its average path length is only O( N)). Specially, Overcast s tree topology is related to path bandwidths and can t be easily decided. On average the node stress of NIE, T, Narada and Overcast are independent of the group size. Only Narada is able to guarantee a certain node stress in the worst case. The maximum node stress of NIE grows with the group size logarithmically, while T and Overcast do not provide any guarantee on the maximum node stress. owever, since the host on the topmost layer is also the leaders of many lower layers, it may become the bottleneck of the system. The routing table sizes of T and NIE are independent of the group size, because only neighborhood information is needed to be maintained. On the other hand, the table sizes of Scribe and Narada grow with the group size logarithmically and linearly, respectively. Specially, the table size of an Overcast node is uncertain, since we don t know the tree topology. Therefore, the scalability of T is much better than that of Narada and Scribe. Besides the above, we have also done simulations on Internet-like topologies to compare the performance of these protocols in terms of relative delay penalty and physical link stresses. In our simulations, we first generate a number () of Transit Stub topologies with Georgia Tech s random graph generator [3]. The generated topologies are a two-layer hierarchy of transit networks (with four transit domains, each with 6 randomly-distributed routers on a 4 4 grid) and stub networks (with 64 domains, each with 5 randomly-distributed routers on a 3 3 grid). A host is connected to a stub router via a LAN (of 4 4 grid). The delays of LAN links are ms while the delays of core links are given by the topology generator. For each protocol, we randomly select a number of hosts (6 4) as group members and choose one of them as the source. We measure the RP and link stress when the source sends packets to all hosts. For T, we take the geographical coordinate of a host as its location. For Scribe, we randomly assign the hosts with key values. There are 8 number of possible keys (corresponding to IPv6 address space), and B = 6. For Narada, add-link/drop-link thresholds are set according to group size, nodes current

18 8 TABLE II A OMPARISON BASE ON PROTOOL ESIGN. Overlay Topology onstruction Narada Add edges of high utility and drop edges of low consensus cost. T The overlay mesh constructed satisfies the T property. Scribe ata delivery tree is an aggregation of Pastry routes from interested hosts to the rendezvous point. NIE Organize hosts in a multiplelayer hierarchical structure, with the highest layer consisting of only one host and the lowest layer consisting of all the hosts. Overcast Incrementally build a tree and try to maximize bandwidth to the root for all nodes. Partition etection and Recovery Every host maintains a complete list of members and probes those silent members. If more than one leader communicates with the T server, it recovers the partition by connecting leaders together. A host rejoins the system if its parent is silent for a long time. A host periodically probes its cluster peers on differen layers. Each node periodically contacts its parent. A node connects to its ancestors one by one, from its grandparent to the root, if its parent is unreachable. Packet Routing Based on the routing table constructed with a shortest widest algorithm, reverse path forwarding algorithm is applied for multicast packet routing. Based on compass routing, reverse path forwarding algorithm is applied for multicast packet routing. ata is disseminated along the tree from the rendezvous point Forward data to all cluster peers except the one sending data to it ata is disseminated along the tree from the root.

19 9 TABLE III OMPARISON OF PAT LENGT, NOE STRESS AN SIZE OF ROUTING TABLE. Protocol Path Length Node Stress Routing table size Average Maximum Average Maximum Narada O(N) O(N) O() O() O(N) T O( N) [5] O(N) 6 O(N) O() Scribe a O(log(M)) O(log(M)) O(N) O(N) O(log(M)) Nice O(log(N)) O(log(N)) O(K) O(K log(n)) O(K) [8] Overcast Undecided O(N) O() O(N) Undecided a Since the bottleneck removal algorithm may introduce looping, we did not implement it. and maximum fanout value. Each node s fanout range (the minimum and maximum number of neighbors each member strives to maintain in the mesh) is 3-6. For Nice, the cluster size parameter, k, is set to 3. For Overcast, the bandwidths of links internal in transit domains are uniformly distributed between and 45 Mbits/s, and that of links connecting stub domains and transit domains uniformly distributed between and.5mbit/s, that of links internal in stub domains uniformly distributed between and Mbit/s. We show in Figures (a) and (b) the average RP versus group size and the cumulative distribution of RP (with 56 group members), respectively, given different protocols. In general RP increases with group size as packets take more hops to reach all the end-hosts. The RP of Scribe and T is substantially higher than that of Narada, NIE and Overcast. The high RP of Scribe is due to random key distribution, which adversely affects the Pastry route. In T, since the geographical location of the hosts does not correlate well with their Internet locations, the overlay mesh built may not reflect the inter-host distance in the underlay network. This leads to a high average RP. Because NIE and Narada continuously measure the round-trip time between hosts to improve their overlay topology, their RPs on average are much lower. The distribution of RP confirms our observations that T and Scribe have much higher RP than NIE, Narada and Overcast. Note that our simulation results on Scribe are different from what is observed in [6]. This is mainly because of the differences in simulation environment. In their simulation,, hosts are attached to only 55 routers. Therefore on average about hosts are attached to a router, as opposed to less than one host per router in our case. The Pastry mesh resulted is hence much denser in their experiments. Since it is more likely to find short or direct paths in a dense mesh, the performance is hence better. ere Overcast s RP is surprisingly low since its tree construction mainly relates to bandwidth not delay. We investigate its tree topologies and find that node stresses are unevenly distributed among members. A

20 8 7 6 Scribe T Narada NIE Overcast Average RP vs. Group Size umulative istribution of RP Average RP Group Size (N) Percentage Scribe T Narada NIE Overcast RP (a) Average RP versus different group sizes. (b) umulative distribution of RP of a group with 56 members. Fig.. A performance comparison among different protocols in terms of RP. few nodes degrees are over while most of the others have degree -5. That is, some nodes have plenty of bandwidth and are able to contain much more children than others. As we know, a star topology s average RP is equal to. Similarly, this partially star topology in Overcast tree helps reduce average RP. We have to claim that although this low average RP is desirable, the system load of Overcast is not evenly distributed. The real capacity of a node (bandwidth or processing capability) may not conform the number of its children. Those nodes with large node stress is likely to be system bottleneck. orrespondingly, we can see later that average link stress of Overcast is much higher, which implies that a few links are frequently used. Those are just links with high bandwidths. We next show in Figures (a) and (b) average link stress versus group size and stress distribution, respectively, given different protocols. In general the link stress increases with group size as packets take more hops to reach all the end-hosts. The link stress of Scribe is the highest, while those of T and NIE are among the lowest. The high link stress of Scribe is due to its unbounded node stress, while the low link stress of T and NIE are due to their rather uniform and low node stress. V. ONLUSION Application-level multicast (ALM) implements multicast-related functionalities at the application level. Such technique promises to overcome the deployment problems associated with IP multicast. In ALM, since packets take more hops to reach all members, it has higher delay and stress. In this paper, we have reviewed a number of application-level protocols. In general, ALM protocols can be classified as mesh-based and tree-based depending on the steps of building data delivery tree. We have chosen Narada, T and Scribe to represent mesh-based protocols,

21 Average Physical Link Stress Scribe T Narada NIE Overcast Average Link Stresses vs. Group Size Group Size (N) (a) Average physical link stress versus different group sizes. Percentage umulative istribution of Physical Link Stress 65 Scribe 6 T Narada 55 NIE Overcast Physical Link Stress (b) umulative distribution of physical link stress of a group with 56 members. Fig.. A performance comparison among different protocols in terms of physical link stresses. and NIE and Overcast to represent tree-based protocols. We describe their delivery mechanisms in detail with illustrative examples. Using Internet-like topologies, we also simulate and compare their performance in terms of stress and relay delay penalty (RP). Narada, though not scalable due to its flat routing protocol, is robust in term of fault tolerance since mesh partitioning can be detected and recovered without the need of a rendezvous point. In contrast, T is more scalable due to its local routing protocol, although the T server may be the single point of failure. Scribe supports applications where the tree spans only a subset of hosts. owever, a host in Scribe may need to forward packets for other multicast groups, which raises some incentive issue in its deployment. In NIE, the maximum path length and node stress grows only logarithmically with the group size. Overcast targets optimal bandwidth allocation and considers latency as a supplement. Our simulations show that Nice s performance in terms of RP and stress is the best among all the schemes, while T would not have good performance since network measurements are not done. REFERENES [] S. E. eering, Multicast routing in internetworks and extended lans, AM SIGOMM omputer ommunication Review, vol. 8, no. 4, pp , Aug [] K. Sripanidkucha, A. Myers, and. Zhzng, A third-party value-added network service approach to reliable multicast, in Proceedings of AM sigmetrics, August 999. [3] Sanjoy Paul, Krishan K. Sabnani, John. Lin, and Supratik Bhattacharyya, Reliable multicast transport protocol (RMTP), IEEE Journal on Selected Areas in ommunications, vol. 5, no. 3, pp. 47 4, April 997. [4] Yang hua hu, Sanjay G. Rao, Srinivasan Seshanand, and ui Zhang, A case for end system multicast, IEEE Journal on Selected Areas in ommunicastions, vol., no. 8, pp , October. [5] Jörg Liebeherr, Michael Nahas, and Weisheng Si, Application-layer multicasting with elaunay triangulation overlays, IEEE Journal on Selected Areas in ommunicastions, vol., no. 8, pp , October.

22 [6] Miguel astro, Peter ruschel, Anne-Marie Kermarec, and Antony I. T. Rowstron, Scribe: a large-scale and decentralized applicationlevel multicast infrastructure, IEEE Journal on Selected Areas in ommunicastions, vol., no. 8, pp , October. [7] imitrios Pendarakis, Sherlia Shi, inesh Verma, and Marcel Waldvogel, ALMI: an application level multicast infrastructure, in Proceeding of 3rd USENIX Symposium on Internet Technology and Systems (USITS, Berkeley, A, USA.,, USENIX Assoc., pp [8] Suman Banerjee, Bobby Bhattacharjee, and hristopher Kommareddy, Scalable application-layer multicast, in AM. omputer ommunication Review, USA, October, number 4, pp [9] John Jannotti, avid K. Gifford, Kirk L. Johnson, M. Frans Kasshoek, and JamesW. O Toole, Overcast: reliable multicasting with an overlay network, in Proceedings of the Fourth Symposium on Operating System esign and Implementation (OSI ). USENIX Assoc., October, pp. 97. [] Y. K. alal and R. M. Metcalfe, Reverse path forwarding of broadcast packets, ommunications of the AM, vol., no., pp. 4 48, ec [] Sibson R., Locally equiangular triangulations, omputer Journal, vol. 3, no., pp , 978. [] A. Rowstron and P. ruschel, Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems, in Proceedings of IFIP/AM International onference on istributed Systems Platforms (Middleware), eidelberg, Germany, November, pp [3] EW. Zegura, KL. alvert, and S. Bhattacharjee, ow to model an internetwork, in Proceedings of INFOOM 96. The onference on omputer ommunications. Fifteenth Annual Joint onference of the IEEE omputer Societies. Networking the Next Generation, Los Alamitos, A, USA., 996, pp