OMNI: An Efficient Overlay Multicast. Infrastructure for Real-time Applications

Transcription

1 OMNI: An Efficient Overlay Multicast Infrastructure for Real-time Alications Suman Banerjee, Christoher Kommareddy, Koushik Kar, Bobby Bhattacharjee, Samir Khuller Abstract We consider an overlay architecture (called OMNI) where service roviders deloy a set of service nodes (called MSNs) in the network to efficiently imlement media-streaming alications. These MSNs are organized into an overlay and act as alication-layer multicast forwarding entities for a set of clients. We resent a decentralized scheme that organizes the MSNs into an aroriate overlay structure that is articularly beneficial for real-time alications. We formulate our otimization criterion as a degreeconstrained minimum average-latency roblem which is known to be NP-Hard. A key feature of this formulation is that it gives a dynamic riority to different MSNs based on the size of its service set. Our roosed aroach iteratively modifies the overlay tree using localized transformations to adat with changing distribution of MSNs, clients, as well as network conditions. We show that a centralized greedy aroach to this roblem does not erform quite as well, while our distributed iterative scheme efficiently converges to near-otimal solutions. I. INTRODUCTION In this aer we consider a two-tier infrastructure to efficiently imlement large-scale media-streaming alications on the Internet. This infrastructure, which we call the Overlay Multicast Network Infrastructure (OMNI), consists of a set of devices called Multicast Service Nodes (MSNs []) distributed in the network and rovides efficient data distribution services to a set of end-hosts. An end-host (client) subscribes with a single MSN to receive multicast data service. The MSNs themselves run a distributed rotocol to organize themselves into an overlay which forms the multicast data delivery backbone. The data delivery ath from the MSN to its clients is indeendent of the data delivery ath used in the overlay backbone, and can be built using network layer multicast alication-layer multicast, or a sequence of direct unicasts. The two-tier OMNI architecture is shown in Figure. In this aer, we resent a distributed iterative scheme that constructs good data distribution aths on the OMNI. Our scheme allows a multicast service rovider to deloy a large number of MSNs without exlicit S. Banerjee is with the Deartment of Comuter Science, University of Wisconsin, Madison, WI 53706, USA ( suman@cs.wisc.edu). C. Kommareddy, B. Bhattacharjee and S. Khuller are with the Deartment of Comuter Science, University of Maryland, College Park, MD 07, USA ( {kcr,bobby,samir}@cs.umd.edu). K. Kar is with the Deartment of Electrical, Comuter and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 80, USA ( koushik@ecse.ri.edu). Similar models of overlay multicast have been roosed in the literature (e.g. Scattercast [] and Overlay Multicast Network []).

2 Source A B MSNs C D E F Service Area of MSNs Clients Fig.. OMNI Architecture. concern about otimal lacement. Once the caacity constraints of the MSNs are secified, our technique organizes them into an overlay toology, which is continuously adated with changes in the distribution of the clients as well as changes in network conditions. Our roosed scheme is most useful for latency-sensitive real-time alications, such as media-streaming. Media streaming alications have exerienced immense oularity on the Internet. Unlike static content, real-time data cannot be re-delivered to the different distribution oints in the network. Therefore an efficient data delivery ath for real-time content is crucial for such alications. The quality of media layback tyically deends on two factors: access loads exerienced by the streaming server(s) and jitter exerienced by the traffic on the end-to-end ath. Our roosed OMNI architecture addresses both these concerns as follows: () being based on an overlay architecture, it relieves the access bottleneck at the server(s), and () by organizing the overlay to have low-latency overlay aths, it reduces the jitter at the clients. For large scale data distributions, such as live webcasts, we assume that there is a single source. The source is connected to a single MSN, which we call the root MSN. The roblem of efficient OMNI construction is as follows: Given a set of MSNs with access bandwidth constraints distributed in the network, construct a multicast data delivery backbone such that the overlay latency to the client set is minimized. Since the goal of OMNIs is to minimize the latencies to the entire client set, MSNs that serve a larger client oulation are, therefore, more imortant than the ones which serve only a few clients. The relative imortance of the corresonding MSNs vary, as clients join and leave the OMNI. This, in turn, affects the structure of the data delivery ath of the overlay backbone. Thus, one of the imortant considerations of the OMNI is its ability to adat the overlay structure based on the distribution of clients at the different MSNs. Our overlay construction objective for OMNIs is related to the objective addressed in [3]. In [3] the authors

3 3 roose a centralized greedy heuristic, called the Comact Tree algorithm, to minimize the maximum latency from the source (also known as the diameter) to an MSN. However the objective of this minimum diameter degree-bounded sanning tree roblem does not account for the difference in the relative imortance of MSNs deending on the size of the client oulation that they are serving. In contrast we formulate our objective as the minimum average-latency degree-bounded sanning tree roblem which weights the different MSNs by the size of the client oulation that they serve. We roose an iterative distributed solution to this roblem, which dynamically adats the tree structure based on the relative imortance of the MSNs. Additionally we show how our solution aroach can be easily augmented to define an equivalent distributed solution for the minimum diameter degree-bounded sanning tree roblem. This is an extended version of an earlier aer by the same authors []. The rest of the aer is structured as follows: In the next section we formalize and differentiate between the definition of these roblems, and ose an Integer Programming based centralized solution. In Section III we describe our distributed solution technique which is the main focus on this aer. In Section IV we study the erformance of our technique through detailed simulation exeriments. In Section V we discuss other alication-layer multicast rotocols that are related to our work. Finally, we resent our conclusions in Section VI. II. PROBLEM FORMULATION In this section we describe the network model and state our solution objectives formally. We subsequently roose an Integer Programming based solution which is useful is evaluating the quality of results obtained by our distributed solution to this roblem. We also outline the ractical requirements that our solution is required to satisfy. A. System Model and Problem Statement The hysical network consists of nodes connected by links. The MSNs are connected to this network at different oints through access links. The multicast overlay network is the network induced by the MSNs on this hysical toology. It can be modeled as a comlete directed grah, denoted by G = (V, E), where V is the set of vertices and E = V V is the set of edges. Each vertex in V reresents an MSN. The directed edge from node i to node j in G reresents the unicast ath from MSN i to MSN j in the hysical toology The latency of an edge i, j in the overlay grah corresonds to the unicast ath latency from MSN i to MSN j, and is denoted by l i,j. The data delivery ath on the OMNI will be a directed sanning tree of G rooted at the source MSN, with the edges directed away from the root. Consider a multicast alication in which the source injects traffic at the rate of B units er second. We will assume that the caacity of any incoming or outgoing access link

4 is no less than B. Let the outgoing access link caacity of MSN i be b i. Then the MSN can send data to at most d i = b i /B other MSNs. This imoses an out-degree bound at MSN i on the overlay tree of the OMNI. The overlay latency L i,j from MSN i to MSN j is the summation of all the unicast latencies along the overlay ath from i to j on the tree, T. The latency exerienced by a client (attached to MSN i) consists of three arts: () the latency from the source to the root MSN, r, () the latency from the MSN i to itself, and (3) the overlay latency L r,i on the OMNI from MSN r to MSN i. The arrangement of the MSNs affects only the overlay latency comonent, and the first two comonents do not deend on the OMNI overlay structure. Henceforth, for each client we only consider the overlay latency L r,i between the root MSN and MSN i as art of our minimization objective in constructing the OMNI overlay backbone. We consider two searate objectives. Our first objective is to minimize is the average (or total) overlay latency of all clients. Let c i be the number of clients that are served by MSN i. Then minimizing the average latency over all clients translates to minimizing the weighted sum of the latencies of all MSNs, where c i denote the MSN weights. The second objective is to minimize the maximum overlay latency for all clients. This translates to minimizing the maximum of the overlay latency of all MSNs. Let S denote the set of all MSNs other than the source. Then the two roblems described above can be stated as follows: P: Minimum average-latency degree-bounded directed sanning tree roblem: Find a directed sanning tree, T of G rooted at the MSN, r, satisfying the degree-constraint at each node, such that i S c il r,i is minimized. P: Minimum maximum-latency degree-bounded directed sanning tree roblem: Find a directed sanning tree, T of G rooted at the MSN, r, satisfying the degree-constraint at each node, such that max i S L r,i is minimized. The minimum average-latency degree-bounded directed sanning tree roblem, as well as the minimum maximum-latency degree-bounded directed sanning tree roblem, are NP-hard [6], [3]. For brevity, in the rest of this aer, we will refer to these roblems as the min avg-latency roblem and the min max-latency roblem, resectively. Note that the max and the avg versions of the roblem have very similar formulations. We focus on the min avg-latency roblem because we believe that by weighting the overlay latency costs by the number of clients at each MSN, this roblem better catures the relative imortance of the MSNs in defining the overlay tree. It should be noted, however, that the tree adatation algorithms resented in this aer for the min avg-latency roblem can be easily alied to the min max-latency roblem, by using the max-oerator instead of the sum-oerator in making tree construction decisions. In this aer, we only Internet measurements have shown that links in the core networks are over-rovisioned, and therefore are not bottlenecks [5].

5 5 Term c i C f i,j, x i,j l i,j L r,i N Meaning Number of clients of MSN i Total number of clients aggregated over all MSNs Variables for the integer-rogram Unicast ath latency from MSN i to MSN j Latency along the overlay ath from the root MSN r to MSN i Total number of MSNs in the OMNI TABLE I GLOSSARY OF NOTATION. resent evaluation results for the avg version of the roblem, and exerimentation for the max case is left as future work. B. Integer-rogramming formulation: Now we resent a linear integer-rogramming formulation for the avg-latency roblem. It is worth noting here that develoing a nonlinear integer-rogramming formulation for this roblem is not difficult. However, nonlinear integer-rograms are usually harder to solve. In contrast, aroximate (and sometimes exact) solutions to linear integer-rograms can be efficiently obtained using common integer-rogram solvers (like CPLEX). In the linear integer-rogramming formulation described below, the number of variables and constraints are also linear in the size of the OMNI, which makes the comutation more feasible. We should note that we do not use this rogramming formulation directly for develoing our solution aroach (for reasons outlined in the next section). The formulation is nevertheless imortant since it allows us to comute the otimal solution efficiently. We use this formulation to comute the otimal tree and comare the erformance of our distributed solution with the otimal tree thus obtained. We demonstrate that the distributed aroach yields a solution that is fairly close to the otimal solution of the integer-rogram osed next. For each edge i, j E in grah G, define two variables: a binary variable x i,j, and a non-negative real (or integer) 3 variable f i,j, where x i,j denotes whether or not the edge i, j is included in the tree, and f i,j denotes the total number of clients served through edge i, j. Then the avg-latency roblem can be formulated as: minimize C i,j E l i,j f i,j 3 We define f i,j as real-valued variables rather than integer variables for reasons of efficiency of solving the integer-rogram. However, in the otimal solution, the variables f i,j will turn out to be integers.

6 6 subject to: f k,i f i,k = c i i V \ {r} () k V \{i} k V \{i} 0 f i,j Cx i,j i, j E () x i,j N (3) i,j E x i,j {0, } i, j E () In Constraint 3, N is the total number of MSNs. In Constraint and in the objective function, C is the total number of clients served by the OMNI. The objective function, as well as Constraint, follow from the definition of the variables f i,j. Constraint ensures that the variable f i,j is zero if x i,j is zero. Constraint 3 is necessary to enforce the tree structure of the OMNI overlay. All the constraints together ensure that the solution is a sanning tree rooted at r. 0 (root) Fig.. An examle grah. MSN i serves c i clients, i =,..., 6. To get a clearer understanding of the above formulation, let us take a look at a feasible solution to the above integer-rogram in a articular roblem instance. Consider the 7-node grah shown in Figure. Each node corresonds to an MSN, and MSN 0 is the root of the OMNI tree. Since this grah is a comlete grah, it has bidirectional edges (i.e., one bidirectional edge corresonding to each air of MSNs). In the figure, however, we show only 9 of these edges. The total number of clients, C, is equal to c + c + c 3 + c + c 5 + c 6. Consider the OMNI tree formed by the six directed edges shown in the figure. For this tree, the variables x i,j and f i,j are obtained as follows. The variable x i,j is equal to if edge i, j belongs to the OMNI tree, and is 0 otherwise. The variable f i,j is 0 if edge i, j does not belong to the OMNI tree. The non-zero f i,j variables are obtained as: f,3 = c 3, f,6 = c 6, f, = c + c 6, f 0, = c + c 3 + c + c 6, f,5 = c 5 and f 0, = c + c 5. It can be verified that these variables satisfy the constraints of the above formulation. Moreover, note that the average latency of clients in this tree is (/C){c l 0, + c l 0, + c 3 (l 0, + l,3 ) + c (l 0, + l, ) + c 5 (l 0, + l,5 ) + c 6 (l 0, + l, + l,6 )}, which is

7 equal to C i,j E l i,jf i,j, the objective function of our formulation. Similarly, it can also be verified that any feasible solution to our rogramming formulation reresents an OMNI tree, with the objective function denoting the average latency of clients for that tree. 7 C. Solution Requirements Note that the integer-rogramming formulation osed above is a fairly comlex roblem. While this formulation may be used to obtain the otimal tree for some small toologies (say u to 6 MSNs), it is ractically infeasible to use such an integer-rogramming solution in a large scale distributed network environment. This motivates us to look for efficient heuristics for solving this roblem. In this aer, we describe an iterative heuristic aroach that can be used to solve the min avg-latency roblem. In the solution descrition, we also briefly highlight the changes necessary to our distributed solution to better solve the min max-latency roblem that has been addressed in rior work [3]. The develoment of the our aroach is motivated by the following set of desirable features that make the solution scheme ractical. Decentralization: We require a solution to be to imlementable in a distributed manner. It is ossible to think of a solution where the information about the client sizes of the MSNs and the unicast ath latencies are conveyed to a single central entity, which then finds a good tree (using some algorithm), and then directs the MSNs to construct the tree obtained. However, the client oulation can change dynamically at different MSNs which would require frequent re-comutation of the overlay tree. Similarly, changes in network conditions can alter latencies between MSNs which will also incur tree re-comutation. Therefore a centralized solution is not ractical for even a moderately sized OMNI. Adatation: The OMNI overlay should adat to changes in network conditions and changes in the distribution of clients at the different MSNs. Feasibility: The OMNI overlay should adat the tree structure by making incremental changes to the existing tree. However at any oint in time the tree should satisfy all the degree constraints at the different MSNs. Any violation of degree constraint would imly an interrution of service for the clients. Therefore, as the tree adats its structure towards an otimal solution using a sequence of otimization stes, none of the transformations should violate the degree constraints of the MSNs. Our solution, as described in the next section, satisfies all the roerties stated above. III. SOLUTION In this section we describe our roosed distributed iterative solution to the roblem described in Section II that meets all of the desired objectives. In this solution descrition, we focus on the min avg-latency roblem and only oint out relevant modifications needed for the min max-latency roblem.

8 8 A. State at MSNs For an MSN i, let Children(i) indicate the set of children of i on the overlay tree and let c i denote the number of clients being directly served by i. We use the term aggregate subtree clients (S i ) at MSN i to denote the entire set of clients served by all MSNs in the subtree rooted at i. The number of such aggregate subtree clients, s i = S i is given by: s i = c i + j Children(i) For examle in Figure, s F = 3, s E = 5, s D =, s C = 6, s B = 8, and s A =. We also define a term called aggregate subtree latency (Λ i ) at any MSN, i, which denotes the summation of the overlay latency of each MSN in the subtree, from MSN i which is weighted by the number of clients at that MSN. This can be exressed as: 0 if i is a leaf MSN Λ i = j Children(i) s jl i,j + Λ j otherwise where, l i,j is the unicast latency between MSNs i and j. In Figure, assuming all edges between MSNs have unit unicast latencies, Λ F s j = Λ E = Λ D = 0, Λ C = 3, Λ B = 6, and Λ A = 3. The otimization objective of the min avg-latency roblem is to minimize the average subtree latency of the root, Λ r, (also called the average tree latency). Each MSN i kees the following state information: The overlay ath from the root to itself: This is used to detect and avoid loos while erforming otimization transformations. The value, s i, reresenting the number of aggregate subtree clients. The aggregate subtree latency: This is aggregated on the OMNI overlay from the leaves to the root. The unicast latency between itself and its tree neighbors: Each MSN eriodically measures the unicast latency to all its neighbors on the tree. Each MSN maintains state for all its tree neighbors and all its ancestors in the tree. If the minimum out-degree bound of an MSN is two, then it maintains state for at most O(degree + log N) other MSNs. We decoule our roosed solution into two arts an initialization hase followed by successive incremental refinements. In each of these incremental oerations no global interactions are necessary. A small number of MSNs interact with each other in each transformation to adat the tree so that the objective function imroves. The maximum subtree latency, λ max i overlay latency from i among the MSNs in the subtree rooted at i, i.e. λ max i at an MSN, i, is the overlay latency from i to another MSN j which has the maximum of the min max-latency roblem is to minimize the maximum subtree latency of the root. = max{l i,j j Subtree(i)}. The otimization objective

9 9 Procedure : CreateInitialTree(r, S) SortedS Sort S in increasing order of dist. from r { Assert: SortedS[] = r } i for j to Ndo while SortedS[i].NumChildren = SortedS[i].DegBd i + + end while SortedS[j].Parent SortedS[i] SortedS[i].NumChildren + + end for Fig. 3. Initial tree creation algorithm for the initialization hase. r is the root MSN, S is an array of all the other MSNs and N is the number of MSNs. B. Initialization In a tyical webcast scenario data distribution is scheduled to commence at a secific time. Prior to this instant the MSNs organize themselves into an initial data delivery tree. Note that the clients of the different MSNs join and leave dynamically. Therefore no information about the client oulation sizes is available a riori at the MSNs during the initialization hase. Each MSN that intends to join the OMNI measures the unicast latency between itself and the root MSN and sends a JoinRequest message to the root MSN. This message contains the tule LatencyToRoot, DegreeBound. The root MSN gathers JoinRequests from all the different MSNs, creates the initial data delivery tree using a simle centralized algorithm, and distributes it to the MSNs. This centralized initialization rocedure is described in seudo-code in Figure 3. We describe this oeration using the examle in Figure. In this examle, all MSNs have a maximum out-degree bound of two. The root, r, sorts the list of MSNs in an increasing order of distance from itself. It then fills u the available degrees of MSNs in this increasing sequence. It starts with itself and chooses the next closest MSNs ( and ) to be its children. It next chooses its closest MSN () and assigns MSNs 3 and (the next closest MSNs with unassigned arents) as its children. Continuing this rocess, the tree shown in Figure is constructed. We can obtain the following worst-case result for the centralized algorithm: Lemma : Assume that unicast latencies are symmetric, and follow triangle inequality. Also assume that the degree bound of each MSN is at least. Then overlay latency from the root MSN to any other MSN, i, is bounded by l r,i log N, where N is the number of MSNs in the OMNI, and l r,i is the direct unicast

10 Source r 3 Fig.. Initialization of the OMNI using Procedure CreateInitialTree. r is the root MSN of the tree. The remaining MSNs are labeled in the increasing order of unicast latencies from r. In this examle, we assume that each MSN has a maximum out-degree bound of two. latency from the root MSN, r, to MSN i. Proof: Consider any MSN i in the OMNI constructed by our initialization rocedure. Note that the MSNs were added in the increasing order of their unicast latencies from the root MSN, r. Therefore, for any MSN j that lies in the overlay ath from r to i, l r,j l r,i. Thus for any two nodes j and k on the overlay ath from r to i, l j,k l j,r + l r,k = l r,j + l r,k l r,i (using symmetry and the triangle inequality). Let E i E be the set of edges in the overlay ath from r to i. Since the minimum out-degree of any MSN is two, it follows that E i log N. Let E i E be the set of edges on the overlay ath from r to i. Thus, L r,i, the latency along the overlay ath from the root MSN r to MSN i, can be bounded as: L r,i = (j,k) E i l j,k l r,i E i l r,i log N. The centralized comutation of this algorithm is accetable because it oerates off-line before data delivery commences. An otimal solution to the min avg-latency roblem is NP-Hard and would tyically require O(N ) latency measurements (i.e. between each air of MSNs). In contrast, the centralized solution rovides a reasonable latency bound using only O(N) latency measurements (one between each MSN and the root MSN). Note that the log N aroximation bound is valid for each MSN. Therefore this initialization rocedure is able to guarantee a log N aroximation for both the min avg-latency roblem as well as the min max-latency roblem. The initialization rocedure, though oblivious of the distribution of the clients at different MSNs, still creates a good initial tree. This data delivery tree will be continuously transformed through local oerations to dynamically adat with changing network conditions (i.e. changing latencies between MSNs) and changing distribution of clients at the MSNs. Additionally new MSNs can join and existing MSNs can leave the OMNI even after data delivery commences. Therefore the initialization hase is otional for the MSNs, which can join the OMNI, even after the initialization rocedure is done.

11 g g 3 Available Degree 3 c c Fig. 5. Child-Promote oeration. g is the grand-arent, is the arent and c is the child. The maximum out-degree of all MSNs is three. MSN c is romoted in this examle. g g 3 c Other MSNs 5 3 c Other MSNs 5 Fig. 6. Parent-Child Swa oeration. g is the grand-arent, is the arent and c is the child. Maximum out-degree is three. C. Local Transformations We define a local transformation as one which requires interactions between nearby MSNs on the overlay tree. In articular these MSNs are within two levels of each other. We define five such local transformation oerations that are ermissible at any MSN of the tree. Each MSN eriodically attemts to erform these oerations. This eriod is called the transformation eriod and is denoted by τ. The oeration is erformed if it reduces the average-latency of the client oulation. Child-Promote: If an MSN g has available degree, then one of its grand-children (e.g. MSN c in Figure 5) is romoted to be a direct child of g if doing so reduces the aggregate subtree latency for the min avg-latency roblem. This is true if: (l g,c l g, l,c )s c < 0 For the min max-latency roblem, the oeration is erformed only if it reduces the maximum subtree latency at g which can be verified by testing the same condition as above. If the triangle inequality holds for the unicast latencies between the MSNs, this condition will always be true. If multile children of are eligible to be romoted, a child which maximally reduces the aggregate

12 g g q q x y x y Fig. 7. Iso-level- Swa oeration. g is the grand-arent, and q are siblings. x and y are swaed. x c y 3 x c y 3 Fig. 8. Aniso-level-- Swa oeration. is the arent of c. x and y are swaed. (maximum) subtree latency for the min avg-latency (min max-latency) roblem is chosen. Parent-Child Swa: In this oeration the arent and child are swaed as shown in Figure 6. Note grand-arent, g is the arent of c after the transformation and c is the arent of. Additionally one child of c is transferred to. This is done if and only if the out-degree bound of c gets violated by the oeration (as in this case). Note that in such a case only one child of c would need to be transferred and would always have an available degree (since the transformation frees u one of its degrees). The swa oeration is erformed for the min avg-latency (min max-latency) roblem if and only if the aggregate (maximum) subtree latency at g reduces due to the oeration. Like the revious case, if multile children of are eligible for the swa oeration, a child which maximally reduces the aggregate (maximum) subtree latency for the min avg-latency (min max-latency) roblem is chosen. Iso-level- Swa: We define an iso-level oeration as one in which two MSNs at the same level swa their ositions on the tree. Iso-level-k denotes a swa where the swaed MSNs have a common ancestor exactly k levels above. Therefore, the iso-level- oeration defines such a swa for two MSNs that have the same grand-arent. As before, this oeration is erformed for the min avg-latency (min max-latency) roblem between two MSNs x and y if and only if it reduces the aggregate (maximum) subtree latency (e.g. Figure 7).

13 3 Number of clients served by each MSN at this level r r 3 3 q 3 q x y x y Fig. 9. Examle where the five local oerations cannot lead to otimality in the min avg-latency roblem. All MSNs have maximum out-degree bound of two. r is the root. Arrow lengths indicate the distance between MSNs. Iso-level- Transfer: This oeration is analogous to the revious oeration. However, instead of a swa, it erforms a transfer. For examle, in Figure 7, Iso-level- transfer would only shift the osition of MSN x from child of to child of q. MSN y does not shift its osition. This oeration is only ossible if q has available degree. Aniso-level-- Swa: An aniso-level oeration involves two MSN that are not on the same level of the overlay tree. An aniso-level-i-j oeration involves two MSNs x and y for which the ancestor of x, i levels u, is also the ancestor of y, j levels u. Therefore the defined swa oeration involves two MSNs x and y where the arent of x is the same as the grand-arent of y (as shown in Figure 8). The oeration is erformed if and only if it reduces the aggregate (maximum) subtree latency at for the min avg-latency (min max-latency) roblem. Following the terminology as described, the Child-Promote oeration is actually the Aniso-level-- transfer oeration. D. Probabilistic Transformation Each of the defined local oerations reduce the aggregate (maximum) subtree latency on the tree for the min avg-latency (min max-latency) roblem. Performing these local transformations will guide the objective function towards a local minimum. However, as shown in the examle in Figure 9, they alone cannot guarantee that a global minimum will be attained. In the examle, the root MSN suorts clients. MSNs in level (i.e. and ) suort 3 clients each, MSNs in level suort clients each and MSNs in level 3 suort a single client each. The arrow lengths indicate the unicast latencies between the MSNs. Initially l,y + l q,x < l,x + l q,y and the tree as shown in the initial configuration was formed. The tree in the initial configuration was the otimal tree for our objective function. Let us assume that due to changes in network conditions (i.e., changed unicast latencies) we now have l,y + l q,x > l,x + l q,y. Therefore the objective function can now be imroved by exchanging the ositions of MSNs x and y in the tree. However, this is an

14 iso-level-3 oeration, and is not one of the local oerations. Additionally it is easy to verify that any local oeration to the initial tree will increase the objective function. Therefore no sequence of local oeration exists that can be alied to the initial tree to reach the global minima. Therefore we define a robabilistic transformation ste that allows MSNs to discover such otential imrovements to the objective function and eventually converge to the global minima. In each transformation eriod, τ, an MSN will choose to erform a robabilistic transformation with a low robability, rand. If MSN i chooses to erform a robabilistic transformation in a secific transformation eriod, it first discovers another MSN, j, from the tree that is not its descendant. This discovery is done by a random-walk on the tree, a technique roosed in Yoid [7]. In this technique, MSN i transmits a Discover message with a time-to-live (TTL) field to its arent on the tree. The message is randomly forwarded from neighbor to neighbor, without re-tracing its ath along the tree and the TTL field is decremented at each ho. The MSN at which the TTL reaches zero is the desired random MSN. Random Swa: We erform the robabilistic transformation only if i and j are not descendant and ancestor of each other. In the robabilistic transformation, MSNs i and j exchange their ositions in the tree. For the min avg-latency (min max-latency) roblem, let denote the increase in the aggregate (maximum) subtree latency of MSN k which is the least common ancestor of i and j on the tree (in Figure 9, this is the root MSN, r). k is identified by the Discover message as the MSN where the message stos its ascent towards the root and starts to descend. For the min avg-latency roblem, can be comuted as follows: = (L k,i L k,i )s i + (L k,j L k,j )s j where, L k,i and L k,j denote the latencies from k to i and j resectively along the overlay if the transformation is erformed, and L k,i and L k,j denotes the same rior to the transformation. Each MSN maintains unicast latency estimates of all its neighbors on the tree. The Discover message aggregates the value of L k,j on its descent from k to j from these unicast latencies. Similarly, a searate TreeLatency message from k to i comutes the value of L k,i. (We use a searate message from k to i since we do not assume symmetric latencies between any air of MSNs.) The L values is comuted from the L values and air-wise unicast latencies between i, j and their arents. Thus, no global state maintenance is required for this oeration. We use a simulated annealing [8] based technique to robabilistically decide when to erform the swa oeration. The swa oeration is erformed: () with a robability of if < 0, and () with a robability e /T if 0, where T is the temerature arameter of the simulated annealing technique. In the min avg-latency (min max-latency) roblem The swa oeration is erformed with a (low) robability even if the aggregate (maximum) subtree latency increases. This is useful in the search for a global otimum in the solution sace. Note that the robability of the swa gets exonentially smaller with increase in.

15 5 JoinRequest n c Join n c Join n c 5 n JoinRequest c : Join at available degree : Slit edge and Join 3: Re-try at next level Fig. 0. Join oeration for a new MSN. At each level there are three choices available to the joining MSN as shown. For each MSN, the maximum out-degree bound is 3. E. Join and Leave of MSNs In our distributed solution, we allow MSNs to arbitrarily join and leave the OMNI overlay. In this section, we describe both these oerations in turn. Join: A new MSN initiates its join rocedure by sending the JoinRequest message to the root MSN. JoinRequest messages received after the initial tree creation hase invokes the distributed join rotocol (as shown in Figure 0). At each level of the tree, the new MSN, n, has three otions. ) Otion : If the currently queried MSN,, has available degree, then n joins as its child. Some of the current children of c (i.e. and ) may later join as children of n in a later Iso-level- transfer oeration. ) Otion : n chooses a child, c, of and attemts to slit the edge between them and join as the arent of c. Additionally some of the current children of c are shifted as children of n. 3) Otion 3: n re-tries the join rocess from some MSN, c. Otion has strict recedence over the other two cases. If otion fails, then we choose the lowest cost otion between and 3. The cost for otion can be calculated exactly through local interactions between n,, c and the children of c. The cost of otion 3 requires the knowledge of exactly where in the tree n will join. Instead of this exact comutation, we comute the cost of otion 3 as the cost incurred if n joins as a child of c. This leads to some inaccuracy which is later handled by the cost-imroving local and robabilistic transformations. Leave: If the leaving MSN is a leaf on the overlay tree, then no further change to the toology is required 5. Otherwise, one of the children of the dearting MSN is romoted u the tree to the osition occuied by the dearting MSN. We show this with an examle in Figure. When MSN 3 leaves, one of its children ( in this case) is romoted. For the min avg-latency (min max-latency) roblem the child is chosen such 5 The clients of the leaving MSNs need to be re-assigned to some other MSN, but that is an orthogonal issue to OMNI overlay construction.

16 6 3 Leaving MSN Fig.. Leave oeration of an MSN. The maximum out-degree of each MSN is two. Overlay of 6 MSNs ( = 0.0 and T = 0.0) Overlay of 6 MNs (T = 0.0, Initialization used) Average Tree Latency (ms) Greedy No Initialization With Initialization Average Tree Latency (ms) No random swa = 0.0 = 0.05 = Time (units of Transformation Period) Time (units of Transformation Period) Fig.. Effect of the initialization hase (6 MSNs). Fig. 3. Varying the robability of erforming the random-swa oeration for the different MSNs (6 MSNs). that the aggregate (maximum) subtree latency is reduced the most. The other children of the dearting MSN join the subtree rooted at the newly romoted child. For examle, 5 attemts to join the subtree rooted at. It alies the join rocedure described above starting from MSN, and is able to join as a child of MSN 7. Note that MSNs are secially managed infrastructure entities. Therefore it is exected that their failures are rare and most deartures from the overlay will be voluntary. In such scenarios the overlay will be aroriately re-structured before the dearture of an MSN takes effect. It is also worth noting here that in the distributed adatation schemes described above, we require that a node does not simultaneously articiate in more than one transformation at a given time. This revents the occurrence of a race condition due to simultaneous transformations. IV. SIMULATION EXPERIMENTS We have studied the erformance of our roosed distributed scheme through detailed simulation exeriments. Our network toologies for these exeriments were generated using the Transit-Stub grah model

17 7 Average Tree Latency (ms) Overlay of 6 MSNs ( = 0.0, Initialization used) T = 5.0 T = 0.0 T = Time (units of Transformation Period) Fig.. Varying the temerature arameter for simulated-annealing (6 MSNs). of the GT-ITM toology generator [9]. All toologies in these simulations had 0, 000 nodes (reresenting network routers) with an average node degree between 3 and. MSNs were attached to a set of these routers, chosen uniformly at random. As a consequence unicast latencies between different airs of MSNs varied between and 00 ms. The number of MSNs was varied between 6 and 5 for different exeriments. In our exeriments we comare the erformance of our distributed iterative scheme to these other schemes: The otimal solution: We comuted the otimal value of the roblem by solving the integer-rogram formulated in Section II-B, using the CPLEX tool 6. Comutation of the otimal value using an IP requires a search over a O(M N ) solution sace, where M is the total number of clients and N is the number of MSNs. We were able to comute the otimal solution for networks with u to 00 clients and 6 MSNs. A centralized greedy heuristic solution: This heuristic is a simle variant of the Comact Tree algorithm roosed in [3]. It incrementally builds a sanning tree from the root MSN, r. For each MSN v that is not yet in the artial tree T, we maintain an edge e(v) = {u, v} to an MSN u in the tree; u is chosen to minimize a cost metric δ(v) = (L r,u + l u,v )/c v where, L r,u is the overlay latency from the root of the artial tree to u and c v is the number of clients being served by v. At each iteration we add one MSN (say v) to the artial tree which has minimum value for δ(v). Then for each MSN w not in the tree, we udate e(w) and δ(w). The centralized greedy heuristic roosed in [3] addresses the min max-latency roblem. Our simle modification to that algorithm only changes the cost metric and is the equivalent centralized greedy heuristic for the min avg-latency roblem as described in Section II. 6 Available from htt://

18 8 Overlay of 56 MSNs ( = 0.0 T = 0.0) Overlay of 56 MSNs (T = 0.0, Initialization used) Average Tree Latency (ms) Greedy No Initialization With Initialization Average Tree Latency (ms) No random swa = 0.0 = 0.05 = Time (units of Transformation Period) Time (units of Transformation Period) Fig. 5. Effect of the initialization hase (56 MSNs). Fig. 6. Varying the robability of erforming the random-swa oeration for the different MSNs (56 MSNs). Overlay of 56 MSNs ( = 0.0, Initialization used) 85 T = 5.0 T = 0.0 T = 0.0 Average Tree Latency (ms) Time (units of Transformation Period) Fig. 7. Varying the temerature arameter for simulated annealing (56 MSNs). A. Convergence We first resent convergence roerties of our solution for OMNI overlay networks. Figures, 3 and show the evolution of the average tree latency, Λr, (our minimization objective) over time for different exeriment arameters for an examle network configuration consisting of 6 MSNs. The MSNs serve between and 5 clients, chosen uniformly at random for each MSN. In these exeriments the set of 6 MSNs join the OMNI at time zero. We use our distributed scheme to let these MSNs organize themselves into the aroriate OMNI overlay. The x-axis in these figures are in units of the transformation eriod arameter, τ, which secifies the average interval between each transformation attemt by the MSNs. The ranges of the axes in these lots are different, since we focus on different time scales to observe the interesting characteristics of these results.

19 9 Figure shows the efficacy of the initialization hase. When none of the MSNs make use of the initialization hase, the initial tree has Λ r = 58.9 ms. In contrast, if the initialization hase is used by all MSNs, the initial tree has Λ r = 33.8 ms, a 6% reduction in cost. In both cases, however, the overlay quickly converges (within < 8 transformation eriods) to a stable value of Λ r.5 ms. The otimal value comuted by the IP for this exeriment was 3.96 ms. Thus, the cost of our solution is about 9% higher than the otimal. We ran different exeriments for different network configurations and found that our distributed scheme converges to within 5 9% of the otimum in all cases. A greedy aroach to this roblem does not work quite as well. The centralized greedy heuristic gives a solution with value 5.59 ms, and is about % higher than the converged value of the distributed scheme. In both these cases we had chosen the robability of a random-swa, rand, at the MSNs to be 0. and the T arameter of simulated-annealing to be 0. In Figure 3 we show how the choice of rand affects the results. The initialization hase is used by MSNs for all the results shown in this figure. The local transformations occur quite raidly and quickly reduces the cost of the tree for all the different cases. The rand = 0 case has no robabilistic transformations and is only able to reach a stable value of 9.5 ms. Clearly, once the objective reaches a local minimum it is unable to find a better solution that will take it towards a global minimum. As rand increases, the search for a global minimum becomes more aggressive and the objective function reaches the lower stable value raidly. Figure shows the corresonding lots for varying the T arameter. A higher T value in the simulated-annealing rocess imlies that a random swa that leads to cost increment is ermitted with a higher robability. For the moderate and high value of T (0 and 0), the schemes are more aggressive and hence the value of Λ r exeriences more oscillations. In the rocess both these schemes are aggressively able to find better solutions to the objective function. The oscillations are restricted to within % of the converged value. Figures 5, 6, and 7 show the corresonding lots for exeriments with 56 MSNs. Note that for the 56 MSN exeriments, the best solution found by different choice of arameters has Λ r = 8.53 ms. Our distributed solution converges to this value after 7607 transformation eriod (τ) units. Since our distributed solution converges to within 5% of the best solution within 5 transformation eriods, the time to convergence really deends on the choice of the transformation eriod. In a deloyed scenario, the transformation eriod is exected to be fairly large (say about 30 secs or so) but can be adatively set to lower values if network and toology roerties change quickly. Figure 7 shows the effect of the temerature arameter for the convergence. As before the oscillations are higher for higher temeratures, but are restricted to less than % of the converged value (the y-axis is magnified to illustrate the oscillations in this lot). This exeriment also indicates that a greedy aroach does not work well for this roblem. The solution found by the greedy heuristic for this network configuration

20 0 Number Distributed Centralized Greedy/Iterative of MSNs Iterative Scheme Greedy Scheme Ratio TABLE II COMPARISON OF THE BEST SOLUTION (IN MS) OF THE AVERAGE TREE LATENCY OBTAINED BY OUR PROPOSED DISTRIBUTED ITERATIVE SCHEME AND THE CENTRALIZED GREEDY HEURISTIC WITH VARYING OMNI SIZES, AVERAGED OVER 0 RUNS EACH. is 3% higher than the one found by our roosed technique. We resent a comarison of our scheme with the greedy heuristic in Table II. We observe that the erformance of our roosed scheme gets rogressively better than the greedy heuristic with increasing size of the OMNI overlay. The control overhead of our aroach is quite low. Under stable conditions based on our exeriments, the number of control messages sent by each MSN in a transformation eriod (say about 30 seconds) is roortional to its degree in the overlay structure. Even under very drastic join-leave scenarios like the one shown in Figure 9 (note that in this exeriment, 6 MSNs join/leave simultaneously in an OMNI with 8 MSNs), the total number of control messages exchanged across all MSNs is observed to be quite small. For the case in Figure 9, the number of messages exchanged is about 00, over 0 transformation eriods. For all ractical uroses, this overhead is negligible. B. Adatability We next resent results of the adatability of our distributed scheme for MSN joins and leaves, changes in network conditions and changing distribution of client oulations. MSNs join and leave: We show how the distributed scheme adats the OMNI as different MSNs join and leave the overlay. Figure 8 lots the average tree latency for a join-leave exeriment involving 8 MSNs. In this exeriment, 8 MSNs join the OMNI during the initialization hase. Every 500 transformation eriods (marked by the vertical lines in the figure), a set of MSNs join or leave. For examle, at time 6000, 6 MSNs join the OMNI and at time 7500, 6 MSNs leave the OMNI. These bulk changes to the OMNI are equivalent to a widesread network outage, e.g. a network artition. The other changes to the OMNI are much smaller, e.g. 8-3 simultaneous changes as shown in the figure. In each case, we let the OMNI

21 Average Tree Latency (ms) Overlay of 8-8 MSNs ( = 0.0 T = 0.0, Initialization used) 8 Join 8 Join 6 Join 3 Join 6 Join Leave Leave Leave Leave Number of transformations Overlay of 8-8 MSNs ( = 0.0 T = 0.0) 8 Join (Time 0) 8 Join (Time 500) 6 Join (Time 6000) 6 Leave (Time 7500) 8 Leave (Time 000) Time (units of Transformation Period) Fig. 8. Join leave exeriments with 8 MSNs. The horizontal lines mark the solution obtained using the greedy heuristic Time (units of Transformation Period) Fig. 9. Distribution of number of transformations in the first 0 transformation eriods after a set of changes haen in the join leave exeriment with 8 MSNs. converge before the next set of changes is effected. In all these changes the OMNI reaches to within 6% of its converged value of Λ r within 5 transformation eriods. In Figure 9 we show the distribution of the number of transformations that haen in the first 0 transformation eriods after a set of changes. (We only lot these distributions for 5 sets of changes initial join of 8 MSNs, 8 MSNs join at time 500, 6 MSNs join at time 6000, 6 MSNs leave at time 7500, and 8 MSNs leave at time 000.) The bulk of the necessary transformations to converge to the best solution occur within the first 5 transformation eriods after the change. Of these a vast majority (more than 97%) are due to local transformations. These results suggest that the transformation eriod at the MSNs can be set to a relatively large value (e.g. minute) and the OMNI overlay would still converge within a short time. It can also be set adatively to a low value when the OMNI is exeriencing a lot of changes for faster convergence and a higher value when it is relatively stable. Changing client distributions and network conditions: A key asect of the roosed distributed scheme is its ability to adat to changing distribution of clients at the different MSNs. In Figure 0, we show a run from a samle exeriment involving 6 MSNs. In this exeriment, we allow a set of MSNs to join the overlay. Subsequently we varied the number of clients served by MSN x over time and observed its effects on the tree and the overlay latency to MSN x. The figure shows the time evolution of the relevant subtree fragment of the overlay. In its initial configuration, the overlay latency from MSN 0 to MSN x is 59 ms. As the number of clients increases to 7, the imortance of MSN x increases. It eventually changes its arent to MSN (Panel ), so that its overlay latency reduces to 5 ms. As the number of clients increases to 9, it becomes a direct

22 Overlay latency for x 3 L0,x = 59 ms L0,x = 5 ms L0,x = 5 ms L0,x = 5 ms L0,x = 59 ms L0,x = 7 ms 0 Other MSN Other MSN Client size at x x x 3 7 x x 9 5 x x Cx = 5 Cx = 7 Cx = 9 Cx = 7 Cx = 3 Cx = Cx = 5 Fig. 0. Dynamics of the OMNI as number of clients change at MSNs (6 MSNs). MSN 0 is the root. MSNs 0,, and 6 had out-degree bound of each and MSNs 7 and x had out-degree bound of 3 each. We varied the number of clients being served by MSN x. The relevant unicast latencies between MSNs are as follows: l 0, = 9 ms, l 0,6 = 5 ms, l 0,7 = ms, l 0,x = 5 ms, l,x = 30 ms, l 6, = ms, l 6,7 = 8 ms, l 6,x = 9 ms, l 7,x = 9 ms. c x indicates the number of clients at MSN x which changes with time. The time axis is not drawn to scale. child of the root MSN (Panel ) with an even lower overlay latency of 5 ms. Subsequently the number of clients of MSN x decreases. This causes x to migrate down the tree, while other MSNs with larger client sizes move u. This examle demonstrates how the scheme rioritizes the MSNs based on the number of clients that they serve. The roosed techniques are fairly resilient to message losses and node failures. Note that our roosed method is comletely distributed and relies on eriodic control messages for all actions. So loss of a message will only lead to temorary inaccuracies. The effect of a node failure will be similar to that of the join-leave exeriments discussed above. The results of these exeriments demonstrate the relatively quick convergence of the roosed scheme in resonse to bulk failures of the MSNs that are art of the OMNI. We also erformed similar exeriments to study the effects of variable unicast latencies on the overlay structure. If the unicast latency on a tree edge between arent MSN x and one of its children, MSN y, goes u, the distributed scheme simly adats the overlay by finding a better oint of attachment for MSN y. Therefore, in one of our exeriments, we icked an MSN directly connected to the root and increased its unicast latencies to all other MSNs (including the root MSN). A high latency edge close to the root affects a large number of clients. In this exeriment, our distributed scheme adated the overlay to reduce the average tree latency by moving this MSN to a leaf osition in the tree, so that it cannot affect a large number of clients. Therefore, our aroach is fairly robust to latency variations in the Internet. V. RELATED WORK A number of other rojects (e.g. Narada [0], NICE [], Yoid [7], Gossamer [],Overcast [],ALMI [3], Scribe [], Bayeux [5], multicast-can [6], ZIGZAG [7]) have exlored imlementing multicast at the alication layer. However, in these rotocols the end-hosts are considered to be equivalent eers and are organized into an aroriate overlay structure for multicast data delivery. Additionally, none of these