Optimal Peer-to-Peer Technique for Massive Content Distribution

Transcription

1 1 Optimal Peer-to-Peer Technique for Maive Content Ditribution Xiaoying Zheng, Chunglae Cho and Ye Xia Computer and Information Science and Engineering Department Univerity of Florida {xiazheng, ccho, Abtract A ditinct trend ha emerged that the Internet i ued to tranport data on a more and more maive cale. Capacity hortage in the backbone network ha become a genuine poibility, which will be more eriou with fiber-baed acce. The problem addreed in thi paper i how to conduct maive content ditribution efficiently in the future network environment where the capacity limitation can equally be at the core or the edge. We propoe a novel peer-to-peer technique a a main content tranport mechanim to achieve efficient network reource utilization. The technique ue multiple tree for ditributing different file piece, which at the heart i a verion of warming. In thi paper, we formulate an optimization problem for determining an optimal et of ditribution tree a well a the rate of ditribution on each tree under bandwidth limitation at arbitrary place in the network. The optimal olution can be found by a ditributed algorithm. The reult of the paper not only provide tand-alone olution to the maive content ditribution problem, but hould alo help the undertanding of exiting ditribution technique uch a BitTorrent or FatReplica. Index Term Content Ditribution, Peer-to-peer Network, Multicat, Optimization, Bandwidth Allocation, Congetion Control I. INTRODUCTION One of the ditinct trend i that the Internet i being ued to tranfer content on a more and more maive cale. The gra root initiative for maive content ditribution by the uer and reearch community ha been a peer-to-peer (PP) networking technique known a warming. In a warming eion, the file to be ditributed i broken into many chunk at the original ource, which are then pread out acro the peer in a fahion that the et of chunk at different peer are ubtantially different. Subequently, the peer can exchange the chunk with each other to peed up the ditribution proce. The above decription of warming doe not pecify the precie manner at which the chunk are initially pread out or the manner at which the peer exchange them later. In fact, many different way of warming have been propoed, uch a BitTorrent [1], FatReplica [], [3], Bullet [4], [5], Chunkcat [6], CoBlitz [7], and Julia [8]. The mot popular one among them i the BitTorrent protocol. Swarming enable content provider with poor capacity to reach a large number of audience, allow rapid deployment of a large ditribution ytem with minimum infratructure upport, and i increaingly ued in maintream and critical network ervice. On the negative ide, warming traffic ha made capacity hortage in the backbone network a genuine poibility, which will become more eriou with fiber-baed acce 1. A an example, with it early adoption of FTTH, by 5, Japan already aw 6% of it backbone network traffic being from reidential uer to uer, which wa conumed by content downloading or PP file haring; the fiber uer were reponible for 86% of the inbound traffic; and the traffic wa rapidly increaing, by 45% that year [11]. The problem addreed in thi paper i how to conduct maive content ditribution efficiently in the future network environment where the capacity limitation can equally be at the core or the edge. The propoed olution i a cla of improved warming technique, known a optimal warming. In thi paper, warming i viewed not jut a a caual technique for end-uer file-haring application. The benefit of warming ret upon the fact it i an advanced form of networking mechanim, more advanced and more powerful than all previou ditribution cheme, including IP or application-level multicat (e.g. [1]), network cache ytem and exiting content ditribution network (e.g., Akamai [13]). A will be een, warming can be thought a ditributing content on multiple multicat tree. When done properly, it provide the mot efficient utilization of the network capacity, a remedy for backbone congetion, or give the fatet ditribution. For illutration of the main idea in thi paper, conider the toy example in Fig. 1. The number aociated with the link are their capacitie. Suppoe a large file i plit into many chunk at ource node 1. We wih to find the fatet way to ditribute all chunk to receiver and 3. We impoe no retriction on how peer can help each other in the ditribution proce. Let u focu on a fixed chunk and conider how it can be ditributed to the receiver. With ome thought, it can be argued that, when the delay i not modeled, the path hould be a tree rooted at the ource and covering both receiver. All poible ditribution tree are hown in Fig.. The quetion become how to aign the chunk to different ditribution tree o that the ditribution time i minimized, ubject to the link capacity contraint. For thi imple example, it i eay to 1 At the preent, telecom companie are aggreively rolling out fiber-to-the home (FTTH) or it variant. Verizon i building a nationwide FTTH network in the US, to be completed by 1. Japan had 5.6 million FTTH ubcriber by June, 6, i on an exponential upward ramp to have 3 million by 1 [9]. The peed of the acce fiber i currently at 1 Mbp or lower, heading to 1 Gbp by and i likely to reach 1 Gbp thereafter [1]. A we will how, the objective of minimizing the ditribution time i equivalent to maximizing the ditribution throughput, which i alo equivalent to minimizing the network congetion.

2 ee that ditributing the chunk in 1: ratio on the econd and third tree, while leaving the firt unued, i optimal. Fig. 1. Node 1 end the file to node and Fig.. All poible ditribution tree for the example in Fig The idea contained in the toy example are alo given in [14]. That work focue on how to compute the maximum throughput, which lead to the fatet ditribution. Our contribution in thi paper i on developing ditributed algorithm to identify and ue the optimal ditribution tree, and at the ame time, allocate correct bandwidth on the elected tree. Our approach illutrated by the toy example can be contrated with exiting PP ditribution ytem or technique [1] [8]. Being originally deigned for end-ytem file-haring application, the current warming ytem leave much room for improvement. Mot ytem only elect and olve part of the problem and do o with poorly undertood heuritic. Not enough i known on how well they work or how much improvement remain poible. They have uer-centric performance objective, but motly ignore the important networkcentric objective, uch a minimizing network congetion. The bandwidth bottleneck i often aumed at the acce link, rather than throughout the network. Our olution will be able to automatically adapt to capacity contraint anywhere in the network. Furthermore, within our framework, we are able to olve four eparate problem jointly and optimally: peer election (from which peer to download), chunk election (which chunk to download from a peer), ditribution tree election and bandwidth allocation. In contrat, other PP ytem olve one or two of thee problem in iolation, typically uing ad-hoc approache. The paper i organized a follow. The model and problem formulation are given in Section II. The ditributed algorithm i given in Section III. In Section IV, we dicu practical iue in applying our algorithm to realitic etting, uch a calability and coping with network dynamic and churn. In Section V, we evaluate the performance of our algorithm, including a comparion with BitTorrent and FatReplica. In Section VI, we dicu additional related work. The concluion i drawn in Section VII II. PROBLEM DESCRIPTION We will tart with a formulation for optimal content ditribution on a generic network. It turn out the problem i difficult on an arbitrary network. However, for overlay content ditribution, the problem i far eaier. We will give formulation for two poible cenario of overlay ditribution. A. Optimal Multicat Tree Packing Let the network be repreented by G = (V,E), where V i the et of node and E i the et of link. The capacity aociated with each link e E i. The utilization of link e, a meaure of link congetion, i denoted by µ e. We define a multicat eion a a group of node (member) exchanging the ame file. In a eion, ome member own ome ditinct chunk of a large file (The cae of overlapping content at different node require a minor extenion, which will be dicued in Section IV-A.) and we call thoe member ource of the eion. A reaonable aumption about a eion i that all member in the eion are intereted in the file, and at the end of file ditribution, every member in the eion will have a complete copy of the file. Let M be the et of all multicat eion. For each eion m M, let V (m) V repreent the et of member in eion m, and let S (m) V (m) be the et of ource in eion m. For each ource S (m), let L (m) be the total ize of the file chunk at ource for eion m. Let the et of all poible multicat tree panning all member in the eion rooted at ource S (m) be denoted by T (m). A multicat tree may contain node not in the eion, in which cae the tree i called a Steiner tree. In the cae where all node on the tree belong to the eion, the multicat tree i called a panning tree, meaning it pan the multicat eion (rather than the whole network V ). For the i th tree t (m) T (m), where the order of indexing i arbitrary, denote to be the ending rate on tree t (m) from the root. A traightforward objective i to minimize the overall downloading time for all multicat eion, which i to minimize the wort downloading time aociated with any ource in any eion m. With ome thought, we ee that no difference i made in term of the achievable downloading time if we aume that all ource in all eion finih their ditribution at the ame time. We can then minimize thi common duration t. Let the total rate on a link e E be denoted by x e. It i equal to the um of all the rate on all the tree paing through link e, acro all eion and all ource. That i, x e = m M S (m) i:e t (m) The optimization problem i a follow.. (1) min t ().t. t T (m) i=1 = L (m), S (m), m M (3) x e, e E (4), i = 1,, T (m), S (m), m M. (5)

3 3 Condition (3) ay that, if one look at all the multicat tree rooted at a ource for a eion m, the um of the ditribution rate on all thee tree, multiplied by the ditribution time, hould be equal to the total ize of all the file chunk tored at ource for eion m. Thi mean that every bit of the file tored at mut be tranmitted exactly once. A moment of thinking reveal that nothing i gained by tranmitting the ame bit more than once. Condition (4) i the link capacity contraint. At each link e, the flow rate on the link hould be no greater than the link capacity,. It turn out the above problem i equivalent to a minimizingcongetion problem. Thi i immediate if we define y (m) t = and make the ubtitution of variable. But, we will do thi a little differently for eae of interpretation. Let z (m) = T (m) i=1 be the total ending rate at a ource node of eion m. Select a et of contant {r (m) }, each being proportional to L (m) with the ame contant proportional factor. Each r (m) i undertood a a rate. Conider a feaible olution { } and t. By (3), z(m) i proportional to L (m), the total ize of the chunk at for eion m. Then, z (m) = γr (m), for ome contant γ >. Next, define µ = 1/γ. We then make the ubtitution of variable by t = µl(m), and r (m) redefine to be µ. We get the following minimizingcongetion formulation. min µ (6).t. T (m) i=1 = r (m), S (m), m M (7) x e µ, e E (8), i = 1,, T (m), S (m), m M. (9) In the above formulation, r (m) can be undertood a the demanded rate, and µ i the maximum link utilization, which alo meaure the wort link congetion. The problem i to minimize the wort link congetion ubject to the fulfillment of all demanded rate. Thu, we have two equivalent view of optimal multicat tree packing. In the firt view, the objective i to minimize the overall ditribution time (or maximize the ditribution throughput) while atifying the link capacity contraint. In the econd view, the objective i to bet balance the network load while atifying the rate demand for all ource and all eion. Another minor reformulation will be helpful later. Let µ e tand for the utilization of link e, and let µ denote the vector of µ e over all link. Let µ denote the maximum norm, i.e., µ = max e E µ e. The above minimizing-congetion formulation i equivalent to the following. min µ (1).t. T (m) i=1 = r (m), S (m), m M (11) x e = µ e, e E (1), i = 1,, T (m), S (m), m M. (13) The optimization problem propoed o far i equivalent to the problem of packing Steiner tree [15], [16], which i computationally intractable. Fortunately, for PP content ditribution, the problem become impler. The main reaon i that the overlay network of each eion conit of exactly thoe node in the eion. Given uch an overlay network, any Steiner tree that cover all node of the eion i in fact a panning tree. We will how later that the algorithm ued to olve the optimization problem involve a minimum-cot panning tree problem in each iteration. Should ome Steiner node exit, it would have involved a minimum-cot Steiner tree problem. The former i far more tractable than the latter NP-hard problem. One hould be reminded that, although the computation for the overlay network cae i far eaier, the achievable performance i ub-optimal ince the overlay edge are determined by the fixed underlay routing. Unlike the inflexible underlay routing, the bandwidth of the overlay edge may have alternative. Next, we conider two cae, both of which can be ueful. B. Fixed Overlay Link Bandwidth In thi cae, the bandwidth of each overlay link i a fixed contant 3 determined by a bandwidth allocation cheme external to our problem. For intance, the bandwidth may be determined by the end-to-end TCP control, or by bandwidth allocation algorithm that enforce other allocation policie uch a the max-min fairne [17]. We aume the overlay node know about the bandwidth on each overlay link. For intance, in the cae of TCP, the overlay link bandwidth can be meaured. When the overlay link bandwidth i fixed, different ditribution eion become decoupled. We then have M totally independent overlay network. The original optimization problem become eparated to M identical but independent optimization problem. The olution to thi problem will require running algorithm only at the overlay node, making deployment eay. We illutrate thi by focuing on one of the overlay network, which correpond to one eion. Let Ĝ = (ˆV,Ê) repreent the overlay network. For all other notation, ince there i no danger of confuing them with earlier definition, we will re-define them. The bandwidth aociated with each overlay link e Ê i, which i allocated already and i a contant. The utilization of overlay link e i denoted by µ e. Aume S ˆV i the et of ource. Let T repreent the et of all poible (overlay) multicat tree rooted at ource panning all overlay node. Let t T be the i th (overlay) multicat tree and z be the aociated ending rate on tree t. The rate on the overlay link e Ê i x e = z (14) S i:e t 3 By fixed overlay bandwidth, we do not mean the bandwidth may not vary over time. We imply mean that the overlay link bandwidth i not determined by our algorithm and i known at the time of running the algorithm.

4 4 The optimization problem i now min µ (15) T.t. i=1 z = r, S x e = µ e, e Ê z, i = 1,, T, S. C. Optimally Allocated Overlay Bandwidth In thi ubection, we conider an alternative cenario with better performance. Intead of relying on TCP to allocate the overlay bandwidth, leading to the partition of the overall network into multiple independent overlay network, we will incorporate overlay bandwidth allocation into the optimization problem. Note that different eion are coupled together by the haring of the underlay link. The olution to thi problem will require cooperation from the phyical link. But, it i poible to modify the problem lightly and run the algorithm at only bottleneck link, uch a the inter-isp low link. Let Ĝ(m) = (ˆV (m),ê(m) ) repreent the overlay network for each eion m. For each overlay link ê Ê(m), the notation e ê for ome underlay link e E mean that link e i on overlay link ê, which i itelf an underlay path. For each eion m, let T (m) be the et of all poible panning tree on Ĝ(m) rooted at ource, and t (m) be the i th tree in T (m). Then, the total rate on a phyical link e E, x e, i given by x e =. (16) m M S (m) ê Ê(m) :e ê i:ê t (m) The optimization problem i exactly written a in (1)-(13). III. DISTRIBUTED ALGORITHM: DIAGONALLY SCALED GRADIENT PROJECTION Note that min µ ha the ame olution a min µ + κ, where κ = (κ,...,κ) for ome mall contant κ. More preciely, we replace the objective function µ by µ + κ in (1) and keep the ame contraint. κ erve a a regularization term. The trictly poitive vector κ (i.e., κ > ) guarantee a global geometric convergence rate of our gradient projection algorithm; if κ =, we can only claim that our gradient projection algorithm converge to one optimal olution globally. A. Fixed Overlay Link Bandwidth The goal of minimizing µ+ κ i to balance the network load. The ame objective can be achieved by minimizing e Ê ˆf e (x e ), where ˆf e i ome convex increaing function on x e. Such an objective function dicourage large link rate. One uch function i the q norm µ + κ q = ( e Ê(µ e +κ) q ) 1/q. In fact, µ+ κ can be approximated by µ + κ q : a q, µ + κ q µ + κ. We will aume q throughout. Since min µ+ κ q i equivalent to min µ + κ q q, after ubtitution of µ e with xe, (15) become min ( x e + κ) q (17) e Ê T.t. i=1 z = r, S (18) z, i = 1,, T, S. (19) For optimization problem with the implex contraint of the form (18), the optimality condition i epecially imple [18]. It ha been hown in [19] and [] that there exit a pecial gradient projection algorithm. For our cae, the gradient projection algorithm can alo be eaily extended to an equally imple caled verion. The latter overcome the iue that our problem may be ill-conditioned, and hence, dratically improve the algorithm convergence time. Our computational experience have hown that the caled gradient algorithm i much fater than the uncaled algorithm or the ubgradient algorithm. The latter i often ued in network optimization problem. Another difficulty i the large number of poible panning tree, and hence, the large number of variable. Fortunately, the algorithm doe not maintain all poible panning tree. The following tep take place for every ource at the overlay network level. The algorithm tart out with one or few panning tree. In each iteration, a cot i aigned to each (overlay) link to reflect the current link congetion. Then, a minimumcot panning tree can be computed. The algorithm hift an appropriate amount of traffic (rate) from each currently maintained panning tree to the minimum-cot tree. The new minimum-cot tree enter the current collection of panning tree. Some previou panning tree may leave the collection if it ditribution rate i reduced to zero. We next illutrate ome detail. Let T = S T be the collection of all multicat tree rooted at any ource. Let z be the vector (z ) where S,i = 1,..., T, with an arbitrary indexing order for the ource. In problem (17), let the feaible et defined by (18) and (19) be denoted by Z. For each overlay link e Ê, recall that x e i the aggregate flow rate it carrie. Let x be the vector (x e ) e Ê. Let H denote the Ê T link-tree incidence matrix aociated with the tree in T (i.e. [H] et = 1 if link e lie on tree t; and [H] et = otherwie). Obviouly, x = Hz. Now define ˆf e (x e ) = (x e / + κ) q and ˆf(x) = ˆf e Ê e (x e ). The objective function, denoted by f(z), i given by f(z) = ˆf(Hz) = e Ê and (17) can be written a ˆf e (x e ) = e Ê( x e + κ) q, () min f(z) = ˆf(Hz) (1).t. z Z. The derivative of the objective function f(z) with repect to z i given by f(z) = ˆf(xe ) = q ( x e + κ) q 1. () z x e t e c e t e

5 5 Note that ˆf(x e) x e i the o-called firt-derivative link cot of link e [18], []. It reflect the current congetion level at link f(z) e. z i the firt-derivative cot of the tree t, which i equal to the um of the firt-derivative cot of the link on the tree. It reflect the current congetion level of the tree t. The firt-derivative tree cot i an important quantity. We will ee later that our algorithm i to hift flow from tree with higher cot to the minimum-cot tree. The econd derivative of f(z) will be ued in the caling of the algorithm. With repect to z and z,j, it i given by f(z) z z,j = q(q 1) c e t t,j e ( x e + κ) q. (3) For each S, let i be the index of a minimum-cot tree rooted at (i.e., with a the ource). That i, i (z) = argmin {i:t T } { f(z) z }. (4) If there are multiple minimum-cot tree, we chooe an arbitrary one. Since the feaible et Z i a convex et and the objective function i a convex function, we can characterize an optimal olution z to the problem (17) by the following optimality condition. S i:t T f(z ) z (z z ), z Z. (5) Thi optimality condition can be equivalently written a, for any ource S, z > only if [ f(z ) z,j f(z ) z, t,j T ]. (6) That i, for every ource, only thoe tree with the minimum firt-derivative cot carry poitive amount of flow. Thi intuitively ugget that, in the algorithm, we hould hift flow to the minimum-cot tree from other tree. It turn out thi i exactly what the gradient projection algorithm doe. We will develop the gradient projection algorithm following the propoal in [] to olve the problem (17). But we will add diagonal caling to peed up the algorithm convergence time. The equality contraint in (18) implie that, for each ource, one of the variable depend completely on the ret of the variable. We can eliminate thi variable and have a problem with fewer variable. To be concrete, at a feaible vector z, let eliminate the variable z for each S. Define a new objective function g(ẑ) on R T S, where ẑ conit of the remaining z after z i eliminated for each. Without lo of generality, uppoe, for each ource, i correpond to the tree with the larget index, i.e., i = T. Alo uppoe the ource are index from 1 to S. Then, ẑ =(z 1,1,...,z 1, T1 1;z,1,...,z, T 1;...;z S,1,...,z S, T S ). We will call the domain of g where ẑ lie the reduced domain. We let T 1 1 g(ẑ) =f(z 1,1,...,z 1, T1 1,r 1 z 1,j ; j=1 T 1 z,1,...,z, T 1,r z,j ;...; j=1 z S,1,...,z S, T S 1,r S T S 1 j=1 z S,j ). The optimization problem in (17)- (19) i equivalent to min g(ẑ) (7).t. ẑ. (8) Thi problem can be olved by the gradient projection algorithm. ẑ(k + 1) = [ẑ(k) δ(k) g(ẑ(k))] +, (9) where δ(k) i a poitive tep ize and [ ] + i the projection operator on ẑ. In thi cae, [y] + jut mean that, if y i i a component of y, we take max(y i,) a the correponding component of the vector [y] +. The key i to compute g(ẑ(k)). Let i (k) be a hort hand for i (z(k)). It i eay to how, for S and i i (k), g(ẑ(k)) z = f(z(k)) z f(z(k)). (3) z (k) The firt derivative are given in (). The algorithm in (9) i actually the contrained teepetdecent algorithm. It i well-known that the teepet-decent algorithm can be low if the optimization problem i illconditioned. It happen that the minimizing-congetion type of network problem i often ill-conditioned. In our cae, the problem become more ill-conditioned when the parameter q in the q norm become larger. An ultimate olution to an ill-conditioned problem i Newton algorithm. However, Newton algorithm i generally very complex ince it require the invere of the Heian matrix of the objective function. For large problem, thi computation i generally impractical. We will next develop the diagonally caled gradient algorithm, which i a much impler alternative and a good approximation of Newton algorithm. The caled gradient projection algorithm can be written a ẑ(k + 1) = [ẑ(k) δ(k)d(k) g(ẑ(k))] +, (31) where D(k) i a poitive definite matrix. For diagonal caling, D(k) i choen to be a diagonal matrix. D(k) = diag[(d (k)) 1 ] S,i i(k). (3) That i, the diagonal entry correponding to z (k) i choen to be (d (k)) 1. For each S and i i (k), the value of d (k) i choen to be d (k) = g(ẑ(k)) z. (33)

6 6 Thi way, the matrix D(k) approximate the invere of the Heian of g at ẑ(k). For each S and i i (k), the econd derivative of g i further given by g(ẑ(k)) z = f(z(k)) z + f(z(k)) z f(z(k)). (k) z z (k) (34) The econd derivative are given in (3). We can now collect different piece of the development above and formally give the diagonally caled gradient projection algorithm in the original domain where z lie. A light generalization i preent in (35). Diagonally Scaled Gradient Projection Algorithm with z(k + 1) = α(k) z(k) + (1 α(k))z(k) (35) [z (k) δ(k) (d (k)) 1 ( f(z(k)) z if i i (k); r 1 j T,j i (k) z,j(k), d (k) = z (k) = (36) q(q 1) c e t t (k) \t e t (k) f(z(k)) z (k) )] +, if i = i (k), ( x e(k) + κ) q. (37) In (35), α(k) i a calar on [a,1], for ome a, < a 1. (35) ay that the new rate vector at the (i + 1) th iteration, z(k + 1), i on the line egment between z(k) and z(k). The main part of the algorithm i expreion (36), which compute the end point of a feaible direction, z(k), entry by entry. There are three cae. cae 1 If a tree t i not the choen minimum-cot tree (with index i (k)) and t ha a poitive flow, it rate will be reduced (more preciely, if t i a minimumcot tree with poitive flow rate but not the choen minimum-cot tree, it rate will keep the ame). cae If a tree t i not the choen minimum-cot tree and the tree ha zero flow rate, then the rate tay at. cae 3 If t i the choen minimum-cot tree, the rate of the tree i increaed o that the total rate of all tree rooted at will be equal to the demanded rate r. Note that the decription in cae 3 enure that z(k) i feaible (in Z). Since z(k) i alo feaible, by (35), the new rate vector z(k+1) i feaible. Hence, if we tart with a feaible olution in Z, z(k) i in Z for all k. 4 What remain to be aid i how much the rate i reduced in cae 1. Note that the expreion f(z(k)) z f(z(k)) z (k) i the difference in the firt-derivative cot between the tree t and the choen minimum-cot tree, and the difference i alway non-negative. Intuitively, the amount of reduction hould be proportional to thi difference. Indeed, if we ignore the factor (d (k)) 1 in (36), the rate reduction i proportional 4 (35) can be replaced with a more general update z(k+1) = A(k) z(k)+ È (I z(k))z(k), where A(k) i a S T È S T diagonal matrix with diagonal entrie in the interval [a,1], for È ome a, < a 1. To enure feaibility of z(k+1) in Z, it i required that 1 i T a (k)( z (k) z (k)) =, where a (k) i a correponding diagonal entry of A(k). to the cot difference with the proportional contant (tep ize) δ (k) >. The factor (d (k)) 1 doe the diagonal caling, which can effectively deal with our ill-conditioned problem. The caling factor (d (k)) 1 can be undertood a allowing different component of the vector z to ue different tep ize. Note that, in the expreion for d (k) in (37), which correpond to the i th tree, the um i over the non-overlapping link between the i th tree and the i (k) th tree, the latter being the minimumcot tree. The algorithm in (35)-(37) i a ditributed one. In order to f(z) compute the tree cot, z in (), and the caling factor, (d (k)) 1 in (37), each link e can independently compute it correponding term baed on the local aggregate rate, x e, paing through the link. Then, the tree cot and the caling factor can be accumulated by the ource baed on the link value along the tree. To find the minimum-cot tree i (k), each ource need to compute the minimum-cot directed panning tree (MDSP). Both centralized and ditributed algorithm exit for computing the MDSP [1] [] [3]. Both achieve O(n ) time complexity for a complete graph with n node. In the ditributed verion, the amount of information exchanged i alo O(n ). In our implementation, each ource collect the (overlay) link cot from all the receiver and ue a centralized algorithm to compute the MDSP. Other than that, the gradient algorithm i completely decentralized. In addition to fat convergence, another trength of thi gradient algorithm lie in that it avoid the enumeration of all poible panning tree. The ource only need to manage the et of active multicat tree, i.e., thoe tree with poitive flow. At each iteration, the ource compute a new minimumcot tree. A non-active tree won t become active unle it i the minimum-cot tree. The ource only adjut the flow rate among the et of active tree. The et of active tree uually i not very large if the algorithm converge fat, ince, at each iteration, at mot one more tree become active. For the original linear model (15), there are at mot Ê + S active tree in any extreme point olution. But ince thi gradient algorithm i a kind of interior point method, trictly peaking, Ê + S i not really an upper bound. Neverthele, it hould give a rough ene on what the bound might be. We tre that the reaon to apply the caling factor i to counter the ill-conditioned problem when q i large. In an ill-conditioned problem, ingle-unit change of different variable have diproportionate effect on the cot (e.g., objective) change. For convergence, the tep ize in the iterative algorithm mut be tuned according to the variable that caue large cot change. However, uch a tep ize can be too mall for other variable, and a a reult, they hardly change from iteration to iteration. The diagonally caled algorithm eentially re-cale the variable o that ingle-unit change in the caled variable have imilar effect on the cot objective. For our problem, the caling ha the imple interpretation that different tree ue different tep ize, each roughly being proportional to a power of the wort link utilization on the tree. The reulting caled algorithm i far uperior to the plain gradient projection algorithm.

7 7 B. Optimally Allocated Overlay Bandwidth The problem decribed in Section II-C can be worked out in a imilar way, leading to a caled gradient projection algorithm. Subtitute µ e with xe, the problem become min ( x e + κ) q (38).t. e E T (m) i=1 = r (m), S (m), m M (39), i = 1,, T (m), S (m), m M. (4) Let T = S (m) m M T (m) be the collection of all multicat tree rooted at any ource for any eion m. Let z be the vector ( ) where S(m),m M,i = 1,..., T (m), with an arbitrary indexing order for the ource. In problem (38)-(4), let the feaible et defined by (39) and (4) be denoted by Z. Let Ê = m M Ê(m) be the collection of all overlay link in all eion. Let Ĥ denote the Ê T overlay link-tree incidence matrix aociated with the tree in T (i.e. [Ĥ] êt = 1 if overlay link ê lie on tree t; and [Ĥ] êt = otherwie). Recall E i the et of underlay link, let H denote the E Ê underlay link-overlay link incidence matrix aociated with the overlay link in Ê (i.e. [H] eê = 1 if underlay link e lie on overlay link (underlay path) ê; and [H] eê = otherwie). For each underlay link e E, recall that x e i the aggregate flow rate it carrie. Let x be the vector (x e ) e E. It i eay to ee x = HĤz. Note that an underlay link e might carry multiple copie of the ame file chunk ditributed by one tree t. Define ˆf e (x e ) = (x e / + κ) q and ˆf(x) = ˆf e E e (x e ). The objective function, denoted by f(z), i given by f(z) = ˆf(HĤz) = e E and (38) can be written a min f(z) =.t. z Z. ˆf e (x e ) = e E( x e + κ) q, (41) ˆf(HĤz) (4) The derivative of the objective function f(z) with repect to i given by f(z) = = ê t (m) ê t (m) e ê e ê ˆf(x e ) x e q ( x e + κ) q 1. (43) For each S (m) in eion m, let i (m) be the index of a minimum-cot tree rooted at (i.e., with a the ource). That i, Let i (m) i (m) (z) = argmin (m) {i:t T(m) (k) be a hort hand of i (m) (z(k)). } { f(z) }. (44) Diagonally Scaled Gradient Projection Algorithm with d (m) [ r (m) z(k + 1) = α(k) z(k) + (1 α(k))z(k) (45) (k) = (46) (k) δ(m) (k) (d (m) (k)) 1 ( f(z(k)) 1 j T (m) if i i (m) (k);,j i (m) (k) z(m) ê t (m) t(m) (m) \t (m) t(m) e ê (k) (m) (k),j (k), f(z(k)) (m) (k) if i = i(m) (k), )] +, (k) = (47) q(q 1) c ( x e(k) + κ) q. e The reulting algorithm i till fully ditributed. C. Convergence Reult ˆf x Ê ] We will how the convergence reult of the ynchronou gradient projection algorithm under contant tep ize, i.e., δ(k) = δ for all k. We will adapt the reult from [4] to find an upper bound on the tep ize δ that guarantee the global convergence of the ynchronou gradient projection algorithm to an optimal olution. Furthermore, with the trictly poitive regularization vector κ (i.e., κ > ), the convergence peed i linear (i.e., geometric). The ame convergence reult can be aid for the cae of optimally allocated bandwidth. In the optimization problem (1), we aume q, o that ˆf e (x e ) i continuou on the interval [, ), tend to a x e approache, and it derivative and econd derivative are continuou and poitive on (, ). Auming the link are indexed from 1 to Ê, the Heian ˆf = diag[ ˆf,, x 1 i an Ê Ê diagonal matrix with nonnegative diagonal entrie. Furthermore, if κ >, the diagonal entrie of ˆf are poitive and bounded below by min e Ê { q(q 1) c κ q }. e We aume there i at leat one feaible olution, i.e., z() Z atifying Hz() e Ê[, ), and define a compact et Z = {z Z f(z) f(z())}. Since thi et i compact, f mut attain a minimum on thi et. Hence, there i a z Z atifying f(z ) = f, where f = min z Z f(z). We call any uch z an optimal olution, and we denote by Z the et of optimal olution. (There may be more than one optimal olution ince, although ˆf i trictly convex, f i not.) That i Z = {z Z f(z) = min z Z f(z)}. For implicity, we aume the caling factor d (k) = 1.. The convergence reult till hold for other caling factor a long a {d (k)} are bounded between two fixed poitive calar [4] [18]. Let δ 1 = a/(lmax S T ), where L > i an upper bound of the norm of f over Z.

8 8 Z Lemma 1: For < δ δ 1, we have for all k that z(k) a f(z(k+1)) f(z(k)) ( δ max S T L ) z(k) z(k). (48) The proof Lemma 1 follow the proof for a imilar lemma in [4]. The only change involve ubtitution of appropriate contant. Therorem : (Globally Convergence) Suppoe κ. For any δ, < δ < δ 1, every limit point of {z(k)} generated by the ynchronou gradient projection algorithm (35)-(37) with z() Z i optimal. a Proof: With the contant tep ize δ < L max S T, the right-hand ide of the inequality (48) i non-poitive. Hence, if {z(k)} ha a limit point, the left-hand ide tend to. The algorithm (35)-(37) can be denoted a a function A(z), i.e., z(k + 1) = A(z(k)). Therefore, z(k) z(k), which implie that for every limit point z of {z(k)} we have z = A( z). It i eay to how if z = A( z), for any S, we have z i f( z) > only if z f( z) z,j, t,j T, which i exactly the optimality condition in (5). So z i tationary (Propoition.3. and Example.1. in [18]). When the regularization vector κ i trictly poitive, the diagonal entrie of ˆf are poitive and bounded below by min e Ê { q(q 1) c κ q } >. When q =, the diagonal entrie e of ˆf are poitive and bounded below by min e Ê { q(q 1) c } > e for all κ. In thee two cae, all condition for global geometric convergence required by [4] are atified. We can tate the global geometric convergence rate for algorithm (35)- (37). Therorem 3: (Globally Geometric Convergence Rate) Suppoe κ >. Let δ atify < δ δ 1. The equence {z(k)} generated by the ynchronou gradient projection algorithm (35)-(37) converge to an element of Z with an initial feaible z() and the convergence rate i linear (i.e., geometric) in the ene that for all k, f(z(k + 1)) f (1 D 5 δ)(f(z(k)) f ). (49) Furthermore, when q =, the above concluion hold for all κ. The contant and parameter are a follow. D 5 = a/(d 4 + δ 1 ), D 4 = ((5L + 1)(D 3 ) δ 1 + 6L(δ 1 ) /a)max S T and D 3 = D max{1,δ 1 } for ome D >. Moreover, D i bounded above by D 1 (D 1 + ( max S T + 1)ˆL H T )/ˆσ, where D 1 = max{ Q 1 Q an invertible ubmatrix of H}. ˆσ ˆL are any two poitive calar uch that the diagonal entrie of ˆf(Hz) lie inide [ˆσ, ˆL] for all z Z, where ˆf(Hz) i a poitive diagonal matrix. IV. PRACTICAL CONSIDERATIONS The problem formulation in the previou ection omit ome detail that are practically required. The purpoe of thi omiion i for eae of preentation. Thee implified formulation contain the technical core, or the mot difficult apect, of the problem. For the mot part, the formulation are without lo of generality. Practical detail can be eaily incorporated into the formulation. We now addre everal of thee. A. Overlapping Content When ome ource hare overlapping chunk, we can create a virtual ource node that connect all thoe ource and move all the overlapping chunk to the virtual ource. The virtual ource ha one outgoing virtual link with infinite capacity connecting to each original ource. We then arrive at an expanded network. Of coure, in actual operation, one of the original phyical ource will act a the virtual ource and run the algorithm aigned to the virtual ource. B. Mixed Architecture of Fixed and Allocated Bandwidth In Section II-B and II-C, we ee two content ditribution cenario with either fixed or optimally-allocated overlay bandwidth. The latter hould achieve better downloading time than the former. However, the latter require the deployment of our algorithm to all network element, i.e. router, which i almot certainly impoible. There i an alternative framework in which ome router or device attached to the router are deployed with our algorithm, while other not. For intance, our algorithm can be deployed at the cro-isp link and acce link, where the bandwidth i more likely to be mall. In thi framework, for thoe phyically directly connected router pair deployed with our algorithm, we model the phyical link between them exactly; while for thoe not directly connected device, router and end-ytem, we let TCP allocate the end-to-end bandwidth between them and model thee end-to-end path a overlay link. Thu, in our graph of the network, ome link are real phyical link and other are overlay link with TCP-allocated bandwidth. The algorithm applie a uual. C. Network Dynamic and Churn Thu far, we aume that the network i table and no member depart or join until all exiting member finih downloading. Since we are conidering the ditribution of maive file or large collection of file, the aumption i reaonable for the mot part. However, we do need to deal with low-degree member churn and network dynamic uch a link capacity variation and failure. If any ource owning unique chunk leave before it finihe dieminating them, thoe chunk are no longer available in the network. To minimize uch rik, in the optimization formulation, we can adjut the requeted ending rate r of the ource. If any ource i expected to leave the network oon, it may requet (or be aigned) a higher ending rate o that it can pread it chunk to network more quickly. Other type of network and member dynamic include link failure, the change of link capacitie, the arrival of new ource, and the departure and arrival of receiver. A argued in [5], a ditributed algorithm ha built-in ability to adapt to variation. A ditributed algorithm can react rapidly to a local diturbance at the point of diturbance with lower fine tuning in the ret of the network. Such adaptive ability i intimately

9 9 connected with the algorithm peed of convergence in the tatic cae. Since our algorithm i the reult of conciou effort to improve the convergence peed (by diagonal caling of the gradient algorithm), we believe it i uperior in coping with network and member dynamic compared to other imilar ditributed algorithm. In addition, our ditributed algorithm i naturally robut becaue of the lack of reliance on a central node that might fail. In Section V, we will how a mall example of how our algorithm uccefully adapt to the departure and arrival of receiver. D. Scalability and Hierarchical Partition of Seion In each overlay network, the ource know the complete information of all overlay link and need to run the expenive centralized MDSP algorithm (There i a ditributed MDSP algorithm [3]. But the price to pay i the potentially lower peed due to the coordination overhead of ditributed operation.). Thu, our gradient algorithm can only deal with ditribution eion with limited ize, ay everal thouand of member in each eion. In order to improve the calability of our algorithm, we hall partition each eion and run the algorithm hierarchically, a mot calable network algorithm would do. Though currently we don t have a well-defined way to partition the eion, we will how one naive approach to partition a large eion in Section V. E. Aynchronou Algorithm Time ynchrony i uually difficult to maintain in a large network. An aynchronou verion of the caled gradient algorithm (35)-(37) (or (45)-(47), repectively) could be developed and the correponding convergence reult could be tated following the approach in [6] [4]. V. PERFORMANCE EVALUATION In thi ection we will compare the performance of our gradient algorithm () with known theoretical bound, Bit- Torrent () and Adaptive FatReplica (AFR) [3]. We elect BitTorrent becaue it technique are intereting and it i the mot popular PP application. We elect AFR becaue it can be thought a uing multiple multicat tree for ditribution. But only a ubet of the tree are allowed, which we call two-level two-phae tree. In each of thee tree, the ource i connected to one receiver at level 1, and then the level 1 receiver i connected to all other receiver at level. It might appear that uch a collection of tree i quite enough for achieving near optimal performance. In an acce-contrained network, thi i indeed true. However, we will how thi i not the cae for network with interior bottleneck. In that cae, a different, maybe larger, collection of tree i needed. Although we are more intereted in network interior bottleneck, our algorithm can equally deal with bottleneck at the acce link, at the ISP backbone or at the cro-isp link. Hence, we will conider all thee cae. The commercial ISP backbone and cro-isp topologie are obtained from the Rocketfuel project [7]. In the terminology of BitTorrent, a eed i a ource, and a leecher i a receiver. In the previou ection, our objective function i the wort network utilization µ. In the evaluation part, we will focu on the ource throughput R = r / µ, where r i the caled ending rate, and the downloading time t = L /R, ince thee are what BitTorrent experiment yield directly. However, recall that the two meaure are the two ide of the ame coin. A. The Performance Evaluation Metric 1) BitTorrent Simulation: We ue the Bittorrent imulator developed by Bharambe et. al. [8]. Since the original imulator only upport acce link contraint, we modified the imulator o that it upport general phyical network topologie. The overlay link bandwidth, which i the per-connection bandwidth at the underlay, i determined by the max-min bandwidth allocation [17]. In the BitTorrent imulation, we ue the following imulation environment. Each peer open 5 uploading connection and one of them i elected by optimitic unchoking, which mean that it connect to a random neighbor. The eed ue the mart eed policy, which i introduced in [8]. Seed are with ditinct file. All the peer join the network at time and continue to be in the network until all of the leecher complete the download. All the other parameter follow the regular BitTorrent environment. Table I ummarize the imulation environment for our tet cae. TABLE I BITTORRENT SIMULATION PARAMETERS #Seed File Size per Seed Neighborhood Size Profile MB 38-8 Profile MB 18-4 Profile 5 64 MB 18-4 Profile MB 38-8 Profile MB 38-8 ) Adaptive FatReplica: AFR upport ingle ource. To compare with AFR, we partition the phyical network into everal overlay network, each for one ource according to max-min fairne allocation. AFR contruct two-phae tree a decribed earlier. From [3], the theoretical throughput of AFR in it bet behavior can be computed a min{c nn i, min {c n in j }}, (5) j=1,,m,j i i=1,,m where n i the ource, n i, i = 1,,m are the receiver, and c nin j i the end-to-end path (overlay link) capacity between n i and n j. 3) Theoretical Bound: Some theoretical reult are ued a performance benchmark in our performance comparion. In everal tudie [9] [31], reearcher have analyzed a model of PP file haring among reidential uer in low accepeed environment. Each participating end-ytem ha an uplink (to the network) and a downlink with limited capacity. The capacity of the network i conidered unlimited. The

10 1 ource i to ditribute a file to L receiver. Let the uplink (downlink) bandwidth of receiver i be u i (d i, repectively), for i = 1,,L. Let the uplink capacity of the ource be u. Then, the maximum ditribution peed i hown to be min(u, min 1 i L d i, u + 1 i L u i ). (51) L In (51), the three term are the optimal peed when the bottleneck i at the ource upload link, at a download link, or due to the aggregate upload bandwidth, repectively. By two-phae ditribution, we mean each ditribution tree ha a depth at mot. The following fact i known to be true. Fact 4: In a network with only acce-peed contraint, the two-phae ditribution [9] achieve the (overlay-network) routing capacity [9], which i (51). In the ingle ource cae, there i a maximum flow from the ource to each receiver. The minimum of all thee maximum flow will be called the max-flow limit (MFL). The max-flow limit i a throughput (total ditribution rate) upper bound. If all node but the ource are receiver, the max-flow limit i achievable. Thi reult i known a Edmond Theorem [3] [33]. B. Bottleneck at the Acce Link (Profile 1 to 4) In thi cae, we aume the network ha infinite capacity but the acce link have finite capacitie. We alo aume that all acce link are deployed with our gradient algorithm. In the four tet cae (profile 1-4), we have a ingle ource with uploading bandwidth u. Let u i and d i be leecher (receiver) i upload and download bandwidth, repectively. Profile 1: u i = d i = 36 Kbp for all 99 receiver, u = 64 Kbp. The download link i the bottleneck. Profile : u i = d i = 36 Kbp for all 99 receiver, u = 8 Kbp. The ource upload link i the bottleneck. Profile 3: d i = 36 Kbp, u i = Kbp for all 99 homogeneou receiver, u = 64 Kbp. The aggregate upload bandwidth i the bottleneck. Profile 4: d i = 36 Kbp for all 1 receiver, u i = 1 Kbp for half of receiver, and u i = 1 Kbp for the ret receiver. u = 1 Kbp. The aggregate upload bandwidth i the bottleneck. TABLE II COMPARISON OF DOWNLOADING TIME (MINUTES) AND NUMBER OF ACTIVE TREES. Profile 1 Profile Profile 3 Profile 4 Optimum (5%) (95%) (1%) AFR #tree AFR #tree In Table II, the optimal downloading time i computed from by (51). The reult indicate that the gradient algorithm obtain near the theoretically optimal olution. 1) Comparion with BitTorrent: Table II how the time when 5%, 95% and 1% receiver finih downloading in BitTorrent, repectively. BitTorrent performance i not bad compared with the optimal value. Thi wa explained in [34], which model the downloading time of BiTtorrent. It how that, in the cae that a flah crowd arrive at the ame time, the bandwidth contraint i at the acce link, and the receiver tay after they finih downloading, BitTorrent achieve near optimal ditribution peed. In Fig. 3 and 4, we how the performance comparion of different ditribution cheme under profile 1 and 4. The reult under profile and 3 are omitted for brevity. The download percentage refer to the total amount of data downloaded at each time intance normalized againt the total data downloaded at the end of the ditribution. Since the two line have different lope, we can extrapolate the line and expect the gradient algorithm to do much better if the file ize become larger. Thi obervation eem to contradict the concluion in [34]. Number of completed leecher Opt. 5 AFR (a) Download percentage Fig. 3. Profile 1: (a) number of leecher that have completed download over time; (b) download percentage over time. Number of completed leecher Opt AFR (a) Download percentage (b) Fig. 4. Profile 4: (a) number of leecher that have completed download over time; (b) download percentage over time. ) Comparion with AFR: AFR downloading time i given by (5). Table II how that AFR two-phae approach achieve the optimal downloading time when the bottleneck i either at the download link or at the ource. But when the bottleneck i due to the aggregate upload bandwidth, AFR fail to achieve the optimum, although we know that ome other two-phae olution i optimal. The reaon i that, in AFR, every receiver i required to relay all chunk it receive from the ource to other receiver. Thi unneceary contraint lead to a ub-optimal olution. AFT doen t allow the breadth-firt earch tree, but an optimal two-phae tree doe. Neverthele, AFR achieve good performance in thi kind of acce-limited ituation. We alo compare the number of active tree AFR and the gradient algorithm eventually ue. The gradient algorithm ue (b)

11 11 1 n 1 n 1 Tree : Breadth Firt Search Tree (BFS) Fig. 5. Tree 1: Depth Firt Search Tree (DFS) n 1 Structure of tree on the overlay network n 3 1 Tree 3: Two phae Tree fewer tree than AFR. We inpected the active tree. With the acce-link contraint, the gradient algorithm ue three kind of tree, the depth-firt earch tree (DFS), the breadthfirt earch tree (BFS), and a two-phae tree. Fig. 5 how the tructure of the three type of tree on the overlay network. In general, there exit an optimal olution that ue only twophae tree for network with uch a tar topology. But it eem that the gradient algorithm prefer the chain-like DFS tree and the fat BFS tree. It may appear counter-intuitive that uch a chain-like ditribution path i preferred becaue the chain eem to involve larget delay. However, thi i in fact not true becaue of our fluid model of traffic and becaue we do not conider propagation delay. The bit that arrive at node 1 from the ource can immediately be tranmitted to node, and to node 3, o on. We leave it to future work on how to incorporate the propagation delay in optimal tree election. Table III how that, when the bottleneck i at either the download link or the ource, the gradient algorithm naturally prefer the DFS tree. When the aggregate upload bandwidth i the bottleneck, we have to ditribute the chunk over the twophae tree. In addition, the more heterogenou the receiver are, the more two-phae tree we need and the more bandwidth i allocated to the two-phae tree. The key concluion here i that the gradient algorithm i able to find the bet ditribution tree for the particular network environment. Without the help of the gradient algorithm, what type of tree are elected i not alway obviou. TABLE III THE DISTRIBUTION OF BANDWIDTH ALLOCATED FOR DIFFERENT TREES Profile 1 Profile Profile 3 Profile 4 DFS 99.4% % 98.91% 1.93% BFS.6%.754%.95% Two-phae.33%.1%.333% 97.1% C. Bottleneck at the Internal of ISP Backbone (Profile 5) In thi cae, we aume all acce link have unlimited bandwidth, and congetion happen at the core network. We wih to ee how our algorithm perform in infratructuremode content ditribution. We did experiment with the ISP Sprintlink backbone obtained from [7]. The network ha 315 backbone node and 1944 link. It i the larget backbone ISP with the highet node degree among the ix commercial backbone network that the RocketFuel project provide. We n 1 n attach 1 peer with unlimited acce bandwidth randomly to ome backbone node, with at mot one peer per backbone node. One may think a peer i a large content ditribution erver cluter. Among the 1 peer, we chooe ource with the normalized ending rate r = 1. (dimenionle value). We did everal experiment with the link capacitie uniformly ditributed in ome range. The actual link capacity data i unavailable. We find the gradient algorithm often give trivial optimal olution. After inpecting the olution and the network graph, it turn out that the ISP backbone i poorly connected. There are many link that lie on all the routing path between one peer and all other peer, which mean if any one of thee critical link i removed, at leat one peer will be diconnected. If thee critical link do not have much larger capacity than other link, they are likely to become the bottleneck. The gradient algorithm i able to locate the bottleneck immediately. Other five ISP backbone how the ame property. Preumably in reality, the ISP are aware of uch link and would enure they have very large bandwidth o that they are never the bottleneck. In order to tet our algorithm in thi non-trivial cenario, we aign the ame bandwidth, 1, to all backbone link. Then, we cale up the bandwidth of all critical link (thoe link that, when removed, will leave ome peer diconnected in the overlay) to be large enough o that they are not the bottleneck. We did three tet on Profile 5. (Tet a) Our algorithm i deployed at all link. Thi i the cae of optimally-allocated overlay bandwidth. (Tet b) Our algorithm i deployed only at the peer. The overlay bandwidth between each pair of peer i fixed by the max-min allocation. The 1 peer form a ingle overlay network. (Tet c) Our algorithm i deployed only at the peer. In order to compare with AFR, we partition the backbone into two overlay network, one for each ource. Note that the max-flow limit i achievable in thi cae. Number of completed leecher AFR of Tet c Opt. of Tet c of Tet a of Tet b (a) Download percentage of Tet a of Tet b Fig. 6. Profile 5. (a) Number of receiver that have completed download over time; (b) download percentage over time. 1) Comparion with BitTorrent: Fig. 6 how that, in Tet a, the downloading time in the gradient algorithm i only 3% of that in BitTorrent. BitTorrent i unable to give good performance when the core network i congeted. But in Tet b, after the overlay bandwidth i fixed, the optimal downloading time i much higher than that of Tet a, and almot equal to BitTorrent time. In the figure, the download percentage refer to the total amount of data downloaded at each time intance (b)