On Dynamic Server Provisioning in Multi-channel P2P Live Streaming

Transcription

1 On Dynami Server Provisioning in Multi-hannel P2P Live Streaming Chuan Wu Baohun Li Shuqiao Zhao Department of Computer Siene Department of Eletrial Multimedia Development Group The University of Hong Kong and Computer Engineering UUSee, In. Hong Kong, China University of Toronto, Canada China Abstrat To guarantee the streaming quality in live peerto-peer (P2P) streaming hannels, it is preferable to provision adequate levels of upload apaities at dediated streaming servers, ompensating for peer instability and time-varying peer upload bandwidth availability. Most ommerial P2P streaming systems have resorted to the pratie of over-provisioning a fixed amount of upload apaity on streaming servers. In this paper, we have performed a detailed analysis on months of run-time traes from UUSee, a ommerial P2P streaming system, and observed that available server apaities are not able to keep up with the inreasing demand by hundre of hannels. We propose a novel online server apaity provisioning algorithm that proatively adjusts server apaities available to eah of the onurrent hannels, suh that the supply of server bandwidth in eah hannel dynamially adapts to the foreasted demand, taking into aount the number of peers, the streaming quality, and the hannel priority. The algorithm is able to learn over time, has full ISP awareness to maximally onstrain P2P traffi within ISP boundaries, and an provide differentiated streaming qualities to different hannels by manipulating their priorities. To evaluate its effetiveness, our experiments are based on an implementation of the algorithm whih replays real-world traes. Index Terms Distributed appliations, peer-to-peer streaming, server bandwidth provisioning, multiple hannels I. INTRODUCTION Large-sale peer-to-peer (P2P) live streaming has reently been suessfully and ommerially deployed [], [2], [3], [4], in whih hundre of media hannels are routinely broadasted to hundre of thousan of users at any given time. The essene of P2P streaming is the use of peer upload bandwidth to alleviate the load on dediated streaming servers [5]. Most existing researh has thus far foused on peer strategies: Should a mesh or tree topology be onstruted? What inentives an be provisioned to enourage peer bandwidth ontribution? How do we ope with peer hurn and maintain the quality of live streams? We reognize the importane of these open researh hallenges, as their solutions seek to maximally utilize peer upload bandwidth, leading to minimized server osts. In this paper, however, we shift our fous to the streaming servers. Suh refousing on servers is motivated by our detailed analysis of months and 8 GB worth of realworld traes from hundre of streaming hannels in UUSee [3], a large-sale ommerial P2P live streaming system in China. As all other ommerial live streaming systems (e.g., PPLive [2], PPStream [4]), in order to maintain a satisfatory and sustained streaming quality, UUSee has so far resorted to the pratie of over-provisioning a fixed amount of server apaity to satisfy the streaming demand from peers in all its hannels, ounterating the impat of volatile peer dynamis and time-varying peer upload bandwidth availability. Nevertheless, ontrary to ommon belief, we have observed that the deployed apaities on streaming servers are not able to keep up with the inreasing demand from hundre of hannels in pratie, leading to degraded streaming quality in all hannels. In response, we advoate to alloate limited server apaities to eah of the hannels based on their popularity and priority in order to maximally utilize dediated servers, and also to dynamially determine the minimum overall amount of server apaity to be deployed in the system. While it is ertainly a hallenge to determine the minimum amount of server bandwidth to provision to aommodate the streaming demand of all onurrent hannels, the hallenge is more daunting when we further onsider the onflit of interest between P2P solution providers and Internet Servie Providers (ISPs). P2P appliations have signifiantly inreased the volume of inter-isp traffi, whih in some ases lea to ISP filtering. We seek to design effetive provisioning algorithms on servers with the awareness of ISP boundaries to minimize inter-isp traffi. Towar these objetives, this paper presents Ration, an online server apaity provisioning algorithm to be arried out on a per-isp basis. Ration dynamially omputes the minimal amount of server apaity to be provisioned to eah hannel inside the ISP, in order to guarantee a desired level of streaming quality for eah hannel, depending on its popularity and priority. With the analysis of our realworld traes, we have observed that the number of peers and their ontributed bandwidth in eah hannel vary dynamially over time, and signifiantly affet the required bandwidth from servers. Ration is designed to atively predit the server bandwidth demand in eah hannel in an ISP with time series foreasting and dynami regression tehniques, utilizing the number of ative peers, the streaming quality, and the server bandwidth usage within a limited window of reent history. It then proatively alloates server bandwidth to eah hannel, respeting the predited demand and priority of hannels. To show the effetiveness of Ration, it has been implemented in streaming servers serving a mesh-based P2P streaming system. In a luster of dual-cpu servers, the system emulates realworld P2P streaming by replaying the senarios aptured by UUSee traes.

2 2 The remainder of this paper is organized as follows. In Se. II, we motivate our fous on servers by showing our analysis of months worth of traes from UUSee. In Se. III, we present the design of Ration. In Se. IV, we disuss how Ration may be deployed with ISP awareness to serve real-world P2P streaming systems. Se. V presents our experimental results evaluating Ration by replaying traes in a P2P streaming system running in a server luster. We disuss related work and onlude the paper in Se. VI and Se. VII, respetively. II. MOTIVATION FROM REAL-WORLD TRACES Why shall we refous our attention to dediated streaming servers in P2P live streaming systems? Starting September 26, we have ontinuously monitored the performane statistis of a real-world ommerial P2P streaming platform, offered by UUSee In., a leading P2P streaming solution provider with legal ontratual rights with mainstream ontent providers in China. As other systems suh as PPLive, UUSee maintains a sizable array of about 5 dediated streaming servers, to support its P2P streaming topologies with hundre of hannels to millions of users, mostly in 4 Kbps media streams. With 8% users in China, UUSee network spans over 2 ISPs in China and around 35 ountries in the world. UUSee streaming protool utilizes the pull-based design on mesh P2P topologies, that allows peers to serve other peers ( partners ) by exhanging media bloks in their playbak buffers, whih represent a sliding window of the stream. When a new peer joins a hannel in UUSee, the initial set of a number of partners (up to 5) is supplied by one of the traker servers by randomly seleting from all the existing peers in the hannel with available upload bandwidth. The peer establishes TCP onnetions with these partners, and buffer availability bitmaps (also alled buffer maps ) are periodially exhanged. The buffer size at eah peer in UUSee is 5 media bloks, and eah blok represents /3 seond of media playbak (about MB in total). To maximally utilize peer upload bandwidth and alleviate server load, UUSee inorporates a number of algorithms in peer seletion. Eah peer applies an algorithm to estimate its maximum upload apaity, and ontinuously estimates its aggregate instantaneous sending throughput to its partners. If its estimated sending throughput is lower than its upload apaity for 3 seon, it will inform one of the traker servers that it is able to reeive new onnetions. The traker servers keep a list of suh peers, and assign them upon requests of partners from other peers. In addition, the number of onseutive bloks reeived and ahed in the urrent playbak buffer, starting from the urrent playbak time, is used in UUSee protool to represent the urrent streaming quality of eah peer, whih is referred to as the buffering level. During the streaming proess, neighboring peers may also reommend partners to eah other based on their urrent streaming quality. A peer may ontat a traker server again to obtain additional peers with better qualities, one it has experiened a low buffering level for a sustained period of time. Besides, UUSee has implemented a number of NAT/firewall traversal tehniques based on lassifiation of different types of user onnetions in its network, in order to maximize peer bandwidth ontribution. To inspet the run-time behavior of UUSee P2P streaming, we have implemented extensive measurement and reporting apabilities within its P2P lient appliation. Eah peer ollets a set of its vital statistis, and reports to dediated trae servers every 5 minutes via UDP. The statistis inlude its IP address, the hannel it is wathing, its buffer map, its buffering level, as well as a list of all its partners, with their orresponding IP addresses, TCP/UDP ports, and urrent sending/reeiving throughput to/from eah of them. Eah dediated streaming server in UUSee utilizes a similar protool as deployed on regular peers, is routinely seleted to serve the peers, and reports its related statistis periodially as well. A detailed desription on UUSee protool and the measurement methodologies for the above metris an be found in our previous work [6], [7]. During a -month period from September 26 to July 27, we have olleted more than 8 GB worth of traes with more than 6 million unique IP addresses, representing time-ontinuous snapshots of the live hannels sustained in UUSee every five minutes in this long period of time. Eah snapshot aptures the information on more than, onurrent peers in the entire UUSee network. A. Insuffiient supply of server bandwidth What have we disovered from the traes? The first observation we made is related to the insuffiient supply of server bandwidth, as more hannels are added over time. Suh insuffiieny has gradually affeted the streaming quality, in both popular and less popular hannels. In order to show bandwidth usage over months and at different times of a day within one figure, we hoose to show all our 5-minute measurements on representative dates in eah month. One suh date, February 7 27, is intentionally hosen to oinide with the Chinese New Year event, with typial flash row due to the broadast of a elebration show on a number of the hannels; April 27 is skipped due to lak of traes in the month aused by an upgrade of the trae servers. Fig. (A) shows the total server bandwidth usage on 5 streaming servers. We may observe that an inreasing amount of server bandwidth has been onsumed over time, but stabilizing starting January 27. This rising trend an be explained by the rapidly inreasing number of hannels deployed during this period, as shown in Fig. (B). The interesting phenomenon that suh bandwidth usage has stabilized, even during the Chinese New Year flash rowd, has led to the onjeture that the total uplink apaity of all servers has been reahed. The daily variation of server bandwidth usage oinides with the daily pattern of peer population. Our onjeture that server apaities have saturated is onfirmed when we investigate the streaming quality in eah hannel. The streaming quality in a hannel at eah time is evaluated as the perentage of high-quality peers in the hannel, where a high-quality peer has a buffering level of more than 8% of the total size of its playbak buffer. The riterion of buffering level (i.e., the number of onseutive bloks reeived and ahed in the urrent playbak buffer of a peer), has been extensively used in UUSee system to

3 3 Server upload apaity usage (Gbps) 3 2 9/5/6/5/6 /5/6 2/5/6/5/72/7/7 3/5/7 5/9/7 6/5/77/5/7 (A) Server apaity usage over time. Total number of hannels 5 9/6 /6 /6 2/6 /7 2/7 3/7 5/7 6/7 7/7 (B) Number of hannels deployed over time..5 9/5/6/5/6 /5/6 2/5/6/5/72/7/7 3/5/7 5/9/76/5/77/5/7 (C)The streaming quality of a popular hannel (CCTV)..5 9/5/6/5/6 /5/6 2/5/6/5/72/7/7 3/5/7 5/9/76/5/77/5/7 (D) The streaming quality of a less popular hannel (CCTV2). Number of peers 3 x 5 2 9/5/6/5/6 /5/6 2/5/6/5/72/7/7 3/5/7 5/9/76/5/77/5/7 (E)The population of a popular hannel (CCTV). Number of peers 5 9/5/6/5/6 /5/6 2/5/6/5/72/7/7 3/5/7 5/9/76/5/77/5/7 (F) The population of a less popular hannel (CCTV2). Fig.. The evolution of server bandwidth, hannels, and streaming quality from September 26 to July 27. evaluate the urrent streaming quality of a peer; and the 8% benhmark has empirially been shown to be effetive in refleting the playbak ontinuity of a peer in the following few minutes, based on an internal performane monitoring system in UUSee. Aordingly, we also use the peer buffering level as our streaming quality riterion. Representative results with a popular hannel CCTV and a less popular hannel CCTV2 are shown in Fig. (C) and (D), respetively, with their population measurements plotted in Fig. (E) and (F), respetively. The streaming quality of both hannels has been dereasing over time, as server apaities are saturated. During the Chinese New Year flash rowd, the streaming quality of CCTV degraded signifiantly, due to the lak of bandwidth to serve a flash rowd of users in the hannel, as illustrated in Fig. (E). Would it be possible that the lak of peer bandwidth ontribution has led to the overwhelming demand of the servers? As we noted, the protool in UUSee uses optimizing algorithms to maximize peer upload bandwidth utilization, whih represents one of the state-of-the-art peer strategies in P2P streaming. The following bak-of-the-envelope alulation with data from the traes may be onvining: At one time on Otober 5, 26, about, peers in the entire network have eah ahieved a streaming rate around 4 Kbps, by onsuming a bandwidth level of 2 Gbps from the servers. The upload bandwidth ontributed by peers an be omputed as, 4 2,, = 38,, Kbps, whih is 38 Kbps per peer on average. This represents quite an ahievement, onsidering that most of the UUSee lientele are ADSL users in China with a maximum of 52 Kbps upload apaity, and that many random fators influene the available Nevertheless, we have evaluated a number of other possible streaming quality metris, suh as the instantaneous streaming download rate of a peer, and still identified the one we use as the most effetive in refleting the streaming quality of peers/hannels. upload bandwidth at the peers. In addition, one may doubt if the downgrade of streaming quality during flash rowd senarios ould have been aused by bandwidth bottleneks within the Internet bakbone at those times. Our previous measurement studies in [7] have revealed that there does not exist signifiant differene between bandwidth availabilities on P2P links, that are deided by Internet bakbone bandwidths, at regular times and during flash rowd senarios, and have onfirmed the ommon belief that bandwidth onstraints in P2P streaming mainly lie at the last-mile upload links at the peers and servers in most ases. All the above observations have led to the onlusion that server apaities have inreasingly beome a bottlenek in real-world P2P live streaming solutions. When the server apaity usage by different hannels is not regulated and is largely random (as in the urrent UUSee protool), the results are less than satisfatory: Taking a typial streaming quality result of.5 for both CCTV and CCTV2, there are many more peers experiening a low buffering level in the popular hannel than in the less popular hannel, onsidering the large differene of their populations. In pratie, we may wish to provide a good streaming experiene to as many peers as possible in the entire system, and therefore advoate to alloate the limited server apaities to eah of the hannels based on their popularity and priority, in order to maximally utilize dediated servers. B. Inreasing volume of inter-isp traffi The urrent UUSee protool is not aware of ISPs. We now investigate the volume of inter-isp traffi during the -month period, omputed as the throughput sum of all links aross ISP boundaries at eah time. For eah IP address in the traes, we derive the AS (Autonomous System) it belongs to using the Whois servie provided by Cymru [8], and then map eah China AS number to its affiliated ISP by making use of the

4 4 Overall inter ISP traffi (Gbps) Fig Peer to peer Server to peer 9/5/6/5/6 /5/6 2/5/6/5/72/7/7 3/5/7 5/9/76/5/77/5/7 The volume of inter-isp traffi inreases over time. offiial mapping data provided by CERNET, China [9]. 2 Fig. 2 reveals that both the inter-isp peer-to-peer and server-to-peer traffi have been inreasing, quadrupled over the -month period, due to the inreased number of hannels and peers. In China, the two nation-wide ISPs, Netom and Teleom, harge eah other based on the differene of inter-isp traffi volume in both diretions, and regional ISPs are harged based on traffi to and from the nation-wide ISPs. Both harging mehanisms have made it important for ISPs to limit inter- ISP traffi. Considering the large and persistent bandwidth onsumption for live streaming, we believe that P2P streaming systems should be designed to minimize inter-isp traffi (to avoid the fate of traffi filtering by ISPs), whih remains one of our objetives in this paper. C. What is the required server bandwidth for eah hannel? To determine the amount of server bandwidth needed to ahieve a speifi level of streaming quality in eah hannel, we wish to explore the relation among server upload bandwidth, the number of peers, and the ahieved streaming quality in eah hannel. Fig. 3(A) illustrates the evolution of the three quantities for hannel CCTV over a period of one week, from February 3 to 9, 27. We an observe a weak positive orrelation between the server bandwidth and the streaming quality, and a weak negative orrelation between the peer population and the streaming quality. To further explore any evident orrelation on a shorter time sale, we plot in Fig. 3(B)- the orrelation between server upload bandwidth usage and the streaming quality on February 3 and that between the number of peers and the streaming quality in Fig. 3(B)-2. We an observe an evident positive relation between the two quantities in the former figure and a negative orrelation in the latter. We have extensively explored suh orrelation in many hannels over different dates and have observed that the orrelation varies from one time to another even in the same hannel, whih an be attributed to the time-varying aggregate peer upload bandwidth in the hannel over time. For example, Fig. 3(B)-3 plots the relation between the number of peers and the streaming quality on February 7, whih approximates a reiproal urve. Another interesting observation we have made here is that: ontrary to ommon belief, we have observed a dereasing trend of streaming quality in a streaming hannel when the number of peers inreases, based on our extensive study of many streaming hannels over many time intervals. We believe the reason lies in that many peers annot ontribute up to the level of their required streaming rate (about 4 Kbps) in suh a pratial P2P system over today s Internet, whih 2 The majority of UUSee users are in China and we fous on trae data of suh users in produing Fig 2. Number of peers Server upload bw (Gbps) 3 x /3 2/4 2/5 2/6 2/7 2/8 2/9 (A) (B) 2/3 Server upload apaity usage (Gbps) Number of peers Number of peers 2/3 2/7 Fig. 3. Relationship among server upload bandwidth, number of peers, and streaming quality in hannel CCTV. further justifies the neessity of deploying server apaity in the system. All of our observations thus far point to the hallenging nature of our problem at hand: How muh server bandwidth should we alloate in eah hannel in eah ISP to assist the peers? III. RATION: ONLINE SERVER CAPACITY PROVISIONING Our proposal is Ration, an online server apaity provisioning algorithm to be arried out on a per-isp basis, that dynamially assigns a minimal amount of server apaity to eah hannel to ahieve a desired level of streaming quality. A. Problem formulation We onsider a P2P live streaming system with multiple hannels (suh as UUSee). We assume that the traker server in the system is aware of ISPs: when it supplies any requesting peer with information of new partners, it first assigns peers (or dediated servers) with available upload bandwidth from the same ISP. Only when no suh peers or servers exist, will the traker server assign peers from other ISPs. The fous of Ration is the dynami provisioning of server apaity in eah ISP, 3 arried out by a designated server in the ISP. In the ISP that we onsider, there are a total of M onurrent hannels to be deployed, represented as a set C. There are n peers in hannel, C. Let s denote the 3 We note that Ration an be extended to ases that it is not feasible to deploy servers in eah ISP, by having servers in one ISP responsible to serve peers from a number of nearby ISPs.

5 5 Symbol U C M F s q p n n θ α β γ G B TABLE I NOTATION IN Ration Definition the total amount of server apaity the set of hannels the number of hannels in C streaming quality funtion of hannel server upload bandwidth assigned to hannel streaming quality of hannel priority level for hannel the number of peers in hannel the estimated number of peers in hannel oeffiient of the random error term in the ARIMA model for hannel exponent of s in streaming quality funtion exponent of n in streaming quality funtion weight parameter in streaming quality funtion the objetive funtion in Provision(t+) the maximal server apaity required for hannel to ahieve q = server upload bandwidth to be assigned to hannel, and q denote the streaming quality of hannel, i.e., the perentage of high-quality peers in the hannel that have a buffering level of more than 8% of the size of its playbak buffer. Let U be the total amount of server apaity to be deployed in the ISP. We assume a priority level p for eah hannel, that an be assigned different values by the P2P streaming solution provider to ahieve servie differentiation aross the hannels. We list important notations in this paper in Table I for ease of referene. At eah time t, Ration proatively omputes the amount of server apaity s t+ to be alloated to eah hannel for time t +, that ahieves optimal utilization of the limited overall server apaity aross all the hannels, based on their priority and popularity (as defined by the number of peers in the hannel) at time t +. Suh an objetive an be formally represented by the optimization problem Provision(t+) as follows ( t =,2,...), in whih a streaming quality funtion F t+ is inluded to represent the relationship among q, s and n at time t + : Provision(t+): max C p n t+q t+ () subjet to C s t+ U, qt+ = Ft+(s t+,n t+), C, (2) q t+,s t+, C. Weighting the streaming quality q t+ of eah hannel with its priority p, the objetive funtion in () reflets our wish to differentiate hannel qualities based on their priorities. With hannel popularity n t+ in the weights, we aim to provide better streaming qualities for hannels with more peers. Noting that n q represents the number of high-quality peers in hannel, in this way, we guarantee that, overall, more peers in the network an ahieve satisfying streaming qualities. The hallenges in solving Provision(t+) at time t to derive 2nd order differene of hannel popularity series.5.5 x 5 lag k 2 3 2/3 2/4 2/5 2/6 2/7 2/8 2/9 2 3 lag k (A) (B) Fig. 4. ARIMA model identifiation for hannel popularity series of CCTV in Fig. 3(A)-. the optimal values of s t+, C are: () the unertainty of the hannel popularity n t+, i.e., the number of peers in eah hannel in the future, and (2) the dynami relationship F t+ among q, s, and n of eah hannel at time t+. In what follows, we present our solutions to both hallenges. B. Ative predition of hannel popularity We first estimate the number of ative peers in eah hannel at the future time t +, i.e., n t+, C. Existing work has been modeling the evolution of the number of peers in P2P streaming systems based on Poisson arrivals and Pareto life time distributions (e.g., []). We argue that these models represent simplifiations of real-world P2P live streaming systems, where peer dynamis are atually affeted by many random fators. To dynamially and aurately predit the number of peers in a hannel, we employ time series foreasting tehniques. We treat the number of peers in eah hannel, i.e., n t,t =,2,..., as an unknown random proess evolving over time, and use the reent historial values to foreast the most likely values of the proess in the future. As the time series of hannel popularity is generally nonstationary (i.e., its values do not vary around a fixed mean), we utilize the autoregressive integrated moving average model, ARIMA(p, d, q), whih is a standard linear preditor to takle non-stationary time series with high auray []. With ARIMA(p,d,q), a time series, z t,t =,2,..., is differened d times to derive a stationary series, w t,t =,2,..., and eah value of w t an be expressed as the linear weighted sum of p previous values in the series, w t,...,w t p, and q previous random errors, a t,...,a t q. The employment of an ARIMA(p,d,q) model involves two steps: () model identifiation, i.e., the deision of model parameters p, d, q, and (2) model estimation, i.e., the estimation of p + q oeffiients in the linear weighted summation. In model identifiation, to determine the degree of differening, d, a standard tehnique is to differene the time series as many times as is needed to produe stationary time series. We have therefore derived d = 2 for our time series n t,t =,2,..., based on the observations that the 2nd-order differene of the original time series for all the hannels is largely stationary. For example, Fig. 4(A) shows the 2nd-order differene of the hannel popularity time series of CCTV, as given in Fig. 3(A)-, whih is stationary with the mean of zero. To identify the values of p and q, the standard tehnique is to study the general appearane of the estimated autoorrelation and partial autoorrelation funtions of the differened time series ([], pp. 87). For example, Fig. 4(B) r k φ kk

6 6 plots the autoorrelation r k and partial autoorrelation ˆφ kk funtion values of the differened hannel popularity series in Fig. 4(A) up to lag k = 3. We observe that only r is nonzero and ˆφ kk tails off, whih identifies p = and q = ([], pp. 86). As we have generally made similar observations regarding hannel popularity series for other hannels, we derive an ARIMA(,2,) model to use in our predition in eah streaming hannel. Having identified an ARIMA(, 2, ) model, the hannel popularity predition in hannel at time t +, n t+ an be expressed as follows: n t+ = 2n t n t + a t+ θ a t, (3) where θ is the oeffiient for the random error term a t and an be estimated with a least squares algorithm. When we use (3) for predition in pratie, the random error at the future time t +, i.e., a t+, an be treated as zero, and the random error at time t an be approximated by a t = n t n t []. Therefore, the predition funtion is simplified to the following. We note that the derived model is desirably simple, with only one oeffiient θ to be estimated: n t+ = 2n t n t θ (n t n t). (4) To dynamially refine the model for an aurate predition of the popularity of hannel over time, we propose to arry out the foreasting in a dynami fashion: To start, the ARIMA(,2,) model for hannel is trained with its hannel popularity statistis in the most reent N time steps, and the value of the oeffiient θ is derived. Then at eah subsequent time t, n t+ is predited using (4), and the onfidene interval of the predited value (at a ertain onfidene level, e.g., 95%) is omputed. When time t + omes, the atual number of peers, n t+, is olleted and tested against the onfidene boun. If the real value lies out of the onfidene interval and suh predition errors have ourred T out of T 2 onseutive times, the foreasting model is retrained, and the above proess repeats. We note that the values of T and T 2 represent a tradeoff between the auray of the model and the omputational overhead inurred. The empirial values of T = 8 and T 2 = work well in our experiments to be presented in Se. V. C. Dynami learning of the streaming quality funtion Next, we dynamially derive the relationship among streaming quality, server bandwidth usage, and the number of peers in eah hannel, denoted as the streaming quality funtion F in (2), with a statistial regression approah. From the trae study in Se II-C, we have observed q (s ) α in eah speifi hannel, where α is the exponent of s, e.g., q (s ).4 in Fig. 3(B)-. 4 We also observed q (n ) β, where β is the exponent of n, e.g., q (n ) in Fig. 3(B)-3. As we have made similar relationship observations from a broad trae analysis of different hannels over different times, we model the streaming quality funtion as 4 A more aurate model for the relation in Fig. 3(B)- (derived using our algorithm that follows in the setion) is q =.5(s ).37. The goodness of fit tests applied over the onverted linear regression model, log q = log log s, validated the signifiane of the oeffiients.5 and.37. q = γ (s ) α (n ) β, (5) where γ > is a weight parameter. Suh a funtion model is advantageous in that it an be transformed into a multiple linear regression problem, by taking logarithm at both sides: log(q ) = log(γ ) + α log(s ) + β log(n ). Let Q = log(q ), S = log(s ), N = log(n ), Γ = log(γ ). We derive the following multiple linear regression problem Q = Γ + α S + β N + ǫ, (6) where S and N are regressors, Q is the response variable, and ǫ is the error term. Γ, α, and β are regression parameters, whih an be estimated with least squares algorithms. In order to aurately apture the relationship among the quantities over time, we dynamially re-learn the regression model in (6) for eah hannel in the following manner. To start, the designated server trains the regression model for hannel with olleted hannel popularity statistis, server bandwidth usage and hannel streaming quality during the most reent N 2 time steps, and derives the values of regression parameters. At eah following time t, it uses the model to estimate the streaming quality based on the used server bandwidth and the olleted number of peers in the hannel at t, and examines the fitness of the urrent regression model by omparing the estimated value with the olleted atual streaming quality. If the atual value exee the onfidene interval of the predited value for T out of T 2 onseutive times, the regression model is retrained with the most reent historial data. In summary, we emphasize that the goal for the modeling of the streaming quality funtion is to aurately apture the relationship among the streaming quality, server apaity usage, and the number of peers in a hannel at eah time. While one may onsider using peer upload bandwidths as variables, we hoose to employ the number of peers instead whose values an be muh easier olleted in pratial systems, and to represent the influene of the supply/demand of peer bandwidths on the streaming quality using both the number of peers n and the dynamially learned parameters γ and β in (5). We further note that the signs of exponents α and β in (5) reflet positive or negative orrelations between the streaming quality and its two deiding variables, respetively. Intuitively, we should always have < α <, as the streaming quality should not be worse when more server apaity is provisioned, and its improvement slows down with more and more server apaity provided, until it finally reahes the upper bound of. On the other hand, the sign of β ould vary over time, depending on the relationship between the demand for streaming bandwidth at peers and the supply of peer upload bandwidth at different times: on one hand, the streaming quality an be improved with more high-ontribution peers (e.g., Ethernet peers) in the hannel (the ase of β > ); on the other hand, if more peers join the hannel with an average upload bandwidth lower than the required streaming rate, a downgrade of the streaming quality would our (the ase of β < ). The different ases of α and β are to be further investigated in our experiments using

7 7 the traes in Se. V-A2. D. Optimal alloation of server apaity Based on the predited hannel popularity and the most reently derived streaming quality funtion for eah hannel, we are now ready to proatively assign the optimal amount of server apaity to eah hannel for time t +, by solving problem Provision(t+) in (). Replaing q with its funtion model in (5), we transform the problem in () into: Provision(t+) : max G (7) subjet to C s t+ U, (8) s t+ Bt+, C, (9) s t+, C, () where the objetive funtion G = C p n t+qt+ = C p γ (n t+) (+β) (s t+) α, and Bt+ = (γ (n t+) β ) α, denoting the maximal server apaity requirement for hannel at time t+, that ahieves qt+ =. The optimal server bandwidth provisioning for eah hannel, s t+, C, an be obtained with a water-filling approah. The impliation of the approah is to maximally alloate the server apaity, at the total amount of U, to the hannels with the urrent largest marginal utility, as omputed with, as long as the upper bound of s t+ indiated t+, t+ in (9) has not been reahed. The marginal utility, represents how muh the streaming quality in a hannel an be enhaned by alloating one unit more server apaity to this hannel, whih is deided by the priority, the number of peers, and the server apaity already alloated into the hannel. In Ration, the server apaity assignment is periodially arried out to adapt to the hanging demand in eah of the hannels over time. To minimize the omputation overhead, we propose an inremental water-filling approah, that adjusts server apaity shares among the hannels from their previous values, instead of a omplete re-omputation from the very beginning. The inremental water-filling approah is given in Table II. To better illustrate the idea of water filling, we utilize the reiproal of marginal utility, i.e., ( t+ ), in our t+ algorithm desription, and maximally assign server apaity to the hannels with the urrent smallest value of ( ), t+ equivalently. We explain the inremental water-filling approah with a 5- hannel example in Fig. 5. In this figure, eah hannel is represented by one bin. The volume of water in bin is (s ) ( α), the width of bin is w = p γ α (n t+) (+β), and thus the water level of the bin represents ( ) = (s ) ( α w for hannel. As < α <, eah bin has a maximal ) height, (B t+ )( α w, whih is represented by the dashed line in eah bin. The inremental water-filling approah starts with the optimal server apaity alloation at the urrent time t, i.e., s = s t, C (Step in Table II). It first omputes whether there exists any surplus of the overall provisioned ) TABLE II INCREMENTAL WATER-FILLING APPROACH. Initialize s s t, C. Bt+ (γ (n t+) β ) α, C. 2. Compute urrent surplus of server apaity surplus = U P C s. for all C if s > Bt+ surplus = surplus + (s Bt+) end if end for 3. Alloate surplus to hannels Compute ( ) (s = ) α p γ α (n t+ )(+β ), C. while surplus > and not all s has reahed Bt+, C find the hannel with the smallest value of ( ) and s < B t+. s s + δ, surplus surplus δ. update ( ). end while 4. Adjust hannel apaity assignment towar the same marginal utility if not all s has reahed Bt+, C find hannel min with the urrent smallest value of ( ) and s < B t+. find hannel max with the urrent largest value of ( ). while ( min ) ( max ) > ǫ s min s min + δ,s max s max δ. update ( ) for hannels min and max. find hannel min with the urrent smallest ( ) and s < B t+. find hannel max with the urrent largest ( ). end while end if 5. s t+ s, C. server apaity, that ours when not all the server apaity has been used with respet to the urrent alloation, i.e., U C s >, or the alloated apaity of some hannel exee its maximal server apaity requirement for time t +, i.e., s > B t+ (e.g., the ase that the water level goes above the maximal bin height for bin in Fig. 5(A).) If so, the overall surplus is omputed (Step 2 in Table II) and alloated to the hannels whose maximal server apaity requirement has not been reahed, starting from the hannel with the urrent maximum marginal utility, i.e., lowest water level (Step 3 in Table II). For the example in Fig. 5, the surplus portion of the water in bin in (A) is realloated to bin 2 and bin 4, with the results shown in (B). After all the surplus has been alloated, the server apaity assignment is further adjusted among the hannels towar a same marginal utility (water level), by repeatedly identifying the hannel with the urrent smallest marginal utility and the hannel with the urrent largest marginal utility, and moving bandwidth from the former to the latter, i.e., the water in the bin with the highest water level (largest value of ( ) ) is flooded into the bin that has the lowest water level (smallest value of ( ) ) and has not reahed its maximal bin height yet (Step 4 in Table II). This proess repeats until all hannels have reahed the same marginal utility or have reahed their

8 8 d G ( d s ) - (s ) -a d G ( d s ) - (s ) -a d G ( d s ) - (s ) -a w w2 w3 w4 w5 w w2 w3 w4 w5 w w2 w3 (A) (B) (C) w4 w5 Fig. 5. An illustration of the inremental water-filling approah with 5 hannels. respetive maximum server bandwidth requirement. For the example in Fig. 5(B), the water from bin 3 and 5 are filled into bin 2 and 4, until bin 4 reahes its maximal height and bins 2, 3, and 5 ahieve the same water level, as shown in Fig. 5(C). Suh an inremental water-filling approah derives the optimal server apaity provisioning for all hannels at time t+, as established by the following theorem: Theorem. Given the hannel popularity predition n t+, C, and the most reent streaming quality funtion qt+ = γ (s t+) α (n t+) β, C, the inremental waterfilling approah obtains an optimal server apaity provisioning aross all the hannels for time t +, i.e., s t+, C, whih solves the optimization problem Provision(t+) in (). We postpone the proof of the theorem to Appendix A. E. Ration: the omplete algorithm Our omplete algorithm is summarized in Table III, whih is periodially arried out on a designated server in eah ISP. The only peer partiipation required is to have eah peer in the ISP send periodi heartbeat messages to the server, eah of whih inludes its urrent buffering level. We note that in pratie, the alloation interval is determined by the P2P streaming solution provider based on need, e.g., every 3 minutes, and peer heartbeat intervals an be shorter, e.g., every 5 minutes. We further remark on the omputational omplexity of the algorithm in Table III. The main omputation for steps 2 and 3 lies in the training of ARIMA or the regression model, with least squares algorithms at O(N 3 ) where N is the size of the training sample set. As both steps are arried out for eah of the M hannels, their omplexity are at most O(MN 3 ) and O(MN 2 3 ), respetively. We generally need no more than 3 5 samples to train an ARIMA(,2,) model, i.e., N < 5, and even fewer to derive the regression model [], [2] (thus only a small amount of historial data nee to be maintained at the server for the exeution of Ration). Further onsidering that the training is only arried out when neessary (i.e., when the auray of the models has fallen), we onlude that the two steps inur low omputational overhead in reality. At step 4, we have designed the inremental waterfilling approah, whih only involves loal adjustments for hannels that are affeted. In summary, the algorithm involves low omputational overhead, and an be arried out in a ompletely online fashion to adapt to the dynamis of P2P systems. IV. PRACTICAL IMPLICATIONS OF Ration We now disuss the pratial appliation of Ration in realworld P2P live streaming systems. In pratial systems with TABLE III RATION: THE ONLINE SERVER CAPACITY PROVISIONING ALGORITHM At time t, the designated server in eah ISP. Peer statistis olletion Collet the number of ative peers in eah hannel, n t, C, with peer heartbeat messages. Collet per-peer buffering level statistis from the heartbeat messages, and derive the streaming quality for eah hannel, q t, C. 2. Channel popularity predition for eah hannel C Test if n t is within the onfidene interval of n t, the value predited at time t. If n t lies out of the onfidene interval for T out of T 2 onseutive times, retrain the ARIMA(, 2, ) model for hannel with the most reent N peer number statistis using least squares algorithm. Predit the hannel popularity at time t+ by n t+ = 2n t n t θ (n t n t), where θ is the parameter in ARIMA(,2,). Channel popularity preditions for all hannels derived. 3. Learning streaming quality funtion for eah hannel C Estimate the streaming quality for time t with the urrent streaming quality funtion model: q t = γ (s t) α (n t) β. Test if the atual q t is within the onfidene interval of q t. If q t lies outside of the onfidene interval for T out of T 2 onseutive times, retrain the regression model in Eq. (6) with statistis in the most reent N 2 times, using the least squares algorithm. Current streaming quality funtions for all hannels derived. 4. Proative server apaity provisioning for all the hannels Adjust server apaity assignment among all the hannels with the inremental water-filling approah in Table II. Optimal server apaity provisioning, s t+, C, derived. unknown demand for server apaity in eah ISP, Ration is able to fully utilize the urrently provisioned server apaity, U, and meanwhile provide exellent guidelines for the adjustment of U, based on different relationships between the supply and demand for server apaity. A. Servie differentiation aross hannels in tight supplydemand relations In most typial senarios, the system is operating in a mode with tight supply-demand relations, i.e., the total server apaity an barely meet the demand from all hannels to ahieve the best streaming qualities. In this ase, Ration guarantees the limited server apaity is most effiiently utilized aross the hannels, respeting their demand and priority. With its water-filling approah, the preferene in apaity assignment is based on the marginal utility of eah hannel, = p n dq, as determined by the priority and popularity of the hannel, and the marginal improvement of its streaming quality upon a unit inrease of server apaity. Given the

9 9 limited server apaity deployed in the ISP, the streaming solution provider an differentiate the streaming quality of the hannels by manipulating the priority levels assigned to the hannels. The following theorem states the proportional servie differentiation aross hannels provided by Ration. Theorem 2. The optimal server apaity provisioning in Ration in () provides the following streaming quality differentiation between any two hannels and, whih do not ahieve the best streaming quality of in the alloation: (qt+) α α (q t+ ) α α = p α (γ ) α (n t+) α +β α p α (γ ) α (n t+ ) α +β α,, C where q t+ < and q t+ <. () We postpone the proof to Appendix B. Priority levels of different hannels an be set aording to Eq. () to ahieve desired relative streaming quality levels aross the hannels. More speifially, if α α α (we have observed similar α values for many different hannels in our trae studies), we have the following approximation: q t+ q t+ = (p ) α α (γ ) α (n t+ ) α+β α (p ) α α (γ ) α (n t+ ) α+β α In this ase, if the streaming solution provider wishes to ahieve a better streaming quality in hannel than that in hannel, i.e., qt+ > q t+, they may set the priorities for the hannels, p and p, to satisfy the following onditions: p p > ( γ γ ) α (n t+ ) α+β α (n t+ ) α+β α. A similar priority assignment method an be applied to determine the priority levels of multiple hannels in the system. B. Total server apaity adjustment in all supply-demand relations Ration is not only useful in optimal utilization of deployed server apaity; the hannel popularity predition and streaming funtion learning modules in Ration an further be used to determine the minimum overall amount of server apaity to be deployed in eah ISP, guaranteeing desired levels of streaming qualities in all the hannels. If the P2P streaming system is operating at the tight supplydemand mode in an ISP and if the streaming solution provider wishes to boost the streaming quality of all the hannels to a best value around, they may ompute how muh more server apaity to be added, by omparing the urrent provisioning with the overall required server apaity to ahieve q t+ =, C. The latter amount is C B t+, where B t+ is the maximum server apaity needed for hannel as in (9) in Provision(t+), derived using the streaming quality funtion in (5) by setting q t+ =. Similarly, if the system is operating with extremely tight supply-demand relations, e.g., the flash rowd senario, most server apaity is assigned to one or few hannels that are involved in the flash rowd, and most of the other hannels are starving with no or very little server bandwidth. The hannel popularity predition in Ration an be used to detet suh a flash rowd, and to trigger the deployment of bakup server. resoures. The extra amount to add an be omputed with the urrent streaming quality funtion derived for the respetive hannels, aording to the targeted streaming quality. If the system is operating at the over-provisioning mode in an ISP, i.e., the total deployed server apaity exee the overall demand from all hannels to ahieve their best streaming quality, Ration alloates the neessary amount of server apaity needed for eah hannel to ahieve its best streaming quality q t+ =. This is guaranteed by (9) in Provision(t+), as the server apaity provisioned to eah hannel may not exeed the amount B t+, that ahieves q t+ =. When the P2P streaming solution provider disovers that the system is always operating at the over-provisioning mode, they may redue their server apaity deployment in the ISP to the neessary overall amount. C. Channel deployment in eah ISP With Ration, the P2P streaming solution provider an dynamially make deisions on hannel deployment in eah ISP, when it is not possible or neessary to deploy every one of the hundre or thousan of hannels in eah ISP. When a hannel is not alloated any server apaity due to very low popularity or priority during a period of time, the hannel is not to be deployed in the ISP during this time. D. Server apaity implementation Finally, we note that the server apaity provisioning in eah ISP an be implemented in pratie with a number of streaming servers deployed, with the number deided by the total apaity to be provisioned and the upload apaity of eah server. Inside eah ISP, the servers an be further deployed in different geographial areas, and the derived server apaity provisioning for eah hannel an be distributed among several servers as well, in order to ahieve load balaning and streaming delay minimization. V. EXPERIMENTAL EVALUATIONS WITH TRACE REPLAY Our evaluation of Ration is based on its implementation in a multi-isp mesh-based P2P streaming system, whih replays real-world streaming senarios aptured by the traes. The P2P streaming system is implemented in C++ over a P2P emulation platform on a high-performane luster of 5 Dell 425SC and Sun v2z dual-cpu servers, deployed in the networking researh laboratory at the University of Toronto [3]. On this platform, we are able to emulate hundre of onurrent peers on eah server, and emulate all network parameters, suh as node apaities, link bandwidth bottleneks, messaging delays, et. Atual media streams are delivered over TCP onnetions among the peers, and ontrol messages are sent by UDP. The platform supports multiple event-driven asynhronous timeout mehanisms with different timeout perio, and peer joins and departures are emulated with events sheduled at their respetive times. The P2P streaming protool we implemented inludes both the standard pull protool and the unique algorithms employed by UUSee, as introdued in Se. II. Without loss of generality,

10 Number of peers 4 x 4 Atual number Predited number 3 95% onfidene interval 2 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (A) CCTV Number of peers 5 5 Atual number Predited number 95% onfidene interval -4 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (B) CCTV2 4 gamma 2 alpha.5.5 beta.5 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (A).5 Atual quality Predited quality 95% onfidene interval 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (B) Fig. 6. Predition of the number of peers with ARIMA(, 2, ). Fig. 7. Dynami learning of the streaming quality funtion for CCTV. we deploy one server for eah ISP, implementing both the traker server and streaming server funtions. Ration is also implemented on eah of the servers, with 8 lines of C++ ode. The server apaity alloation for eah hannel is implemented by limiting the total number of bytes sent over the outgoing onnetions from the server for the hannel in eah unit time. Our experiments are arried out on realisti replays of the traes. We emulate peer dynamis based on the evolution of the number of peers in eah hannel from the traes: when the number of peers rises between two onseutive time intervals, we shedule a orresponding number of peer join events during the interval; when the number of peers dereases, peer departure events are sheduled for a orresponding number of randomly seleted peers. Upon arrival, eah peer aquires 3 initial upstream peers, and the P2P topology evolves based on the same peer seletion protool as UUSee employs. The node upload apaities are emulated using values from the traes, whih follow a heavy-tail distribution in the major range of 5 Kbps to Mbps. The streaming rate of eah hannel is 4 Kbps, with the streams divided into -seond bloks for distribution. The size of the playbak buffer on the peers is set to 3 seon. Eah peer reports its buffering level to the server in its ISP every 2 seon with heartbeat messages, and the server proesses them and adjusts the apaity alloation every 6 seon. We note that this represents muh expedited settings, as these intervals an be muh longer in real-world systems. With this setting, we are not only able to aelerate our experiments that emulate real-world streaming over a long period of time, but also to demonstrate the speed and effiieny of our algorithm even when it is exeuted in very short time sales. A. Performane of Ration omponents We first examine the effetiveness of eah omposing algorithm in Ration. In this set of experiments, we fous on the streaming inside one ISP, with one server of 8 Mbps upload apaity and 5 hannels. We use the peer number statistis of 5 hannels from the traes, CCTV, CCTV4, CCTV2, CCTV7, and CCTV2, in one ISP (China Teleom) during the week of February 3 9, The 5 hannels have a regular instantaneous number of peers at the sale of 2, 5, 4, 5 and, respetively. The statistis of CCTV and 5 To expedite our experiments, eah peer number time series from the traes is sampled, suh that the evolution of the P2P system in eah day is emulated within half an hour. CCTV4 also aptured the flash rowd senario on February 7, when the Chinese New Year elebration show was broadast on the two hannels. ) Predition of the number of peers: Fig. 6 presents the results of predition with ARIMA(, 2, ) for the popular hannel CCTV and the less popular hannel CCTV2, respetively. In the dynami predition, the training set size is N = 3, and the error ount parameters are T = 8 and T 2 =. The predited numbers for both hannels largely oinide with the atually olleted number of peers, both at regular times and during the flash rowd, no matter whether the predition onfidene interval is large or small at different times. This validates the orretness of our model identifiation, as well as the auray of our dynami predition. 2) Dynami streaming quality funtion: Fig. 7(A) plots the derived parameter values for the dynami streaming quality funtion of CCTV. In the dynami regression, the training set size is N 2 = 2, the error ount parameters are T = 8 and T 2 =. We see that γ is all positive, the values of α are always within the range of, and β may be positive or negative at different times. We have observed similar results with the derived streaming quality funtions of other hannels. This validates our analysis in the last paragraph of Se. III-C. During the flash rowd senario, whih hereinafter is marked with a vertial line in the figures, β is signifiantly below zero, revealing a negative impat on the streaming quality with a rapidly inreasing number of peers in the hannel. Fig. 7(B) plots the atually measured streaming quality in the hannel against its estimated value, alulated with the derived streaming quality funtion at eah time. The atual streaming quality losely follows the predited trajetory at most times, inluding the flash rowd senario, whih exhibits the effetiveness of our dynami regression. 3) Optimal provisioning among all hannels: We now investigate the optimal server apaity provisioned to different hannels over time. In this experiment, we fous on examining the effets of hannel popularity on apaity alloation, and set the priorities for all 5 hannels to the same value of. Fig. 8(A) and (B) show the server apaity alloated to eah of the 5 hannels, and their atually ahieved streaming quality at different times. We observe that, generally speaking, the higher the hannel s popularity is, the more server apaity it is assigned. This an be explained by the marginal utility of the hannels used in the water-filling alloation of Ration, = p n dq = p γ α (n ) (+β ) (s ). As α β > is observed in our previous experiment, the marginal utility is positively

11 Server apaity provisioning (Mbps) Objetive funtion value /3 2/4 2/5 2/6 2/7 2/8 2/9 (A) CCTV CCTV4 CCTV2 CCTV7 CCTV2 Optimal provisioning Proportional provisioning UUSee 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (D).5.5 CCTV CCTV4 CCTV2 CCTV7 CCTV2 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (B) CCTV CCTV4 CCTV2 CCTV7 CCTV2 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (C) Fig. 8. Server apaity provisioning for 5 non-prioritized hannels: (A) Server apaity provisioning with Ration, (B) ahieved with Ration, (C) ahieved with proportional alloation, (D) Comparison of objetive funtion values. orrelated with the number of peers, and thus the more popular hannel is assigned more server apaity. On the other hand, in Fig. 8(B), we do not observe evident orrelation between the hannel popularity and its ahieved streaming quality, as the latter is deided by both alloated server apaity (positively) and the number of peers (positively or negatively at different times). Nevertheless, we show that our water-filling assignment ahieves the best utilization of the limited overall server apaity at all times, with a omparison study to a proportional alloation approah. The proportional alloation approah implements the same protool as employed in UUSee, exept its server apaity alloation, whih goes as follows: At eah time t, the server apaity is proportionally alloated to the hannels, based only on their predited number of peers for time t +. Fig. 8(C) shows that the most popular hannel, CCTV, ahieves better streaming quality with this proportional alloation as ompared to that in Fig. 8(B), at the prie of downgraded quality for the other hannels, espeially during the flash rowd. This is beause CCTV now obtains more than half of the total server apaity at regular times, and almost all during the flash rowd senario. With the streaming quality results in Fig. 8(B) and (C), we ompute the values of the objetive funtion of Provision(t+) in (), and plot them in Fig. 8(D). Given the same priority for all the hannels, the value of the objetive funtion at eah time represents the total number of peers in all the hannels that ahieve satisfying buffering level at the time. The values from the proportional alloation are onsistently lower than those ahieved with our water-filling approah, exhibiting the optimality of the server apaity utilization with Ration. In our experiments, we have also ompared Ration with the original UUSee protool, in whih no alloation aross hannels is done at all. We have observed more unstable and lower streaming qualities in all the hannels than those shown in Fig. 8(B) and (C), and lower objetive funtion values than Server apaity provisioning (Mbps) Server apaity provisioning (Mbps) /3 2/4 2/5 2/6 2/7 2/8 2/9 (A) CCTV CCTV4 CCTV2 CCTV CCTV4 CCTV2 CCTV7 CCTV2 CCTV7 CCTV2 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (C).5.5 CCTV CCTV4 CCTV2 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (B) CCTV CCTV4 CCTV2 CCTV7 CCTV2 CCTV7 CCTV2 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (D) Fig. 9. Server apaity provisioning for 5 prioritized hannels with Ration: (A)(B) 5 streaming quality levels; (C)(D) 3 streaming quality levels. those ahieved by the proportional provisioning approah, as shown in Fig. 8(D). 4) Effetiveness of hannel prioritization: In the next experiment, we study the effet of hannel prioritization on server apaity provisioning with Ration. We investigate two ases: () We wish to ahieve differentiated streaming qualities aross all 5 hannels, with q tv > q tv4 > q tv2 > q tv7 > q tv2 ; (2) We wish to ahieve three streaming quality levels, with q tv > q tv4 q tv2 > q tv7 q tv2. We set the following hannel priority levels in the two ases, respetively: () p tv = 5,p tv4 = 25,p tv2 = 2,p tv7 = 75,p tv2 = ; (2) p tv = 5,p tv4 = 22,p tv2 = 2,p tv7 = 75,p tv2 =. These priority values are derived based on Eqn. () in Se. IV to guarantee the targeted streaming quality differentiation, using the popularity, γ, α and β derived for respetive hannels. The experimental results are plotted in Fig. 9. Comparing Fig. 9(A) and (C) to Fig. 8(A), we observe further differentiated server apaities among the hannels, where the hannels with higher priority and popularity are alloated more apaity. In Fig. 9(B) and (D), we observe differentiated streaming quality levels aross the hannels, whih meet our expetations: we an observe 5 different streaming quality levels in Fig. 9(B) and 3 in Fig. 9(D). These demonstrate the effetiveness of hannel prioritization in Ration, whih failitates the streaming solution provider to differentiate servies aross hannels, when the supplydemand relation of server apaity is tight in the system. B. Effetiveness of ISP-aware server apaity provisioning Next, we evaluate Ration in multi-isp streaming senarios. 4 ISPs are emulated by tagging servers and peers with their ISP IDs. Again, 5 hannels, CCTV, CCTV4, CCTV2, CCTV7, CCTV2, are deployed in the ISPs, with peer number statistis in eah ISP extrated from those in 4 China ISPs, Teleom, Netom, Uniom and Tietong, from the traes. While a fixed overall server apaity is used in the previous experiments, in

12 2 Overall server apaity (Mbps) Fig.. 2/3 2/4 2/5 2/6 2/7 2/8 2/9 (A) Overall inter ISP P2P traffi (Gbps) /3 2/4 2/5 2/6 2/7 2/8 2/9 (B) P2P live streaming for 5 hannels in 4 ISPs: without ISP awareness. the following experiments, we do not ap the server apaity, but derive with Ration the minimal amount of overall server apaity needed to ahieve the best streaming qualities for all the hannels in the system (i.e., q =, C), whih is referred to as U B hereinafter. At eah time during the dynami provisioning, U B is derived by summing up the upper bound of server apaity required for eah of the hannels, Bt+, as given in (9), at the time. Our fous is to ompare the total server apaity U B required when ISP awareness is in plae and not, and the inter-isp traffi that is aused. The hannels are not prioritized in this set of experiments. ) Without ISP awareness: In the first experiment, we deploy one server in the system, and stream with a peer seletion protool that is not ISP-aware, i.e., eah peer is assigned partners that an be any other peers in the entire network. The overall server apaity U B used on the server over time is shown in Fig. (A), and the total inter-isp P2P traffi in the system is plotted in Fig. (B). 2) With full ISP awareness: In the seond experiment, we deploy one server in eah ISP, and onstrain all streaming traffi inside eah ISP by fully ISP-aware peer seletion, i.e., peers are only assigned partners inside the ISP. There is no inter-isp traffi in this ase. The server apaity used on the server in eah ISP is illustrated with the area plot in Fig.. Comparing Fig. to Fig. (A), we an see that more overall server apaity is needed in the system when the traffi is ompletely restrited inside eah ISP with per-isp server apaity deployment, as peers now have fewer hoies of supplying neighbors and may have to resort more to the server in their respetive ISPs. However, the inrease in the total server apaity usage is non-signifiant. The differene is only larger during the flash rowd, when it beomes very diffiult for peers to identify enough supplying peers with available bandwidth inside the ISP. 3) Tradeoff between server apaity and inter-isp traffi: In the final experiment, we provision a total server apaity in the system that is between the amount used in ) and that used in 2), and examine the resulting inter-isp traffi. Speifially, let the overall server apaity usage shown in Fig. (A) be U B min and that shown in Fig. be U B max. We redue the server apaity provisioned on eah server in eah ISP, suh that the overall server apaity at eah time is at the value of U B min + φ(u B max U B min ) at the time. In this ase, peers are allowed to onnet to servers/peers aross ISPs if they fail to aquire suffiient streaming bandwidth within the ISP. The experiment is repeated by setting φ to 3 4, 2, 4 or, that represent different levels of the total server apaity. The Overall server apaity (Mbps) ISP ISP2 ISP3 ISP4 2/3 2/4 2/5 2/6 2/7 2/8 2/9 2/3 2/4 2/5 2/6 2/7 2/8 2/9 Fig.. P2P live streaming for Fig. 2. Server apaity provisioning vs. inter-isp traffi: a tradeoff. 5 hannels in 4 ISPs: with full ISP awareness. Overall inter ISP P2P traffi (Gbps) Phi=3/4 Phi=/2 Phi=/4 Phi= results in Fig. 2 show an inrease of inter-isp traffi with the derease of server apaity provisioning. Further omparing the φ = ase in Fig. 2 to Fig. (B), we observe that while the total server apaity is the same U B min in both ases, a smaller amount of inter-isp P2P traffi is involved with the ISP-aware peer seletion than without any ISP awareness. VI. RELATED WORK With the suessful Internet deployment of mesh-based P2P live streaming systems [4], [2], [3], [4], signifiant researh efforts have been devoted to their measurements and improvements. With respet to measurements, existing studies [5], [6], [7], [8], [9], [2] mostly fous on the behavior of peers, with little attention devoted to the streaming servers, whih nevertheless ontribute signifiantly to the stability of P2P live streaming. Sine the seminal work of Coolstreaming [4], various improvements of peer strategies in suh mesh-based P2P live streaming have been proposed, e.g., the enhanement of the blok pulling mehanism [2], the optimization of peer onnetivity for ontent swarming [22], and the exploration of inter-overlay ooperation [23]. To the best of our knowledge, this paper presents the first detailed measurements of server apaity utilization in a live P2P streaming system, and the first online server apaity provisioning mehanism to address the dynami demand in multiple onurrent hannels. A preliminary report of this work appeared in INFOCOM 28 [24]. This paper represents a substantial revision and extension, with solid studies of the UUSee server apaity utilization over a longer trae period and omplete disussions on the design, analysis and appliation of the online server apaity provisioning algorithm. With respet to analytial work related to the subjet of server apaity, Das et al. [25] have shown, with a fluid model, the effets of server upload apaities on the average peer download time in BitTorrent-like P2P file sharing appliations. Also based on fluid theory, Kumar et al. [] modeled streaming quality in a mesh-based P2P streaming system in terms of both server and peer upload apaities. As ompared to these studies, our work fouses entirely on the pratiality of a dynami server apaity provisioning mehanism. Other than using simplified modeling assumptions suh as Poisson arrivals, we employ time series foreasting tehniques to derive the evolution of the number of peers, and use dynami regression approahes to learn the relation among

13 3 the streaming quality, server apaity and the number of peers at different times. There have reently emerged a number of disussions on the large amount of inter-isp traffi brought by P2P appliations, with respet to BitTorrent file sharing [26], [27], P2P Video on Demand [5], and P2P software update distribution [28]. Approahes for the loalization of P2P traffi inside ISP boundaries have been proposed, whih mostly fous on ISPaware peer seletion strategies[27], [29], [3]. In ontrast, our study is the first to investigate the impat and evolution of inter-isp P2P live streaming traffi, and our proposal emphasizes on the dynami provisioning at the server side on a per-isp basis to maximally guarantee the suess of ISPaware P2P streaming. VII. CONCLUDING REMARKS This paper fouses on dynami server apaity provisioning in multi-isp multi-hannel P2P live streaming systems. In pratie, we believe that it is important to refous our attention on dediated streaming servers: based on our detailed analysis of months worth of traes from a large-sale P2P streaming system, available server apaities are not able to keep up with the inreasing demand in suh realworld ommerial systems, leading to a downgrade of peer streaming quality. Emphasizing on pratiality, our proposed algorithm, Ration, is able to dynamially predit the demand in eah hannel, using an array of dynami learning tehniques, and to proatively provision optimal server apaities aross different hannels. With full ISP awareness, Ration is arried out on a per-isp basis, and is able to guide the deployment of server apaities and hannels in eah ISP to maximally onstrain P2P traffi inside ISP boundaries. Our performane evaluation of Ration is highlighted with the replay of realworld streaming traffi from our traes. We show that Ration lives up to our expetations to effetively provision server apaities aording to the demand and hannel priority over time. APPENDIX A PROOF OF THEOREM Proof: Let s t+, C, be an optimal solution to the optimization problem in (7). Introduing Lagrangian multiplier λ for the onstraint in (8), ν = (ν, C) for the onstraints in (9) and µ = (µ, C) for the onstraints in (), we obtain the KKT onditions for the problem as follows (pp. 244, [3]): X C s t+ U, (2) λ, (3) s t+ Bt+, ν, µ, C, (4) λ ( X s t+ U) =, C (5) µ s t+ =, C, (6) ν (s t+ Bt+) =,, C, (7) + λ µ + ν =, C. t+ (8) For s t+ >, we have µ = from (6), and then t+ λ + ν from (8). Therefore, if further we have s B t+, we derive ν = from (7) and then t+ t+ = t+ < = λ. As = p γ α (n ) t+ )(+β (s, γ > and < α <, we an t+ ) α derive, C, s t+ = B t+ if ( p γ α (n t+ )+β λ ) λ α if λ < (B t+ )( α ) p γ α (n t+ )(+β ), (B t+ )( α ) p γ α (n t+ )(+β ). Notie that for all the hannels with < s t+ < Bt+, we have λ = ( ), whih is the final water level for those t+ bins whose maximal heights are not ahieved, as illustrated in Fig. 5(C) 6 (B. We also notie that t+ )( α ) p γ α (n t+ )(+β ) is the maximal height for bin. Therefore, the key to derive s t+ is to derive the optimal water level λ. If a bin s maximal height is below the optimal water level, the optimal server apaity share for the orresponding hannel is its maximal server apaity requirement, i.e., s t+ = Bt+; otherwise, its alloated server apaity is what ahieves (s t+ )( α ) p γ α (n t+ )(+β ) = λ. To derive the optimal water level, from the starting water levels in the bins deided by the server apaity assignment at time t, we first make sure the overall server apaity at the amount of U is maximally filled into the bins, while no bin s maximal height is exeeded. Then, we derease the high water levels by dereasing the server apaity assigned to the orresponding bins (as α <, ( t+ ) dereases with the derease of s t+), and inrease the low water levels by inreasing the server apaity assigned to the orresponding bins, while guaranteeing the maximal height of eah bin is never exeeded. When all bins reah the same water level, exept those whose maximal heights have been reahed, we have derived the optimal server apaity alloation for all hannels for time t +, as given in (9). APPENDIX B PROOF OF THEOREM 2 Proof: Both the inremental water-filling algorithm in Table II and the proof of theorem in Appendix I show that at optimality, the marginal utility for any hannels, that are not alloated their maximum server apaity requirement Bt+ (i.e.,,qt+ < ), is the same, i.e., where qt+ < and q t+ <. t+ Sine = p γ α (n ) t+ )(+β and t+ (s t+ ) α = p γ α (n ) t+ )(+β, we have t+ (s ) α t+ = t+ p γ α (n t+) (+β ) p γ α (n (s = t+ ) (+β ) t+ ) α,, C, (s t+ ) α. (9) 6 Note that s t+ = only ours at very speial ases, suh as n t+ or α. In this ase, the width of the bin orresponding to hannel is zero, and thus no water (bandwidth) will be assigned to the bin. We omit this speial ase in our disussions. (9)

14 4 From qt+ = γ (s t+) α (n t+) β, we derive s t+ = q ( t+ γ (n ) t+ )β α. Substitute s t+ with this formula in Eq. (9), we an obtain p γ α (n t+) (+β ) ( qt+ α ) γ (n α t+ )β p γ α (n = t+ ) (+β ). (2) ( q t+ α α ) γ (n t+ )β A transformation of (2) will give Eq. (). REFERENCES [] J. Liu, S. G. Rao, B. Li, and H. Zhang, Opportunities and Challenges of Peer-to-Peer Internet Video Broadast, Proeedings of the IEEE, Speial Issue on Reent Advanes in Distributed Multimedia Communiations, 27. [2] PPLive, [3] UUSee, [4] ppstream, [5] C. Huang, J. Li, and K. W. Ross, Can Internet Video-on-Demand Be Profitable? in Pro. of ACM SIGCOMM 27, August 27. [6] C. Wu, B. Li, and S. Zhao, Exploring Large-Sale Peer-to-Peer Live Streaming Topologies, ACM Transations on Multimedia Computing, Communiations and Appliations, vol. 4, no. 3, August 28. [7], Charaterizing Peer-to-Peer Streaming Flows, IEEE Journal on Seleted Areas in Communiations, vol. 25, no. 9, pp , Deember 27. [8] Cymru Whois servie, [9] Cernet BGP View, /hinaasnlist.shtml. [] R. Kumar, Y. Liu, and K. W. Ross, Stohasti Fluid Theory for P2P Streaming Systems, in Pro. of IEEE INFOCOM, May 27. [] G. E. P. Box, G. M. Jenkins, and G. C. Reinsel, Time Series Analysis: Foreasting and Control (Third Edition). Prentie Hall, 994. [2] D. C. Montgomery, e. A. Pek, and G. G. Vining, Introdution to Linear Regression Analysis, Third Edition. John Wiley and Sons, In., 2. [3] M. Wang, H. Shojania, and B. Li, Crystal: An Emulation Framework for Pratial Peer-to-Peer Multimedia Streaming Systems, in Pro. of the 28th International Conferene on Distributed Computing Systems (ICDCS 28), June 28. [4] X. Zhang, J. Liu, B. Li, and T. P. Yum, CoolStreaming/DONet: A Data- Driven Overlay Network for Live Media Streaming, in Pro. of IEEE INFOCOM 25, Marh 25. [5] X. Hei, C. Liang, J. Liang, Y. Liu, and K. W. Ross, A Measurement Study of a Large-Sale P2P IPTV System, IEEE Trans. on Multimedia, vol. 9, no. 8, pp , Deember 27. [6] X. Hei, Y. Liu, and K. W. Ross, Inferring Network-Wide Quality in P2P Live Streaming Systems, IEEE Journal on Seleted Areas in Communiations, Speial Issue on Advanes in Peer-to-Peer Streaming Systems, vol. 25, no. 9, pp , Deember 27. [7] A. Ali, A. Mathur, and H. Zhang, Measurement of Commerial Peer-To-Peer Live Video Streaming, in Pro. of Workshop in Reent Advanes in Peer-to-Peer Streaming, August 26. [8] T. Silverston and O. Fourmaux, Measuring P2P IPTV Systems, in Pro. of the 7th International workshop on Network and Operating Systems Support for Digital Audio & Video (NOSSDAV 7), June 27. [9] B. Li, S. Xie, G. Y. Keung, J. Liu, I. Stoia, H. Zhang, and X. Zhang, An Empirial Study of the CoolStreaming+ System, IEEE Journal on Seleted Areas in Communiations, Speial Issue on Advanes in Peerto-Peer Streaming Systems, vol. 25, no. 9, pp , Deember 27. [2] B. Li, Y. Qu, Y. Keung, S. Xie, C. Lin, J. Liu, and X. Zhang, Inside the New Coolstreaming: Priniples, Measurements and Performane Impliations, in Pro. of IEEE INFOCOM, April 28. [2] M. Zhang, J. Luo, L. Zhao, and S. Yang, A Peer-to-Peer Network for Live Media Streaming - Using a Push-Pull Approah, in Pro. of ACM Multimedia 25, November 25. [22] N. Magharei and R. Rejaie, PRIME: Peer-to-Peer Reeiver-drIven MEsh-based Streaming, in Pro. of IEEE INFOCOM 27, May 27. [23] X. Liao, H. Jin, Y. Liu, L. M. Ni, and D. Deng, AnySee: Peer-to-Peer Live Streaming, in Pro. of IEEE INFOCOM 26, Marh 26. [24] C. Wu, B. Li, and S. Zhao, Multi-hannel Live P2P Streaming: Refousing on Servers, in Pro. of IEEE INFOCOM 28, April 28. [25] S. Das, S. Tewari, and L. Kleinrok, The Case for Servers in a Peer-to-Peer World, in Pro. of IEEE International Conferene on Communiations (ICC 26), June 26. [26] T. Karagiannis, P. Rodriguez, and K. Papagiannaki, Should Internet Servie Providers Fear Peer-Assisted Content Distribution, in Pro. of the Internet Measurement Conferene (IMC 25), Otober 25. [27] R. Bindal, P. Cao, W. Chan, J. Medval, G. Suwala, T. Bates, and A. Zhang, Improving Traffi Loality in BitTorrent via Biased Neighbor Seletion, in Pro. of the 26th IEEE International Conferene on Distributed Computing Systems (ICDCS 26), July 26. [28] C. Gkantsidis, T. Karagiannis, P. Rodriguez, and M. Vojnovi, Planet Sale Software Updates, in Pro. of ACM SIGCOMM, September 26. [29] V. Aggarwal, A. Feldmann, and C. Sheideler, Can ISPs and P2P systems ooperate for improved performane? in Pro. of ACM CCR, July 27. [3] H. Xie, Y. R. Yang, A. Krishnamurthy, Y. Liu, and A. Silbershatz, P4P: Provider Portal for Appliations, in Pro. of ACM SIGCOMM, August 28. [3] S. Boyd, Convex Optimization. Cambridge University Press, 24. Chuan Wu. Chuan Wu reeived her B.Engr. and M.Engr. degrees in 2 and 22 from Department of Computer Siene and Tehnology, Tsinghua University, China, and her Ph.D. degree in 28 from the Department of Eletrial and Computer Engineering, University of Toronto, Canada. She is urrently an assistant professor in the Department of Computer Siene, the University of Hong Kong, China. Her researh interests inlude measurement, modeling, and optimization of large-sale peer-topeer systems and online/mobile soial networks. She is a member of IEEE and ACM. Baohun Li. Baohun Li reeived the B.Engr. degree from the Department of Computer Siene and Tehnology, Tsinghua University, China, in 995 and the M.S. and Ph.D. degrees from the Department of Computer Siene, University of Illinois at Urbana-Champaign, Urbana, in 997 and 2. Sine 2, he has been with the Department of Eletrial and Computer Engineering at the University of Toronto, where he is urrently a Professor. He hol the Nortel Networks Junior Chair in Network Arhiteture and Servies from Otober 23 to June 25, and the Bell University Laboratories Endowed Chair in Computer Engineering sine August 25. His researh interests inlude large-sale multimedia systems, loud omputing, peer-to-peer networks, appliations of network oding, and wireless networks. Dr. Li was the reipient of the IEEE Communiations Soiety Leonard G. Abraham Award in the Field of Communiations Systems in 2. In 29, he was a reipient of the Multimedia Communiations Best Paper Award from the IEEE Communiations Soiety, and a reipient of the University of Toronto MLean Award. He is a member of ACM and a senior member of IEEE. Shuqiao Zhao. Shuqiao Zhao reeived his B.Engr. degree from Department of Computer Siene, Shandong University, China, in 2. He is urrently the managing diretor in the Multimedia Development group in UUSee In., Beijing, China, where he is in harge of the development of the ore tehniques of UUSee s peer-to-peer live streaming solution.