FORESIGHTED JOINT RESOURCE RECIPROCATION AND SCHEDULING STRATEGIES FOR REAL-TIME VIDEO STREAMING OVER PEER-TO-PEER NETWORKS

Transcription

1 FORESIGHTED JOINT RESOURCE RECIPROCATION AND SCHEDULING STRATEGIES FOR REAL-TIME VIDEO STREAMING OVER PEER-TO-PEER NETWORKS Sunghoon Ivan Lee, Hyunggon Park, and Mhaela van der Schaar Electrcal Engneerng Department, UCLA Emal: hgpark, ABSTRACT We consder peer-to-peer (P2P) networks, where multple heterogeneous and self-nterested peers are sharng multmeda data. In ths paper, we propose a novel schedulng algorthm for real-tme vdeo streamng over dynamc P2P networks. The proposed schedulng algorthm s foresghted, snce t enables each peer to maxmze ts long-term vdeo qualty by effcently utlzng ts lmted resources (e.g., uploadng bandwdth) over tme, whle explctly consderng the tme-varyng resource recprocaton behavors of ts assocated peers. To successfully desgn the schedulng algorthm, we consder a dstnct buffer structure that allows the peers to model the resource recprocaton behavor as a recprocaton game. Then, each peer can determne ts foresghted decsons based on a Markov Decson Process (MDP). The smulaton results show that the proposed algorthm sgnfcantly mproves the average vdeo qualty, compared to other exstng schedulng strateges. Moreover, smulaton results also show that the proposed algorthm can flexbly and effectvely operate n heterogeneous P2P networks. Index Terms Peer-to-peer (P2P) networks, real-tme vdeo streamng, foresghted schedulng strategy, resource recprocaton game. 1. INTRODUCTION Multmeda streamng over Internet, such as Vdeo-on- Demand (VoD), broadcast of specal events, remote educaton, etc. [1], has recently ganed ncreased popularty. However, multmeda streamng over Internet does not scale easly wth the number of users, as t requres the deployment of numerous servers, havng consderable storage capacty [1] n order to support the Qualty of Servce. As an alternatve soluton to ths challenge, P2P-based approaches have been recently proposed. A P2P system conssts of multple partcpants, referred to as peers, whch act as both clents and servers [2]. The peers n P2P networks are nterconnected wth each other and share ther resources such as multmeda content, avalable bandwdth, CPU cycles, storage, etc. [2]. Several approaches for general fle sharng over P2P networks have been proposed [3 7], as well as for multmeda streamng and broadcastng over P2P networks (see e.g., [8 10]). In ths paper, we focus on developng a schedulng algorthm for real-tme P2P vdeo streamng, whch consders how pre-encoded segments (or chunks [11]) of multmeda contents can be optmally downloaded. For example, a schedulng algorthm should be able to determne when or from whch peer the segments should be downloaded. Unlke general fle sharng algorthms deployed n several P2P applcatons such as BtTorrent [6], a schedulng algorthm for real-tme vdeo streamng over P2P networks must take nto account the tmng constrants of each segment (e.g., the tolerable maxmum playback delay) [8], [11]. Moreover, the negotaton of downloadng and uploadng rates among the assocated peers becomes a key ssue, because the downloadng rates of each peer must be greater or equal to the mnmum requred decodng rate. Fnally, the order of downloadng contents must be addressed, as the segment that s to be mmedately dsplayed should have hgher prorty than the other segments. To address these challenges, several schedulng algorthms for multmeda streamng applcatons over P2P networks have been proposed [8], [10], [12]. In [8] the algorthm decdes a supplyng peer 1 that contrbutes the hghest bandwdth and that does not volate the playback delay. Alternatvely, the schedulng algorthm n [10] deploys the rarest frst search. Whle these schedulng algorthms provde smple and heurstc approaches, they do not consder the tme-varyng behavors of supplyng peers towards the resource recprocaton. Thus, these approaches provde only sub-optmal performances for contnuously nteractng peers. The nteractons among the nterested peers that have tmevaryng resource recprocaton behavors are modeled as a resource recprocaton game, and each peer can make foresghted decsons on ts resource dstrbuton based on an MDP n the game [11]. However, the resource recprocaton strategy based on an MDP n [11] does not consder the tme constrants such as the playback deadlnes, whch are crtcal for real-tme vdeo streamng applcatons. 1 A supplyng peer s defned as a peer delverng the content of nterest. A set of supplyng peers can be swarm n [6], [7], partnershp n [8], or group n [11].

2 The man contrbuton of ths paper s to ntroduce a schedulng algorthm for P2P real-tme vdeo streamng applcatons that explctly consders the dynamc resource recprocaton behavors of peers. The proposed algorthm also explctly consders the tme constrants (.e., playback deadlnes) of the vdeo segments. To successfully mplement the algorthm, we desgn a new buffer structure, whch enables each peer to play the resource recprocaton game based on an MDP [11]. Hence, peers can make foresghted decsons that maxmze ther long-term vdeo qualty, whle ensurng that segments n the buffer are contnuously dsplayed n a tmely manner. Ths paper s organzed as follows. In Secton 2, we descrbe the proposed schedulng algorthm for real-tme vdeo streamng over P2P networks. In Secton 3, we descrbe the resource recprocaton game played by peers based on an MDP. In Secton 4, we show the expermental results about the proposed algorthm compared aganst other exstng schedulng schemes. Fnally, the conclusons are drawn n Secton A FORESIGHTED SCHEDULING ALGORITHM FOR REAL-TIME STREAMING OVER P2P NETWORKS 2.1. Prelmnares: Schedulng algorthms The goal of real-tme multmeda streamng applcatons over P2P networks s to dsplay multmeda contents n a tmely manner, whle maxmzng the multmeda qualty, by effcently aggregatng content from peers. To acheve ths goal, peers n P2P streamng applcatons should deploy schedulng algorthms that determne the order of segment downloads, whle effcently allocatng a fxed amount of resources (e.g., upload bandwdth). Segments n the buffer are already encoded and are provded by the orgnal supplyng peer 2.An llustratve buffer snapshot for a multmeda streamng P2P system s shown n Fg. 1, based on [1]. In Fg. 1, the buffer s represented as a collecton of segments, where multmeda data can be stored. For example, n [1], buffer conssts of a total 120 segments, where each segment contans multmeda data of 1 second vdeo. The shaded and empty segments n Fg. 1 represent chunks that are completely downloaded and that are scheduled to be downloaded n the future, respectvely. The dotted lne shows a snapshot of the buffer, whch acts as a sldng wndow on the multmeda bt-stream. In the rest of Secton 2, we propose a schedulng algorthm wth a dstnct buffer structure that enables the peers to make foresghted decsons, by consderng the mpact of ther mmedate actons on ther long-term performance. 2 The orgnal supplyng peer s referred to as the orgn node n [8]. It s the orgnal source of the vdeo, whch could be a server, or a peer dstrbutng the vdeo data. Multmeda Chunks Playback pont Sldng wndow Fg. 1. Example of buffer snapshot. The dotted lne represents the sldng wndow and the shaded segments represent completely downloaded segments. past segment GOS 1 GOS 2 GOS 3 Pnt (t) playback sldng wndow future segment Fg. 2. An llustratve example of the proposed buffer structure Buffer structure for real-tme P2P streamng To successfully desgn the schedulng algorthm, we propose an alternatve buffer structure. The proposed buffer structure enables peers to () play the resource recprocaton game (Secton 2.4) and () to deploy the system at the desred computatonal complexty and utlty par (Secton 3.7). Frst, we dvde the buffer nto a fnte number of Group of Segments (GOS). We denote the total number of segments n the buffer as N total and the number of GOSs n a buffer as. Thus, the number of segments n one GOS can be expressed as N seg = N total /. For example, f there are 9 segments n the buffer and there are 3 GOSs, then one GOS has 3 segments,.e., N seg = N total / = 3. Ths s llustrated n Fg. 2. In Fg. 2, playback pont at tme t s denoted by Pnt (t) playback. In the proposed buffer structure, the segments n a GOS should be completely downloaded when the GOS s located n the frst GOS n the buffer (.e., GOS 1). However, the other segments n the rest of GOSs are scheduled to be downloaded completely before ther GOSs are postoned n the frst GOS. For example, when a GOS newly enters the buffer, t s placed at the end of the buffer, e.g., poston of GOS 3 n Fg. 2. As tme t ncreases, the sldng wndow wll move along the bt-stream, whch ncludes the future segments n the buffer, and thus, the temporal dstance between the new GOS and the playback pont decreases. Fnally, the new GOS wll reach the playback pont and the contents of that GOS wll be dsplayed. The llustratve example s depcted n Fg. 3. In the next secton, we dscuss how we allocate the total avalable upload bandwdth to groups of supplyng peers n dfferent GOSs.

3 past segments GOS 1 past segments Pnt (t+3) playback GOS 2 GOS 2 Pnt (t+4) playback buffer at tme t + 3 buffer at tme t + 4 Fg. 3. Illustratve example showng that GOS 2 completes ts downloadng process as t reaches the playback pont Bandwdth allocaton to peers n GOS Suppose BW denotes the total amount of avalable uploadng bandwdth of a peer. We equally dvde the total uploadng bandwdth and allocate them to the group of supplyng peers n each of GOSs n the buffer. Thus, we can express the amount of bandwdth dstrbuted to the supplyng peers n each GOS as, BW 1 = = BW NGOS = BW, (1) where BW k represents the uploadng bandwdth of peer allocated to supplyng peers n GOS k. The amount of cumulated bandwdth that supplyng peers can receve, untl ther GOS reaches the playback pont, can be expressed as, NGOS BW k = N GOS BW = BW. (2) The above equaton tells us that each GOS n the buffer enjoys the full amount of bandwdth that a peer may provde. We may consder allocatng dfferent amount of bandwdth to supplyng peers n dfferent GOSs because a GOS temporally close to the playback pont should have hgher prorty compared to the one that s far away. Suppose that dfferent amount of bandwdth s dstrbuted to supplyng peers n dfferent GOSs accordng to ts temporal dstance from the playback pont. In ths case, the bandwdth allocated to the frst GOS wll be the largest and the bandwdth allocated to N th GOS GOS wll be the smallest. Then, the bandwdth allocated to supplyng peers n each GOS can be expressed as, BW k ρ k = NGOS m=1 ρ BW, m where ρ k represents the prorty that GOS k holds, e.g. ρ k = 1/k. As tme goes by, the temporal dstance between the playback pont and GOS k wll decrease and thus, the prorty for GOS k wll ncrease accordngly. The total amount of bandwdth that supplyng peers n GOS k receve from peer can be expressed as, N GOS BW k = = N GOS } ρ k NGOS m=1 ρ BW m NGOS ρ k NGOS m=1 ρ BW m = BW. Hence, the amount of the upload bandwdth from one GOS s constant regardless of prortzaton. In ths paper, we equally dvde the total bandwdth of peer and allocate t to supplyng peers n each GOS. Ths bandwdth allocaton scheme enables peers to deploy MDP, whch wll be dscussed n detal n Secton 3. In the next secton, we dscuss how we recprocate the equally allocated bandwdth among supplyng peers n each GOS Resource recprocaton n GOS As dscussed n Secton 2.3, the total bandwdth of peer s equally dstrbuted to group of supplyng peers n each GOS n the buffer. Then, the allocated bandwdth s also dstrbuted among supplyng peers of segments n each GOS. Note that the supplyng peers dynamcally respond to the bandwdth dvsons of peer, by allocatng dfferent amount of bandwdth to peer. We term ths nteracton as resource recprocaton n ths paper. The resource recprocatons between a peer and ts assocated supplyng peers wthn each GOS are modeled as a resource recprocaton game, and each peer determnes ts optmal bandwdth allocaton based on MDP, smlar to [11]. The bandwdth allocaton determned based on MDP allows each peer to optmally utlze the avalable upload bandwdth wthn each GOS, such that all segments n each GOS need to be downloaded wthn ther playback deadlnes. An llustratve example showng how MDP s deployed n each GOS s depcted n Fg. 4. The MDP-based bandwdth allocaton of each peer n the resource recprocaton game s dscussed n detal n Secton 3. Note that MPD needs to be solved (Secton 3) only once when a GOS newly enters the buffer unless the supplyng peers n that GOS leave the network or change ther recprocaton behavors before the GOS reaches the playback pont. We study the mpact of supplyng peers leavng before ther GOSs reach the playback pont, and quantfy how the proposed schedulng algorthm can be optmzed n such dynamc network condtons n Secton Other ssues for the proposed schedulng algorthm In ths secton, we dscuss several remanng consderaton for mplementng the proposed schedulng algorthm.

4 GOS 1 GOS 2 (BW 2 ) GOS 3 (BW 3 ) GOS 1 GOS 2 GOS 3 New segment Buffer Pnt (t) playback sup 2 sup 1 sup 3 sup 5 Pnt (t) playback Fg. 5. Illustraton of resdual bandwdth created n the buffer. MDP sup 4 sup 6 Start MDP Fg. 4. Illustraton of the buffer snapshot, where BW k represents the allocated bandwdth to GOS k and sup m represent the supplyng peer m assocated wth ts correspondng segment. In the proposed buffer structure, BW 1 = BW 2 = BW Resdual bandwdth In the proposed schedulng algorthm, the resdual bandwdths are created n the buffer as the playback pont moves toward the last segment n the frst GOS. An llustratve example s depcted n Fg. 5. In the proposed approach, the resdual bandwdth can be used to estmate the resource recprocaton behavor of supplyng peers to be ncluded n the new GOS. We deploy the resource recprocaton models proposed n [11] n order to effcently analyze the resource recprocaton behavors of supplyng peers Past segments The segments already passed the playback pont are not useful to dsplay a vdeo content. However, hgher chance that other peers may be nterested n downloadng the segments. Therefore, a peer would lke to keep the past segments. We know that tme lags between playback ponts of assocated peers are usually less than 1 mnute [8] Incompletely flled buffer The performance of the proposed algorthm can be maxmzed f all segments n the buffer are flled. Thus, when a peer newly enters the network and starts to recprocate ts resources, t can ntally employ a smple heurstc schedulng algorthm such as that n [8], untl the buffer s flled up. After the peer fnds supplyng peers for all segments, t wll run the proposed schedulng algorthm. Note that the duraton of ths ntal phase s very short compared to the overall runnng tme of the system and thus, the mpact of ths heurstc algorthm on the overall performance s mnmal. The proposed schedulng algorthm s summarzed n Fg. 6. Is buffer flled? N Heurstc schedulng algorthm (Secton 2.5) Y New GOS created? N Allocate the gven BW among supplers n each GOS (Secton 3) Y Insert the new GOS n the buffer (Secton 2.2) Intalze MDP for the new GOS (Secton 3) The proposed schedulng algorthm Fg. 6. The proposed schedulng algorthm for real-tme vdeo streamng over P2P networks. 3. RESOURCE RECIPROCATION GAME I In ths secton, we model the resource recprocatons between a peer and ts supplyng peers wthn each GOS as a resource recprocaton game. The group of supplyng peers of peer n ts GOS k s denoted by C k. It can be easly computed that the total number of supplyng peers n C k s equal to the total number of segments n one GOS, e.g., N C k = N seg.the group of all supplyng peers n the buffer s denoted by C and t can be expressed as, C = N GOS C k. Note that every supplyng peer j n C has also ts own group of peers (.e., supplyng peers) C j. In ths paper, we assume that one segment s downloaded only by a dstnct peer n the

5 buffer. However, several segments can also be downloaded from one peer smultaneously, whch leads to the resource recprocaton behavor changes of the peer. In ths case, the schedulng algorthm needs to update the state transton probablty and the correspondng optmal polcy for ths peer. We now model the resource recprocaton game wthn one GOS played by a peer and ts supplyng peers [11]. Snce the same amount of the bandwdth s allocated to group of supplyng peers n each GOS, all GOSs n the buffer wll employ the dentcal method for the resource recprocaton strategy. Every peer partcpatng n resource recprocaton game employs MDP, whch s a 4-tuple S, A,P,R. S denotes the state space of peer. A represents the acton space whch contans sets of actons that a peer may take n the game. P s state transton probablty of peer. Suppose that a peer s at a state s (t) S and t takes an acton a (t) A at tme t. Then, the probablty that the peer wll be at the next state S at tme t +1can be found based } on the state transton probablty, P =Pr. R : S R s (t+1) s (t+1) s (t),a (t) represents the reward functon that maps the beneft of beng n a certan state nto a real number State space The state of a peer s defned as a set of bandwdths (e.g. the download rate) that each supplyng peer n a GOS provdes to peer. The set of bandwdths receved can be expressed as, (x k 1,,,x k N seg,) 1 x k j, BW j, j C k, 1 k }, (3) where x k j, represents the provded bandwdth by peer j n C k to peer, and BW j s the maxmum upload bandwdth of peer j. In ths paper, we assume that x k j, s quantzed by peer usng a functon f,j ( ), whch results n a quantzed dscrete value. For smplcty, we assume that f,j ( ) s a unform quantzaton functon,.e., f,j (x j, )=s l,j f (l 1) BW j n,j x j, <l BW j n,j, (4) where s l,j 3 s a representatve output value of the l th bn, and n,j s the total number of bns. The state of peer n GOS k can be correspondngly defned as, S k = s k =(s k,1,,s k,n seg ) s k,j = f,j (x k j,), j C k, 1 k }. (5) Therefore, the state of a peer of GOS k can be descrbed as a set of quantzed upload bandwdth that each supplyng peer n GOS k provdes. Note that, throughout ths paper, we 3 Note that l n ths notaton does not represent the GOS number as n (5) sometmes omt the subscrpton for the GOS number n the notaton of the state for smplcty, snce dentcal recprocaton strateges are employed n all GOSs Acton space An acton of a peer of GOS k can be defned as a set of bandwdth dvsons that peer contrbutes to each supplyng peer n GOS k. Thus, the acton space of peer n GOS k can be expressed as, A k = a k =(a k,1,,a k,n seg ) 0 a k,j BW k, j C k, } a k j C k,j = BW k, (6) where a k,j Ak represents the allocated bandwdth from peer to ts assocated supplyng peer j n GOS k and BW k denotes the maxmum total avalable bandwdth of peer n GOS k. The total bandwdth of wthn GOS k, BW k,s dstrbuted to ts supplyng peers n the GOS. Therefore, BW,j k = BW k. (7) j C k We can now descrbe the resource recprocaton wthn GOS ) k usng a par, (a k,sk ((a )= k 1, ak N seg ), (s k 1,,sk N seg ) where a k,j s the acton of peer n GOS k and sk,j s the quantzed resource obtaned from supplyng peer j n GOS k, whch s determned by f,j (x k j, ). For example, let us consder a stuaton where peer s n a state s (t) at tme t and the peer takes an acton a k,j Ak. By takng an acton ak, peer allocates a certan amount of bandwdth to each supplyng peer j n C k. Ths acton affects the states of all supplyng peers n C k and they wll response wth a new set of bandwdths, x k j,, back to peer Ths chan reacton wll eventually lead peer nto a new state, s (t+1),attmet +1. An llustratve explanaton about the resource recprocaton s provded n Fg State transton probablty State transton probablty (STP) s defned as a set of probabltes whch specfes how a peer would evolve from one state to another state (wthn a GOS) gven that the peer takes a certan acton. Suppose that a peer s n state s (t) wthn GOS k at tme t, and t takes an acton a k. Then, peer can transt to another state s (t+1) at tme t +1wth probablty 4, P a k,s (t+1) )=Pr(s (t+1) s (t),a k ). Wealsohaveanacton, whch s a set of bandwdth dvsons to each supplyng peer n GOS k, a (t) =(a (t),1,,a(t),n seg ). In order to calculate 4 Note that, for smplcty, we omt the subscrpton for GOS number n the notaton of the state. The subscrpton ndcatng tme, (t), s used nstead.

6 a k 1 GOS k a k 2 a k 3 a k 4 C k = p 1,, p 4} k a = a k 1,,a k 4} x 1 GOS k t x 2 x 3 x 4 s (t+1) = s 1,,s 4 } =f 1 (x 1 ),, f 4 (x 4 )} t+1 tme Fg. 7. An example of recprocaton game between peer and ts assocated supplyng peers, 1, 2, 3, and 4. Ths llustraton shows how the state evolve from t to t +1dependng on peer s acton at tme t and the reacton from each supplyng peer. the STP, P a k,s (t+1) ), we frst compute the STP for each supplyng peer n GOS k, denoted as P a k,j,j,s(t+1),j ) = Pr(s (t+1),j s (t),j,ak,j ), for each component state, s(t),j s(t). Then, the STP of peer n GOS k can be expressed as, P a k,s (t+1) )= P j C k a k,j,j,s(t+1),j ). (8) The above equaton s vald snce we assume that STP for each supplyng peer n a GOS s ndependent from each other as t s suggested n [11]. The STP for supplyng peer j n ), can be estmated based on supplyng peers behavors n the past usng an algorthm proposed n [11]. GOS k, P a k 3.4. Reward,s (t+1) We defne the utlty of peer acheved from GOS k as, = U,j(x k j, ), (9) U k j C k where U,j k s the utlty of peer acheves from the segment that s downloaded from supplyng peer j n GOS k at the rate x j,. U,j k can be defned as, 0, f U,j k xj, <BW,j =, ρ,j Q (x j, ), otherwse, (10) where ρ,j s the constant representng the preference that peer has on segment assocated wth peer j. Note that havng ρ,j n (10) allows the proposed schedulng algorthm to work wth scalable vdeo coders where each segment or packet has dfferent mpact on the output vdeo qualty. Q (x j, ) s the derved qualty of the downloadng rate x j,, whch can be represented n PSNR [13]. BW,j represents the theoretcal constant bandwdth (.e. rate) that j must provde n order to complete the download wthn the playback deadlne. It can also be defned as the mnmum download rate requred n order to meet the mnmum requred utlty of segment obtaned from supplyng peer j. We can compute the value of BW,j as follows. Suppose that a new GOS enters the buffer. If we assume that each segment contans data of t seg second vdeo (.e. t seg =1second n [8]), we can calculate the playback deadlne from the tme that a GOS newly enters the buffer as, t deadlne =( 1) N seg t seg, (11) where s the number of GOSs n the buffer and N seg s the number of segments (or number of supplyng peers) n each GOS. We also defne B j as the sze (bts) of each segment that supplyng peer j must transmt. Then, we calculate the theoretcal constant bandwdth (.e. rate) that j must mantan as, BW,j = B j/t deadlne. The utlty of peer acheved from GOS k, U k, s a nondecreasng functon of the x whch s the sum of download rate from supplyng peers n GOS k, e.g., x = j C x k j,. Thus, we defned the reward of peer wthn GOS k, R k(sk ), as the total resource receved from supplyng peers. Snce a state of peer s a collecton of quantzed value of the receved resource from supplyng peers n GOS k, we defne the reward of a state to be a random varable. The reward can be mathematcally represented as, R k (s k )=R (s k,1,,s k,n seg )= ρ,j r,j (s k,j), j C k (12) where r,j (s k,j ) s a random varable representng the receved resources from peer j, e.g., x j,. Thus, the resultng utlty of peer wthn GOS k can be expressed as, U k = ( j C U k k,j r,j (s k,j )) Optmal resource recprocaton polcy A resource recprocaton polcy determnes an acton that a peer can take when t s n a certan state. The optmal resource recprocaton strategy determned by MDP enables each peer to maxmze ts long-term utlty. Specfcally, optmal polcy π of peer for GOS k determnes the optmal acton π (s )=a for every state s S, such that the optmal actons maxmze not only the mmedate expected reward, but also the future rewards. The cumulated expected reward (.e., mmedate expected reward and the future rewards) for a peer that s n a state s (t) =(s 1,,s Nseg ) at tme t = t c can be express as [11], R fore t =t c+1 (s (tc) ) γ (t t c) P a (s (tc) s (tc+1) S P a (t +1) s (t +1) S,s (tc+1) )R (s (tc+1) )+ (s (t ),s (t +1) )R (s (t +1) ), (13) where γ s a dscount factor, whch s ntroduced n order to provde dfferent weghts between the mmedate and the

7 future rewards, because the mmedate reward s worth more than the future rewards. The optmal actons that maxmze (s (tc) ) n (13) are referred to as foresghted actons,.e., R fore a = max arg a A s (t+1) P a S,s (t+1) } )R (s (t+1) ). Moreover, the optmal polcy can be always found based on well-known algorthms such as value teraton or polcy teraton [14]. Usng our buffer structure, the optmal polcy computed by MDP s performed multple tmes, whle supplyng peers perform the download n the buffer (.e., from the tme a GOS enters the buffer untl t meets the playback pont). Thereby, t s mportant to maxmze both the mmedate and the future rewards because a GOS stays n the buffer for a long-term. As a result, output vdeo qualty acheved by segments n a GOS wll be mproved as we perform foresghted actons. Smulaton results for the mpact of foresghted actons are provded n Secton 4.1. The performance of the overall schedulng algorthm may also vary dependng on the value of the dscount factor, γ. We can estmate the approprate value for the dscount factor by predctng the number of tmes that MDP wll update the state of n a GOS. We prevously calculated the playback deadlne for all segments n a GOS,.e., t deadlne n (11). We can realze that t deadlne s actually equal to the amount of tme that GOS stays n the buffer before t reaches the playback pont. For example, n [8], the buffer s assumed to consst 120 segments, and each segment contans data of 1 second vdeo. Hence, t deadlne = 120 second n ths case. Suppose that the MDP updates the state of n GOS (.e., perform the optmal acton) every t update seconds. Then the total number of updates of the state of s gven by, tdeadlne N update =. t update We can now choose a value of γ such that the future reward after N update s neglgbly small,.e., γ (N update+1) γ thresh, (14) where γ thresh s a threshold value to represent the level of accuracy. Smulaton results usng dfferent dscount values wll be provded n Secton Complexty analyss for optmal resource recprocaton polcy The optmal resource recprocaton polcy can be computed based on the well-known value teraton algorthm, and t s shown that ths algorthm requres O( A S 2 )complexty for each teraton [15], where A and S denote the number of actons and the number of states. Recall that, n the proposed schedulng algorthm, N seg segments are ncluded n a GOS k, and the segments can be downloaded from dfferent supplyng peers. Therefore, the complexty requred to solve an MDPnGOSk can be expressed as, O A k N seg j C k n,j 2, (15) where n,j s the number of the state descrpton as n (4). As shown n [15], the maxmum number of teratons requred to fnd the optmal polcy s determned by several pre-determned parameters such as the number of bts used to descrbe each state, the dscount factor, etc. Hence, the total complexty requred to fnd the optmal polcy s prmarly determned by the number of actons, the number of states, and the number of segments n a GOS as shown n (15). Therefore, the complexty analyss shown n (15) provdes a gudelne how peer can flexbly adjust the number of ts actons, states, and the segments n a GOS, whle explctly consderng ts avalable computatonal capacty, avalable tme to fnd the polcy, etc. For example, f the computaton complexty s a concern, peer can adaptvely determne the number of states by adjustng f,j ( ) n the proposed algorthm. Alternatvely, the number of segments N seg can also be determned. Hence, each peer can explctly consder the tradeoffs between the complexty requred to mplement the proposed approach and the correspondng performance mprovement Determnng the number of segments n a GOS In ths secton, we consder the tradeoffs between computatonal complexty and utlty. Thus, a peer can determne the number of GOSs n ts buffer and the number of segments n each GOS, thereby deployng ts schedulng algorthm at the desred complexty and utlty par. For nstance, we consder a peer assocated wth a fnte number of assocated peers. We label the acton of peer as a =(a,1,,a,ntotal ) as n (6). As an llustratve example, we assume that assocated peers of peer recprocate resources lnearly wth respect to the acton of peer (.e., x j, = α,j a,j ) for all j C. We also assume that peer has ts optmal acton, a,j, whch maxmzes the utlty (e.g. output vdeo qualty n PSNR) of peer. The range of the optmal bandwdth allocated from peer to peer j s 0 a,j BW. When the buffer s dvded nto GOSs, the range of the bandwdth that peer can allocate to peer j changes accordng to (6). We denote the actual allowable bandwdth from peer to peer j wth consderng the buffer structure as a,j, and the range of a,j can be defned as 0 a,j BW /. Snce we assume that peer j provdes ts bandwdth lnearly proportonal to a,j, the reducton n the bandwdth provded from peer j due to

8 the dfference between a,j and a,j can be expressed as ) α,j (a,j Δx j, = BW BW, f a,j BW 0, f 0 a,j. BW We can compute the expected value of Δx j, under an assumpton that a,j has a unform probablty dstrbuton over ts range [0,BW ] (.e., f a,j =1/BW )as E Δx j, } = = = BW 0 BW 0 BW f a,j (a) Δx j, da 1 α,j (a a,j ) da BW BW / 1 BW α,j = 1 2 α,jbw ( NGOS 1 ( a BW ). ) da Usng the dstorton-rate based utlty functon [16] for the state-of-the-art vdeo coders [17], we can compute the expected dstorton for a fxed based on E x j, }. Snce E x j, } s an ncreasng functon of, the dstorton functon wll also be an ncreasng functon of and thus, the utlty functon of peer wll be a decreasng functon of. Fg. 8 llustrates an example of the utlty functon of peer wth a buffer of 12 segment. We express the utlty functon of peer n PSNR and the testng vdeo sequence s a Foreman sequence wth CIF resoluton, temporal level of 4, and the frame rate of 30 Hz under the assumpton that the rate x j, = α,j a,j gves the best performance (.e., α,j BW = 250Kbps). The computatonal complexty can also be computed as a functon of accordng to (15), snce N C k = N total /. Thus, a user can choose the number of GOSs n the buffer accordng to the desred utlty and complexty par. 4. SIMULATION RESULTS 4.1. Comparson of varous schedulng algorthms on PSNR performance As dscussed n Secton 2 and 3, the buffer can be dvded nto several GOSs and the MDP-based foresghted decsons can effcently allocate the avalable bandwdth. In ths secton, we compare the multmeda qualty acheved based on the proposed schedulng algorthm as well as the other exstng schedulng strateges - Tt-for-Tat (TFT) strategy n Bt- Torrent system [6], [7] and BToS systems [12]. A peer wth the TFT strategy selects a fxed number of leechers (supplyng peers) that contrbute the hghest bandwdths to the peer, and recprocates the equally dvded bandwdth to the selected leechers. The TFT s deployed on the Fg. 8. An llustratve example of acheved PSNR value at dfferent. entre buffer that s not dvded nto GOSs. We assume that the buffer only serves the downloadng processes and also assume a fxed set of supplyng peers n the buffer. The BToS system s a varant of the BtTorrent system, whch can support streamng applcatons [12]. In ths system, the schedulng algorthm categorzes the segments n the buffer nto two groups: hgh prorty group and remanng group. Hgh prorty group contans segments that have not been downloaded and are close to the playback deadlne. The remanng group contans segments that has not been downloaded and that are not a member of the hgh prorty group. Ths algorthm has a parameter p whch represents the probablty that a segment n the hgh prorty group s selected to be downloaded. Thus, ths schedulng algorthm explctly consders the real-tme constrants by focusng on downloadng segments that are closer to the playback deadlne. In ths smulaton, the buffer of peer may contan at most 18 segments and the multmeda bt-streams of 50 segments are provded to the system n order to emulate the real-tme envronment. For smplcty, we assume that each segment s separated by I-frames, thereby havng ts own vdeo qualty. We also assume that each supplyng peer can provde maxmum allowable bandwdth of 125 Kbps. For the smulaton, we deploy the dstorton-rate based utlty functon for the state-of-the-art vdeo coders [17], whch was ntroduced n [16]. Peer downloads Foreman vdeo sequence wth CIF resoluton and the frame rate of 30 frames/second. The vdeo sze s 1.6 Mbytes. The smulaton results are shown n Fg. 9. In the smulatons, the number of the assocated peers s 18, whch s same as the number of segments n the buffer. The number of slots for downloadng s 4, and thus, each peer equally dvdes ts avalable upload bandwdth and allocates t to 4 dfferent supplyng peers that contrbute the hghest bandwdth. Each peer has 10 avalable actons and 4 state descrptons,.e., a,j a 1,j,a2,j,,a10,j } and s,j s 1,j,s2,j,s3,j,s4,j }, respectvely. For BTos system, the hgh prorty group contans 4 segments that are closest to the play-

9 40 35 Receved Vdeo Qualty (PSNR) 4.2. Impact of dscount factor on heterogeneous network condtons PSNR [db] Proposed Algorthm BTos lke TFT strategy BtTorrent TFT strategy State Index Fg. 9. An llustratve example for vdeo qualty n PSNR, acheved by dfferent schedulng algorthms. back deadlne and the value of p s set to 1. Thus, the second test-bench system equally dvdes the total bandwdth and provde them to peers assocated wth the 4 segments that are about to be dsplayed. In the proposed algorthm, the buffer s dvded nto GOSs, where each GOS contans 3 segments. Hence, the number of GOSs n the buffer s 18/3 =6.However, because we assume 10 acton descrptons, havng 6 GOSs s not reasonable to show the result of our foresghted schedulng algorthm. Thus, n ths smulaton, the proposed schedulng algorthm performs the foresghted schedulng for the 6 segments closest to the playback deadlne. A state of peer s represented as s =(s,1,s,2,,s,6 ) buttsdvded nto 2 GOSs. Therefore, we have two sub-states for each GOS represented as s k =(s k,1,sk,2,sk,3 ) for 1 k = 2. The total number of possble state of peer can be calculated as 4 6 = 4096, and the smulaton s performed for all possble 4096 ntal state of peer. The bandwdth allocated to each GOS, denoted as BW k, s equal to BW / as n (2). The dscount factor s set to 0.8,.e., γ =0.8. The smulaton results n Fg. 9 clearly show that the proposed foresghted schedulng algorthm outperforms the heurstc TFT strateges. State ndex n Fg. 9 represents an ntal state of a peer. In Fg. 9, n order to help vsualze the results, we only dsplay a randomly selected porton of the ntal state ndces,.e., from state ndex 1 to 100. The overall smulaton results over the entre state ndces are smlar to what we have n Fg. 9. On average, the proposed algorthm mproves 3.61 db over the BToS system and 5.33 db over the BtTorrent-lke TFT strategy across all state ndces. In ths secton, we quanttatvely compare the mpact of the dscount factor on the vdeo qualty n heterogeneous P2P networks. In our consdered P2P real-tme streamng scenaro, the sze of the buffer s fxed, and thus, the optmal polcy should not maxmze the reward cumulated up to nfntely far away n the future. We can calculate the value of the dscount factor based on (14) usng nformaton about the buffer sze. Ths allows the schedulng algorthm to consder the future reward lmted to the sze of the buffer. However, when the network s heterogeneous and the supplyng peers n GOS leave the network before the GOS reaches the playback pont, the dscount factor computed based on the buffer sze s not optmal anymore. When one or more of the supplyng peers leave the network, the schedulng algorthm must fnd new supplyng peers and compute the optmal polcy based on the recprocaton atttude of the new supplyng peers as dscusses n Secton 3. In ths case, f we assume that we know the average number of supplyng peers leavng as a GOS performs the download process n the buffer, we can compute the value of the optmal dscount factor based on the average tme duraton that a GOS remans n the buffer wthout changes n supplyng peers atttudes. For example, let us assume that the scheduler performs the optmal polcy 20 tmes per GOS untl the GOS reaches the playback pont. We also assume that, on average, recprocaton behavors of supplyng peers change once as a GOS progresses n the buffer. In other words, the optmal acton s performed 10 tmes on average before the recprocaton behavors change. In ths scenaro, the scheduler can compute the dscount factor based on the average number of optmal actons to be performed wthout changes n supplyng peers atttudes, e.g. 10 tmes. In the example above, the dscount factor computed based on the buffer sze s γ = and the dscount factor based on the average number of optmal actons performed s γ = In Fg. 10, the parameter settngs, such as the number of assocate peers and the buffer szes are dentcal to the smulaton for the proposed algorthm performed n Secton 4.1, except that dfferent values of the dscount factors are consdered. We dsplay the multmeda vdeo qualty over a randomly selected porton of the state ndces,.e., from state ndex 400 to 500. The results n Fg. 10 show that the system deployng the optmal dscount factor based on the average number of acton perform wthout changes n supplyng peers recprocaton behavors gves hgher and more stablzed PSNR performance compared to the system wth dscount factor computed based on the buffer sze. On average, the system wth the optmal dscount factor shows advantages of 0.44 db over the system wth dscount factor computed based on the buffer sze.

10 PSNR [db] Acheved Multmeda Qualty usng Dfferent Dscount Factor 28 Optmal dscount factor Dscount factor based on buffer sze State Index Fg. 10. Illustratve example of mpact of dscount factor on heterogeneous network condtons. 5. CONCLUSIONS In ths paper, we propose a schedulng strategy for vdeo streamng over P2P networks formed by heterogeneous and self-nterested peers. We propose a buffer structure whch dvdes the buffer nto a fnte number of GOSs. Ths buffer desgn enables the proposed schedulng algorthm to successfully delver vdeo segments wthn the playback deadlne. Moreover, ths buffer structure allows us to model the resource recprocaton among peers n each GOS as a recprocaton game, and thus, peers can make foresghted decsons based on MDP, whch lead them to maxmze ther long-term vdeo qualtes. Smulaton results show that the proposed foresghted schedulng algorthm results n mproved vdeo qualtes (.e., acheves a hgher PSNR) compared to other exstng schedulng strateges such as TFT strategy. Moreover, the proposed schedulng algorthm enables the peers to flexbly operate n heterogeneous and dynamc P2P networks by computng the optmal value of the dscount factor. [7] A. Legout, N. Logkas, E. Kohler, and L. Zhang, Clusterng and sharng ncentves n BtTorrent systems, SIGMETRICS Perform. Eval. Rev., vol. 35, no. 1, pp , [8] X. Zhang, J. Lu, B. L, and T. S. P. Yum, CoolStreamng/DONet: A data-drven overlay network for effcent lve meda streamng, n Proc. IEEE Conf. of Comput. and Commun. Soc. (INFOCOM 05), vol. 3, Mar. 2005, pp [9] Z. Xang, Q. Zhang, W. Zhu, Z. Zhang, and Y.-Q. Zhang, Peer-to-peer based multmeda dstrbuton servce, IEEE Trans. Multmeda, vol. 6, no. 2, pp , Apr [10] V. Pa, K. Kumar, K. Tamlman, V. Sambamurthy, and A. E. Mohr, Chansaw: Elmnatng trees from overlay multcast, n Proc. 4th Int. Workshop on Peer-to-Peer Systems (IPTPS), Feb [11] H. Park and M. van der Schaar, A framework for foresghted resource recprocaton n P2P networks, IEEE Trans. Multmeda, vol. 11, no. 1, pp , Jan [12] M. I. A. Vlavanos and M. Faloutsos, Btos: Enhancng bttorrent for supportng streamng applcatons, n Global Internet Workshop n conjuncton wth IEEE INFOCOM 2006, Apr [13] M. van der Schaar and P. A. Chou, Eds., Multmeda over IP and Wreless Networks. Academc Press, [14] D. P. Bertsekas, Dynamc Programmng and Stochastc Control. Academc Press, [15] M. L. Lttman, T. L. Dean, and L. P. Kaelblng, On the complexty of solvng Markov decson problems, n Proc. 11th Conf. on Uncertanty n Artfcal Intellgence, May [16] H. Park and M. van der Schaar, Barganng strateges for networked multmeda resource management, IEEE Trans. Sgnal Process., vol. 55, no. 7, pp , Jul [17] Y. Andreopoulos, A. Munteanu, J. Barbaren, M. van der Schaar, J. Cornels, and P. Schelkens, In-band moton compensated temporal flterng, Sgnal Processng: Image Communcaton (specal ssue on Subband/Wavelet Interframe Vdeo Codng ), vol. 19, no. 7, pp , Aug REFERENCES [1] J. Lu, S. G. Rao, B. L, and H. Zhang, Opportuntes and challenges of peer-to-peer nternet vdeo broadcast, Proceedngs of the IEEE, Specal Issue on Recent Advances n Dstrbuted Multmeda Communcatons, [2] S. Androutsells-Theotoks and D. Spnells, A survey of peerto-peer content dstrbuton technologes, ACM Comp. Surveys, vol. 36, no. 4, pp , Dec [3] Gnutella. [Onlne]. Avalable: [4] KaZaA. [Onlne]. Avalable: [5] Napster. [Onlne]. Avalable: [6] B. Cohen, Incentves buld robustnessn BtTorrent, n Proc. P2P Economcs Workshop, Berkerly, CA, 2003.