Scalable Download Protocols

Transcription

1 Scalable Download Protocols A Thess Submtted to the College of Graduate Studes and Research In Partal Fulfllment of the Requrements For the Degree of Doctor of Phlosophy In the Department of Computer Scence Unversty of Saskatchewan Saskatoon, Saskatchewan By Nklas Carlsson Copyrght Nklas Carlsson, December All rghts reserved.

2 Permsson to Use In presentng ths thess n partal fulflment of the requrements for a Postgraduate degree from the Unversty of Saskatchewan, I agree that the Lbrares of ths Unversty may make t freely avalable for nspecton. I further agree that permsson for copyng of ths thess n any manner, n whole or n part, for scholarly purposes may be granted by the professor or professors who supervsed my thess work or, n ther absence, by the Head of the Department or the Dean of the College n whch my thess work was done. It s understood that any copyng or publcaton or use of ths thess or parts thereof for fnancal gan shall not be allowed wthout my wrtten permsson. It s also understood that due recognton shall be gven to me and to the Unversty of Saskatchewan n any scholarly use whch may be made of any materal n my thess. Requests for permsson to copy or to make other use of materal n ths thess n whole or part should be addressed to: Head of the Department of Computer Scence Unversty of Saskatchewan Saskatoon, Saskatchewan, Canada S7N 5C9

3 Abstract Scalable on-demand content delvery systems, desgned to effectvely handle ncreasng request rates, typcally use servce aggregaton or content replcaton technques. Servce aggregaton reles on one-to-many communcaton technques, such as multcast, to effcently delver content from a sngle sender to multple recevers. Wth replcaton, multple geographcally dstrbuted replcas of the servce or content share the load of processng clent requests and enable delvery from a nearby server. Prevous scalable protocols for downloadng large, popular fles from a sngle server nclude batchng and cyclc multcast. Analytc lower bounds developed n ths thess show that nether of these protocols consstently yelds performance close to optmal. New hybrd protocols are proposed that acheve wthn 20% of the optmal delay n homogeneous systems, as well as wthn 25% of the optmal maxmum clent delay n all heterogeneous scenaros consdered. In systems utlzng both servce aggregaton and replcaton, well-desgned polces determnng whch replca serves each request must balance the objectves of achevng hgh localty of servce, and hgh effcency of servce aggregaton. By comparng classes of polces, usng both analyss and smulatons, ths thess shows that there are sgnfcant performance advantages n usng current system state nformaton (rather than only proxmtes and average loads) and n deferrng selecton decsons when possble. Most of these performance gans can be acheved usng only local (rather than global) request nformaton. Fnally, ths thess proposes adaptatons of already proposed peer-asssted download technques to support a streamng (rather than download) servce, enablng playback to begn well before the entre meda fle s receved. These protocols splt each fle nto peces, whch can be downloaded from multple sources, ncludng other clents downloadng the same fle. Usng smulatons, a canddate protocol s presented and evaluated. The protocol ncludes both a pece selecton technque that effectvely medates the conflct between achevng hgh pece dversty and the n-order requrements of meda fle playback, as well as a smple on-lne rule for decdng when playback can safely commence.

4 Acknowledgements Ths work has been carred out under the supervson of Derek Eager. Throughout my tme n department he has been an excellent teacher, mentor, and frend. Learnng from hm has been extremely rewardng and s an experence whch I wll always treasure. My research has also greatly benefted from feedback and support provded by Dr. Mary Vernon. My advsory commttee, consstng of Wnfred Grassmann, Mark Kel, Ralph Deters, and Chrs Soteros, has provded thoughtful comments and feedback, makng every meetng a great learnng experence. Thank you to my external examner Dr. Janelle Harms, who traveled from Edmonton to attend my defence. I would also lke to thank all the graduate students, staff, and faculty who have made the atmosphere wthn the Department a comfortable place to work. In partcular, I would lke to thank Jan Thompson for makng lfe easer on graduate students; Gord McCalla for makng sure that I have human contact even at odd hours; Dwght Makaroff for hs wllngness to talk hockey; and fnally, Anrban Mahant for provdng me wth gudance through lfe as a graduate student and beyond. I would lke to thank my parents Olle and Lena who have provded me wth more help and better gudance than I could have ever asked for, my sster Maln, as well as my grandparents. Your support and love made ths degree worth a lot more. A specal thank you goes to my partner Alexa Brggs. In addton to supportng me n my pursut of ths degree, she has been a great teacher of the Englsh language, my best frend, and a great source of joy and laughter. Fnally, I would lke to send a thought to my very specal four legged frend Bobby Lue. Although you are no longer wth us, you have a bg part n ths thess. She was a great lstener and would always keep me company when I needed t the most. Thank you all for your support!

5 Table of Contents Permsson to Use... Abstract... Acknowledgements... Table of Contents...v Lst of Tables...v Lst of Fgures...v Lst of Common Notaton... x Lst of Acronyms...x Glossary...x Introducton.... Scalable Content Delvery Servce Aggregaton Replcaton Proxy Cachng Content Dstrbuton Networks Peer-to-Peer and Peer-asssted Systems Problem Descrpton Contrbutons Scalable Download from a Sngle Server Scalable Download from Multple Servers On-demand Streamng usng Scalable Download....4 Thess Organzaton Background Multcast IP Multcast Applcaton-level Multcast Multcast-based Sngle Server Scalable Download Protocols Batchng Protocols Push-based Protocols Pull-based Protocols Cyclc Multcast Protocols Hybrd Protocols Clent Heterogenety Server Placement and Selecton Replca Placement Replca Selecton Peer-asssted Content Delvery Peer-asssted Download Peer-asssted Streamng Scalable Download from a Sngle Server Baselne Polces Batchng Cyclc Multcast Lower Bounds v

6 3.2. Unconstraned Clent Recepton Rates Constraned Clent Recepton Rates Lower Bound Comparsons Near-optmal Protocols Protocols Mnmzng Maxmum Delay Protocols Mnmzng Average Delay Worst-case Performance Heterogeneous Clents Class-specfc Servce Requrement and Maxmum Delay Lower Bound (slp) Bandwdth Approxmaton Extenson of Cyclc/cd,bot Class-specfc Recepton Rates Heterogeneous Lower Bound Protocols Summary Scalable Download from Multple Replcas System Model Dynamc vs. Statc Polces Analyss for Sngle Replca Systems Delmtng the Achevable Performance wth Statc Polces Delmtng the Achevable Performance wth Dynamc Polces Batched Servce Fountan Servce Performance Comparsons Deferred Selecton vs. At-arrval Selecton Delmtng the Achevable Performance wth At-arrval Polces Batched Servce Fountan Servce Performance Comparsons Local State vs. Global State Canddate Local State Polces Batched Servce Fountan Servce Canddate On-lne Global State Polces Performance Comparson Average Delay Batchng Polcy Comparson Delmtng the Achevable Performance wth Statc Polces A Canddate Dynamc Global State Polcy A Canddate At-arrval Polcy Performance Comparsons Summary On-demand Streamng usng Scalable Download Smulaton Model Pece Selecton Canddate Polces Performance Comparsons...22 v

7 5.3 Usng a Dynamc Startup Rule Smple Startup Polces Performance Comparsons Summary Conclusons Thess Summary Thess Contrbutons Future Work References Appendx A: Proof of Sngle Server Heterogeneous Lower Bound Appendx B: Asymptotc Analyss of Dynamc vs. Statc Replca Selecton Appendx C: Analyss of Heterogeneous Batched At-arrval Polcy v

8 Lst of Tables Table 3.: Notaton used n Chapter Table 3.2: Summary of Worst-case Performance ( f>0 = f f > 0 and 0 otherwse) Table 3.3: Notaton for Heterogeneous Lower Bound Algorthm Table 4.: Notaton used n Chapter Table 4.2: Average and Maxmum Number of States usng the Optmal Offlne Algorthm (N = 6, c = 0.5, L =, λ = for all ) Table 5.: Notaton used n Chapter Table 5.2: Example Smulaton Executon Tmes for a Flash Crowd of Sze N v

9 Lst of Fgures Fgure 3.: Operaton of the Baselne Protocols for an Example Request Sequence Fgure 3.2: Lower Bound Approxmaton (% relatve error contours; unt of data volume s the fle, unt of tme s the tme requred to download the fle at maxmum rate:.e., L =, b = ) Fgure 3.3: Lower Bounds on Clent Delay (unt of data volume s the fle, unt of tme s the average tme between requests:.e., L... 5 Fgure 3.4: Maxmum Delay wth Baselne Protocols Relatve to Lower Bound (L = ) Fgure 3.5: Average Delay wth Baselne Protocols Relatve to Lower Bound (L = ) Fgure 3.6: Examples Scenaros for Improved Protocols Fgure 3.7: Maxmum Delay wth Improved Protocols Relatve to Lower Bound (L =, Fgure 3.8: Average Delay wth Improved Protocols Relatve to Lower Bound (L = ) Fgure 3.9: Impact of Class-specfc Maxmum Delays (L =, D 2 = 5D, varyng arrval 2 }) Fgure 3.0: Heterogeneous Lower Bound Algorthm Fgure 3.: Maxmum Delay wth Heterogeneous Clent Protocols Relatve to Lower Bound (L D values nversely proportonal to maxmum achevable recepton rates) Fgure 4.: Best Potental Performance wth Statc Polces Relatve to that wth Dynamc Polces, for Batched Servce Fgure 4.2: Best Potental Performance wth Statc Polces Relatve to that wth Dynamc Polces, for Fountan Servce... 9 Fgure 4.3: Confdence Intervals for Fgure 4.2(a) Fgure 4.4: Best Potental Performance wth At-arrval Polces Relatve to that wth General Dynamc Polces, for Batched Servce Fgure 4.5: Best Potental Performance wth At-arrval Polces Relatve to that wth General Dynamc Polces, for Fountan Servce Fgure 4.6: Performance wth Local State Polcy Relatve to the Best Potental Performance wth General Dynamc Polces, for Batched Servce Fgure 4.7: Performance wth Local State Polcy Relatve to the Best Potental Performance wth General Dynamc Polces, for Fountan Servce (L = L/2) v

10 Fgure 4.8: Performance wth Local State Polcy Relatve to the Performance wth the Canddate Global State Polcy, for Batched Servce Fgure 4.9: Best Potental Performance wth Statc Polces and wth At-arrval Polces, and Actual Performance wth a Local State Polcy, Relatve to the Best Potental Performance for General Dynamc Polces, for Batched Servce Fgure 4.0: Best Potental Performance wth Statc Polces and Actual Performance wth an At-arrval Polcy, Relatve to the Actual Performance for a Global Dynamc Polcy, for Batched Servce (each polcy s evaluated based on average delay) Fgure 5.: Max-mn Far Bandwdth Allocaton Algorthm used to Calculate the Rates of the Unchoked Peer Connectons Fgure 5.2: Average Achevable Startup Delay under a Steady State Posson Arrval Process wthout Early Departures: The Impact of Clent Bandwdth Capacty (b/u = 3, λ = 64, and ϕ = 0) Fgure 5.3: Cumulatve Dstrbuton Functon of the Best Achevable Startup Delay under a Steady State Posson Arrval Process wthout Early Departures (u = 2, b = 6, λ = 64, and ϕ = 0) Fgure 5.4: Average Achevable Startup Delay under Steady State Posson Arrval Process: Example Scenaros Fgure 5.5: Average Achevable Startup Delay under a Steady State Posson Arrval Process wth Early Departures (u = 2, b = 6, λ = 64) Fgure 5.6: Average Achevable Startup Delay under Exponentally Decayng Arrval Rates (u = 2, b = 6, λ(t) = λ 0 e ηt, λ 0 = 28η / ( e 2η )) Fgure 5.7: Average Achevable Startup Delay under a Steady State Posson Arrval Process wth both Hgh and Low Bandwdth Clents (λ = 64, ϕ = 0, u L = 0.4, b L =.2, u H = 2, b H = 6) Fgure 5.8: Exponentally Decayng Arrval Rates (u = 2, b = 6, λ(t) = λ 0 e ηt, λ 0 = 28η / ( e 2η )) Fgure 5.9: Heterogeneous Scenaro wth Posson Arrvals wth both Hgh and Low Bandwdth Clents (λ = 64, ϕ = 0, u L = 0.4, b L =.2, u H = 2, b H = 6) x

11 Lst of Common Notaton Ths table summarzes the most commonly used notaton n ths thess. Symbol Page Defnton A 40, 07 Average clent delay B 40, 8 Average rate at whch servce cost s ncurred (equal to the average server bandwdth) b 40, 9 Maxmum sustanable clent recepton rate (equal to download bandwdth capacty of a clent/peer) m b j 9 Maxmum sustanable download rate for any of j s download connectons C 8 Total servce delvery cost c 8 Access cost per unt of servce receved for all clent groups and replcas j, j D 40, 8 Maxmum clent delay f 4, 08 Batchng delay parameter (fracton of tmes n+ should be used as a threshold, rather than n) L 40, 80, 22 Servce requred by each requestng clent (equal to the fle sze) N 80 Number of replcas n 4, 08 Batchng delay parameter (threshold value on the number of requests) q 8 Average fracton of ts servce that a group clent receves from other than replca P, p Probabltes r 40, 80 Transmsson rate on a multcast channel r j 9 Transfer rate on a connecton from peer to peer j T, t Tme u 9 Upload bandwdth capacty of peer m u 9 Maxmum sustanable upload rate for any of s upload connectons α 90, 2 Zpf parameter 40 Batchng delay parameter (threshold value on the maxmum tme untl servce) δ j 9 f s transmttng to j (.e., j s nterested n and has been unchoked by ); 0 otherwse f >0 63, 09 or 0, based on logc comparson ϕ 22 Rate at whch peers defect from the system before havng completed download η 27 Exponental decay factor λ 40, 80, 22 Fle request rate x

12 Lst of Acronyms Ths table summarzes acronyms used n ths thess. Acronym Page Defnton AS 4 Autonomous System batchng/cbd 40 Batchng/constant batchng delay batchng/rbd 4 Batchng/request-based delay BGP 5 Boarder Gateway Protocol CBT 5 Core-Based Trees CDN 3 Content Dstrbuton Network cyclc/cd,l 55 Cyclc/constant delay, lsteners cyclc/cd,bot 57 Cyclc/constant delay, bounded on-tme cyclc/l 42 Cyclc/lsteners cyclc/rbd,l 60 Cyclc/request-based delay, lsteners cyclc/rbd,cot 6 Cyclc/request-based delay, controlled on-tme cyclc/w2,l 48 Cyclc/wat for second, lsteners DNS 6 Doman Name System DVMRP 5 Dstance Vector Multcast Routng Protocol EWMA 30 Expected weghted movng average FCFS 23 Frst Come Frst Serve 7 Heterogeneous lower bound IGMP 4 Internet Group Membershp Protocol ISP 4 Internet Servce Provders KBR 9 Key-based routng LTA 30 Long term average LWF 23 Longest Wat Frst MBGP 5 Multprotocol Boarder Gateway Protocol MOSPF 5 Multcast Open Shortest Path Frst MRF 23 Most Request Frst MRFL 23 Most Request Frst Lowest MSDP 6 Multcast Source Dscovery Protocol NRU 20 Normalzed Resource Usage PIM-SM 5 Protocol Independent Multcast Sparse Mode RP 5 Rendezvous pont sa 50 Shfted arrvals slp 46 Send as late as possble s-cyclc/l 73 Shared cyclc/lsteners x

13 Glossary Batchng protocol A type of servce aggregaton protocol, n whch clents havng requested a fle, wat to begn recevng the fle untl the begnnng of a multcast (or broadcast) transmsson, whch collectvely serves a set of watng clents. BtTorrent A peer-asssted download protocol, n whch a fle s splt nto smaller peces that can be downloaded n parallel from dfferent peers. Bulk data Data or fles for whch there s no advantageous order n whch data should be retreved. Choke algorthm Algorthm used, by BtTorrent, to determne whch peers to upload (and not to upload) peces of a fle to. Clent recepton rate The rate at whch data s receved by the clent. Content Dstrbuton Network (CDN) Interconnected servers dstrbuted across the network, whch allows the content to be effectvely replcated, clents to be served by nearby replcas, and the content dstrbutor to mantan control over the content. Contnuous meda fles Meda fles, such as audo and vdeo, that contnuously must be rendered at specfed rates. Cyclc multcast protocol A type of servce aggregaton protocol, n whch the fle data s cyclcally transmtted on a multcast channel, whch clents begn lstenng to at an arbtrary pont n tme, and contnue lstenng to untl all of the fle data has been receved. Dgtal fountan A cyclc multcast protocol, n whch the fle data s erasure coded such that a clent lstenng to the channel can recreate the orgnal content after havng retreved an arbtrary set of data equal (or slghtly larger) n sze as the orgnal fle. Download bandwdth capacty The maxmum sustanable rate at whch data can be receved by the clent. Download protocol Protocol used to transfer bulk data to clents. The man metrc of these protocols s the tme untl the entre fle s fully downloaded. Erasure codng Codng technque used to accommodate packet losses. Leecher BtTorrent peer whch does not have a complete copy of a fle, and currently s downloadng peces of the fle. x

14 Multcast Famly of technques used to set up forwardng trees and to forward the content (through these dstrbuton trees), from one or more source to multple recevers. Multcast channel The server and network resources used to delver each multcast to each member of a multcast group. Multcast group A collecton of nodes nterested n recevng the same multcast transmsson. Upload bandwdth capacty The maxmum sustanable rate at whch data can be transferred by the clent. Peer-asssted protocol Protocol n whch peers contrbute to the collectve power of the system by makng (part of) ther resources avalable. Peer-to-peer system Systems consstng of peers. Pece selecton polcy Polcy used by BtTorrent peer to determne the next pece to request for download. Proxy caches Content caches located at servers embedded between clents and the orgn server, whch ntercept clent requests and (n the case they have a stored copy) serves them on behalf of the orgn server. Posson arrval process A memoryless arrval process wth constant arrval rate, or equvalently, an arrval process wth nter-arrval tmes that are ndependent and exponentally dstrbuted. Replca A server whch has a copy of the replcated fle (or servce). Replca selecton polcy Polcy used to determne whch replca should serve a gven clent request. Seeder A BtTorrent peer whch has a complete copy of a fle (hence not requrng addtonal data to be downloaded), yet uploadng peces of the fle to other peers. Server bandwdth The rate at whch data s transferred by a server. Servce aggregaton technque Technque that allows multple clent requests to be served together n a manner that s more effcent than ndvdual servce. Streamng protocol Fle transfer protocol for contnuous meda fles that allows playback to begn before a fle s completely retreved. Tt-for-tat polcy Polcy used by BtTorrent, gvng upload preference to peers that provde the hghest download rates. x

15 Chapter Introducton Wth tremendous mprovements n network bandwdth and computer capabltes many new hgh-bandwdth applcatons have emerged n the entertanment, busness, and scentfc communtes. In contrast to tradtonal content dstrbuton systems, such as TV and rado channels, many of these new applcatons operate on an on-demand bass and only serve clents when explct requests for servce are made. As on-demand applcatons are becomng more popular, content provders are faced wth the problem of dstrbutng enormous amounts of data to a growng populaton of clent requests. For example, the sze of a full length move may be on the order of ggabytes. On-demand dssemnaton of such fles to many dfferent clents, potentally wdely dstrbuted across the Internet, requres sgnfcant server and network resources. Therefore, the rate at whch a system can serve clent requests s often lmted by the server (and/or network) bandwdth avalable for dssemnaton, where bandwdth refers to the amount of data that can be transferred per tme unt by the server (and/or across some network connecton). Two basc servce models commonly used for on-demand delvery of stored data are download and streamng. Wth download, clents download the entre fle before makng use of t. In ths context the man performance metrc s the tme untl the entre fle s downloaded. Streamng, on the other hand, utlzes the n-order playback characterstcs of meda fles, such as vdeo, to allow playback to begn well before all of the fle data s retreved. To ncrease the lkelhood that each part of the meda fle s retreved before ts playback tme, streamng technques generally requre that some ntal porton of the fle s retreved, and stored nto a buffer, before startng playback. Mantanng a buffer of fle data s especally mportant n envronments wth wdely varyng (playback and/or retreval) rates. Wth streamng, the prmary metrc of nterest s the startup delay untl playback can safely begn.

16 For content delvery systems to handle hgh request rates, t s mportant that protocols are desgned such that ether the resource requrements ncrease sub-lnearly wth ncreasng request rate, or the resources avalable for content delvery ncrease lnearly wth request rate. Usng scalable technques can allow a content dstrbutor wth lmted resources to provde ts customers wth better servce, handle a hgher request rate, and/or reduce ts resource requrements (and hence also ts delvery costs). Throughout ths thess the scalablty and resource requrements of dfferent delvery protocols and archtectures are consdered. Of partcular nterest s the best achevable delvery servce, gven some avalable resources. The remander of ths chapter s organzed as follows. Secton. provdes an overvew of exstng scalable content delvery approaches. Secton.2 defnes the objectves of the thess. The prmary contrbutons are outlned n Secton.3. Secton.4 gves the organzaton of the remander of the thess.. Scalable Content Delvery Before dscussng scalable delvery archtectures and protocols, consder the lmtatons of the basc clent-server model, n whch a content provder hosts all ts content at a sngle server, and clent requests are served ndvdually. Such systems, ndependently of the schedulng algorthm used, requre resource usage drectly proportonal to the number of requests. Wth lmted resources ths can easly result n unbounded clent delays or dropped requests. Further, both the server tself and the network connectvty to the server wll be potental bottlenecks and act as sngle ponts of falure. Scalablty can be acheved usng servce aggregaton (e.g., [2, 6, 0, 2, 26, 3, 57, 46, 50, 76, 80, 82]) or replcaton (e.g., [93, 94, 96, 33]) technques. Wth aggregaton, multple clent requests for the same fle are collectvely served. These technques often rely on one-to-many delvery multcast technques, whch buld effcent dssemnaton trees from a sngle sender to multple recevers. Wth replcaton, multple geographcally dstrbuted replcas of the content share the load of processng clent requests, offload the orgn content server, and enable delvery from nearby replca servers. For example, replcas may be proactvely pushed out to replca 2

17 servers across a Content Dstrbuton Network (CDN) or reactvely replcated at proxy caches n response to clent requests. Replcaton s also utlzed n peer-asssted systems, n whch other clents havng obtaned, or are currently obtanng, some content are wllng to serve as addtonal replca servers... Servce Aggregaton Rather than servng each request ndvdually, servce aggregaton technques attempt to serve multple requests smultaneously, n a manner that s more effcent than ndvdual servce. These technques often utlze multcast, n whch the server can use a sngle send operaton to delver content to all of the requestng clents. Multcast servce employs a multcast delvery tree to dssemnate the content. Ths tree can be constructed ether by network routers (e.g., [78, 23, 9, 8, 60, 6, 87, 38]) or by applcaton-level software (e.g., [45, 42, 95, 93, 53, 39, 39, 40]). When servng multple requests smultaneously, multcast can sgnfcantly decrease the bandwdth requrements at the server, as well as the total bandwdth requred throughout the network. Multcast-based servce aggregaton technques have been proposed both n the context of download and streamng. Prevous scalable protocols for downloadng large, popular fles from a sngle server nclude batchng [66, 82] and cyclc multcast [46, 0, 26, 76, 3, 50, 27]. Wth batchng, clents wat to begn recevng a requested fle untl the begnnng of ts next multcast transmsson, whch collectvely serves all of the watng clents that have accumulated up to that pont. Wth cyclc multcast, the fle data s contnually beng multcast. Clents can begn lstenng to the multcast at an arbtrary pont n tme, and contnue lstenng untl all of the fle data has been receved. In the context of streamng, scalable servce aggregaton protocols nclude perodc broadcast protocols [5, 77, 90, 9, 88] and mmedate servce protocols [79, 36, 89, 67, 68, 69, 76]. To allow playback to begn quckly, wth mmedate servce protocols, a new stream s started for each clent request, delverng the begnnng of the fle. To allow later clents to catch up wth earler clents, wth respect to the porton of the fle that has been receved, clents may also lsten to earler streams. At the pont a stream s no longer needed (snce the clents lstenng to t have already receved the data t s delverng, by lstenng to earler streams) t can be termnated. Wth perodc 3

18 broadcast protocols, segments of fles are perodcally multcast accordng to some schedule. Clents are provded wth a schedule for lstenng to the varous multcasts that ensures that all data s receved n tme. To accommodate packet losses many servce aggregaton technques, ncludng some cyclc multcast and perodc broadcast protocols, utlze erasure codes [48, 3, 64, 2]. Wth erasure codng, a fle of sze N blocks s encoded nto M blocks (M > N) such that recepton of only a subset of the M blocks (of total number N or slghtly larger) s suffcent to allow recreaton of the fle. For example, a clent lstenng to a cyclc multcast can recover from a packet loss by contnung to lsten untl a suffcent amount of erasure-coded data has been receved [46, 3]. Erasure codes also smplfy content delvery n systems utlzng replcaton [30]. A clent s able to download erasure coded blocks from multple servers wth mnmal duplcate block receptons, as long as M >> N. The rato M/N s called the stretch factor of the codng scheme...2 Replcaton The frst and smplest replcaton approach s for the content dstrbutor to nvest n a server farm consstng of a set of mrror servers among whch an ntellgent content swtch can drect requests. However, wthout servce aggregaton ths approach requres server and network resources to scale lnearly wth the number of requests and may result n a sngle pont of falure f all replcas are located n the same sub-network or behnd a common network bottleneck. Three alternatve replcaton strateges are () proxy cachng, () Content Dstrbuton Networks (CDNs), and () peer-to-peer networks...2. Proxy Cachng Internet Servce Provders (ISPs) or user communtes (such as busnesses or unverstes) often embed proxy servers or organzaton level caches at the boundary of ther networks. By redrectng clent requests through a proxy server, that caches prevously requested fles, these archtectures allow requests to be served by a nearby proxy server, rather than the orgn content source. Specfcally, f the proxy has a cached copy of the requested content, the proxy can serve the request tself, otherwse the proxy frst retreves a copy from the server. Proxy cachng can reduce network bandwdth usage as well as mprove the perceved clent performance. 4

19 To determne whch fles a proxy cache should retan copes of, varous replacement polces have been developed [47,, 34]. Such polces may explot () the hghly varable populartes of web objects, () short-term temporal localty n the object request stream, whereby an object may experence several closely spaced requests, and () the correlatons among requests for dfferent objects. However, wth relatvely cheap storage, dsk caches can be made large enough to make replacement polces less pertnent. The effectveness of proxy cachng s largely determned by the proporton of cacheable objects, and the rate these objects are updated, n comparson wth the request rate of each object [8]. To ncrease the probablty that a copy of the requested fle can be found close to the requestng clent, many systems have been proposed that use cooperatve cachng, whereby proxy caches close to each other cooperate n servng clent requests [7, 8]. Common for all cachng technques s that they work best f the content s statc; however, as data s pulled from the orgn server and stored at ndvdual proxy caches the content provder loses control over the content and can not provde servce guarantees. Therefore, the orgn server may nclude a drectve wth data that t sends to the proxy cache, requestng a short maxmum cache lfetme, forcng caches to refresh ther content relatvely frequently, and thus reducng ther effectveness Content Dstrbuton Networks Content Dstrbuton Networks (CDNs) [62, 66] are provsoned by content brokers. By dstrbutng servers across the network and nterconnectng them at the applcaton-level the content broker can create a dstrbuted overlay nfrastructure, whch t can use to provde content dstrbuton servces for content provders. Sellng ther servces to content provders, these networks are generally desgned to provde attractve servces such as relable and hgh qualty delvery to the content provder s customers. These systems releve content provders from nvestng n nfrastructure and offload the orgn content servers. Wth control over the entre delvery archtecture the content broker s able to allow the content provder to mantan full control of ts content. Ths added control also allows CDNs to be used to delver dynamc content and streamng meda [62] 5

20 In practce, CDNs use both reactve and proactve replcaton. In contrast to reactve approaches, used by proxy caches, proactve replcaton s extremely benefcal for networks that may suffer from substantal network delays, low bandwdth, or even undrectonal lnks. For example, n a network wth a undrectonal satellte lnk and no uplnk, data may be pushed to a local server, from whch local clents can retreve the data; or a move that s about to be released can be proactvely replcated to multple servers (avodng a sngle server to become overloaded at the release). Ths approach can be further mproved by dssemnatng content to servers at tmes when the network s less utlzed. Akama s the largest and best known CDN. In August 2006 t deploys 20,000 servers, spread over,000 networks, located n 7 dfferent countres. However, there are other commercal CDNs deployed that use many fewer servers. Also, some larger corporatons choose to set up prvate CDNs over whch they provde tranng, dstrbute tools, nformaton and software, as well as provde an nfrastructure for effcent wdearea meetngs, whle savng consderable network resources [66]. In an attempt to scale beyond the lmtatons set by ndvdual content brokers, wthout mpactng the prvacy of each CDN, some research efforts have nvestgated nterconnecton of ndependent CDNs [58, 80]. A common goal for all CDNs s to provde an archtecture that mprove the overall clent experence. When redrectng requests t s therefore mportant that content brokers provde an nfrastructure and mechansm that s transparent for the end user. In partcular, clent requests should be transparently redrected to an approprate server. Optmally, the clents should beneft from beng redrected whle nteractng wth the system n exactly the same way as f there were only a sngle server. Many redrecton technques have been proposed for CDNs [2]; however, most commercal systems use some form of Doman Name System (DNS) redrecton [03]. For example, Akama mplements ts own DNS servce usng a two level server herarchy [7]. Akama, August

21 ..2.3 Peer-to-Peer and Peer-asssted Systems In peer-to-peer and peer-asssted systems peers dstrbuted across the network contrbute to the collectve resources of the system. Even though the orgnal Internet was desgned on peer-to-peer prncples, t s not untl the last few years that peerasssted systems have been consdered for content dstrbuton. As more peers choose to share ther content and resources, the capacty of these archtectures grows. Wth approprate technques to dscover nearby replcas, these systems also have the potental to reduce network bandwdth usage. The man applcaton of current peer-to-peer systems s fle sharng among peers. Ths applcaton s not only the most wdespread applcaton, but also the most controversal applcaton. Systems such as Napster [24], Gnutella 2, Freenet [47, 88], Kazaa 3, and many of the stes provdng support for the BtTorrent [48] download protocol have ganed a multtude of attenton from authortes, copyrght protectors, and meda due to the enormous amount of copyrghted musc and moves that are shared among users across these systems. Other applcatons nclude dstrbuted computaton, computer gamng, and other collaboraton applcatons. Measurement studes have observed that peer-to-peer traffc s responsble for a large porton of the bytes transferred across the Internet (e.g., [57]). Wth ncreasng peer-to-peer traffc localty aware mechansms, whch allow content to be retreved from nearby rather than far-away peers becomes more mportant [45, 98]. Peer-asssted content dstrbuton systems and algorthms have been proposed for both lve streamng [39, 45, 95, 02] and for on-demand streamng of stored meda fles [24, 53, 6]. To acheve streamng, these protocols typcally establsh relatvely long-duraton streams from the content source and between peers, as organzed nto some form of overlay topology. In contrast, wth BtTorrent [48] and smlar download protocols (e.g., [78, 62]) a clent may download a fle from a large and changng set of peers, usng connectons of heterogeneous and tme-varyng bandwdths. Ths flexblty s acheved by breakng the fle nto many small peces, each of whch may be 2 Gnutella, August Kazaa, August

22 downloaded from dfferent peers. Ths approach has also been found benefcal n the context of lve streamng [86, 9, 90, 2]. Other work has proposed mechansms to replcate content [6, 5, 55, 52], search for content (or nformaton) [6, 84, 49, 50, 7, 05, 7, 72], as well as route data (or queres) [38, 39, 93, 68] n varous types of overlay peer-to-peer structures..2 Problem Descrpton Ths thess consders the scalablty and performance of download protocols, used to effectvely dssemnate data to a large number of requestng clents. In partcular, new protocols and polces are desgned and evaluated for three dfferent contexts, each achevng scalablty through servce aggregaton and/or replcaton. Frst, ths thess consders the problem of devsng sngle server protocols that mnmze the average or maxmum clent delay for downloadng a sngle fle, as a functon of the average server bandwdth used for delvery of that fle. An equvalent problem s to mnmze the average server bandwdth requred to acheve a gven average or maxmum clent delay. Ths equvalent perspectve s sometmes adopted. Although delvery of multple fles s not explctly consdered, note that use of a download protocol that mnmzes the average server bandwdth for delvery of each fle wll mnmze the average total requred server bandwdth for delverng all fles as well. Secondly, ths thess consders the problem of devsng polces to select whch replca should serve each request, n systems explotng both servce aggregaton and replcaton. Such polces must take nto consderaton the basc tradeoff between localty of servce (maxmzed by selectng the nearest replca), and effcency of use of server resources (maxmzed by selectng the replca at whch servce can be shared among the largest number of clents). Fnally, a peer-asssted envronment s consdered n whch the content s replcated but peers do not utlze servce aggregaton technques. For ths context, scalable download protocols, such as BtTorrent [48] have already been proposed, successfully deployed, and have shown to provde good performance [, 34]. Ths thess consders the problem of usng adaptatons of these download protocols to provde on-demand streamng of stored meda. 8

23 .3 Contrbutons The man contrbutons of ths thess are as follows. New scalable download protocols are desgned for download of a large fle from a sngle server, usng a multcast based approach, and ther performance evaluated aganst new analytc bounds on the best achevable performance. The relatve performance of classes of replca selecton polces, of varyng complextes, are compared n a context where a large fle may be downloaded from multple replca stes, each usng multcast. A peer-asssted protocol s desgned that splts a large meda fle nto small peces, uses a pece selecton polcy to determne whch pece to be downloaded next from multple content server(s) and/or other clents havng retreved part of the fle, n an order that allows streamng, as well as a rule to determne when playback can safely begn. For each of the above contexts a number of abstractons are developed, wthn whch protocols and polces are evaluated. The followng sectons elaborate on the contrbutons made n each context..3. Scalable Download from a Sngle Server To evaluate the performance of exstng and new protocols lower bounds on the average and maxmum clent delay for completely downloadng a fle, as functons of the average server bandwdth used to serve requests for that fle, are developed for systems wth homogeneous clents. The results show that nether optmzed versons of cyclc multcast nor batchng consstently yeld performance close to optmal. New, relatvely smple, scalable download protocols are proposed that acheve wthn 5% of the optmal maxmum delay and 20% of the optmal average delay n homogeneous systems. Smlar to cyclc multcast, these protocols allow clents to start lstenng to on-gong multcasts at the tme of ther arrval, but lmt server transmssons to tme perods n whch (probablstcally) there are more clents lstenng. For heterogeneous systems n whch clents have wdely-varyng achevable recepton rates, an addtonal desgn queston concerns the use of hgh-rate 9

24 transmssons, whch can decrease delay for clents that can receve at such rates, n addton to use of low-rate transmssons that can be receved by all clents. A new scalable download protocol for such systems s proposed, and ts performance s compared to that of alternatve protocols as well as to new lower bounds on maxmum clent delay. The new protocol acheves wthn 25% of the optmal maxmum clent delay n all scenaros consdered. Throughout ths analyss t s assumed that each requestng clent receves the entre fle (.e., clents never abort ther request whle watng for servce to begn or after havng receved only a porton of the fle). The analyss and protocols presented are compatble wth erasure-coded data. Each clent s assumed to have successfully receved the fle once t has lstened to multcasts of an amount of data L (termed the fle sze, although wth packet loss and erasure codng, L may exceed the true fle sze). Posson request arrvals are typcally assumed, although generalzatons are dscussed n some cases. Note that Posson arrvals can be expected for ndependent requests from large numbers of clents (durng tme perods wth constant arrval rates). Furthermore, multcast delvery protocols that have hgh performance for Posson arrvals, have even better performance under the more bursty arrval processes that are typcally found n contexts where clent requests are not ndependent [68]..3.2 Scalable Download from Multple Servers In large dstrbuted systems mplementng both replcaton and servce aggregaton, a basc tradeoff s between localty of servce (maxmzed by selectng the nearest replca), and effcency of use of server resources (maxmzed by selectng the replca at whch servce can be shared among the largest number of clents). Rather than propose a specfc polcy to medate ths tradeoff, classes of polces of dfferng complextes are compared wthn the context of a smple cost model, capturng both the servce requrements of the ndvdual replca servers, and the addtonal cost assocated wth retrevng servce at remote replcas. A large popular fle s assumed to be replcated at multple servers across the network, from whch the fle can be downloaded. The set of servers wth a replca may be determned based on expectatons of future demands, avalablty, or some other system requrements. Here, the set of servers wth a replca of the fle s assumed to be 0

25 predetermned. It s further assumed that each server mplements some form of servce aggregaton technque allowng multple clent requests to be served together, rather than ndvdually. Wthn each class of polces, lmts on the best achevable performance are determned (or representatves defned) for both batchng and cyclc multcast aggregaton approaches. When usng cyclc multcast the fle s assumed to be erasure encoded. Smlar to the analyss used for the sngle server case, ths analyss assumes that requests arrve accordng to a Posson process, and no clent aborts ther request whle watng for servce to begn or after havng receved only a porton of the fle. It s concluded that () selecton usng current system state nformaton (rather than only proxmtes and average loads) can yeld large mprovements n performance, () when t s possble to defer selecton decsons (e.g., when requests are delayed and served n batches), deferrng decsons as late as possble can yeld addtonal large mprovements, and () relatvely smple polces usng only local (rather than global) request nformaton are able to acheve most of the potental performance gans..3.3 On-demand Streamng usng Scalable Download Based on the desgn of the relatvely smple and flexble BtTorrent download protocol, ths thess proposes a peer-asssted BtTorrent-lke approach to meda fle delvery whch s able to acheve a form of streamng delvery, n the sense that playback can begn well before the entre meda fle s receved. Achevng ths goal requres: () a pece selecton strategy that effectvely medates the conflct between the goals of hgh pece dversty (acheved n BtTorrent usng a rarest-frst polcy), and the n-order requrements of meda fle playback, and () an on-lne rule for decdng when playback can safely commence. Canddate protocols ncludng both of these components are presented and evaluated usng event-based smulatons, n whch each peer s assumed to be bottlenecked by ether ts upload or download rate. Localty s not consdered n ths part of the thess. It s further (very conservatvely) assumed that no peer, except the orgn content source, shares peces once t has receved the whole fle. In a real system, peers are lkely to contnue servng other peers as long as they are stll playng out the meda fle, whle other peers may (gracously) choose to upload to other peers beyond

26 that tme. Wth the hgher avalablty of rare peces and download bandwdth n such systems, the benefts of more aggressve pece selecton technques (gvng prorty to earler peces rather than rare peces) are lkely to be even greater than presented here. It s found that smple probablstc pece selecton polces, gvng preference to earler peces, allow peers to begn playback well before the entre fle s downloaded. Further, whereas no on-lne strategy for selectng startup delays s expected to gve close to optmal startup delays (wthout sgnfcant chance of playback nterrupton), promsng results are obtaned usng a smple startup rule. Before startng playback, the rule requres the retreved number of peces to exceed some (small) threshold, and the rate at whch n-order peces are beng accumulated to exceed a value suffcent to allow contnuous playback wthout nterrupton, f that rate was to be mantaned..4 Thess Organzaton The remander of ths thess s organzed as follows. Chapter 2 revews related work, outlnng the current state of scalable download protocols and settng exstng solutons nto context. Chapter 3 develops lower bounds and new scalable download protocols that acheve close to optmal performance when downloadng large fles from a sngle server. Chapter 4 consders the problem of replca selecton n systems explotng both replcaton and servce aggregaton. Chapter 5 proposes adaptatons of exstng scalable peer-asssted download protocols, n a way that allows on-demand streamng. Conclusons and drectons for future work are presented n Chapter 6. 2

27 Chapter 2 2 Background Rather than attemptng to provde a complete survey of all exstng scalable content delvery protocols and archtectures, ths chapter focuses on the technques most relevant for the three contexts consdered n ths thess. Secton 2. presents an overvew of varous approaches to mplement multcast. Secton 2.2 surveys prevous work on scalable sngle server download protocols that use multcast-based servce aggregaton. Secton 2.3 dscusses replca placement and selecton technques applcable for the context of scalable download from multple servers. Fnally, related work on peer-asssted content delvery protocols s surveyed n Secton Multcast When dstrbutng content to multple clents across the Internet, content servers have tradtonally used multple concurrent uncast connectons. Ths approach suffers from hghly redundant usage of network resources and hgh server overhead. Wth multcast, n contrast, a sngle transmsson of the content can be receved by multple recevers. A collecton of nodes nterested n recevng the same multcast transmsson s called a multcast group, and the server and network resources used to transmt each multcast to each member of the group s called a multcast channel. Throughout ths thess, lstenng to a channel refers to lstenng to a partcular on-gong or ntermttent multcast transmsson. Content s dssemnated usng a multcast delvery tree, and n contrast to replcaton strateges, multcast does not requre any persstent storage capacty n the network. Multcast sgnfcantly decreases the bandwdth requrements at the server, and decreases the total bandwdth requred throughout the network. Despte frst beng mplemented as an overlay system mplemented at the applcaton-level [70], multcast was orgnally envsoned as a network-level functonalty ( IP multcast ) supported by the network routers [59]. In theory ths 3

28 would gve shorter paths and use less total network bandwdth; however, for numerous reasons IP multcast has seen slow commercal deployment [64, 87]. Ths has prompted much research on mplementng multcast at the applcaton level. Ths secton wll dscuss both IP multcast and applcaton-level multcast. 2.. IP Multcast Ths secton dscusses how the current multcast soluton has evolved and concludes wth a dscusson of current deployment ssues and alternatve network-level solutons that have been proposed. In the tradtonal IP multcast servce model, a multcast group s formed by a set of clents that have all expressed nterest n recevng transmssons sent to some partcular multcast address (used as the group dentfer). Whle only the nodes currently n the group receve data sent to the multcast address, any node, ncludng nodes that are not members of the group, can send data to the group by addressng transmssons to the multcast address. Implemented at the network-level, the content s delvered wthout guarantees of n-order or loss-free delvery. The Internet Group Membershp Protocol (IGMP) [32] provdes the functonalty to handle group membershp. It operates between clents and ther drectly attached routers. Group members use ths protocol to nform ther nearest router about multcast groups whch they wsh to jon or leave. Note that IGMP s only used for group membershp, and other protocols called multcast routng protocols are needed to buld and mantan delvery trees for each group. To acheve scale and admnstratve autonomy the Internet s organzed nto domans or regons, each called an Autonomous System (AS). Routng protocols are generally categorzed as ether ntradoman routng protocols, responsble for routng wthn a doman, or nterdoman routng protocols, responsble for routng between dfferent domans. Many dfferent ntradoman protocols have been proposed for multcast routng wthn an AS. The man dfferences among these protocols concern how they buld and mantan the multcast tree structure. These routng protocols are normally categorzed as ether dense-mode protocols or sparse-mode protocols. Dense-mode protocols are desgned to perform best when multcast transmssons must pass through most of the 4

29 network routers and normally use some form of broadcast and prune mechansm. Sparse-mode protocols are desgned to perform best when multcast transmssons need to pass through only a small fracton of the network routers and rely on recevers explctly sendng requests to jon the multcast group. Dense-mode ntradoman multcast protocols nclude Dstance Vector Multcast Routng Protocol (DVMRP) [78] and Multcast Open Shortest Path Frst (MOSPF) [23], whle sparse-mode protocols nclude Core-Based Trees (CBT) [9, 8] and Protocol Independent Multcast Sparse Mode (PIM-SM) [60, 6]. PIM-SM s also an ntegral component of the current nterdoman multcast archtecture. PIM-SM forms a reverse shortest path tree, rooted at a rendezvous pont (RP) assocated wth a multcast group, by settng up routng states at routers when propagatng the explct jon messages towards the RP. The tree s a reverse shortest path tree n the sense that the path from each recever to the RP uses the shortest IP path; however, wth the asymmetry of path lengths, these paths are not necessarly the shortest paths from the RP to each recever. A novel feature of PIM-SM s ts ablty to let recevers swtch from group-shared trees (n whch all content s forwarded through the RP) to source-specfc trees (n whch the multcast tree s rooted at the content source). Ths ablty can mprove performance for clents and offload the RP. Whle all routers n a specfc AS generally deploy the same multcast routng protocol, routers n dfferent domans may use dfferent protocols. Therefore, nterdoman routng protocols are generally requred to acheve nteroperablty among domans usng dfferent routng protocols. Up untl the begnnng of 999, DVMRP was almost exclusvely the only protocol deployed for nterdoman routng. However, as a dense-mode protocol (usng a broadcast and prune approach) t s not suted for sparse sets of partcpatng routers, and as observed by Rajvadya and Almeroth [37], DVMRP was almost entrely replaced n March In the replacement multcast archtecture, PIM-SM s used for routng and the Multprotocol Boarder Gateway Protocol (MBGP) [22], whch extends the Boarder Gateway Protocol (BGP) [44], s PIM-SM s currently the only multcast routng protocol used for nterdoman routng and t (or other PIM versons) s also typcally used for ntradoman routng [60]. 5