Swarming on optimized graphs for n-way broadcast

Transcription

1 Swarming on optimized graphs for n-way broadcast GEORGIOS SMARAGDAKIS NIKOLAOS LAOUTARIS PIETRO MICHIARDI AZER BESTAVROS JOHN W. BYERS MEMA ROUSSOPOULOS Abstract In an n-way broadcast application each one of n overlay nodes wants to psh its own distinct large data file to all other n- destinations as well as download their respective data files. BitTorrent-like swarming protocols are ideal choices for handling sch massive data volme transfers. The original BitTorrent targets one-to-many broadcasts of a single file to a very large nmber of receivers and ths, by necessity, employs an almost random overlay topology. n-way broadcast applications on the other hand, owing to their inherent n-sqared natre, are realizable only in small to medim scale networks. In this paper, we show that we can leverage this scale constraint to constrct optimized overlay topologies that take into consideration the end-to-end characteristics of the network and as a conseqence deliver far sperior performance compared to random and myopic (local) approaches. We present the Max-Min and Max- Sm peer-selection policies sed by individal nodes to select their neighbors. The first one strives to maximize the available bandwidth to the slowest destination, while the second maximizes the aggregate otpt rate. We design a swarming protocol sitable for n-way broadcast and operate it on top of overlay graphs formed by nodes that employ Max-Min or Max-Sm policies. Using trace-driven simlation and measrements from a PlanetLab prototype implementation, we demonstrate that the performance of swarming on top of or constrcted topologies is far sperior to the performance of random and myopic overlays. Moreover, we show how to modify or swarming protocol to allow it to accommodate selfish nodes. I. INTRODUCTION -OUTLINE Motivation: The BitTorrent protocol [] has established swarming, i.e., parallel download of a file from mltiple peers with concrrent pload to other reqesting peers, as one of the most efficient methods for mlticasting blk data. A fndamental characteristic of the existing BitTorrent is that the overlay graph reslting from its bootstrap and choke/nchoke algorithms is mostly ad-hoc, in the sense that it is the otgrowth of random choices of neighboring peers. This is jstified given the scale of PP file swapping networks. PP file swapping, is not the be all and end all for swarming. In this work we consider n-way broadcasting another class of applications, in which each one of n overlay nodes mst psh a very large chnk of data (a distinct file) to all other n peers, as well as pll the n files pshed by these other peers. Once completed, this psh-pll cycle may be repeated with new sets of files. Compter Science Dept, Boston University, Boston, MA. School of Engineering and Applied Sciences, Harvard University, Cambridge, MA. Networking and Secrity Dept, Institt Erecom, Sophia-Antipolis, France. A. Bestavros and J. Byers are spported in part by a nmber of NSF awards, inclding CNS CSR #0700, CNS Cybertrst #077, CNS NeTS #00, CNS ITR #009, and EIA RI #0007. N. Laotaris and M. Rossopolos are spported in part by NSF CAREER Grant #0. P. Michiardi is spported in part by the Integrated Project CASCADAS (FET Proactive Initiative, IST Sitated and Atonomic Commnications) within the th IST Framework Program. Applications sing n-way broadcasting wold involve small/medim-sized networks, as they are inherently of n natre. Examples inclde: distribtion of large scientific data-sets in grid compting, distribtion of large traffic log files for network-wide distribted intrsion/anomaly detection schemes [], synchronization of distribted databases [], and several other enterprise applications. Contrary to the prevailing assmption nderlying the design of BitTorrent, the nodes that make p sch networks are basically cooperative (at an extreme case they belong to the same administrative athority). Even for relatively small networks, n parallel broadcasts of distinct large files can create data volmes that are impossible to handle via centralized soltions: ploading each file to a centralized server and then copying it back to all destinations in a point-to-point manner means that the same file is transmitted O(n) times over the same link, i.e., imposing an O(n) stress on the physical links. n-way broadcast via swarming: Swarming is clearly an attractive approach to spporting n-way broadcast applications. The obvios soltion is to otsorce the psh-pll fnctionality to BitTorrent: set-p n different torrents, each one seeded by a different node. In this paper, we qestion the effectiveness of BitTorrent for n-way broadcasting (which is not what it is primarily designed to spport). In particlar, we note that BitTorrent rns on the topologies that reslt from the composition of its bootstrapping and choke/nchoke algorithms. These topologies are mostly noptimized. Indeed, the only topological optimization in BitTorrent is a local one: nder the choke/nchoke algorithm, fast peers are matched p with other fast peers from within the same randomly bootstrapped neighborhood. By virte of the relatively small size of neighborhoods compared to the entire network, the reslting topology is close to being random. While randomly-bootstrapped graphs may possess desirable theoretical properties (sch as small diameters), they are likely to be inefficient when compared to graphs that are systematically constrcted to optimize a specific application. Notice that BitTorrent s matching of fast nodes is mostly in the protocol as an efficient tool against free-riding, rather than as a conscios attempt to optimize the overall overlay topology for applications sch as n-way broadcast. Or work Swarming over optimized overlays: For n-way broadcast applications (as well as for other potential classes of applications), the overriding goal is to optimize the efficiency of the entire overlay as opposed to creating a tit-for-tat environment to reign in selfish, free-riding behavior of individal nodes. Also, the scale of the applications we envision makes it /08/$ IEEE 8

2 possible/practical to optimize the constrction of the overlay, especially if distribted optimization is sed. Armed with this realization, or goal will be to constrct highly efficient topologies to be sed by swarming protocols for n-way broadcast. Specifically, we constrct an optimized, common overlay network, pon which swarming is sed. In order to control the stress of the physical links spporting the overlay, we impose an pper bond on the degree of the nodes in the constrcted overlay network. Next we present jstification for several of the salient featres of or soltion featres that will be developed and presented flly later in the paper. Why swarming on top of an overlay? Becase hop-by-hop relay of the entire file over a shortest-path tree embedded on the overlay topology and rooted at the seed node wold take too long. We want to harness the power of parallel downloads as exemplified in BitTorrent. Why se a common overlay? Becase a topological optimization reqires monitoring the performance of overlay links, and we want to amortize the cost of sch monitoring pay it only once per link and rese the reslt for the benefit of all n transmissions (and avoid monitoring the same link p to n times as can happen if one bilds n independent overlays). How cold swarming benefit from an end-to-end optimized overlay? Or overlays are optimized for end-to-end performance over mlti-hop paths, e.g., by maximizing the minimm available bandwidth to any destination over mltiple paths, or by maximizing the total available bandwidth to all destinations over all available paths. From a single node s perspective, swarming involves point-to-point transfers within the neighborhood of that node. Each node, however, has in its neighborhood nodes that also belong to other adjacent neighborhoods. Noting this, one can see that, throgh swarming, data chnks eventally reach their destinations throgh mlti-hop paths formed throgh single hop transfers between neighborhoods. If these mlti-hop paths are end-to-end optimized, then swarming will be more effective in operating pon them as compared to pon noptimized paths. Why optimize the overlay based solely on network characteristics, withot consideration of data availability? Argably, one cold conceive of more general overlay constrctions in which neighbors are selected based on criteria involving both the network characteristics and the availability of chnks at each candidate connection point. In or work, we adopt a bandwidth-centric/data-agnostic approach to the constrction of the overlay for two main reasons: () for large objects it is high bandwidth that leads to small delivery completion times and high object throghpt; () the global state in terms of available chnks per node changes too freqently (with each sccessfl chnk exchange between two nodes), reslting in an optimized topology that changes too freqently to be of practical se. The fact that we do not consider data availability in the constrction of the overlay does not mean that data availability does not play a role in or approach: it does, bt not at the overlay constrction time-scale. Specifically, we advocate a two-pronged approach operating at two distinct time scales: at a coarse time scale, we address isses related to network characteristics throgh the constrction of a dynamic, distribtively optimized overlay, and at a finer time scale, we address isses related to data availability throgh the pload/download schedling algorithms employed in the swarming protocol that rns on top of the overlay. II. RELATED WORK This work is the fsion of two very recent thrsts in networking research: network creation games and swarming protocols. Network creation games appeared in compter science with the work of Fabrikant et al. [] in which a set of nodes forms a network in a distribted manner driven by self-interest each node pays for the creation of a nmber of links to other neighbors so as to minimize a hybrid cost that captres the prchase cost of these links and the delay for roting packets to all other destinations sing own and remote links. The model targeted the creation of physical telecommnication networks throgh peering agreements between ISPs (hence the explicit modeling of the cost of bying a link). Laotaris et al. [], stdied the capacitated version of the above problem, targeting the constrction of overlay roting networks each node is given a bond on the nmber of immediate peering relationships that it can establish (defined by the protocol that implements sch an overlay network) and selects the best neighbors so as to minimize its sm of distances to all destinations throgh shortest-path rotes over the reslting overlay topology. These works differ fndamentally from ors in that they target roting, i.e., they assme that a packet from v to is of interest only to. Intermediate nodes w that lay on the overlay path from v to are there jst to assist in the roting of the packet. In the crrent work, each node is broadcasting a file to all destinations and ths intermediate nodes are also receivers in addition to being relay points. More fndamentally, in or case the delivery of information from v to occrs not throgh a single path bt (potentially) throgh all the available connected paths between the two end-points (becase the file is ct into chnks which travel in parallel along different paths on the overlay). For this reason we employ max-flows as bilding blocks for designing the overlay (as opposed to shortest-paths which are sed in pointto-point roting [], []). Max-flows reflect better the natre of or application (broadcasting) as well as the natre of the employed techniqe for implementing it (swarming). The BitTorrent protocol [] has established swarming as one of the most fresh and promising ideas in contemporary networking research and ths has kicked-started a (tidal) wave of research articles in the area. Or fndamental difference from this body of work, whether analytic, e.g., Qi and Srikant [], Massolie and Vojnovic [7], Kmar and Ross [8], experimental, e.g., Bharambe [9], or measrement based, e.g., Izal et al. [0], Legot et al. [], is that we have sbstitted the (close to) random graph reslting from BitTorrent s bootstrap and choke/nchoke algorithms with a highly efficient distribtively optimized graph. As we show later on, sch a switch boosts the performance of a swarming protocol rnning on top of it. We are able to obtain sch highly efficient graphs becase or interest is on smaller networks. We show that at sch scales one can do mch better than close to random. Some other relevant works are the following ones. Massolie et al. [] recently showed that a simple distribted 9

3 randomized algorithm can achieve the theoretical optimal broadcast rate given by Edmond s theorem [] for a sorce node in a flow network. Compared to this work, we let each node select its neighbors and ths participate in the constrction of the flow network, as opposed to taking it for granted. Gkantsidis and Rodrigez [] have proposed the se of network coding as an alternative to BitTorrent s chnk schedling algorithm. The performance benefit/added complexity ratio of employing network coding is not yet generally agreed pon []. Althogh we focs on BitTorrent-like swarming here, or optimized topologies shold also benefit network-coding based swarming becase they are oblivios to whether network coding is sed or not. Go et al. [] and Tien et al. [] look at the design of mlti-torrent systems. Their contribtion is mostly on the measrement and the design of inter-peer incentive mechanisms for peers that participate in mltiple torrents concrrently. They do not look at overlay constrction isses. Interestingly, Tien et al. [] provide jstification for one of or design choices, which is to enforce that at any time there shold be only one active torrent between any two nodes (more in Sect. IV). They show that deviating from this choice and allowing transferring between two nodes mltiple chnks in parallel (one for each torrent), slows down the system by overpartitioning the pload bandwidth of nodes. Other end-system mlticast systems sch as SplitStream from Castro et al. [7] and Bllet from Costic et al. [8] cold be sed to spport n-way broadcasting by creating a separate overlay for each sorce. The problem with this approach is that there is no coordination across different overlays and ths there can be performance inefficiencies as well as significant overheads de to the redndant monitoring of the same physical paths mltiple times from different overlays. Or approach is to constrct one overlay for all sorces and ths jointly optimize as well as share the monitoring cost. The only work we are aware of on the intersection of overlay creation and BitTorrent is a very recent one from Zhang et al. [9]. It looks at the formation of Nash eqilibria topologies in view of download-selfish peers that participate in a single torrent. Or overlay formation, althogh distribted and based on local tility fnctions is: () primarily targeting the optimization of the social tility of the network, meaning that all nodes are assmed to be nder common control, and () considering both pload and download performance for mltiple torrents, one at each node. We examine selfishness isses and how these cold be addressed towards the end of or article, bt this is jst a spplement of or main contribtion. III. PEER-SET SELECTION Let V = {v,v,...,v n } denote a set of nodes. Node v i selects k other nodes to be in its peer-set s i = {v i,v i,...,v ik } and establishes bidirectional links to them. Let S = {s,s,...,s n } denote the edge set of the overlay graph G = (V,S) reslting from the sperposition of the individal peer-sets. Each link of G is annotated with a capacity c ij which captres the available bandwidth [0] (availbw) on the the nderlying IP layer path that goes from v i to v j. Capacities can be asymmetric, meaning that c ij c ji in the general case. Let MF(v i,v j,s) denote the reslting maxflow from v i to v j nder S. Let also Φ(v i,s) and Ψ(v i,s) denote the minimm max-flow from v i to any other node nder S, and the sm of max-flows from v i to all other nodes nder S, respectively, i.e.: Φ(v i,s)= min MF(v i,v j,s), Ψ(v i,s)= MF(v i,v j,s) v j V i v j V i In the above definitions, each max-flow from v i to an individal destination is compted independently of other max-flows from the same node to different destinations (i.e., each one is compted on an empty flow network G). These definitions shold not be confsed with mlti-commodity flow problems in which mltiple distinct flows co-exist. Definition : (Max-Min and Max-Sm peer-sets) A peerset s i is called Max-Min if it maximizes the minimm maxflow of node v i, i.e., Φ(v i, {s i } + S i ) Φ(v i, {s i } + S i ), s i s i, where S i denotes the sperposition of the peer-sets of all nodes bt v i. Similarly, a peer-set is called Max-Sm if Ψ(v i, {s i } + S i ) Ψ(v i, {s i } + S i ), s i s i. Lemma : Finding a Max-Min or Max-Sm peer-set for v i given S i is an NP-hard problem. Proof: See Appendices A and B. These peer-set selection policies optimize the connectivity of a given node to the remaining network. One cold say that this constittes selfish behavior. This is indeed the case if the nodes se this connectivity to only disseminate their own file. However, when they also indiscriminately relay the files of others, which is the assmption for the applications we consider, then optimizing one s connectivity boosts the aggregate social performance of the network. Later on, in Sect. VI we discss what happens when the swarming protocol (rnning above the overlay) ceases to be indiscriminate with respect to the pload qality it gives to local and remote files. Why Max-Min and Max-Sm? Given a flow network G, the broadcast problem asks what is the maximm (broadcast) rate at which a sorce v i can deliver its stream concrrently to all other nodes. Edmonds showed in [] that the broadcast rate is eqal to min vj V i minct(v i,v j ), which in view of the max-flow/min-ct theorem is eqal to min vj V i MF(v i,v j ). Therefore, the Max-Min peer-set is the peer-set that maximizes the broadcast rate of a node, or conversely the delivery rate to the slowest receiving peers. It does so by placing the links so as to boost the max-flow to these slowest peers. Of corse for this to be possible there mst be available bandwidth to be tilized at the IP level (this is reflected on the c ij s which steers the peer-set selection, and which are obtained throgh measrements as explained in Sect IV). Edmonds gave an exponential time centralized algorithm for achieving the broadcast rate, which was later improved to a small polynomial time by Lovasz, Gabow and others []. Recently, Massolie et al. [] showed that a simple randomized decentralized algorithm can achieve a delivery rate that is arbitrarily close to the broadcast rate. A Max-Sm peer-set on the other hand is a peer-set that maximizes the theoretical maximm aggregate transmission rate from a node. Contrary to the Max-Min peer-set that maximizes a provably attainable broadcasting rate, the Max- Sm maximizes only an pper bond on the aggregate rate 0

4 which, in the general case, is not attainable de to contention for link bandwidth when max-flows from the same sorce to different destinations share common overlay links. We elaborate with an example. Consider the flow network of Fig. (top-left) in which all links have nit capacity and node is the sorce. Compting each max-flow on an empty network we get that the max-flow from the sorce to nodes,, and is eqal to whereas that to nodes and is eqal to, thereby Ψ() = 7. Consider now the maximm real flows that can exist concrrently from the sorce to nodes and (top-center). Breaking the file into two eqal parts A and B the sorce can transmit A at fll rate over the dotted paths ( and ) and B at fll rate over the dashed path (only once over link (,)) and achieve concrrent real flows that match the capacity of corresponding max-flows on an empty graph, i.e., RF(,)=MF(,)= and RF(,)=MF(,)=. This is possible becase a single transmission of B on the edge (,) sffices for contribting to both RF(,) and RF(,). Ths the two flows don t compete for bandwidth on the shared link and can achieve the same capacity as the corresponding maxflows on empty networks. This is not, however, generally possible. On the top-right part of the figre we depict the sitation when sending from the sorce to all destinations (nodes -) concrrently. In this case the entire file (both A and B) has to go over links (,), (,), and (,) and ths RF(,)=RF(,)=<MF(,)=MF(,)= leading to a real aggregate rate Ψ() = smaller than the bond Ψ() = 7. Generally, the bond becomes less tight with increasing link density k/n. On the bottom-left part of Fig. we add to the previos network two new links: (,) and (,). It is easy to verify that the max-flow from the sorce to nodes,,, and is now and to node is, leading to Φ() = 9. As before, if we consider only the flows to and, it is easy to see that their max-flow vales can co-exist. Considering, however, the flows to all destinations, we see that any partition of the file into parts will inevitably lead again to all real flows being, whereas the corresponding max-flows with the exception of MF(,) are now. In other words, althogh the new links increased both MF(,) and MF(,) by compared to the previos network, they cannot increase any of the real flows and ths widen the gap between the bond (Φ() = 9) and the maximm attainable aggregate rate ( Φ() = ). To sm p, we propose and stdy these peer selection policies for the following reasons: () Max-flows are sed to captre the fact that in a swarming protocol the chnks of a sorce node v i travel towards a sink node v j over (potentially) all the available paths of the overlay graph of point-to-point peer relationships. () The gap between the bond on the aggregate rate Ψ(v i,s) given by a Max-Sm peer-set and the actal maximm attainable aggregate rate Ψ(v i,s) which factors in the sharing of overlay links from mltiple max-flows The contention between max-flows from different sorces does not come explicitly in these objective fnctions. It is captred in or framework throgh the measred availbw c ij: the availbw on a direct overlay link from v i to v j depends on the capacity of the nderlying physical path and the amont of this capacity already captred by the competing max-flows from other sorces. At this level the problem is indeed a mlticommodity flow. The fact that the entire file has to go over the edge (,), eliminates any chance for increasing the real flows to nodes,,, and beyond. A B A,B A,B A A,B A A B Fig.. Mixing max-flows. Left: empty network. Middle: RF(,) and RF(,) co-existing. Right: RF(,), RF(,), RF(,), RF(,), RF(,) co-existing. Top: initial network. Bottom: Initial agmented with edges, (,) and (,). to different destinations, is redced by the fact that swarming protocols garantee that any chnk is transmitted at most once between any two peers; therefore, Ψ(v i,s) can se an overlay link mltiple times (for different max-flows) bt wold seize bandwidth only once, thereby redcing its gap from the bond Ψ(v i,s) that assmes that the entire flow network is available to each individal max-flow from v i. () The overlay network has to be rather sparse (small k) so as to limit the stress on the physical links. Ths the bond Max-Sm won t be very mch off from the actal achievable aggregate rate and it makes sense optimizing the peer-set based on it. Regarding Max-Min, this is provably attainable, and optimal for broadcast rate as discssed earlier. Since a node cares to both pload its local file to all other nodes as well as download from them all remote files, we combine the previos definitions in the following objective fnctions: Φ(v i,s i)=αφ(v i, {s i} + S i)+( α)minmf(v j,v i, {s i} + S i), v j V i Ψ(v i,s i)=αψ(v i, {s i} + S i)+( α) MF(v j,v i, {s i} + S i) v j V i In the above fnctions, the parameter α reglates the relative importance between pload and download qality in selecting a peer-set. If the link capacities are symmetric, then optimizing Φ or Ψ redces to optimizing Φ or Ψ, independently of α. IV. NODE ARCHITECTURE Nodes consist of the following components: a peer selection modle implementing the peer-set selection algorithms described in Sect. III; a downloader modle, responsible for issing reqests to neighboring nodes and downloading missing chnks; and an ploader modle, responsible for sending back local and in-transit chnks (an in-transit chnk is a chnk that does not belong to the local sorce file). In this section we describe these three modles nder the assmption that nodes are cooperative (therefore we don t need mechanisms like choke/nchoke). Later on, in Sect. VI we discss the necessary changes for dealing with selfishly behaving nodes. A. Peer Selection Modle Every time period T, a node: () measres its available bandwidth to all other nodes sing pathchirp [], () exectes a peer-set selection algorithm from Sect. III and connects to the corresponding nodes (incoming links are left ntoched). Since both Max-Min and Max-Sm are NP-hard, we se fast local-search heristics to compte approximately A,B A,B A,B

5 optimal peer-sets (which we verified to be always within % of the exact optimal for all problem sizes on which we were able to se integer linear programming to compte the latter). Once links are established, the node keeps monitoring them (inclding the incoming ones) and relays their capacity to all other nodes throgh an overlay link-state annoncement protocol. Remote nodes need this information to compte their own peer-sets. Althogh each node measres O(n) overlay links every re-wiring epoch T, the monitoring and annoncement overhead is only O(kn) and not O(n ) since only the O(k) established links are monitored and annonced in between the (infreqent) rewiring epochs, where k n. B. The Downloader Modle The downloader modle monitors the available chnks on the peer-set and isses reqests for downloading missing ones. The selection is based on the well established Local Rarest First (LRF) heristic [] that looks at the peer-set and isses a reqest for any missing chnk that is among the least replicated ones in the peer-set. New reqests are triggered either pon the completion of a download, or if an overlay link is inactive, pon the detection on the other side of the link of a missing chnk. C. The Uploader Modle The ploader receives reqests and sends back chnks. Or baseline ploader allows for p to active pload (chnk) per overlay link (neighbor). It implements this by maintaining a FIFO qee for each overlay connection. This choice bonds the nmber of concrrent ploads by the nmber of neighbors thereby avoiding excessive fragmentation (over partitioning) of the pload bandwidth of the local (physical) access link of a node (this choice is backed-p by reslts appearing in []). We also experimented with an ploader that allows p to active chnk per sorce file per connection, bt this can lead to p n parallel ploads per overlay link, which becomes problematic as n increases. Indeed, over-partitioning the pload bandwidth defeats the entire concept of swarming: it takes too mch time to pload an entire chnk, and dring this time the downloading node is nder tilizing its pload bandwidth as it cannot relay the chnk before it completes the reception. We want to note, however, that or baseline design is by no means claimed to be optimal. For an example consider a node that can pload to its first k neighbors with rate x and to the last one with rate larger than k x. Then as long as this last neighbor can always find k missing chnks from or node, and can also itself disseminate them frther down in the network faster than the k slow neighbors, then the system wold be better off allowing p to k parallel ploads to the fast one at the expense of the slow ones. Sch sitations thogh are rather pecliar and even if they arise, it is difficlt to check the necessary conditions for taking advantage of them, so we leave their investigation to ftre work and stick to the simple one-chnk-per-connection policy. V. PERFORMANCE EVALUATION In this section we compare the performance of Max-Min and Max-Sm peer selection policies against three reference selection policies: Random (node v i selects k peers at random from the set of all nodes in V i ); k-widest (node v i selects Normalized time average finish time per topology MaxMin MaxSm kw Rnd kw Random Normalized time.. 0. worst finish time per topology 0 MaxMin MaxSm kw Fig.. PlanetLab experiment, mid Jne 007. Rnd kw Random node v j if c ij is among the k largest ones across all nodes in V i ); Rand k-widest (v i performs k-widest on a random sbset of V i of size β k). Rand k-widest is inclded in the evalation to mimic the effect of combining random bootstrapping with choke/nchoke in BitTorrent. We compare these policies in terms of (node,remote file) finish times. We denote f(j, i) the time that the sink v j completes downloading the file of sorce v i, assming that all exchanges start at time 0. In all experiments we assme that nodes are flly cooperative (they belong to the same athority) and ths follow exactly and trthflly the peerselection policies of Sect. III and the swarming protocol of Sect. IV (i.e., no choke/nchoke mechanism is employed). We discss the impact of selfishly behaving nodes in Sect. VI. Or performance evalation is done in two settings. In the first, we assme that the n-way broadcast is to be carried over the Internet. We do so by evalating the performance of a prototype implementation of or architectre on PlanetLab. In the second, we assme that the n-way broadcast is to be carried on a closed (controlled/isolated) network. We do so by evalating the performance of a prototype implementation of or architectre on a discrete event simlator of the closed network. [] contains additional reslts and discssion that do not fit here. A. Case Stdy : A PlanetLab Prototype In this setting, we compare the performance of different overlay topologies when the nderlying physical network is the Internet and the overlay nodes are single-homed, i.e., all overlay links of a node go over the same physical access link. For this prpose we selected n =PlanetLab nodes (0 in North America, in Soth America, in Erope, in Asia) and let each one of them disseminate a different 00MBytes file and allow it to connect to k =neighbors (and accept additional incoming links). Notice that we limited or experiment to only nodes and only 00MBytes per node so as to keep the amont of exchanged traffic on PlanetLab at reasonable levels, while also allowing s to monitor the network throghot the experiment. Notice that if data were to be transfered in a point-to-point manner, then it wold amont to over a Terabyte for each exection of the entire experiment: different peer-set selection policies, each one generating 00MBytes of data at each rn, and repeated 0 times to get confidence intervals. We let the re-wiring epoch be T =0mintes and the measrement/annoncement epoch for existing links be mintes. Also we set α =0. to indicate that nodes care eqally for download and pload qality. For a node v j, we compte its maximm finish time max(j) = max i j f(j, i), i.e., the time at which it has completed downloading all n remote files, as well as

6 its average finish time avg(j) =/(n ) i j f(j, i). For peer-set selection policy X,weletmax(X ) = max j max(j) denote its maximm finish time across all nodes, and avg(x )= /n j avg(j) denote its average finish time across all nodes. On the left-hand-side of Fig. we present the normalized average finish time of each policy with respect to the average finish time of the Max-Sm policy. On the right-hand-side, we present the normalized maximm finish time of each policy with respect to the maximm finish time of the Max- Min policy. These reslts show that the varios policies perform qite similarly with respect to average finish time. When looking at maximm finish times thogh, the pictre is completely different. Max-Min manages to complete all downloads anywhere between 0% and 0% faster than the heristics and almost 0% faster than Max-Sm. This can be very significant for Blk Synchronos Parallel (BSP) applications [], in which the global progress depends on the finish time of the slowest node. It is worth noting that optimizing the worst case finish time is mch more difficlt than optimizing the average, and ths it shold come as no srprise that the heristics perform well on average bt fail to improve the worst case. B. Case Stdy : A Dedicated Network Prototype In this setting, we examine overlay networks whose links are dedicated, meaning that they do not compete for bandwidth on the nderlying physical network. This model is plasible for (mlti-homed) networks set-p in spport of an enterprise throgh the acqisition of dedicated links to connect its varios locations. Sch link acqisitions cold be done throgh SLA contracts with ISPs, or throgh virtalization technologies sch as those envisioned for GENI. In either cases, a dedicated link cold be set p between two enterprise nodes i and j for a given price. Any sch dedicated link will have a nominal capacity c ij, which may depend on any nmber of factors (e.g., physical constraints of the nderlying technology, the demand at the ISP for carrying traffic between these two locations, or the price paid for varios links. Since setting p a complete network to connect all n nodes directly to each other may not be feasible (especially for systems of moderate sizes), designers of sch enterprise networks are likely to constrct the network so as to maximize its tility with respect to some objective fnction. Independent of which process/strategy is sed to constrct the optimized overlay, the reslting network wold allow all enterprise nodes to commnicate either directly or throgh overlay paths. The constrction we propose for optimizing the overlay for n-way broadcast proceeds as follows. First, we order the nodes according to their ids. Next, we proceed in ronds in which nodes take trns in selecting their peer-sets (as discssed in Sect. III). This process is repeated ntil we converge by reaching a rond that does not introdce changes in the constrcted topology. It is worth noting that the convergence of the above procedre relates to a qestion regarding the existence of pre Nash eqilibria, and their reachability throgh local improvement paths, in a strategic game with Max-Min or Max-Sm as its payoff fnction. Althogh interesting from a theoretical standpoint, the qestion is not directly relevant here as we have assmed that nodes forward indiscriminately local and in-transit chnks. In all or experiments we got fast convergence bt cold also stop prematrely after a maximm nmber of iterations so as to deal with inexistence, nreachability, or slow convergence to stable topologies. time (secs) Fig.. average finish time per link density Max-Min Max-Sm k-widest Rnd k-widest Random k/n time (secs) worst finish time per link density Max-Min Max-Sm k-widest Rnd k-widest Random k/n Simlation of a closed network based on Sprint s topology. Towards or goal of evalating the impact of varios peer selection policies on the performance of n-way broadcast in this setting, we developed a discrete-event simlator that is able to rn over dedicated overlay networks. We constrcted the dedicated overlay (enterprise) network sing the procedre described above, sing the pblicly available trace of Sprint s physical topology taken from Rocketfel []. In particlar, we assmed that the dedicated capacity that cold be acqired from the ISP (Sprint) wold reflect an eqal-share partitioning, which we approximated as follows. We conted the nmber of shortest-paths (for all physical node pairs) that go over a physical link and set the available bandwidth of that link to be its real capacity divided by this nmber. Then, for an overlay link (i, j) we set c i,j to be eqal to the available bandwidth of the tightest physical link on the indced shortestpath over the physical topology. This prodces the amont of available bandwidth that the ISP can garantee for the new application if it admits it into its network and treats it eqally with pre-existing ones. One advantage of simlations (compared to PlanetLab prototyping) is that it allows s to consider a bigger network. In particlar, in the experiments that follow, we stdy overlays of size n =0nodes, which are randomly selected from the physical Sprint network each node holding a 00Mbytes file. In Fig. we compare the average and maximm finish times of different policies for different link densities k/n. Compared to the previos reslts from PlanetLab, we observe a qalitatively similar behavior. The gap, however, between Max-Min and the rest in terms of maximm finish time widens sbstantially: Max-Min is able to finish - times faster in this setting, even for relatively large k/n ( 0%). The reason is that Max-Min has more real bandwidth to work with in this case: When it places a link (i, j), the capacity (both pload and download) of the two end-points increases by the capacity of the newly-added dedicated overlay link, whereas in PlanetLab the physical bandwidth is fixed, so when Max-Min places an overlay link it can only benefit by whatever nsed bandwidth exists on the nderlying physical network. VI. DEALING WITH SELFISH BEHAVIOR Up to now we have assmed that nodes are flly cooperative, which is a realistic assmption for the applications enmerated in the introdction. In this section we will try to explore ways to accommodate applications that involve selfish nodes. We will focs on the following definition of selfishness: The idea is that each pair of physical nodes represents a different application that is assigned an eqal share of the physical capacity of all links on which it competes with other applications.

7 Definition : (Upload-selfishness) An pload-selfish node is a node that wants to se as mch of its pload capacity as possible for forwarding its local chnks and avoid wasting it in relaying the in-transit chnks that it holds. A. A Brief Taxonomy of Deterrence Mechanisms The amont of extra benefit for an pload-selfish node (and potential harm to others) depends on the mechanisms that the network deploys for discoraging sch behavior. We examine the following cases. Case 0 (netral): Here the network stays netral and does not deploy any deterrence mechanism. In sch a setting, the pload-selfish node cold simply pload its own chnks and ignore all other reqests. The harm to cooperative nodes can easily be qantified for this case, so we don t discss it frther; it will be proportional to the nmber of pload-selfish nodes, and cooperative nodes will be slowed down and at an extreme case will be nable to receive some files (e.g., when all their neighbors are pload-selfish, which is similar to the case of an eclipse attack []). Case (oblivios retribtion): A network can employ several retribtion mechanisms to pnish a node that fails to deliver a chnk after a reqest. The choke/nchoke [] mechanism of BitTorrent, or modified versions based on bit-level tit-fortat [9], [] are two established existing proposals. Contrary to the original BitTorrent, sch mechanisms are marginally sefl here becase they are oblivios to whether a node ploads local or in-transit chnks. An pload-selfish node will appear to be contribting by the mere fact that it is certainly ploading its own chnks. Ths oblivios strategies fail to pnish nodes that free-ride by not ploading in-transit chnks. Case (non-oblivios retribtion): Now, let s assme that there exists a non-oblivios retribtion mechanism that pnishes a node that fails to service reqests for in-transit chnks that it holds. What can a selfish node do against sch mechanism? The simplest strategy is to hide (by not annoncing) the availability of in-transit chnks it holds, and ths get rid of the brden of having to service reqests for these chnks. This can be addressed with a simple two-hop annoncement strategy in which a node that ploads to another node annonces on its behalf the availability of the chnk (sing HAVE messages []) to downloaders belonging to the peer-set of the receiving node. This reqires obtaining pon bootstrap (and re-wiring) second hop neighbors. Assming that the retribtion is severe enogh, the pload-selfish node will have to honor all reqests. Despite that, the pload-selfish node still has some room to game the system by changing the ploader and the downloader as follows. The pload-selfish node can sbstitte each FIFO qee at its ploader with a selfish FIFO (S-FIFO) that gives priority (preemptive or non preemptive) to reqests for local chnks. The pload-selfish node can switch from Least Replicated First to Most Replicated First (MRF) downloads. Highly replicated chnks receive fewer reqests and ths redce the We do not want to pnish nodes that don t have enogh in-transit content for whatever reason (slow local link or peer-set) bt wold relay if they had, so we only pnish when a reqest exists and is not honored. node id file id file id Fig.. Maximm finish time for all nodes and all files nder Random with cooperative nodes and Max-Min with pload-selfish slowest node ( k n =0.0). waste of pload bandwidth for sending in-transit chnks, is smaller (most nodes already have these chnks, and any reqests for these chnks will be divided over many peers). Since it is difficlt to detect sch deviations from the protocol, we instead qantify their impact. B. Qantifying the Impact of S-FIFO/MRF We qantify the advantage for a single pload-selfish node by looking at the ratio between the time it takes to pload its file to all other nodes when it is selfish and when it is cooperative, granted that all other nodes are cooperative. We examined this ratio for different overlays bilt on the Sprint trace and for different choices with respect to the choice of selfish node. We consider three cases, where the selfish node is : () the slowest node, i.e., the one whose adjacent links have the minimm aggregate pload capacity; () the fastest node; or () a typical node (median pload capacity). On the Max-Min overlay the selfish node redced its maximm pload finish time by 0% when it was the slowest one []. When it was a typical (or the fastest one), then it got almost no benefit, since in these cases the bottleneck is at the downloading nodes (so a local selfishness behavior cannot help). In all other overlays, the selfish node got almost no benefit, even when it was the slowest node. Unlike the Max- Min, the other overlays are not optimized for the slowest node, so even if this bottleneck node tries to selfishly pload its file, it cannot really benefit becase it has very limited bandwidth. From the above, it is clear that there exist cases in which pload-selfishness pays sbstantially. Granted that ploadselfishness is hard to detect, we also look at its impact on the cooperative nodes. We consider again a single selfish node (one can easily extrapolate for mltiple selfish nodes). The impact depends on the considered metric and on the identity of the selfish node. If we care abot the worst-case download time of cooperative nodes and let the selfish node be the slowest node, then conter-intitively, the impact on the cooperative nodes is positive. This is simply becase by being selfish, the slowest node helps all other nodes improve their (bottleneck) downloads from it. To get a feeling of this we show a scatterplot on the left of Fig. with the download time for each pair (node,remote file) when the topology is random and all the nodes are cooperative. The solid black line that stands ot corresponds to the slowest node (node 9), whose file is the last one to be downloaded by all others. To contrast this, we plot on the right the corresponding times when the topology is Max-Min and the slowest node is pload-selfish. As it can be seen, the combination of Max-Min topology and ploadselfish schedling on the slowest node does a pretty good job node id

8 at smoothing ot the differences in maximm finish times. If, on the other hand, the selfish node is a typical node, or the fastest node, then its effect on the download qality of others is rather marginal. First, its own file is not a bottleneck. Second, the relay of in-transit chnks is largely carried by the other n nodes. Third, S-FIFO and MRF impact primarily firsthop neighbors and have small impact on nodes frther away. Overall, pload-selfishness, nlike its name sggest, is not necessarily bad. A socially inclining global schedling policy, for example, wold certainly make slow nodes pload only their own chnks so as to redce the severity of the bottlenecks that they case. More generally, for social optimality, one shold split the pload bandwidth of a node between local and in-transit chnks according to the relative speed of the node. Nodes who are fast shold contribte heavily in relaying in-transit chnks. Nodes who are slow, shold focs only on ploading their own chnks so as to avoid becoming bottleneck points. Stated differently, a single ploading policy across all nodes cannot be socially optimal. We postpone the investigation of node-dependent pload schedling for ftre work (see Sect. VI of [7] for a similar discssion based on or previos work on selfish caching). C. Download-Selfishness and Other It is tempting to ask whether a notion of downloadselfishness wold make sense. Or answer leans towards the negative. First, there is no contention between local and in-transit chnks in the incoming direction towards a node only in-transit chnks flow there. Second, as long as the downloader keeps all its overlay connections bsy by immediately identifying and reqesting missing chnks, its download-finish time will be the same, so it gets no foreseeable benefit by deviating from LRF. Finally, trying to maniplate the system by advertising false c ij s for the established links can be disclosed by having nodes periodically adit others by measring some remote c ij s and comparing with the advertised vales on the link-state protocol. Sch methods are qite elaborate and fall otside the scope of the crrent work. VII. CONCLUSIONS AND FUTURE WORK In this article we showed that swarming protocols for blk data transfers perform mch better when operating over optimized overlay topologies that take into consideration the end-toend performance characteristics of the nderlying network. Sch topologies improve the aggregate transmission capacity of nodes, bt where they make a hge difference compared to existing heristic approaches, is on relieving bottleneck points. Random and myopic heristics sed in practice lack the reqired sophistication for overcoming sch bottlenecks. Or optimized topologies are oblivios to the details of the swarming protocol that rns on top. They leverage the available bandwidth of the nderlying network and abstract the swarming protocol by viewing it as a series of max-flows. Ths they can benefit a variety of swarming protocols with different pload/download schedling characteristics. Since or topologies are data-blind, it is the job of the swarming protocol to make the best se of the end-to-end bandwidth that they offer. To that end, we have shown that a commonly parameterized swarming protocol is far from being optimal. Designing swarming protocols tailored to the characteristics of individal nodes is on or ftre research agenda. REFERENCES [] B. Cohen, Incentives bild robstness in BitTorrent, in Proc. of First Workshop on Economics of Peer-to-Peer Systems, Berkeley, CA, 00. [] X. Li, F. Bian, M. Crovella, C. Diot, R. Govindan, and G. Iannaccone, Detection and identification of network anomalies, in Proc. of IMC 0, Rio de Janeriro, Brazil, 00. [] P. A. Bernstein and N. Goodman, Concrrency control in distribted database systems, ACM Compt. Srv., vol., no., 98. [] A. Fabrikant, A. Lthra, E. Maneva, C. H. Papadimitrio, and S. Shenker, On a network creation game, in Proc. of ACM PODC 0, Boston, MA, 00. [] N. Laotaris, G. Smaragdakis, A. Bestavros, and J. Byers, Implications of selfish neighbor selection in overlay networks, in Proc. of IEEE INFOCOM 07, Anchorage, AK, 007. [] D. Qi and R. Srikant, Modeling and performance analysis of bittorrent-like peer-to-peer networks, in Proc. of ACM SIGCOMM 0, Portland, Oregon, USA, 00. [7] L. Massolie and M. Vojnovic, Copon replication systems, in Proc. of ACM SIGMETRICS 0, Banff, Alberta, Canada, 00, pp.. [8] R. Kmar and K. W. Ross, Optimal peer-assisted file distribtion: Single and mlti-class problems, sbmitted, Agst 00. [9] A. R. Bharambe, C. Herley, and V. N. Padmanabhan, Analyzing and improving a bittorrent networks performance mechanisms, in Proc. of IEEE INFOCOM 0, Barcelona, Spain, 00. [0] M. Izal, G. Urvoy-Keller, E. W. Biersack, P. A. Felber, A. A. Hamra, and L. Garces-Erice, Dissecting bittorrent: Five months in a torrent s lifetime, in Proc. of PAM 0, Antibes Jan-les-Pins, France, 00. [] A. Legot, G. Urvoy-Keller, and P. Michiardi, Rarest first and choke algorithms are enogh, in Proc. of ACM IMC 0, Rio de Janeriro, Brazil, 00. [] L. Massolie, A. Twigg, C. Gkantsidis, and P. Rodrigez, Randomized decentralized broadcasting algorithms, in Proc. of IEEE INFOCOM 07, Anchorage, AK, 007. [] J. Edmonds, Edge-disjoint branchings, in Proc. of the 9th Corant Compter Science Symposim on Combinatorial Algorithms, Algorithmics Press, 97, pp [] C. Gkantsidis and P. Rodrigez, Network coding for large scale content distribtion, in Proc. of IEEE INFOCOM 0, Miami, FL, 00. [] L. Go, S. Chen, Z. Xiao, E. Tan, X. Ding, and X. Zhang, Measrements, analysis, and modeling of bittorrent-like systems. in Proc. of ACM IMC 0, Berkeley, CA, 00. [] Y. Tian, D. W, and K.-W. Ng, Analyzing mltiple file downloading in bittorrent, in Proc. of ICPP 0, Washington, DC, 00. [7] M. Castro, P. Drschel, A.-M. Kermarrec, A. Nandi, A. Rowstron, and A. Singh, SplitStream: high-bandwidth mlticast in cooperative environments, in Proc. of ACM SOSP 0, Bolton Landing, NY, 00. [8] D. Kostic, A. Rodrigez, J. Albrecht, and A. Vahdat, Bllet: high bandwidth data dissemination sing an overlay mesh, in Proc. of ACM SOSP 0, Bolton Landing, NY, USA, 00. [9] H. Zhang, G. Neglia, D. Towsley, and G. L. Presti, On nstrctred file sharing networks, in Proc. of IEEE INFOCOM 07, Anchorage, AK, 007. [0] M. Jain and C. Dovrolis, End-to-end available bandwidth: measrement methodology, dynamics, and relation with tcp throghpt, IEEE/ACM Trans. Netw., vol., no., pp. 7 9, 00. [] S. M. Hedetniemi, S. T. Hedetniemi, and A. L. Liestman, A srvey of gossiping and broadcasting in commnication networks, Networks, vol. 8, pp. 9 9, 988. [] V. Ribeiro, R. Riedi, R. Baranik, J. Navratil, and L. Cottrell, pathchirp: Efficient Available Bandwidth Estimation for Network Paths, in Proc. of PAM 0, La Jolla, CA, 00. [] G. Smaragdakis, N. Laotaris, P. Michiardi, A. Bestavros, J. Byers, and M. Rossopolos, Swarming on optimized graphs for n-way broadcast, Boston University, Tech. Rep. BUCS-TR , Jly 007. [] R. H. Bisseling, Parallel Scientific Comptation: A Strctred Approach sing BSP and MPI. Oxford University Press, 00. [] N. Spring, R. Mahajan, D. Wetherall, and T. Anderson, Measring ISP topologies with rocketfel, IEEE/ACM Trans. Netw., vol., no., pp., 00. [] A. Singh, M. Castro, P. Drschel, and A. Rowstron, Defending against eclipse attacks on overlay networks, in Proc. of ACM SIGOPS Eropean Workshop, Leven, Belgim, 00.