P2P Content Distribution

Transcription

1 , University of Hannover Wolf-Tilo Balke and Wolf Siberski *With slides from K. Wehrle (RWTH), A. Bharambe (CMU), P. Rodriguez, P. Chou (MSRC), Chiu (CUHK) Peer-to-Peer Systems and Applications, Springer LNCS Early approach to download Napster/Gnutella/Fasttrack Download whole file from one peer If download fails: repeat search, resume download from alternative Issues No load distribution Poor performance Main reason: Asymmetric uplink/downlink bandwidth (ADSL) Low reliability (except for small files) 2

2 Swarming Approach Chunks Split large files into small chunks Identify/protect chunks via hash values 0x9A3C 0x7C23 0x194F 0xDE6A Parallelization Download different chunks from different sources Utilize upload capacity of multiple sources Sources: Destination: 3 Swarming Properties Advantages Peer failures: no loss of files, only chunks Increased throughput Strategiest Chunk selection Avoid scarcity Best overall availability? Fairness Systems Free-Riding Bandwidth allocation BitTorrent Avalanche 4

3 BitTorrent Overview Bittorrent or BitTorrent Torrent = big stream Author: Bram Cohen, 2003 Only for file distribution, no search features Designed for Content providers Flash crowds Central components Web server for search Tracker for peer coordination 5 BitTorrent Parts Peers Torrent Contains metadata about the files Contains the address of a tracker Specification of backup trackers possible Swarm All peers sharing a torrent are called a swarm Tracker Keeps track of which peers are in a swarm Coordinates communication between the peers 6

4 BitTorrent joining a torrent new leecher 2 join peer list 3 tracker torrent 1 data request 4 website seed/leecher Peers divided into: seeds: have the entire file leechers: still downloading 1. obtain the torrent 2. contact the tracker 3. obtain a peer list (contains seeds & leechers) 4. contact peers from that list for data 7 BitTorrent exchanging data leecher B leecher A I have! seed leecher C Verify pieces using hashes Download sub-pieces in parallel Advertise received pieces to the entire peer list Look for the rarest pieces 8

5 BitTorrent - unchoking leecher B leecher A seed leecher D leecher C Periodically calculate data-receiving rates Upload to (unchoke) the fastest downloaders Optimistic unchoking periodically select a peer at random and upload to it continuously look for the fastest partners 9 Torrent A Torrent file Passive component Files are typically fragmented into 256KB pieces Typically hosted on a web server Metadata file structure Describes the files in the torrent URL of tracker File name File length Piece length SHA-1 hashes of pieces Allow peers to verify integrity Creation date 10

6 Tracker Peer cache IP, port, peer id State information Completed Downloading Clients report status periodically to tracker Returns random list 50 random leechers/seeds Client first contacts of them and more if some do not respond 11 Tracker 12

7 Tracker-less approaches Tracker issues Single point of failure Scalability Piratebay tracker nearly overloaded (>5 Mio. Peers) Decentralized tracker Replace with DHT (Kademlia) Does not tackle distributed search Currently not widely used 13 Chunk Selection Which chunk next? 1. Strict Priority Finish active chunks 2. Rarest First Improves availability of rare chunks Delays download of common chunks 3. Random First Chunk Get first chunk quickly (rarest chunk probably slow to get) 4. Endgame Mode Send requests for last sub-chunks to all known peers End of download not stalled by slow peers 14

8 Choking Choking mechanism Ensures that nodes cooperate Eliminates the free-rider problem Cooperation involves uploaded sub-pieces that you have on your peer Choking Temporary refusal to upload Downloading occurs as normal Connection is kept open No Setup costs TCP congestion control Based on game-theoretic concepts Tit-for-tat strategy in repeated games 15 Game Theory Basic Ideas of Game Theory Studies situations where players choose different actions in an attempt to maximize their returns Studies the ways in which strategic interactions among rational players produce outcomes with respect to the players preferences The outcomes might not have been intended by any of them Game theory offers a general theory of strategic behavior Described in mathematical form Plays an important role in Modern economics Decision theory Multi-agent systems 16

9 Game Theory Developed to explain the optimal strategy in two-person interactions. von Neumann and Morgenstern Initially John Nash Zero-sum games Works in game theory and differential geometry Nonzero-sum games Nash equilibrium 1994 Nobel Prize in Economics Harsanyi, Selten Incomplete information 17 Definitions Games Situations are treated as games. Rules The rules of the game state who can do what And when they can do it. Player's Strategies Plan for actions in each possible situation in the game Player's Payoffs Is the amount that the player wins or looses in a particular situation Dominant Strategy If players best strategy doesn t depend on what other players do 18

10 Prisoner's Dilemma Famous example of game theory A and B are arrested by the police They are questioned in separate cells Unable to communicate with each other. They know how it works If they both resist interrogation and proclaim their mutual innocence, they will get off with a three year sentence for robbery. If one of them confesses to the entire string of robberies and the other does not, the confessor will be rewarded with a light, one year sentence and the other will get a severe eight year sentence. If they both confess, then the judge will sentence both to a moderate four years in prison 19 Prisoner's Dilemma B Confess Not Confess A Confess 4 years each 1 year for B and 8 years for A Not Confess 8 years for A and 1 year for B 3 years each 20

11 A s Decision Tree There are two cases to consider If B Confesses If B Does Not Confess A A Confess Not Confess Confess Not Confess 4 Years in Prison 8 Years in Prison 1 Year in Prison 3 Years in Prison Best Strategy Best Strategy The dominant strategy for A is to confess No matter what B does, confessing is better choice Nash equilibrium: both A and B will confess 21 Repeated Games A repeated game Game that the same players play more than once Differ from one-shot games because people's current actions can depend d on the past behavior of other players. Cooperation is encouraged Book recommendation Thinking strategically by A.Dixit and B Nalebuff German translation: Spieltheorie für Einsteiger 22

12 Tit for tat Tit for tat Highly effective strategy An agent using this strategy will initially cooperate Then respond in kind to an opponent's previous action If the opponent previously was cooperative, the agent is cooperative. If not, the agent is not. Dependent on four conditions Unless provoked, the agent will always cooperate If provoked, the agent will retaliate The agent is quick to forgive The agent must have a good chance of competing against the opponent more than once 23 Bittorrent Choking mechanism Ensures that nodes cooperate Eliminates the free-rider problem Cooperation involves uploaded sub-pieces that you have to your peer Based on game-theoretic concepts Tit-for-tat strategy in repeated games Goal is to have several bidirectional connections running continuously Upload to peers who have uploaded d to you recently Unutilized connections are uploaded to on a trial basis to see if better transfer rates could be found using them 24

13 Choking Details A peer always unchokes a fixed number of its peers Default of 4 Choking decision based on current download rates Evaluated on a rolling 20-second average Choking evaluation performed every 10 seconds Prevents wastage of resources by rapidly choking/unchoking peers Optimistic Unchoking Each BitTorrent peer has a single optimistic unchoke which is uploaded regardless of the current download rate from it. This peer rotates every 30s Reason: To discover currently unused connections that are better than the ones being used 25 Anti-Snubbing Choking policy When over a minute has gone by without receiving a single sub-piece from a particular peer, do not upload to it except as an optimistic unchoke Problem A peer might find itself being simultaneously choked by all its peers that it was just downloading from Download will lag until optimistic unchoke finds better peers Solution If choked by everyone, increase the number of simultaneous optimistic unchokes to more than one 26

14 Choking for Seeds Open issue: upload-only choking Once download is complete, a peer has no download rates to use for comparison nor has any need to use them The question is, which h nodes to upload to? Policy Upload to those with the best upload rate. Advantages Ensures that t pieces get replicated faster Peers that have good upload rates are probably not being served by others 27 Experimental Evaluation 1 Comprehensive analysis of BitTorrent to assess its performance "Dissecting BitTorrent: Five Months in a Torrent's Lifetime" by M. Izal, G. Urvoy-Keller, E.W. Biersack, P.A. Felber, A. Al Hamra, and L. Garces-Erice, Passive and Active Network Measurement 2004 Obtained the "tracker" log Case study of 1.77GB Linux Redhat 9 distribution For its first 5 months of activity More than 180,000 clients More than 50,000 clients initiating a download in the first five days 28

15 Experiments Number of active peers over time 29 Experiments Number of active peers in first 5 days 30

16 Experiments Data Volume uploaded by seeds and leachers 31 Experiments Proportions of seeds and leachers 32

17 Experiments Session types and their characteristics 33 Experimental Evaluation 2 Simulation-based evaluation Analyzing and Improving a BitTorrent Network s Performance Mechanisms by A. Bharambe, C. Herley, and V. N. Padmanabhan, SIGCOMM2006 Motivating Questions Are download rates optimal? Can we do better? Does rate-based Tit-for-tat (TFT) work? Is ste the Rarest aestfirst stpocy policy really eaybeeca beneficial? Must nodes continue seeding after downloading? 34

18 Experimental Setup Discrete-event simulator Models BitTorrent joins, leaves, block exchanges Models queuing delays, no propagation p delay Assumes bandwidth bottlenecks only at the edge Common parameters 100 MB file; 400 blocks of 256 KB 1 seed always on, flash-crowd: 100 joins/sec Seed-uplink = 6 Mbps, Nodes = 1500/400 kbps #nodes = 1000, #neighbors = Scalability: Uplink Utilization Upload utilization is constantly very high 36 36

19 Problem Case: Slow Seed Node capacities Uplink: 400 kbps Downlink: 1500 kbps Seed capacity Uplink: varies from 200 kbps 1000 kbps Scenario: seed uplink = 400 kbps If BitTorrent is performing optimally, we should see near 100% uplink utilizatoin Problem Case: Slow Seed The seed node decides which blocks to serve Vanilla BitTorrent: Connected nodes decide which blocks to request from seed Avoid sending duplicate blocks from seed at all costs 38 38

20 Improving Fairness Goal: ensure nodes upload as much as they download ISPs have begun to charge heavy P2P users Uploaders will bear the brunt of the charges BitTorrent s rate-based TFT and optimistic unchoke can result in high unfairness Proposed solution: pair-wise block-based TFT Bound the difference between blocks uploaded and downloaded Improving Fairness Questions: In the worst case, how many blocks does a node serve? Measure as ratio to #blocks downloaded What is the overall uplink utilization? TFT advocates blocking a link even when there is data to send Can hurt link utilization 40 40

21 Improving Fairness: Blocks served Vanilla BitTorrent results in high unfairness Block-level TFT effective Matching Tracker useful Improving Fairness: Uplink Utilization Matching Tracker helps increase utilization Pairwise TFT needs higher node degrees for better utilization 42 42

22 Fairness / Efficiency Tradeoff in Swarming Protocols Equal download bandwidth for all peers Download bandwidth proportional to upload contribution "The Delicate Tradeoff of BitTorrent-like File Sharing Protocol Design", by B Fan, DM chiu and JCS Lui,,IEEE ICNP BitTorrent: Summary Optimized file transfer system No file search, no fancy GUI, etc. Very effective High throughput & scalability Nearly perfect utilization of bandwidth Fairness and load distribution not optimal, but good enough Commercially successful Distribution of RedHat distribution BBC evaluates the distribution ib i of TV content (not in real-time) Centralized Easier to take down than other approaches 44

23 Avalanche Developed at Microsoft Research "Network Coding for Large Scale Content Distribution by C. Gkantsidis and P. Rodriguez, INFOCOM 2005 Main application scenario: software update distribution Cope with huge flash crowds Evolution of swarming approach Better control of chunk distribution Network coding Keep BitTorrent advantages 45 Goals Improve on following issues Rare blocks are hard to obtain Tit-for-tat incentive mechanisms decrease speeds Arrival of new users slows down old users Heterogeneous nodes do not interact well Same information travels repeatedly over bottleneck links Too much dependency from seeds Sudden departures can prevent peers from finishing 46

24 Network Coding Term Network Coding coined in "Distributed Source Coding for Satellite Communications by R.W. Yeung, Z. Zhang, IEEE Transactions on Information Theory,1999 Basic ideas Optimize transfer via multiple paths Instead of directly transferring data blocks, use linear combinations 47 Network Coding can Increase Throughput sender s 1 receiver t b 2 rate r 1 XOR a XOR b a b sender s XOR receiver t 1 2 rate r 2 a r 2 r 2 Routing only: Capacity region? Network Coding: r 1 r 1 48

25 Network Coding Capacity a,b a b a b a b a+b a+b a+b a,b b,a optimal routing throughput = 1 network coding throughput = 2 Network coding achieves multicast capacity sender receiver coding node Network information flow by R. Ahlswede, N. Cai, S. R. Li, R. W. Yeung:. IEEE Transactions on Information Theory, Network Coding and Latency a b a b b a b a+b b a a a+b optimal routing delay = 3 network coding delay = 2 Network coding minimizes delay A comparison of network coding and tree packing by Y. Wu, P.A. Chou, K. Jain, ISIT

26 Network Coding - Theory vs. Practice Theory: Symbols flow synchronously throughout network Edges have unit (or known integer) capacities Centralized knowledge of topology assumed to compute encoding and decoding functions Practice: Information travels asynchronously in packets Packets subject to random delays and losses Edge capacities often unknown, time-varying Difficult to obtain centralized knowledge, or to arrange reliable broadcast of functions Need simple technology, applicable in practice 51 Linear Network Coding Assume file: F = [x 1 x 2 ], where x i is a block. Define code E i (a i,1, a i,2 ) = a i,1 *x 1 + a i,2 *x 2, where a i,1, i( i,1 i,2 ) i,1 1 i,2 2 i,1 a i,2 are numbers. Infinite number of E i s. Any two linearly independent E i (a i,1, a i,2 ) can recover [x 1 x 2]. Similar as solving a system of linear equations. 52

27 Making Network Coding Practical Packetization Include matrix coefficients in chunk header Decoding becomes simple matrix operation Header removes need for centralized knowledge of graph topology and encoding/decoding functions Overhead: ~ 3% Buffering Allows asynchronous packets arrivals & departures with arbitrarily varying rates, delay, loss 53 Avalanche Coding Content initially encoded at the server Clients can produce new encoded packets out of partial files File B 1 B 2 B n Server α 1 α 2 β 1 β 2 βn α n Client A E 1 E 2 ω 1 ω 2 Client B E 3 54

28 Avalanche Robustness If server goes down (after serving the full file once) All Avalanche users are able to complete the download Only 10% of users using typical file-swarming techniques are able to complete. # of Peers Finished Avalanche NC FEC LR Typical file-swarming systems Time 55 Avalanche Download Time Much lower and predictable download times Finish Times Avalanche NC Typical swarming Random Peers that did not finish Nodes (sorted by order of arrival) 56

29 No need for nodes to stay around No need for nodes to stay after they finish the download Performance remains unchanged Finish Times Nodes stay for ever Nodes leave immediately Nodes (sorted by order of arrival) 57 Decoding Performance Avalanche tradeoff decision Impose more processing load on peers Optimize network utilization Decoding time less than 4% of the total download File Size (MB) Blocks Time sec sec m 21 sec m 38 sec Measured on Pentium III, 650MHz, 512MB RAM. 58

30 Avalanche: Summary Optimized version of BitTorrent Advantages Improves chunk availability by network coding No endgame required, because many different chunks are usable Reduces need for seed significantly Keeps BitTorrent advantages (scalability, etc.) Disadvantage Increased local load Processing Memory Harddisk 59 Swarming: Summary Solves the problem of efficient file distribution Scalable Handles flash crowds Areas for optimization Incentive models Tracker-less approaches Further endgame improvements Next step: content streaming Real-time constraints Chunk order 60