Distributed Data Management

Size: px
Start display at page:

Download "Distributed Data Management"

Transcription

1 Distributed Data Management Profr. Dr. Wolf-Tilo Balke Institut für Informationssysteme Technische Universität Braunschweig

2 Network Models and Content Provisioning Network Models (2 nd Part) 7.4 Scale-Free Networks 7.5 Comparing Graphs 7.6 Models in P2P Content Distribution 8.1 Swarming 8.2 BitTorrent 8.3 Anonymous P2P Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 2

3 7.4 Scale-Free Networks In 1999, Albert-László Barabási (Univ. of Notre Dame) crawled parts of the WWW to investigate its actual structure The node degree is power-law distributed i.e., the probability that a node in the network is connects to k other nodes is P k ~ k γ (usually with 2 < γ 3) Most nodes have a small degree of around 1 to 2 Few nodes have an extremely high node degree High-degree vertices are called hubs Albert-László Barabási. Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life. Plume ISBN Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 3

4 7.4 Scale-Free Networks Definition: Graphs with a power-law node degree distribution form scale-free networks Also called power-law networks What kind of network model can generate this more realistic degree distribution? Barabási Albert model builds a certain subset of scale-free networks Albert-László Barabási & Réka Albert."Emergence of scaling in random networks". Science, 1999 doi: /science Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 4

5 7.4 Barabási Albert Graphs Barabási Albert model: Basic Idea In its simplest form denoted as g_ba n,m n is the number of nodes in the graph m is the number of edges added per time step The total number of edges is thus n m Start with any initial graph of size n 0 n 0 2 and degree of any node deg(v) 1 Often, just m connected nodes are used as default initial network If initial network is not connected, the result network cannot be guaranteed to be connected Barabási Albert graph is constructed iteratively by adding new nodes one by one until target size n is reached Represents one time step in a simulated network growth i.e. Discrete Time Modeling Add nodes until target size n is reached Each new node is connected to m existing nodes Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 5

6 7.4 Barabási Albert Graphs New edges are not added randomly, but favor higher-degree nodes The rich get richer Preferential attachment to higher-degree nodes The higher the degree of a possible target node, the higher the probability that the new node will attach to it Preferential attachment defines the probability (v) for vertex v to get an edge to a new node In general, is proportional to the node degree, i.e. v ~ deg(v) Most common definition is deg v v = w V deg(w) Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 6

7 7.4 Barabási Albert Graphs Example: g_ba 5,1 t = 0 t = 1 ε Initial graph Add new node v 3 Probability for connecting any old node v to v 3 is given by v = deg v w V deg w e.g., connect to v 1 Random decision steered by preferential attachment t = 1 v 1 v 2 v 1 v 2 v 1 v 2 (v 1 ) = 1 2 (v 2 ) = 1 2 v 3 v 3 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 7

8 7.4 Barabási Albert Graphs Example: g_ba 5,1 t = 2 ε Add new node v 4 Evaluate preferential attachment e.g. connect to v 1 t = 3 ε Add new node v 5 Evaluate preferential attachment e.g. connect to v 1 v 4 v 4 (v 4 ) = 1 6 (v 1 ) = 1 2 (v 1 ) = 1 2 v 1 v 2 (v 2 ) = 1 4 v 1 v 2 v 5 (v 2 ) = 1 6 (v 3 ) = 1 4 v 3 v 3 (v 3 ) = 1 6 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 8

9 7.4 Barabási Albert Graphs Comparing Barabási Albert Graphs n = 50, ~50 edges coloring by node degree Erdős-Rényi Graph Barabási Albert Graphs Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 9

10 7.4 Barabási Albert Graphs Comparing Barabási Albert Graphs n = 100, ~100 edges Erdős-Rényi Graph Barabási Albert Graphs Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 10

11 7.4 Barabási Albert Graphs Comparing Barabási Albert Graphs n = 100, ~150 edges Erdős-Rényi Graph Barabási Albert Graphs Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 11

12 7.4 Barabási Albert Graphs Histogram of node coefficients Single sample 100 nodes 300 edges Random Generally lower degree Small World Homogeneous degree Scale-Free Power-law Hubs visible Number of Nodes Dampening factor for decreasing strength of preferential attachment Barabási(pa=0.5) Watts-Strogatz(p=0.05) Random Node Degree Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 12

13 relative frequency 7.4 Barabási Albert Graphs Node degree for larger Barabási Albert graphs 200k nodes 400k edges Logarithmic Scale degree Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 13

14 7.4 Barabási Albert Graphs Histogram of cluster coefficients (C) Same sample Random Low C Small World Homogeneous high C Scale-Free Also power-law Lower than SW Number of Nodes Barabási(pa=0.5) Watts-Strogatz(p=0.05) Random Cluster Coefficient Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 14

15 7.4 Scale-Free Networks Important property of scale-free networks is robustness against random failures Removing a random vertex v will likely hit a low-degree node Expected damage to network is small A failing high-degree node can severely damage a network Better fail-safety necessary for high-degree node to ensure overall robustness Thus, scale-free networks are very sensitive against attacks If a malevolent attacks explicitly target the highest degree nodes, the network can easily decompose Note: random graphs are not resilient against random failures, but also not particularly prone to attacks Most vertices more or less have the same degree Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 15

16 7.5 Comparing Graphs Random Graph: 50 nodes, 50 edges Color by degree Property Value Connected No Diameter (conn.) 9 Avg. Path Length 4.39 #Clusters 6 Largest Cluster 39 k-connectedness 0 Avg. Cluster Coeff Avg. Degree 2 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 16

17 7.5 Comparing Graphs Watts-Strogatz Graph: 50 nodes, 50 edges Property Value Connected No Diameter (conn.) 35 Avg. Path Length #Clusters 2 Largest Cluster 38 k-connectedness 0 Avg. Cluster Coeff. 0 Avg. Degree 2 p = 0.05 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 17

18 7.5 Comparing Graphs Barabási-Albert Graph: 50 nodes, 49 edges Property Value Connected Yes Diameter 12 Avg. Path Length 5.14 k-connectedness 1 Avg. Cluster Coeff. 0 Avg. Degree 1.96 pa = 0.8 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 18

19 7.5 Comparing Graphs Random Graph: 50 nodes, 100 edges Property Value Connected No Diameter (conn.) 6 Avg. Path Length 2.88 #Clusters 2 Largest Cluster 49 k-connectedness 0 Avg. Cluster Coeff Avg. Degree 4 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 19

20 7.5 Comparing Graphs Watts-Strogatz Graph: 50 nodes, 100 edges Property Value Connected Yes Diameter (conn.) 10 Avg. Path Length 4.6 k-connectedness 2 Avg. Cluster Coeff Avg. Degree 4 p = 0.05 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 20

21 7.5 Comparing Graphs Barabási-Albert: 50 nodes, 98 edges Property Value Connected Yes Diameter 4 Avg. Path Length 2.55 k-connectedness 2 Avg. Cluster Coeff Avg. Degree 3.92 pa = 0.8 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 21

22 7.6 Models in P2P What do real Peer-To-Peer Networks look like? Depends on the used protocols Some P2P networks like e.g. Freenet evolve voluntarily in a small-world with a high clustering coefficient and a small diameter Analogously, some protocols, e.g., Gnutella, will implicitly generate a scale-free degree distribution Implied by boot-strapping and Ping-Pong Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 22

23 7.6 Models in P2P What should Peer-to-Peer networks look like? It depends If it should be navigable in a decentralized fashion, Make it a small-world and implement Kleinberg s routing algorithm (or a variant, e.g., Symphony) If the peer-to-peer network could be under attack also make it a small-world, where most vertices have the same (low) degree If it is peer-to-peer network in a small and secure context, e.g. an intranet in a company, Make it a scale-free network. This allows to buy only a small number of servers with a high bandwidth. These will work as 'hubs' of the network Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 23

24 8.1 Swarming Sometimes large amounts of data have to be distributed over networks Software updates, video on demand, etc. Early approaches: Napster, Gnutella, Fasttrack, Kaazaa, Use P2P network to locate a node offering the requested content Download whole file from a single peer If download fails: repeat search, resume download from alternative source Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 24

25 8.1 Swarming Issues Poor performance due to asymmetric uplink/downlink bandwidth Most common network home network connection technology: ADSL Asynchronous Digital Subscriber Line e.g. ADSL kb/sec download, 1024 kb/sec upload No load distribution Popular files may have extremely low download speed due to congestion of the offering node Low reliability (except for small files) Connected glitches may severely hamper download Frequent re-connects and resumes necessary Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 25

26 8.1 Swarming Idea: Chunks Split large files into small chunks Assign hash values to chunks Identification: simple and deterministic labeling of chunks Transfer Protection: download chunk and compute hash Compare computed hash with hash provided by offering peer If comparison fails, a transfer error occurred reload chunk Original File 0x9A3C 0x7C23 0x194F 0xDE6A Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 26

27 8.1 Swarming Parallelization Locate the swarm of all peers hosting the required file (and thus the required chunks) Download different chunks from different sources simultaneously Utilize upload capacity of multiple sources Overall download speed may thus exceed upload capabilities of individual sources Usually, upload capacity is the bottleneck due to asymmetrical connections Sources: Destination: Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 27

28 8.1 Swarming So called swarming strategy Download chunks in parallel from a swarm of peers Swarming Advantages Peer failures: no loss of files, only chunks Discard unfinished chunks and download them new No complicated resume mechanism necessary Increased throughput Download chunks in parallel from different sources Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 28

29 8.1 Swarming Swarming Issues Chunk selection: in which order should chunks be requested from which peer Avoid scarcity Best overall availability? Fairness: how can the protocol ensure fair usage of bandwidth Avoid free-riding: all peers should contribute to the networks Bandwidth allocation: single pairs should not be overwhelmed by request while others are idling Systems implementing swarming BitTorrent Avalanche Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 29

30 8.2 BitTorrent BitTorrent Torrent = big stream Author: Bram Cohen, 2001 Protocol for swarming file distribution, no search features Designed for Protocol to quickly and decentrally distribute large content Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 30

31 8.2 BitTorrent Implements swarming strategy for content distribution Especially suited for flash crowds, i.e. content which is high in demand for a short period of time Central components Web server for search (torrent site) Classic web server maintains list of available content (so called torrents ) Provides search functionalities Content is represented as a torrent file containing required meta data e.g. address of the tracker Tracker for peer coordination A tracker is a centralized service maintaining the peer swarms i.e. which peers have which chunks of which torrents Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 31

32 8.2 BitTorrent Workflow for download A user uses a torrent site to obtain a torrent file Torrent file contains content meta data The user s node connects to the responsible tracker Tracker URL and content identifiers ( info hash ) are provided by torrent file Node registers itself with the tracker and the corresponding torrent i.e. joins the swarm Node obtains a list of all peers offering the torrent Contact some swarm peers Obtain a list of available chunks Download chunks from peers Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 32

33 8.2 BitTorrent Each offered file is split in chunks between 32KB and 4MB each Each chunk is identified by an 160-Bit SHA-Hash With respect to a certain torrent, each peer can fulfill one of the following roles Seeders Have all chunks of the torrent and are actively seeding (uploading) those chunks to the swarm Leechers Do not have all chunks of a torrent Download missing chunks from other leechers or seeders Upload chunks to other leecher There is no download-only role Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 33

34 8.2 BitTorrent Torrent File metadata structure Describes the files in the torrent URL of tracker File name File length Piece length SHA-1 hashes of pieces Allow peers to verify integrity Creation date An info hash is created from some fields of the torrent file This hash uniquely identifies a torrent Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 34

35 8.2 BitTorrent Fields being hashed in InfoHash Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 35

36 8.2 BitTorrent torrent site tracker new leecher peer other swarm peers torrent file register swarm peer list request info chunk information request chunk send chunk Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 36

37 8.2 BitTorrent :-) Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 37

38 8.2 BitTorrent: piece selection Which chunk next? Priority Actives Finish active chunks Rarest First Improves availability of rare chunks Delays download of common chunks Random First Piece Get first chunk quickly (rarest chunk probably slow to get) Endgame Mode Send requests for last chunks to all known peers End of download not stalled by slow peers Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 38

39 Game Theory Basic Ideas of Game Theory Game theory offers a general theory of strategic behavior Described in mathematical form Situations in which players may choose different actions to maximize their returns Situations in which strategic interactions among rational players produce outcomes with respect to the players preferences The outcomes might not have been intended by any of them Plays an important role in Modern economics Decision theory Multi-agent systems Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 39

40 Game Theory Early game theory tries to explain the optimal strategy in two-person interactions. von Neumann and Morgenstern, 1944 Initially: zero-sum games Expected utility hypothesis Players will rationally decide for the option with the highest expected outcome Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 40

41 Game Theory John Nash Worked in game theory and differential geometry Non-zero-sum games Nash equilibrium 1950 Strategic equilibrium in which no player gains any advantage when changing strategies (while knowing the opponents strategy) 1994 Nobel Prize in Economics Harsanyi and Selten Incomplete information Also 1994 Nobel Prize in Economics Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 41

42 Game Theory Games Situations are treated as games. Rules The rules of the game state which actions and decisions are possible Player's Strategies Plan for actions in each possible situation in the game Player's Payoffs A player s expected gain or loss when winning or loosing in a particular situation Dominant Strategy If players best strategy doesn t depend on what other players do Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 42

43 Game Theory Famous example: Prisoners Dilemma A and B are arrested by the police during a robbery They are interrogated in separate cells Unable to communicate with each other Following conditions are known If they both resist interrogation and proclaim their mutual innocence, they both will get off with a three year sentence for robbery If one of them confesses to the entire string of robberies and the other does not, the confessor will be rewarded with a light, one year sentence and while the other will get a severe eight year sentence If they both confess, then the judge will sentence both to a moderate four years in prison Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 43

44 Game Theory Prisoner Dilemma Possible outcomes A - Confession A - No Confession B - Confession 4 years each 8 years for A 1 year for B B - No Confession 1 year for A 8 years for B 3 years each Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 44

45 Game Theory Decision Tree of A A B Confesses B does not confess A: Confess A: Does Not Confess A: Confess A: Does Not Confess 4 Years in Prison 8 Years in Prison 1 Year in Prison 3 Years in Prison Best strategy The dominant strategy for A is to confess No matter what B does, confessing is better choice Nash equilibrium: both A and B will confess Also, dominant strategy of B is to confess Best strategy Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 45

46 Game Theory A repeated game Game that the same players play more than once Differ from one-shot games because people's current actions can depend on the past behavior of other players. Cooperation is encouraged Book recommendation Thinking strategically by A.Dixit and B Nalebuff German translation: Spieltheorie für Einsteiger Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 46

47 8.2 BitTorrent Game theory for designing a swarming protocol Swarming Game Decisions While seeding, who should get the chunks? While leeching, from whom to download chunks? Goal Available resources should be optimally used Free-riding should be prevented Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 47

48 8.2 BitTorrent Possible strategy in repeated games: Tit for Tat A player using this strategy will initially cooperate Player will adapt to opponent If the opponent previously was cooperative, the agent is cooperative. If not, the agent is not. Depends on four conditions Unless provoked, the agent will always cooperate If provoked, the agent will retaliate The agent is quick to forgive The agent must have a good chance of competing against the opponent more than once Get-to-know each other Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 48

49 8.2 BitTorrent BitTorrent uses a so-called choking mechanism for distributing chunks Basic idea Prefer uploading chunks to peers which also offered chunks for download i.e. aim for bi-directional communication channels Bi-directional communication will benefit the whole swarm most Tit-for-tat Punish peers which seem to be free-riding i.e. who only download but provide no upload Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 49

50 8.2 BitTorrent Choking as leecher Open a bi-directional transfer to another leecher Mutually exchange missing chunks If a peer does not upload any chunks for more than a minute, choke it: Temporary refuse to upload Downloading continues as usual TCP Connection is kept open No Setup costs TCP congestion control Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 50

51 8.2 BitTorrent Using only this choking mechanism may endanger the health of the swarm New leechers will automatically be choked because they cannot offer upload Two nodes which would have a good connection won t use it as no node takes initiative A node may be choked by all other nodes due to unlucky circumstances / network weaknesses Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 51

52 8.2 Solution: Optimistic Unchoking Randomly initiate a new connection to a currently unconnected leecher in the swarm and start uploading Take initiative Hope for a good cooperation e.g. that this new node provides a high upload rate in the bi-directional transfer Allows finding better peers Allows new peers to integrate themselves in the swarm Other peers voluntarily start uploading to them Randomly unchoke some currently choked connections Quick to forgive Helps locked-out peers to return to the swarm If a leecher is currently choked by all its peers, it initiates even more unchoking connection Anti-Snubbing Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 52

53 8.2 BitTorrent Open issue: upload-only choking Once a download is complete, no bi-directional transfer connections are required anymore by that peer Peer becomes a seeder Which nodes to upload to? Seeding Policy Upload to those peers with the best upload / download ratio Probable Advantages Ensures that chunks are replicated faster within the swarm Leechers that have a good upload rates are probably not being served by others But: Upload / Download ratio hard to determine Central bookkeeping? Bookkeeping own ratio? What about cheating? Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 53

54 8.2 BitTorrent Download chunks in parallel Look for the rarest pieces Verify each chunk by checking hash, download again if hash fails Advertise received pieces to all connected peers leecher B leecher A I have! seed leecher C Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 54

55 8.2 BitTorrent Periodically calculate data-receiving rates Upload to (unchoke) the fastest downloaders Optimistic unchoking Periodically select a peer at random and upload to it Continuously look for the fastest partners leecher B leecher A seed leecher D leecher C Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 55

56 8.2 Descentralized tracker Remember: Bit Torrent uses centralized trackers for managing peer lists Tracker Issues Single Point of Failure and Attack Scalability PirateBay tracker nearly overloaded (>5 Mio. Peers) Solution: Decentralized Tracker Replace with DHT (Kademlia) Does not tackle distributed search Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 56

57 Kademlia As a generic DHT protocol: Kademlia also uses a SHA Bit addressed ring hashing data and nodes Similar to Pastry, but uses a more sophisticated routing (tree-based) mechanism requiring less maintenance Each key-value pair is stored redundantly on multiple nodes. Every node maintains information about files, keywords close to itself. The closeness between two objects measured as their bitwise XOR interpreted as an integer distance(a, b) = a XOR b Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 57

58 Kademlia binary tree Subtrees of interest for a node 0011 Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 58

59 Kademlia node state For each i (0 i <160) every node keeps a list of nodes of distance between 2 i and 2 (i+1) from itself. Call each list a k-bucket. The list is sorted by time last seen. The value of k is chosen so that any given set of k nodes is unlikely to fail within an hour. The list is updated whenever a node receives a message. k = system-wide replication parameter, usually 20 head Least recenly seen Most recenly seen Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 59

60 8.2 BitTorrent Kademlia DHT Tracker Each torrent is identified by its infohash All BitTorrent nodes using an compatible client may join the DHT tracker Not part of the core BitTorrent protocol Authors of client usually also provide bootstrapping nodes to the DHT tracker The DHT takes over the trackers responsibility DHT Key-Value pairs Key: infohash Value: swarm peer listing Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 60

61 8.2 BitTorrent Peer Exchange (PEX) and Multi-Tracker To further increase the performance and resilience of a torrent, multiple trackers can be used Easiest case Torrent is registered with multiple trackers which are all explicitly specified in the torrent file More complex solution: Peer Exchange Start connecting to any one tracker Ask other connected peers in the swarm for additional peers and / or trackers Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 61

62 8.2 BitTorrent PEX Example Obtain a torrent and connect to tracker i.e. official Ubuntu torrent providing two trackers Obtain peer lists Additionally, connect to DHT tracker Obtain even more peers Peers using the same torrent but are trackerless or use DHT and another tracker Start peer exchange Again, obtain more peers i.e. receive peers not using DHT from a DHT node which is using, e.g. the openbittorrent-tracker Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 62

63 8.3 Anonymous P2P BitTorrent provides absolutely no anonymity Peer list of a torrent contain all participating nodes IP address, port, sometimes upload/download ratio, etc. Thus, it is very easy to identify all people downloading / uploading to a torrent User behavior can be tracked quite easily by simply introducing a spy node into the swarm Privacy implication Also, no pure downloading is feasible in BitTorrent as download-only nodes will be choked Possibly legal implication for copyrighted content Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 63

64 8.3 Anonymous P2P Real anonymity is hard to achieve Nodes need to communicate and contact each other Identities (addresses must be known) However, the number of nodes knowing a nodes identity can be limited to only trusted nodes So-called dark nets Basic idea Only connect to a few trusted friend nodes Never communicate directly with a non-friend Friends forward any message anonymously to their friends If network is designed correctly, most parts should be reachable via friend-of-a-friend routing Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 64

65 8.3 Anonymous P2P Two notable systems Freenet: Pure P2P network using small worldproperties and anonymous routing OneSwarm: BitTorrent extension based on friend-to-friend-routing Friend-To-Friend Routing e.g. B passes a message from A to C C does not know that message originated from A A does not know that B passes message to C A requester B C provider Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 65

66 8.3 Anonymous P2P If the request s time-to-live expires or a node does not have neighbors to send the file to, a backtracking request failed message is sent If the request is successful, the file is sent back via the routing nodes and each node saves the file and adds the sending node s address to its local routing table i.e., frequently requested files are replicated If the routing table is full, the least recently used (LRU) entry is evicted Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 66

67 Content Provisioning Swarming & BitTorrent Segment a file into multiple chunks Download chunks from multiple peers in parallel Seeder and leecher peers form a swarm Increased throughput Faster dissemination of new content i.e. for countering flash crowds Main question: which chunks should be downloaded / uploaded to best benefit the whole swarm? Prevent free-riding Discourage parasitic behavior Reward cooperation Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 67

68 Content Provisioning Solution: use concepts from game theory Tit-for-tat strategy Encourages strong bi-directional links among leecher Choking If a node in a bidirectional pipe is not cooperative (provides upload bandwidth), choke it by refusing further uploads to that peer Optimistic Unchoke Randomly unchoke some choked connections Take initiative and voluntarily upload to an unconnected node May discover better and more reliable partners Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 68

69 Next Lecture Load Balancing & Data Durability Data caching, replication, etc. P2P and Databases Building database link systems on top of P2P Toward cloud storage Distributed Data Management Profr. Dr. Wolf-Tilo Balke IfIS TU Braunschweig 69

Distributed Data Management. Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig

Distributed Data Management. Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig Distributed Data Management Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 8.0 Content Provisioning 8.0 Content Distribution 8.1 Swarming

More information

Distributed Data Management

Distributed Data Management Distributed Data Management Wolf-Tilo Balke Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 8.0 Content Provisioning 8.0 Content Distribution

More information

P2P Content Distribution

P2P Content Distribution , University of Hannover Wolf-Tilo Balke and Wolf Siberski 28.11.2007 *With slides from K. Wehrle (RWTH), A. Bharambe (CMU), P. Rodriguez, P. Chou (MSRC), Chiu (CUHK) Peer-to-Peer Systems and Applications,

More information

Content Distribution and BitTorrent [Based on slides by Cosmin Arad]

Content Distribution and BitTorrent [Based on slides by Cosmin Arad] ID2210 - Distributed Computing, Peer-to-Peer and GRIDS Content Distribution and BitTorrent [Based on slides by Cosmin Arad] Today The problem of content distribution A popular solution: BitTorrent Underlying

More information

Lecture 17: Peer-to-Peer System and BitTorrent

Lecture 17: Peer-to-Peer System and BitTorrent CSCI-351 Data communication and Networks Lecture 17: Peer-to-Peer System and BitTorrent (I swear I only use it for Linux ISOs) The slide is built with the help of Prof. Alan Mislove, Christo Wilson, and

More information

Extreme Computing. BitTorrent and incentive-based overlay networks.

Extreme Computing. BitTorrent and incentive-based overlay networks. Extreme Computing BitTorrent and incentive-based overlay networks BitTorrent Today we will focus on BitTorrent The technology really has three aspects A standard that BitTorrent client systems follow Some

More information

BitTorrent. Masood Khosroshahy. July Tech. Report. Copyright 2009 Masood Khosroshahy, All rights reserved.

BitTorrent. Masood Khosroshahy. July Tech. Report. Copyright 2009 Masood Khosroshahy, All rights reserved. BitTorrent Masood Khosroshahy July 2009 Tech. Report Copyright 2009 Masood Khosroshahy, All rights reserved. www.masoodkh.com Contents Contents 1 Basic Concepts 1 2 Mechanics 3 2.1 Protocols: Tracker and

More information

Content Overlays (continued) Nick Feamster CS 7260 March 26, 2007

Content Overlays (continued) Nick Feamster CS 7260 March 26, 2007 Content Overlays (continued) Nick Feamster CS 7260 March 26, 2007 Administrivia Quiz date Remaining lectures Interim report PS 3 Out Friday, 1-2 problems 2 Structured vs. Unstructured Overlays Structured

More information

Distributed Data Management. Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig

Distributed Data Management. Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig Distributed Data Management Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 7.0 Network Models 7.0 Introduction 7.1 Graph Model Basics 7.2

More information

Distributed Data Management. Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig

Distributed Data Management. Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig Distributed Data Management Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 6.4 Other DHTs: CAN CAN is another early DHT implementation S.

More information

Distributed Data Management

Distributed Data Management Distributed Data Management Christoph Lofi José Pinto Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 7.0 Network Models 7.0 Introduction 7.1 Graph Model

More information

Peer-to-Peer Data Management

Peer-to-Peer Data Management Peer-to-Peer Data Management Wolf-Tilo Balke Sascha Tönnies Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 10. Networkmodels 1. Introduction Motivation

More information

P2P Applications. Reti di Elaboratori Corso di Laurea in Informatica Università degli Studi di Roma La Sapienza Canale A-L Prof.ssa Chiara Petrioli

P2P Applications. Reti di Elaboratori Corso di Laurea in Informatica Università degli Studi di Roma La Sapienza Canale A-L Prof.ssa Chiara Petrioli P2P Applications Reti di Elaboratori Corso di Laurea in Informatica Università degli Studi di Roma La Sapienza Canale A-L Prof.ssa Chiara Petrioli Server-based Network Peer-to-peer networks A type of network

More information

internet technologies and standards

internet technologies and standards Institute of Telecommunications Warsaw University of Technology 25 internet technologies and standards Piotr Gajowniczek Andrzej Bąk Michał Jarociński Internet application layer peer-to-peer systems overview

More information

BitTorrent Optimization Techniques. (from various online sources)

BitTorrent Optimization Techniques. (from various online sources) BitTorrent Optimization Techniques (from various online sources) Announcement No recitation next week! Final review session Next Sunday (5/2) 5-7pm, GHC 4215 Let us know what you want at http://www.doodle.com/6qvsnubhmam2zkxp

More information

CompSci 356: Computer Network Architectures Lecture 21: Overlay Networks Chap 9.4. Xiaowei Yang

CompSci 356: Computer Network Architectures Lecture 21: Overlay Networks Chap 9.4. Xiaowei Yang CompSci 356: Computer Network Architectures Lecture 21: Overlay Networks Chap 9.4 Xiaowei Yang xwy@cs.duke.edu Overview Problem Evolving solutions IP multicast Proxy caching Content distribution networks

More information

Peer-to-Peer Applications Reading: 9.4

Peer-to-Peer Applications Reading: 9.4 Peer-to-Peer Applications Reading: 9.4 Acknowledgments: Lecture slides are from Computer networks course thought by Jennifer Rexford at Princeton University. When slides are obtained from other sources,

More information

Department of Computer Science Institute for System Architecture, Chair for Computer Networks. File Sharing

Department of Computer Science Institute for System Architecture, Chair for Computer Networks. File Sharing Department of Computer Science Institute for System Architecture, Chair for Computer Networks File Sharing What is file sharing? File sharing is the practice of making files available for other users to

More information

Do incentives build robustness in BitTorrent?

Do incentives build robustness in BitTorrent? Do incentives build robustness in BitTorrent? ronghui.gu@yale.edu Agenda 2 Introduction BitTorrent Overview Modeling Altruism in BitTorrent Building BitTyrant Evaluation Conclusion Introduction 3 MAIN

More information

CSE 486/586 Distributed Systems Peer-to-Peer Architectures

CSE 486/586 Distributed Systems Peer-to-Peer Architectures CSE 486/586 Distributed Systems eer-to-eer Architectures Steve Ko Computer Sciences and Engineering University at Buffalo CSE 486/586 Last Time Gossiping Multicast Failure detection Today s Question How

More information

Introduction to P P Networks

Introduction to P P Networks Introduction to P P Networks B Sc Florian Adamsky florianadamsky@iemthmde http://florianadamskyit/ cbd Internet Protocols and Applications SS B Sc Florian Adamsky IPA / Outline Introduction What is P P?

More information

6. Overview. L3S Research Center, University of Hannover. 6.1 Section Motivation. Investigation of structural aspects of peer-to-peer networks

6. Overview. L3S Research Center, University of Hannover. 6.1 Section Motivation. Investigation of structural aspects of peer-to-peer networks , University of Hannover Random Graphs, Small-Worlds, and Scale-Free Networks Wolf-Tilo Balke and Wolf Siberski 05.12.07 * Original slides provided by K.A. Lehmann (University Tübingen, Germany) 6. Overview

More information

Lecture 8: Application Layer P2P Applications and DHTs

Lecture 8: Application Layer P2P Applications and DHTs Lecture 8: Application Layer P2P Applications and DHTs COMP 332, Spring 2018 Victoria Manfredi Acknowledgements: materials adapted from Computer Networking: A Top Down Approach 7 th edition: 1996-2016,

More information

P2P Applications. Reti di Elaboratori Corso di Laurea in Informatica Università degli Studi di Roma La Sapienza

P2P Applications. Reti di Elaboratori Corso di Laurea in Informatica Università degli Studi di Roma La Sapienza P2P Applications Reti di Elaboratori Corso di Laurea in Informatica Università degli Studi di Roma La Sapienza Versione originale delle slides fornita da Dora Spenza e Marco Barbera P2P Paradigm Late 80

More information

CS5412: TORRENTS AND TIT-FOR-TAT

CS5412: TORRENTS AND TIT-FOR-TAT 1 CS5412: TORRENTS AND TIT-FOR-TAT Lecture VII Ken Birman BitTorrent 2 Widely used download technology Implementations specialized for setting Some focus on P2P downloads, e.g. patches Others focus on

More information

DISTRIBUTED SYSTEMS CSCI 4963/ /4/2015

DISTRIBUTED SYSTEMS CSCI 4963/ /4/2015 1 DISTRIBUTED SYSTEMS CSCI 4963/6963 12/4/2015 2 Info Quiz 7 on Tuesday. Project 2 submission URL is posted on the web site Submit your source code and project report (PDF!!!) in a single zip file. If

More information

P2P content distribution Jukka K. Nurminen

P2P content distribution Jukka K. Nurminen P2P content distribution Jukka K. Nurminen 1 V1-Filename.ppt / yyyy-mm-dd / Initials BitTorrent content downloading Efficient content distribution Bram Cohen, 2001 File divided into pieces Each recipient

More information

BitTorrent from swarms to collectives

BitTorrent from swarms to collectives BitTorrent from swarms to collectives Some on-going research in the tribler team at TU Delft name: David Hales date: Sept. 21, 2009 event: ECCS 2009 Tribler team: 35 + collaborators tribler.org Tribler

More information

Game Theory. Presented by Hakim Weatherspoon

Game Theory. Presented by Hakim Weatherspoon Game Theory Presented by Hakim Weatherspoon Game Theory Main Question: Can we cheat (and get away with it)? BitTorrent P2P file distribution tool designed with incentives for contribution Users contribute

More information

Last Lecture SMTP. SUNY at Buffalo; CSE 489/589 Modern Networking Concepts; Fall 2010; Instructor: Hung Q. Ngo 1

Last Lecture SMTP. SUNY at Buffalo; CSE 489/589 Modern Networking Concepts; Fall 2010; Instructor: Hung Q. Ngo 1 Last Lecture SMTP SUNY at Buffalo; CSE 489/589 Modern Networking Concepts; Fall 2010; Instructor: Hung Q. Ngo 1 This Lecture Peer-to-Peer (P2P) Applications SUNY at Buffalo; CSE 489/589 Modern Networking

More information

Scalability of the BitTorrent P2P Application

Scalability of the BitTorrent P2P Application Scalability of the BitTorrent P2P Application Kolja Eger, Ulrich Killat Hamburg University of Technology 5.Würzburger Workshop 8.-9. July 2005 Overview File dissemination in peer-to-peer (p2p) networks

More information

A Case For OneSwarm. Tom Anderson University of Washington.

A Case For OneSwarm. Tom Anderson University of Washington. A Case For OneSwarm Tom Anderson University of Washington http://oneswarm.cs.washington.edu/ With: Jarret Falkner, Tomas Isdal, Alex Jaffe, John P. John, Arvind Krishnamurthy, Harsha Madhyastha and Mike

More information

Peer to Peer Networks

Peer to Peer Networks Sungkyunkwan University Peer to Peer Networks Prepared by T. Le-Duc and H. Choo Copyright 2000-2018 Networking Laboratory P2P Applications Traditional P2P applications: for file sharing BitTorrent, Emule

More information

Peer-to-Peer Networks 12 Fast Download

Peer-to-Peer Networks 12 Fast Download Peer-to-Peer Networks 12 Fast Download Arne Vater Technical Faculty Computer Networks and Telematics University of Freiburg IP Multicast Motivation - Transmission of a data stream to many receivers Unicast

More information

Peer to Peer Networks

Peer to Peer Networks Sungkyunkwan University Peer to Peer Networks Prepared by T. Le-Duc and H. Choo Copyright 2000-2017 Networking Laboratory Presentation Outline 2.1 Introduction 2.2 Client-Server Paradigm 2.3 Peer-To-Peer

More information

Reti P2P per file distribution: BitTorrent

Reti P2P per file distribution: BitTorrent Reti P2P per file distribution: BitTorrent Corso di Applicazioni Telematiche A.A. 2008-09 Lezione n.22 Prof. Roberto Canonico Università degli Studi di Napoli Federico II Facoltà di Ingegneria P2p file-sharing

More information

CS 3516: Advanced Computer Networks

CS 3516: Advanced Computer Networks Welcome to CS 3516: Advanced Computer Networks Prof. Yanhua Li Time: 9:00am 9:50am M, T, R, and F Location: Fuller 320 Fall 2017 A-term 1 Some slides are originally from the course materials of the textbook

More information

BitTorrent. Internet Technologies and Applications

BitTorrent. Internet Technologies and Applications BitTorrent Internet Technologies and Applications Aims and Contents Aims Show benefits of P2P for file downloads Explain concepts of BitTorrent as a P2P file sharing application Show how BitTorrent protocol

More information

416 Distributed Systems. Mar 3, Peer-to-Peer Part 2

416 Distributed Systems. Mar 3, Peer-to-Peer Part 2 416 Distributed Systems Mar 3, Peer-to-Peer Part 2 Scaling Problem Millions of clients server and network meltdown 2 P2P System Leverage the resources of client machines (peers) Traditional: Computation,

More information

Peer to Peer Systems and Probabilistic Protocols

Peer to Peer Systems and Probabilistic Protocols Distributed Systems 600.437 Peer to Peer Systems & Probabilistic Protocols Department of Computer Science The Johns Hopkins University 1 Peer to Peer Systems and Probabilistic Protocols Lecture 11 Good

More information

γ : constant Goett 2 P(k) = k γ k : degree

γ : constant Goett 2 P(k) = k γ k : degree Goett 1 Jeffrey Goett Final Research Paper, Fall 2003 Professor Madey 19 December 2003 Abstract: Recent observations by physicists have lead to new theories about the mechanisms controlling the growth

More information

P2P content distribution

P2P content distribution P2P content distribution T-110.7100 Applications and Services in Internet, Fall 2010 Jukka K. Nurminen 1 V1-Filename.ppt / yyyy-mm-dd / Initials Steps of content sharing Share content Find content Transfer

More information

IMPLEMENTATION AND EVALUATION OF THE MULTICAST FILE TRANSFER PROTOCOL (MCFTP)

IMPLEMENTATION AND EVALUATION OF THE MULTICAST FILE TRANSFER PROTOCOL (MCFTP) IMPLEMENTATION AND EVALUATION OF THE MULTICAST FILE TRANSFER PROTOCOL (MCFTP) Masterarbeit der Philosophisch-naturwissenschaftlichen Fakultät der Universität Bern vorgelegt von Alican Gecyasar Dezember,

More information

Distributed Systems. peer-to-peer Johan Montelius ID2201. Distributed Systems ID2201

Distributed Systems. peer-to-peer Johan Montelius ID2201. Distributed Systems ID2201 Distributed Systems ID2201 peer-to-peer Johan Montelius 1 Idéa use resources in edge of network computing storage communication 2 Computing 3 seti@home central server millions of clients hundred of thousands

More information

P2P. 1 Introduction. 2 Napster. Alex S. 2.1 Client/Server. 2.2 Problems

P2P. 1 Introduction. 2 Napster. Alex S. 2.1 Client/Server. 2.2 Problems P2P Alex S. 1 Introduction The systems we will examine are known as Peer-To-Peer, or P2P systems, meaning that in the network, the primary mode of communication is between equally capable peers. Basically

More information

Advanced Distributed Systems. Peer to peer systems. Reference. Reference. What is P2P? Unstructured P2P Systems Structured P2P Systems

Advanced Distributed Systems. Peer to peer systems. Reference. Reference. What is P2P? Unstructured P2P Systems Structured P2P Systems Advanced Distributed Systems Peer to peer systems Karl M. Göschka Karl.Goeschka@tuwien.ac.at http://www.infosys.tuwien.ac.at/teaching/courses/ AdvancedDistributedSystems/ What is P2P Unstructured P2P Systems

More information

BitTorrent and CoolStreaming

BitTorrent and CoolStreaming BitTorrent and CoolStreaming Jukka K. Nurminen Data Communications Software (DCS) Lab, Department of Computer Science and Engineering, Aalto University Jukka K. Nurminen Aalto University P2P Networks BitTorrent

More information

Boosting Market Liquidity of Peer-to-Peer Systems Through Cyclic Trading

Boosting Market Liquidity of Peer-to-Peer Systems Through Cyclic Trading Boosting Market Liquidity of Peer-to-Peer Systems Through Cyclic Trading Raphael Eidenbenz Thomas Locher Stefan Schmid Raphael+Roger: ETH Zurich DCG Thomas: ABB Research Roger Wattenhofer T-Labs Stefan:

More information

inria , version 1-6 Sep 2006

inria , version 1-6 Sep 2006 Rarest First and Choke Algorithms Are Enough Arnaud Legout I.N.R.I.A. Sophia Antipolis France arnaud.legout@sophia.inria.fr G. Urvoy-Keller and P. Michiardi Institut Eurecom Sophia Antipolis France {Guillaume.Urvoy,Pietro.Michiardi}@eurecom.fr

More information

Lecture 21 P2P. Napster. Centralized Index. Napster. Gnutella. Peer-to-Peer Model March 16, Overview:

Lecture 21 P2P. Napster. Centralized Index. Napster. Gnutella. Peer-to-Peer Model March 16, Overview: PP Lecture 1 Peer-to-Peer Model March 16, 005 Overview: centralized database: Napster query flooding: Gnutella intelligent query flooding: KaZaA swarming: BitTorrent unstructured overlay routing: Freenet

More information

Middleware and Distributed Systems. Peer-to-Peer Systems. Peter Tröger

Middleware and Distributed Systems. Peer-to-Peer Systems. Peter Tröger Middleware and Distributed Systems Peer-to-Peer Systems Peter Tröger Peer-to-Peer Systems (P2P) Concept of a decentralized large-scale distributed system Large number of networked computers (peers) Each

More information

Cooperation in Open Distributed Systems. Stefan Schmid

Cooperation in Open Distributed Systems. Stefan Schmid Cooperation in Open Distributed Systems Stefan Schmid T-Labs, Berlin, July 2, 2009 Distributed Systems 2008/9 Wireless: Many mobile phones today have WLAN (and even Skype) P2P: Olympic games 2008 live-broadcast

More information

Traffic Patterns in Peer-to-Peer-Networking. Christian Schindelhauer. joint work with Amir Alsbih Thomas Janson

Traffic Patterns in Peer-to-Peer-Networking. Christian Schindelhauer. joint work with Amir Alsbih Thomas Janson Traffic Patterns in Peer-to-Peer-Networking Christian Schindelhauer joint work with Amir Alsbih Thomas Janson to be presented at ITA Albert-Ludwig University Freiburg Department of Computer Science Computer

More information

Scalable overlay Networks

Scalable overlay Networks overlay Networks Dr. Samu Varjonen 1 Lectures MO 15.01. C122 Introduction. Exercises. Motivation. TH 18.01. DK117 Unstructured networks I MO 22.01. C122 Unstructured networks II TH 25.01. DK117 Bittorrent

More information

INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY

INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY A PATH FOR HORIZING YOUR INNOVATIVE WORK PEER-TO-PEER FILE SHARING WITH THE BITTORRENT PROTOCOL APURWA D. PALIWAL 1, PROF.

More information

Lesson 9 Applications of DHT: Bittorrent Mainline DHT, the KAD network

Lesson 9 Applications of DHT: Bittorrent Mainline DHT, the KAD network Lesson 9 Applications of DHT: Bittorrent Mainline DHT, the KAD network 20/03/2017 1 IMPLEMENTATION OF KADEMLIA Kademlia is the protocol used by the largest public DHTs. Vuze ( before known as Azureus),

More information

Modeling and Simulating Social Systems with MATLAB

Modeling and Simulating Social Systems with MATLAB Modeling and Simulating Social Systems with MATLAB Lecture 7 Game Theory / Agent-Based Modeling Stefano Balietti, Olivia Woolley, Lloyd Sanders, Dirk Helbing Computational Social Science ETH Zürich 02-11-2015

More information

Distributed Data Management. Profr. Dr. Wolf-Tilo Balke Institut für Informationssysteme Technische Universität Braunschweig

Distributed Data Management. Profr. Dr. Wolf-Tilo Balke Institut für Informationssysteme Technische Universität Braunschweig Distributed Data Management Profr. Dr. Wolf-Tilo Balke Institut für Informationssysteme Technische Universität Braunschweig 6 Structured P2P Networks 6.1 Hash Tables 6.2 Distributed Hash Tables 6.3 CHORD

More information

Overview. Peer-to-Peer Systems and Security IN2194. Chapter 1 Peer-to-Peer Systems 1.2 Unstructured Networks. Unstructured networks

Overview. Peer-to-Peer Systems and Security IN2194. Chapter 1 Peer-to-Peer Systems 1.2 Unstructured Networks. Unstructured networks Chair for Network Architectures and Services Department of Informatics TU München Prof. Carle Overview Peer-to-Peer Systems and Security IN2194 Chapter 1 Peer-to-Peer Systems 1.2 Unstructured Networks

More information

CS 43: Computer Networks BitTorrent & Content Distribution. Kevin Webb Swarthmore College September 28, 2017

CS 43: Computer Networks BitTorrent & Content Distribution. Kevin Webb Swarthmore College September 28, 2017 CS 43: Computer Networks BitTorrent & Content Distribution Kevin Webb Swarthmore College September 28, 2017 Agenda BitTorrent Cooperative file transfers Briefly: Distributed Hash Tables Finding things

More information

Application Layer: P2P File Distribution

Application Layer: P2P File Distribution Application Layer: P2P File Distribution EECS 3214 Slides courtesy of J.F Kurose and K.W. Ross, All Rights Reserved 29-Jan-18 1-1 Chapter 2: outline 2.1 principles of network applications 2.2 Web and HTTP

More information

Introduction to Distributed Computing Systems

Introduction to Distributed Computing Systems Introduction to Distributed Computing Systems Alexandru Iosup Parallel and Distributed Systems Group Delft University of Technology Some material kindly provided by the PDS group. 1 What is a Distributed

More information

Chapter 2: Application layer

Chapter 2: Application layer Chapter 2: Application layer 2. Principles of network applications app architectures app requirements 2.2 Web and HTTP 2.4 Electronic Mail SMTP, POP3, IMAP 2.5 DNS 2.6 P2P applications 2.7 Socket programming

More information

CS 3516: Computer Networks

CS 3516: Computer Networks Welcome to CS 3516: Computer Networks Prof. Yanhua Li Time: 9:00am 9:50am M, T, R, and F Location: AK219 Fall 2018 A-term 1 Some slides are originally from the course materials of the textbook Computer

More information

Overlay and P2P Networks. Unstructured networks. Prof. Sasu Tarkoma

Overlay and P2P Networks. Unstructured networks. Prof. Sasu Tarkoma Overlay and P2P Networks Unstructured networks Prof. Sasu Tarkoma 20.1.2014 Contents P2P index revisited Unstructured networks Gnutella Bloom filters BitTorrent Freenet Summary of unstructured networks

More information

BitTorrent for Really Selfish Peers T4T Trading over Cycles

BitTorrent for Really Selfish Peers T4T Trading over Cycles Semester Thesis BitTorrent for Really Selfish Peers T4T Trading over Cycles Roger Odermatt oroger@student.ethz.ch Professor: Dr. Roger Wattenhofer Advisor: Raphael Eidenbenz Department of Information Technology

More information

08 Distributed Hash Tables

08 Distributed Hash Tables 08 Distributed Hash Tables 2/59 Chord Lookup Algorithm Properties Interface: lookup(key) IP address Efficient: O(log N) messages per lookup N is the total number of servers Scalable: O(log N) state per

More information

CSCI-1680 P2P Rodrigo Fonseca

CSCI-1680 P2P Rodrigo Fonseca CSCI-1680 P2P Rodrigo Fonseca Based partly on lecture notes by Ion Stoica, Sco5 Shenker, Joe Hellerstein Today Overlay networks and Peer-to-Peer Motivation Suppose you want to write a routing protocol

More information

CMSC 332 Computer Networks P2P and Sockets

CMSC 332 Computer Networks P2P and Sockets CMSC 332 Computer Networks P2P and Sockets Professor Szajda Announcements Programming Assignment 1 is due Thursday Where are we? What sorts of problems are we having? 2 Recap SMTP is the language that

More information

The Design Trade-offs of BitTorrent-like File Sharing Protocols

The Design Trade-offs of BitTorrent-like File Sharing Protocols The Design Trade-offs of BitTorrent-like File Sharing Protocols Bin Fan John C.S. Lui Dah-Ming Chiu Abstract The BitTorrent BT file sharing protocol is very popular due to its scalability property and

More information

Application-Layer Protocols Peer-to-Peer Systems, Media Streaming & Content Delivery Networks

Application-Layer Protocols Peer-to-Peer Systems, Media Streaming & Content Delivery Networks COMP 431 Internet Services & Protocols Application-Layer Protocols Peer-to-Peer Systems, Media Streaming & Content Delivery Networks Jasleen Kaur February 14, 2019 Application-Layer Protocols Outline Example

More information

15-744: Computer Networking P2P/DHT

15-744: Computer Networking P2P/DHT 15-744: Computer Networking P2P/DHT Overview P2P Lookup Overview Centralized/Flooded Lookups Routed Lookups Chord Comparison of DHTs 2 Peer-to-Peer Networks Typically each member stores/provides access

More information

Peer-to-Peer (P2P) Systems

Peer-to-Peer (P2P) Systems Peer-to-Peer (P2P) Systems What Does Peer-to-Peer Mean? A generic name for systems in which peers communicate directly and not through a server Characteristics: decentralized self-organizing distributed

More information

Performance Analysis and Improvement for BitTorrent-like File Sharing Systems

Performance Analysis and Improvement for BitTorrent-like File Sharing Systems CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE Concurrency Computat.: Pract. Exper. 2007; 00:1 7 [Version: 2002/09/19 v2.02] Performance Analysis and Improvement for BitTorrent-like File Sharing

More information

Design Space Analysis for Modeling Incentives in Distributed Systems

Design Space Analysis for Modeling Incentives in Distributed Systems Design Space Analysis for Modeling Incentives in Distributed Systems by Rameez Rahman, Tamas Vinko, David Hales, Johan Pouwelse, and Henk Sips Delft University of Technology 1 Incentives in Distributed

More information

6 Structured P2P Networks

6 Structured P2P Networks 6 Structured P2P Networks Distributed Data Management Wolf-Tilo Balke Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 6.1 Hash Tables 6.3

More information

Peer-to-Peer Systems and Distributed Hash Tables

Peer-to-Peer Systems and Distributed Hash Tables Peer-to-Peer Systems and Distributed Hash Tables CS 240: Computing Systems and Concurrency Lecture 8 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Selected

More information

Cooperative End-to-end content distribution. Márk Jelasity

Cooperative End-to-end content distribution. Márk Jelasity Cooperative End-to-end content distribution Márk Jelasity Content distribution So far we looked at search Content distribution is about allowing clients (peers) to actually get a file or other data after

More information

(Social) Networks Analysis III. Prof. Dr. Daning Hu Department of Informatics University of Zurich

(Social) Networks Analysis III. Prof. Dr. Daning Hu Department of Informatics University of Zurich (Social) Networks Analysis III Prof. Dr. Daning Hu Department of Informatics University of Zurich Outline Network Topological Analysis Network Models Random Networks Small-World Networks Scale-Free Networks

More information

Scalable overlay Networks

Scalable overlay Networks overlay Networks Dr. Samu Varjonen 1 Lectures MO 15.01. C122 Introduction. Exercises. Motivation. TH 18.01. DK117 Unstructured networks I MO 22.01. C122 Unstructured networks II TH 25.01. DK117 Bittorrent

More information

5. Bi&orrent and coopera0on

5. Bi&orrent and coopera0on 5. Bi&orrent and coopera0on Modelling Social Interac0on in Informa0on systems h&p://davidhales.com/msiis David Hales, University of Szeged dave@davidhales.com Quick recap of previous lecture on the evolu0on

More information

Further extensions to the Wigan architecture are discussed in this thesis, including possible means of handling data updates.

Further extensions to the Wigan architecture are discussed in this thesis, including possible means of handling data updates. COMPUTING SCIENCE A BitTorrent-based Peer-to-Peer Database Server J. Colquhoun TECHNICAL REPORT SERIES No. CS-TR-1135 January, 2009 TECHNICAL REPORT SERIES No. CS-TR-1135 January, 2009 A BitTorrent-based

More information

Attacks and exploits targeting BitTorrent and other P2P file sharing networks

Attacks and exploits targeting BitTorrent and other P2P file sharing networks Attacks and exploits targeting BitTorrent and other P2P file sharing networks Andreas Hegenberg Betreuer: Benedikt Elser Seminar Future Internet WS 09/10 Lehrstuhl Netzarchitekturen und Netzdienste Fakultät

More information

Today. Why might P2P be a win? What is a Peer-to-Peer (P2P) system? Peer-to-Peer Systems and Distributed Hash Tables

Today. Why might P2P be a win? What is a Peer-to-Peer (P2P) system? Peer-to-Peer Systems and Distributed Hash Tables Peer-to-Peer Systems and Distributed Hash Tables COS 418: Distributed Systems Lecture 7 Today 1. Peer-to-Peer Systems Napster, Gnutella, BitTorrent, challenges 2. Distributed Hash Tables 3. The Chord Lookup

More information

Flooded Queries (Gnutella) Centralized Lookup (Napster) Routed Queries (Freenet, Chord, etc.) Overview N 2 N 1 N 3 N 4 N 8 N 9 N N 7 N 6 N 9

Flooded Queries (Gnutella) Centralized Lookup (Napster) Routed Queries (Freenet, Chord, etc.) Overview N 2 N 1 N 3 N 4 N 8 N 9 N N 7 N 6 N 9 Peer-to-Peer Networks -: Computer Networking L-: PP Typically each member stores/provides access to content Has quickly grown in popularity Bulk of traffic from/to CMU is Kazaa! Basically a replication

More information

Peer-peer and Application-level Networking. CS 218 Fall 2003

Peer-peer and Application-level Networking. CS 218 Fall 2003 Peer-peer and Application-level Networking CS 218 Fall 2003 Multicast Overlays P2P applications Napster, Gnutella, Robust Overlay Networks Distributed Hash Tables (DHT) Chord CAN Much of this material

More information

INVESTIGATING AND IMPROVING BITTORRENT S PIECE AND NEIGHBOR SELECTION ALGORITHMS

INVESTIGATING AND IMPROVING BITTORRENT S PIECE AND NEIGHBOR SELECTION ALGORITHMS INVESTIGATING AND IMPROVING BITTORRENT S PIECE AND NEIGHBOR SELECTION ALGORITHMS by Cameron Dale B.Sc., Simon Fraser University, 2000 a thesis submitted in partial fulfillment of the requirements for the

More information

Lesson 4. Random graphs. Sergio Barbarossa. UPC - Barcelona - July 2008

Lesson 4. Random graphs. Sergio Barbarossa. UPC - Barcelona - July 2008 Lesson 4 Random graphs Sergio Barbarossa Graph models 1. Uncorrelated random graph (Erdős, Rényi) N nodes are connected through n edges which are chosen randomly from the possible configurations 2. Binomial

More information

Problems in Reputation based Methods in P2P Networks

Problems in Reputation based Methods in P2P Networks WDS'08 Proceedings of Contributed Papers, Part I, 235 239, 2008. ISBN 978-80-7378-065-4 MATFYZPRESS Problems in Reputation based Methods in P2P Networks M. Novotný Charles University, Faculty of Mathematics

More information

Page 1. How Did it Start?" Model" Main Challenge" CS162 Operating Systems and Systems Programming Lecture 24. Peer-to-Peer Networks"

Page 1. How Did it Start? Model Main Challenge CS162 Operating Systems and Systems Programming Lecture 24. Peer-to-Peer Networks How Did it Start?" CS162 Operating Systems and Systems Programming Lecture 24 Peer-to-Peer Networks" A killer application: Napster (1999) Free music over the Internet Key idea: share the storage and bandwidth

More information

Exercise set #2 (29 pts)

Exercise set #2 (29 pts) (29 pts) The deadline for handing in your solutions is Nov 16th 2015 07:00. Return your solutions (one.pdf le and one.zip le containing Python code) via e- mail to Becs-114.4150@aalto.fi. Additionally,

More information

LECT-01, S-1 IAD99S, Javed I.

LECT-01, S-1 IAD99S, Javed I. A Course on Foundations of Peer-to-Peer Systems & Applications LECT-01, S-1 CS 6/75995 Foundation of Peer-to-Peer Applications & Systems Kent State University Dept. of Computer Science www.cs.kent.edu/~javed/class-p2p08/

More information

Some Peers Are More Equal than Others! Stefan Schmid

Some Peers Are More Equal than Others! Stefan Schmid Some Peers Are More Equal than Others! Stefan Schmid Telefónica, March 12, 2009 Peer-to-Peer Technology Well-known p2p systems - Internet telephony: Skype - File sharing: BitTorrent, emule,... - Streaming:

More information

Improving Performance of BitTorrent Network through Incentive Mechanism

Improving Performance of BitTorrent Network through Incentive Mechanism San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Summer 2011 Improving Performance of BitTorrent Network through Incentive Mechanism Mingzhe Li San Jose

More information

Telematics Chapter 9: Peer-to-Peer Networks

Telematics Chapter 9: Peer-to-Peer Networks Telematics Chapter 9: Peer-to-Peer Networks Beispielbild User watching video clip Server with video clips Application Layer Presentation Layer Application Layer Presentation Layer Session Layer Session

More information

Peer-to-Peer Systems and Security IN2194. Chapter 1 Peer-to-Peer Systems 1.2 Unstructured Systems

Peer-to-Peer Systems and Security IN2194. Chapter 1 Peer-to-Peer Systems 1.2 Unstructured Systems Chair for Network Architectures and Services Department of Informatics TU München Prof. Carle -to- Systems and Security IN2194 Chapter 1 -to- Systems 1.2 Unstructured Systems Dipl.-Inform. Heiko Niedermayer

More information

Overlay and P2P Networks. Unstructured networks. PhD. Samu Varjonen

Overlay and P2P Networks. Unstructured networks. PhD. Samu Varjonen Overlay and P2P Networks Unstructured networks PhD. Samu Varjonen 25.1.2016 Contents Unstructured networks Last week Napster Skype This week: Gnutella BitTorrent P2P Index It is crucial to be able to find

More information

Overview Computer Networking Lecture 16: Delivering Content: Peer to Peer and CDNs Peter Steenkiste

Overview Computer Networking Lecture 16: Delivering Content: Peer to Peer and CDNs Peter Steenkiste Overview 5-44 5-44 Computer Networking 5-64 Lecture 6: Delivering Content: Peer to Peer and CDNs Peter Steenkiste Web Consistent hashing Peer-to-peer Motivation Architectures Discussion CDN Video Fall

More information

Resilient Networking. 6: Resilient Peer-to-Peer Networking Thorsten Strufe. What is P2P? Types of P2P Systems P2P Security

Resilient Networking. 6: Resilient Peer-to-Peer Networking Thorsten Strufe. What is P2P? Types of P2P Systems P2P Security 6: Resilient Peer-to-Peer Networking Thorsten Strufe What is P2P? Types of P2P Systems P2P Security How do you define peer-to-peer? 2 What do you consider to be the main problems? 3 What do you think are

More information

WIRELESS ACCESS IN VEHICULAR ENVIRONMENTS USING BIT TORRENT AND BARGAINING

WIRELESS ACCESS IN VEHICULAR ENVIRONMENTS USING BIT TORRENT AND BARGAINING WIRELESS ACCESS IN VEHICULAR ENVIRONMENTS USING BIT TORRENT AND BARGAINING by Barsha Shrestha A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Electrical

More information