DISTRIBUTED SYSTEMS CSCI 4963/ /4/2015

Transcription

1 1 DISTRIBUTED SYSTEMS CSCI 4963/ /4/2015

2 2 Info Quiz 7 on Tuesday. Project 2 submission URL is posted on the web site Submit your source code and project report (PDF!!!) in a single zip file. If you haven t done so already, me your top 3 choices for demo timeslots today.

3 P2P File Sharing 3

4 4 Review from Last Time Napster Central replicated index servers used for file search File downloaded directly from peer Gnutella Queries for files propagated through networks in the Gnutella network overlay, File downloaded directly from peer Kazaa / FastTrack Network of super-peers store indexes of files File downloaded directly from peer Challenge with all of these freeloading A peer can download files from the network without uploading any files.

5 5 Hash Functions A hash function is a function that maps input data of any size to a fixed size. Desirable properties of a hash function 1. Determinism: a given input always generates the same output 2. Uniformity: map the expected inputs uniformly to the outputs

6 6 Cryptographic Hash Functions A cryptographic hash function is a hash function. The input is called the message. The output is called the digest (or the hash). Desirable properties for cryptographic hash functions. 1.Easy to compute the digest for any given message. 2. Infeasible to find two different messages with the same digest. (collision resistant) 3. Infeasible to identify characteristics of the message from the digest. Examples: MD5, SHA-1

7 7 Functionality File pieces distributed to peers Collaborative download Mechanism - Swarming Publish: make piece information known and make pieces available. Search: Out-of-band Join: get list of peers Fetch: Direct piece exchange with peers

8 8 BitTorrent Publishing a File Publisher divides file into identically-sized pieces (usually 32kB 16MB each). Publisher creates.torrent file that contains File metadata: Names of file(s), lengths of file(s)f, piece size SHA-1 hash for each piece URL of the Tracker Publisher makes.torrent file available for download (not part of BitTorrent protocol). Publisher also needs to make the file available for download by the peers. Provides a seed.

9 9 Bit Torrent Downloading a File A Tracker is a server that maintains a list of all of the peers currently participating in a torrent. This group of peers is called the swarm. To download a file: The new peer connects to the tracker listed in the.torrent file. The tracker sends a list of peers to the new peer. The new peer connects to the other peers to download pieces. Pieces are broken in sub-pieces and pipelines the downloads. Each downloaded piece can be verified using the cryptographic hash in the.torrent file.

10 10 BitTorrent Downloading Pieces When selecting piece to download from peers, a peer will select the rarest piece first. Exception: when peer first starts downloading, it selects a random piece. Peer choking. Peers will upload to other peers that upload to them tit for tat. To stop cooperating (uploading), a peer chokes another peer. Once a peer has downloaded the file, it switches to preferring peers that have better upload rates.

11 11 BitTorrent - Trackers Tracker keeps track of what peers are downloading/ uploading a file. Help coordinate peers in collaborative downloaded. Once a peer starts downloading, it doesn t really need the tracker anymore. Trackers sometimes ordered to shutdown for copyright violations. Solution: a decentralized mechanism to locate peers with a desired file. Trackerless torrents.

12 12 Distributed Hash Tables (DHTs) A distributed hash table is a distributed system that provides a service similar to a hash table. Keys are partitioned over a collection of nodes (servers). The system provides a single operation: lookup(key): returns the node responsible for the key Desirable features of DHTs: Load balancing Efficient lookups Handles churn

13 13 Consistent Hashing (Karger et. al) A consistent hash function assigns each node and key an m-bit identifiers, e.g., by using SHA-1. Node s ID chosen by hashing its IP address Key s ID chosen by hashing the key itself (e.g., a file name) IDs are ordered on a ring, mod 2 m Key ID k assigned to first node with ID k successor(k)

14 14 Consistent Hashing (cont.) For a node n, successor(n) is the node immediately following it on the ring (clockwise). When a new node n joins, certain keys assigned to successor(n) are assigned to n. When a node n leaves, all of its keys are assigned to successor(n).

15 15 Properties of Consistent Hashing Theorem: For any set of N nodes and K keys, with high probability: 1. Each node is responsible for (1 + ε)k/n keys 2. When an (N+1) th node joins or leaves the network, O(K/N) keys are reassigned. For implementation on previous slide, ε = O(logN)

16 16 Implementing lookup(key) Consistent hashing can be used just for data partitioning. E.g., Amazon Dynamo, Apache Cassandra Can use a centralized server for lookup. In DHTs, want a distributed implementation for lookup(key)

17 17 The Chord DHT (Stoica, et al. 2001/2003) Chord is a ring-based DHT. Uses the consistent hashing scheme on the previous slides. Also provides: distributed lookup mechanism Support for churn This is not the DHT that BitTorrent uses. BitTorrent uses Kademlia, a tree-based DHT

18 18 Chord: lookup(key) Simple key location Each node stores contact info for its immediate successor on the ring. To execute lookup(key) Client contacts any node on the ring Nodes forward lookup request around ring (using successor pointers), until responsible node is located. Result is returned along reverse path through ring, and eventually to the client. Pro: each node only need know its successor Con: Message complexity = O(N)

19 19 Efficient Lookups in Chord (1) N1 N51 N N8 +1 N14 Finger table N8 + 1 N14 N8 + 2 N14 N8 + 4 N14 N8 + 8 N21 N8 +16 N32 N8 +32 N42 N42 N21 N38 N32

20 20 Efficient Lookups in Chord (2) N1 lookup(54) K54 N56 N8 N51 N14 N48 N42 N21 N38 N32

21 21 Efficient Lookups in Chord (3) Each node stores information about log N other nodes. Theorem: With high probability, the number of nodes that must be contacted in a lookup is O(logN)

22 22 Chord: Supporting Churn When node n joins the network Invokes lookup(n) to find its successor on the ring. Notifies its successor that it has joined (for key redistribution) Populates finger table by executing lookup(n+2 i-1 ) for i=1 m To handle failures, each node periodically Verifies that it knows its correct successor and predecessor Repopulates finger table. Theorem: if there is no churn for long enough, the successor pointers will form a cycle.

23 23 Chord: Other Details If the ring breaks, Chord will not work. Each node stores pointers to its r immediate successors on the ring. If its successor fails, it can contact next successor. For file storage systems, common to replicate a file at k of these r successors (also done in Dynamo) To get better load balancing, can use virtual nodes. A single physical node is mapped to multiple virtual nodes on the ring.

24 24 BitTorrent: Distributed Tracker BitTorrent provides a distributed tracker implementation for peers to find other peers to download from. Uses the Kademlia DHT Node IDs are SHA-1 hash of IP address + secret Key IDs are SHA-1 hash of.torrent file