08 Distributed Hash Tables

Transcription

1 08 Distributed Hash Tables

2 2/59 Chord Lookup Algorithm Properties Interface: lookup(key) IP address Efficient: O(log N) messages per lookup N is the total number of servers Scalable: O(log N) state per node Robust: survives massive failures Simple to analyze

3 3/59 Chord IDs Key identifier = SHA-1(key) Node identifier = SHA-1(IP address) SHA-1 distributes both uniformly How to map key IDs to node IDs?

4 4/59 Consistent Hashing [Karger 97] Key 5 Node 105 K5 N105 K20 Circular 7-bit ID space N32 N90 K80 A key is stored at its successor: node with next higher ID

5 5/59 Basic Lookup N120 N10 Where is key 80? N105 N90 has K80 K80 N90 N60 N32

6 6/59 Simple lookup algorithm Lookup(my-id, key-id) n = my_successor if my-id < n < key-id call Lookup(id) on node n // next hop else return my successor // done Correctness depends only on successors

7 7/59 Finger Table Allows log(n)-time Lookups ¼ 1/8 1/16 1/32 1/64 1/128 N80 ½

8 8/59 Finger i Points to Successor of n+2i N ¼ 1/8 1/16 1/32 1/64 1/128 N80 ½

9 9/59 Lookup with Fingers Lookup(my-id, key-id) look in local finger table for highest node n s.t. my-id < n < key-id if n exists call Lookup(id) on node n // next hop else return my_successor // done

10 10/59 Lookups Take O(log(N)) Hops N5 N10 N110 K19 N20 N99 N32 K90 Lookup(K19) Lookup(K90) N80 N60

11 11/59 Joining: Linked List Insert N25 N36 1. Lookup(36) N40 K30 K38

12 12/59 Join (2) N25 2. N36 sets its own successor pointer N36 N40 K26 K30 K38

13 13/59 Join (3) N25 3. Copy keys from N40 to N36 N36 N40 K38 K26 K30

14 14/59 Join (4) N25 4. Set N25 s successor pointer N36 N40 K38 Predecessor pointer allows link to new host Update finger pointers in the background Correct successors produce correct lookups K26 K30

15 15/59 Failures Might Cause Incorrect Lookup N120 N113 N10 N102 N85 Lookup(90) N80 N80 doesn t know correct successor, so incorrect lookup

16 16/59 Solution: Successor Lists Each node knows r immediate successors After failure, will know first live successor Correct successors guarantee correct lookups Guarantee is with some probability

17 17/59 Lookup with Fault Tolerance Lookup(my-id, key-id) look in finger_table + succ_list for highest node n s.t. my-id < n < key-id if n exists call Lookup(id) on node n // next hop if call failed remove n from finger table return Lookup(my-id, key-id) else return my successor // done

18 18/59 Experimental Overview Quick lookup in large systems Low variation in lookup costs Robust despite massive failure Experiments confirm theoretical results

19 19/59 DHash Properties Builds key/value storage on Chord Replicates blocks for availability What happens when DHT partitions, then heals? Which (k, v) pairs do I need? Caches blocks for load balance Authenticates block contents

20 20/59 Replication Data at r Successors N5 N10 N110 N20 N99 N40 Block 17 N50 N80 N68 N60 Replicas are easy to find if successor fails Hashed node IDs ensure independent failure

21 21/59 Pastry

22 Pastry Node IDs are assigned randomly With high probability nodes with adjacent IDs are diverse Considers network locality Seeks to minimize distance messages travel Scalar proximity metric #IP routing hops RTT

23 Pastry: Object distribution O Consistent hashing [Karger et al. 97] objid nodeids 128 bit circular id space * nodeids (uniform random) objids (uniform random) Invariant: node with numerically closest nodeid maintains object. (Recall Chord)

24 Pastry: Node Soft State Storage requirement in each node = O(log N) Immediate Neighbors in ID space (Used for routing) Used for routing Nodes closest according to locality (Used to update routing table) Similar to successor and predecessor Similar to finger table entries

25 Pastry: Node Soft State Leaf Set Neighborhood Set Contains L nodes, closest in the ID space Contains M nodes, closest according to proximity metric Routing Table Entries of row n, shares exactly the first n digits with the local node Nodes are chosen according to proximity metric

26 Pastry: Routing Case I Key within leaf set Case II (Prefix Routing) Key not within leaf set Route to the node in the leaf set with ID closest to key Route a node in the routing table, such that the new node shares one more digit with the key than the local node Case III Key not within leaf set Case II not possible Route to a node which shares at least same number of digits with the key, but is closer to the key than the local node

27 Routing from To To By To Owner By To By

28 Routing Example d471f1 d467c4 d46a1c 65a1fc Cuts the ID space into 1/(2^b) Number of hops needed is log2^bn d462ba d4213f lookup(d46a1c) d13da3

29 Self-organization How are the routing tables and leaf sets initialized and maintained? Node addition Node departure (failure)

30 Self Organization: Node Join d471f1 Z=d467c4 X=d46a1c d462ba d4213f New node: X=d46a1c A is X s neighbor Route(d46a1c) The new node X asks node 65a1fc to route a message to it. Nodes in the route share their routing tables with X A = 65a1fc Node X splits key with d4771f and d467c4 B = d13da3

31 Self Organization: Node Join Leaf set (X) = leaf set (Z) Neighborhood set (A) = neighborhood set (X) d471f1 Z=d467c4 X=d46a1c d462ba d4213f New node: X=d46a1c Route(d46a1c) Routing Table Row zero of X = row zero of A Row one of X = row one of B A = 65a1fc d13da3

32 Pastry: Node departure (failure) Leaf set repair (eager all the time): Routing table repair (lazy upon failure): Leaf set members exchange keep-alive messages Request set from furthest live node in set Get table from peers in the same row, if not found from higher rows Neighborhood set repair (eager)

33 Pastry: Node State Update X informs any nodes that need to be aware of its arrival X also improves its table locality by requesting neighborhood sets from all nodes X knows Optimistic approach

34 CAN Content-Addressable Network

35 System Goals CAN: A distributed infrastructure that provides hash table-like functionality on Internet-like scales. Scalable Fault-tolerant Self-organizing

36 Basic Design A virtual d-dimensional Coordinate space Each node owns a Zone in the virtual space Data is stored as (key, value) pair Hash(key) --> a point P in the virtual space (key, value) pair is stored on the node within whose Zone the point P locates

37 Routing For routing purpose, each node only need to maintain the information of those nodes that hold coordinate zone adjoining its own zone (neighbors) Routing: greedy algorithm if P is within the Zone of current node else return(key, value) or failure (if no such key) forward the query to the neighbor with coordinates closest to P

38 Example: Routing (0, 4) (4, 4) 7 (0,0) (4, 0)

39 Node Insertion A new node N1 is going to join the network: Find a node N2 already in the CAN Randomly choose one point P in the space Send a JOIN request destined for P (P resides in the Zone of N3) N3 splits its Zone and assigns half zone to N1, and N3 sends (key, values) pairs from the half zone to N1 N3 sends neighbors information N3 notices all the neighbors the reallocation of space. Neighbors change their corresponding data

40 Example: Insertion

41 Node departure Explicit departure Hand over its zone to another node to produce a valid single zone Or merge with a smallest zone Node failure Periodic update messages between Neighbors A takeover mechanism merges the zone Prolonged absence of an update message from a neighbor<=>failure with the smallest adjacent zone There is also a background zone-reassignment algorithm to smooth the zone allocation

42 Example: Node Departure

43 Kademlia

44 Kademlia Nodes, files and key words, deploy SHA-1 hash into a 160 bits space. Every node maintains information about files, key words close to itself. The closeness between two objects measured as their bitwise XOR interpreted as an integer. distance(a, b) = a XOR b

45 Kademlia Binary Tree Treat nodes as leaves of a binary tree. Start from root, for any given node, dividing the binary tree into a series of successively lower subtrees that don t contain the node.

46 Kademlia Binary Tree Subtrees of interest for a node 0011

47 Kademlia Binary Tree Every node keeps touch with at least one node from each of its subtrees. (if there is a node in that subtree.) Corresponding to each subtree, there is a k-bucket. Every node keeps a list of (IP-address, Port, Node id) triples, and (key, value) tuples for further exchanging information with others.

48 Kademlia Search An example of lookup: node 0011 is searching for 1110 in the network

49 The XOR Metric d (x,x) = 0 d (x,y) > 0 if x y d (x,y) = d (y,x) d (x,y) + d (y,z) d (x, z) For each x and t, there is exactly one node y for which d (x,y) = t

50 Node state For each i (0 i <160) every node keeps a list of nodes of distance between 2i and 2(i+1) from itself.. Call each list a k-bucket. Least recenly seen The list is sorted by time last seen. The value of k is chosen so that any give set of k nodes is unlikely to fail within an hour. head Most recenly seen The list is updated whenever a node tail receives a message. Gnutella showed that the longer a node is up, the more likely it is to remain up for one more hour k = system-wide replication parameter

51

52

53 Node state The nodes in the k-buckets are the stepping stones of routing. By relying on the oldest nodes, k-buckets promise the probability that they will remain online. Least recenly seen Most recenly seen DoS attack is prevented since the new nodes find it difficult to How is the bucket updated? get into the k-bucket

54 Kademlia RPC PING: to test whether a node is online STORE: instruct a node to store a key FIND_NODE: takes an ID as an argument, a recipient returns (IP address, UDP port, node id) of the k nodes that it knows from the set of nodes closest to ID (node lookup) FIND_VALUE: behaves like FIND_NODE, unless the recipient received a STORE for that key, it just returns the stored value.

55 Kademlia Lookup Kademlia employs a recursive algorithm for node lookups. The lookup initiator starts by picking α nodes from its closest non-empty k-bucket. The initiator then sends parallel, asynchronous FIND_NODE to the α nodes it has chosen. α is a system-wide concurrency parameter, such as 3.

56 Kademlia Lookup When α = 1, the lookup resembles that in Chord in terms of message cost, and the latency of detecting the failed nodes. However, unlike Chord, Kademlia has the flexibility of choosing any one of the k nodes in a bucket, so it can forward with lower latency.

57 Kademlia Lookup The initiator resends the FIND_NODE to nodes it has learned about from previous RPCs. If a round of FIND_NODES fails to return a node any closer than the closest already seen, the initiator resends the FIND_NODE to all of the k closest nodes it has not already queried. The lookup terminates when the initiator has queried and gotten responses from the k closest nodes it has seen.

58 Kademlia Keys Store To store a (key,value) pair, a participant locates the k closest nodes to the key and sends them STORE RPCs. Additionally, each node re-publishes (key,value) pairs as necessary to keep them alive. For Kademlia s file sharing application, the original publisher of a (key,value) pair is required to republish it every 24 hours. Otherwise, (key,value) pairs expire 24 hours after publication.

59 New node join Each node bootstraps by looking for its own ID Search recursively until no closer nodes can be found The nodes passed on the way are stored in the routing table