CS 268: Lecture 22 DHT Applications

Transcription

1 CS 268: Lecture 22 DHT Applications Ion Stoica Computer Science Division Department of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA (Presentation based on slides from Robert Morris and Sean Rhea) 1 Outline Cooperative File System (CFS) Open DHT 2 Page 1

2 Target CFS Uses Serving data with inexpensive hosts: - open-source distributions - off-site backups - tech report archive - efficient sharing of music 3 How to mirror open-source distributions? Multiple independent distributions - Each has high peak load, low average Individual servers are wasteful Solution: aggregate - Option 1: single powerful server - Option 2: distributed service But how do you find the data? 4 Page 2

3 Design Challenges Avoid hot spots Spread storage burden evenly Tolerate unreliable participants Fetch speed comparable to whole-file TCP Avoid O(#participants) algorithms - Centralized mechanisms [Napster], broadcasts [Gnutella] CFS solves these challenges 5 CFS Architecture Each node is a client and a server Clients can support different interfaces - File system interface - Music key-word search 6 Page 3

4 Client-server interface Files have unique names Files are read-only (single writer, many readers) Publishers split files into blocks Clients check files for authenticity 7 Server Structure QHR1S T QR(S T T T U M U K! " #%$&('*)+-,.-/0 $+1'#2$3-, $&('9070:8$&)-,;"-0=<?>($&A@*"B#2CED GFH I-J-JAKLLMN"O,;"-0 $E)-,;"-0P< 8 Page 4

5 Chord Hashes a Block ID to its Successor Q U Q N10 (MM K(MK L"!#(M L (ML L N100 S Q S N32 (MM L $%& 'L N80 N60 LK "-C-$+ /CO),6" 0=< & :$E# /C-", C 4;! # 4 )->H$+C >80 0 $+1(" #9/"-C $ 4 /$ 4 $+ 9 DHash/Chord Interface ( Q*),+ - Q. 0/ S &1 QHR1S T &2 S 43 T5+ Q 0/ S 1 T lookup() returns list with node IDs closer in ID space to block ID - Sorted, closest first 10 Page 5

6 M K DHash Uses Other Nodes to Locate Blocks N5 N99 N110 N80 N10 N20 N40 N50 N68 N60 ( Q "%) 11 Storing Blocks S T &2- S 2 Long-term blocks are stored for a fixed time - Publishers need to refresh periodically Cache uses LRU 12 Page 6

7 M K M K Replicate blocks at r successors N5 N110 N10 N99 N20 N40 M&' N80 N50 N68 N60 "-C-$ -#%$ B " CC #%$&( /8>-#%$+ 4 /8C-$ 38$ /C-$/= #2$3-, ,5>-#2$ 13 Lookups find replicas N5 / F S + T & T N110 N99 N80 N68 N60 ( Q +M') N10 N20 N40 N50 M&' 14 Page 7

8 M K M K First Live Successor Manages Replicas N5 N110 N10 N99 M&' N20 N40 M&' N80 N50 N68 N60, 4 4 4, 4 "-C-$0 -/E,;"-07-, C-$: $ #& /$ + 4;+$ #%! +$ H>0 07$&(1"B# 15 DHash Copies to Caches Along Lookup Path N5 / T T + T S T N110 N99 N80 N68 ( N60 Q ) N10 N20 N40 N50 16 Page 8

9 Caching at Fingers Limits Load N32 /-,,;"# /"-C $& :$ 4 / -$#% 3" 45/=4 / " -4;, 4 4 $ H4 /,;$ ),6" 0=<,;"C " / 17 Virtual Nodes Allow Heterogeneity N10 N60 N101 N5 U U Hosts may differ in disk/net capacity Hosts may advertise multiple IDs - Chosen as SHA-1(IP Address, index) - Each ID represents a virtual node Host load proportional to # v.n. s Manually controlled 18 Page 9

10 Why Blocks Instead of Files? Cost: one lookup per block - Can tailor cost by choosing good block size Benefit: load balance is simple - For large files - Storage cost of large files is spread out - Popular files are served in parallel 19 Outline Cooperative File System (CFS) Open DHT 20 Page 10

11 Questions: How many DHTs will there be? Can all applications share one DHT? 21 Benefits of Sharing a DHT Amortizes costs across applications - Maintenance bandwidth, connection state, etc. Facilitates bootstrapping of new applications - Working infrastructure already in place Allows for statistical multiplexing of resources - Takes advantage of spare storage and bandwidth Facilitates upgrading existing applications - Share DHT between application versions 22 Page 11

12 The DHT as a Service 23 The DHT as a Service OpenDHT 24 Page 12

13 The DHT as a Service OpenDHT Clients 25 The DHT as a Service OpenDHT 26 Page 13

14 The DHT as a Service What is this interface? OpenDHT 27 It s not lookup() lookup(k) Challenges: 1. Distribution 2. Security What does this node do with it? k 28 Page 14

15 1. Storage - CFS, UsenetDHT, PKI, etc. How are DHTs Used? 2. Rendezvous - Simple: Chat, Instant Messenger - Load balanced: i3 - Multicast: RSS Aggregation, White Board - Anycast: Tapestry, Coral 29 What about put/get? Works easily for storage applications Easy to share - No upcalls, so no code distribution or security complications But does it work for rendezvous? - Chat? Sure: put(my-name, my-ip) - What about the others? 30 Page 15

16 Protecting Against Overuse Must protect system resources against overuse - Resources include network, CPU, and disk - Network and CPU straightforward - Disk harder: usage persists long after requests Hard to distinguish malice from eager usage - Don t want to hurt eager users if utilization low Number of active users changes over time - Quotas are inappropriate 31 Fair Storage Allocation Our solution: give each client a fair share - Will define fairness in a few slides Limits strength of malicious clients - Only as powerful as they are numerous Protect storage on each DHT node separately - Must protect each subrange of the key space - Rewards clients that balance their key choices 32 Page 16

17 The Problem of Starvation Fair shares change over time - Decrease as system load increases Starvation! Client 1 arrives fills 50% of disk Client 2 arrives fills 40% of disk Client 3 arrives max share = 10% time 33 Preventing Starvation Simple fix: add time-to-live (TTL) to puts - put (key, value) put (key, value, ttl) Prevents long-term starvation - Eventually all puts will expire 34 Page 17

18 Preventing Starvation Simple fix: add time-to-live (TTL) to puts - put (key, value) put (key, value, ttl) Prevents long-term starvation - Eventually all puts will expire Can still get short term starvation Client A arrives fills entire of disk Client B arrives asks for space Client A s values start expiring B Starves time 35 Preventing Starvation Stronger condition: Be able to accept r min bytes/sec new data at all times This is non-trivial to arrange! max space TTL Sum must be < max capacity Reserved for future puts. Slope = r min size Candidate put 0 now time max 36 Page 18

19 Preventing Starvation Stronger condition: Be able to accept r min bytes/sec new data at all times This is non-trivial to arrange! Violation! max max space TTL size space TTL size 0 now time max 0 now time max 37 Preventing Starvation Formalize graphical intuition: f(τ) = B(t now ) - D(t now, t now + τ) + r min τ D(t now, t now + τ): aggregate size of puts expiring in the interval (t now, t now + τ) To accept put of size x and TTL l: f(τ) + x < C for all 0 τ < l Can track the value of f efficiently with a tree - Leaves represent inflection points of f - Add put, shift time are O(log n), n = # of puts 38 Page 19

20 Fair Storage Allocation Queue full: reject put Not full: enqueue put Per-client put queues Select most underrepresented Wait until can accept without violating r min Store and send accept message to client The Big Decision: Definition of most under-represented 39 Defining Most Under-Represented Not just sharing disk, but disk over time - 1 byte put for 100s same as 100 byte put for 1s - So units are bytes seconds, call them commitments Equalize total commitments granted? - No: leads to starvation - A fills disk, B starts putting, A starves up to max TTL Client A arrives fills entire of disk Client B arrives asks for space B catches up with A Now A Starves! time 40 Page 20

21 Defining Most Under-Represented Instead, equalize rate of commitments granted - Service granted to one client depends only on others putting at same time Client A arrives fills entire of disk Client B arrives asks for space B catches up with A time A & B share available rate 41 Defining Most Under-Represented Instead, equalize rate of commitments granted - Service granted to one client depends only on others putting at same time Mechanism inspired by Start-time Fair Queuing - Have virtual time, v(t) - Each put gets a start time S(p ci ) and finish time F(p ci ) F(p ci ) = S(p ci ) + size(p ci ) ttl(p ci ) S(p ci ) = max(v(a(p ci )) - ε, F(p c i-1 )) v(t) = maximum start time of all accepted puts 42 Page 21

22 FST Performance 43 Page 22