CS 268: DHTs. Page 1. How Did it Start? Model. Main Challenge. Napster. Other Challenges

Transcription

1 How Did it Start? CS : DHTs A killer application: Naptser - Free music over the Internet Key idea: share the content, storage and bandwidth of individual (home) users Scott Shenker and Ion Stoica April, Internet Model Main Challenge Each user stores a subset of files Each user has access (can download) files from all users in the system Find where a particular file is stored F E D E? A B C Other Challenges Napster Scale: up to hundred of thousands or millions of machines Dynamicity: machines can come and go any time Assume a centralized index system that maps files (songs) to machines that are alive How to find a file (song) - Query the index system return a machine that stores the required file Ideally this is the closest/least-loaded machine - ftp the file Advantages: - Simplicity, easy to implement sophisticated search engines on top of the index system Disadvantages: - Robustness, scalability (?) Page

2 Napster: Example Gnutella m F E? E m A E? m m E m A m B m C m D m E m F m B D m C m Distribute file location Idea: flood the request Hot to find a file: - Send request to all neighbors - Neighbors recursively multicast the request - Eventually a machine that has the file receives the request, and it sends back the answer Advantages: - Totally decentralized, highly robust Disadvantages: - Not scalable; the entire network can be swamped with request (to alleviate this problem, each request has a TTL) Gnutella Ad-hoc topology Queries are flooded for bounded number of hops No guarantees on recall Distributed Hash Tables (DHTs) Abstraction: a distributed hash-table data structure - insert(id, item); - item = query(id); (or lookup(id);) - Note: item can be anything: a data object, document, file, pointer to a file Proposals - CAN, Chord, Kademlia, Pastry, Tapestry, etc 9 DHT Design Goals Make sure that an item (file) identified is always found Scales to hundreds of thousands of nodes Handles rapid arrival and failure of nodes Structured Networks Distributed Hash Tables (DHTs) Hash table interface: put(key,item), get(key) O(log n) hops Guarantees on recall (K,I ) put(k,i ) get (K ) Page

3 Content Addressable Network (CAN) CAN Example: Two Dimensional Space Associate to each node and item a unique id in an d- dimensional Cartesian space on a d-torus Properties - Routing table size O(d) - Guarantees that a file is found in at most d*n /d steps, where n is the total number of nodes Space divided between nodes All nodes cover the entire space Each node covers either a square or a rectangular area of ratios : or : Example: - Node n:(, ) first node that joins cover the entire space n CAN Example: Two Dimensional Space CAN Example: Two Dimensional Space Node n:(, ) joins space is divided between n and n Node n:(, ) joins space is divided between n and n n n n n n CAN Example: Two Dimensional Space CAN Example: Two Dimensional Space Nodes n:(, ) and n:(,) join n n n Nodes: n:(, ); n:(,); n:(, ); n:(,);n:(,) Items: f:(,); f:(,); f:(,); f:(,); n n n f n n n f f n f Page

4 Q Q CAN Example: Two Dimensional Space CAN: Query Example Each item is stored by the node who owns its mapping in the space f n f n n n f n f Each node knows its neighbors in the d-space Forward query to the neighbor that is closest to the query id Example: assume n queries f Can route around some failures f n f n n n f n f 9 CAN: Node Joining CAN: Node Joining 9: *!#"%$'& (!) (*+ -,!./ ; < = ; >? <! % (!! -( CAN: Node Joining CAN: Node Joining 9: * ; < = ; < = ; >? A-B > C D< E-D>GF'HJI K A IL?M EN!>O< B EP; >? <RQ S A ET U M DVQ!W EYX>Z;Z<[M ;G\Z]ZU ^%_`; < = ;Z>J? <a> =b;zec> ; <G\] U ^ Page

5 Node departure Chord Node explicitly hands over its zone and the associated (key,value) database to one of its neighbors Incase of network failure this is handled by a take-over algorithm Problem : take over mechanism does not provide regeneration of data Solution: every node has a backup of its neighbours Associate to each node and item a unique id in an unidimensional space.. m - Goals - Scales to hundreds of thousands of nodes - Handles rapid arrival and failure of nodes Key design decision - Decouple correctness from efficiency Properties - Routing table size O(log(N)), where N is the total number of nodes - Guarantees that a file is found in O(log(N)) steps Identifier to Node Mapping Example Lookup Node maps [,] Node maps [9,] Node maps [, ] Node maps [9, ] Each node maintains its successor Route packet (ID, data) to the node responsible for ID using successor pointers lookup() Each node maintains a pointer to its successor node= Joining Operation Joining Operation Each node A periodically sends a stabilize() message to its successor B Upon receiving a stabilize() message node B - returns its predecessor B =pred(b) to A by sending a notify(b ) message Upon receiving notify(b ) from B, - if B is between A and B, A updates its successor to B - A doesn t do anything, otherwise Node with id= joins the ring Node needs to know at least one node already in the system - Assume known node is succ=nil pred=nil succ= pred= succ= pred= 9 Page

6 Joining Operation Joining Operation Node : send join() to node Node : returns node Node updates its successor to succ= succ=nil pred=nil succ= pred= join() Node : send stabilize() to node Node : - update predecessor to - send notify() back succ= pred=nil notify(pred=) succ= pred= pred= stabilize() succ= pred= succ= pred= Joining Operation (cont d) Joining Operation (cont d) Node sends a stabilize message to its successor, node Node reply with a notify message Node updates its successor to succ= stabilize() pred=nil notify(pred=) succ= pred= Node sends a stabilize message to its new successor, node Node sets its predecessor to node succ= pred= pred=nil succ= pred= Stabilize() succ= succ= pred= succ= pred= Joining Operation (cont d) Achieving Efficiency: finger tables This completes the joining operation! succ= pred= succ= pred= Finger Table at i ft[i] ( + ) mod = Say m= i m ith entry at peer with id n is first peer with id >= n + (mod ) Page

7 Achieving Robustness CAN/Chord Optimizations To improve robustness each node maintains the k (> ) immediate successors instead of only one successor In the notify() message, node A can send its k- successors to its predecessor B Upon receiving notify() message, B can update its successor list by concatenating the successor list received from A with A itself Reduce latency - Chose finger that reduces expected time to reach destination - Chose the closest node from range [N+ i-,n+ i ) as successor Accommodate heterogeneous systems - Multiple virtual nodes per physical node Conclusions Two Other Papers Distributed Hash Tables are a key component of scalable and robust overlay networks CAN: O(d) state, O(d*n/d) distance Chord: O(log n) state, O(log n) distance Both can achieve stretch < Simplicity is key Services built on top of distributed hash tables - persistent storage (OpenDHT, Oceanstore) - pp file storage, i (chord) - multicast (CAN, Tapestry) Boon Loo Tau et al, Enhancing PP File-Sharing with an Internet-Scale Query Processor, VLDB Krishna Gummadi et al, The Impact of DHT Routing Geometry on Resilience and Proximity, SIGCOMM 9 Goal Keyword Search using DHTs PP Search - What is the best design? Target Environment - Typical PP file-sharing environments. - Items stored in PP network, queried by keywords - Replicas of items follow a long-tailed distribution Popular items at head of distribution Rare items at tail of distribution Inverted Lists hashed by keyword (term) in the DHT %& %. * +,-. + -* %'& % )( * +,- ( + -/.*! " #$ #$ %& % /* +,-/ + -* Page

8 Solution Space & Proposal Gnutella Measurements ;^_ V \]' %'&('(')+*-,'././ %..9,&:;9< = >?!"$# U V W X ZY$ Q[W; \]' (/.A.'9,&:;9 ` Quality of Searches: - Recall (% of all relevant items retrieved) - Distinct Recall (% of all relevant distinct items retrieved) - Response Time (Latency) to st result ` Computed Query Recall - Each query issued simultaneously from ultrapeers - Union of results from ultrapeers Union-of- is our approximation of perfect answer BC DFE'&.G/H&:;./ I'H:JCK.L:(F.C:M'./G(GEN:C DC O C I'P+,I'9Q:/,'RQ:,/.') (/9Q:J/,'RQ:J,/.)+I'. : (G/ S9 T Result Size CDF ab cdef g hfi j k l cbmcn m o n p qr g hfi j vz ww{- }~ y ~w w{z }w t~ $ Query Latency vg $ yt z ˆy ƒ } u; yt zˆy satufvwxty Š_Œ Ž Œ L _ 'Œ_ Š+ 'ŒF Œ LŠ+š _ LœFŒ _ž+ž Ÿ ŒF _ž _ $Œ Ž; ŒF Hybrid Search Challenges Hybrid = Best of both worlds vz w w{ y { ˆw u ¼½ ujxg ¾w¾Ãt z u ˆ À]y'Á ' ' ± -²'³ µ ³ µ ³ ³'¹ ² º;¹» ÄÅÆ -²'³ µ Ç ª$«¼½ ujxg $ ¾uJ [ ˆ Ày'Á ³A ³¹ ² º;¹ È Identifying Rare Items - Query Results Size (QRS) Publish items from previous queries that return few results - Term Frequency Statistics Single Term (TF) Term Pairs (TPF) - Sample items on neighboring nodes (SAM) È Network Churn (IPTPS ) - Use Ultrapeers as DHT nodes - Avoid publishing items from short-lived nodes Page

9 ; ; Identifying Rare Items ³ ² ' ¹'³ L³_ º ³ º;³ ' ' _ ³' ³! '³º G " ³G ² ' #$ %'&( ) ¹'³ L³ Ä* + ¹ + ³ º² '¹-,¹L ³º;³ _¹-,G ³ ²' ¹'³' /. ¹'³ _³'¹ ³º ³'³ #'$%'&( ), ' Results Improved Response Time: - PIER (DHT) returns first result in seconds seconds in aggregate including seconds timeout - Gnutella (Flood) queries returns first result in seconds - seconds (%) reduction in latency Improved Recall Analysis: - % reduction in #queries with empty results Using the naïve QRS rare-item selection scheme - Opportunity to do far better Recall: % potential reduction based on Union-of- Approaches: larger-scale deployments and using better rare-item schemes. 9 Two Other Papers Motivation 9 Boon Loo Tau et al, Enhancing PP File-Sharing with an Internet-Scale Query Processor, VLDB : Krishna Gummadi et al, The Impact of DHT Routing Geometry on Resilience and Proximity, SIGCOMM New DHTs constantly proposed - CAN, Chord, Pastry, Tapestry, Plaxton, Viceroy, Kademlia, Skipnet, Symphony, Koorde, Apocrypha, Land, ORDI ; Each is extensively analyzed but in isolation Each DHT has many algorithmic details making it difficult to compare Goals: a) Separate fundamental design choices from algorithmic details b) Understand their affect reliability and efficiency Our approach: Component-based analysis Three aspects of a DHT design 9 Break DHT design into independent components 9 Analyze impact of each component choice separately - compare with black-box analysis: benchmark each DHT implementation rankings of existing DHTs vs. hints on better designs 9 Two types of components - Routing-level : neighbor & route selection - System-level : caching, replication, querying policy etc. ) Geometry: a graph structure that inspires a DHT design, with its exciting properties - Tree, Hypercube, Ring, Butterfly, Debruijn ) Distance function: captures a geometric structure - d(id, id) for any two node identifiers ) Algorithm: rules for selecting neighbors and routes using the distance function Page 9

10 Chord DHT has Ring Geometry Chord Distance function captures Ring d(, ) = 9 Nodes are points on a clock-wise Ring 9 d(id, id) = length of clock-wise arc between ids = (id id) mod N Chord Neighbor and Route selection Algorithms One Geometry, Many Algorithms d(, ) = 9 Algorithm : exact rules for selecting neighbors, routes - Chord, CAN, PRR, Tapestry, Pastry etc. d(, ) = - inspired by geometric structures like Ring, Hyper-cube, Tree 9 Geometry : an algorithm s underlying structure d(, ) = 9 Neighbor selection: i th neighbor at i distance - Distance function is the formal representation of Geometry - Chord, Symphony => Ring - many algorithms can have same geometry Why is Geometry important? 9 Route selection: pick neighbor closest to destination Insight: Geometry => Flexibility => Performance Route selection flexibility allowed by Ring Geometry 9 Geometry captures flexibility in selecting algorithms 9 Flexibility is important for routing performance - Flexibility in selecting routes leads to shorter, reliable paths - Flexibility in selecting neighbors leads to shorter paths 9 Chord algorithm picks neighbor closest to destination 9 A different algorithm picks the best of alternate paths 9 Page

11 Neighbor selection flexibility allowed by Ring Geometry Geometries we compare Geometry Algorithm Ring Chord, Symphony Hypercube CAN Tree Plaxton 9 Chord algorithm picks i th neighbor at i distance 9 A different algorithm picks i th neighbor from [ i, i+ ) Hybrid = Tree + Ring XOR d(id, id) = id XOR id Tapestry, Pastry Kademlia Metrics for flexibilities Flexibility in neighbor selection (FNS) for Tree 9 FNS: Flexibility in Neighbor Selection = number of node choices for a neighbor 9 FRS: Flexibility in Route Selection = avg. number of next-hop choices for all destinations h = h = h = 9 Constraints for neighbors and routes - select neighbors to have paths of O(logN) - select routes so that each hop is closer to destination 9 logn neighbors in sub-trees of varying heights 9 FNS = i- for i th neighbor of a node Flexibility in route selection (FRS) for Hypercube Summary of our flexibility analysis d(, ) = Flexibility Ordering of Geometries d(, ) = Neighbors (FNS) Hypercube << Tree, XOR, Ring, Hybrid (logn) ( i- ) 9 Routing to next hop fixes one bit 9 FRS =Avg. (#bits destination differs in)=logn/ d(, ) = d(, ) = Routes Tree << XOR, Hybrid < Hypercube < Ring (FRS) () (logn/) (logn/) (logn) How relevant is flexibility for DHT routing performance? Page

12 Analysis of Static Resilience Two aspects of robust routing Dynamic Recovery : how quickly routing state is recovered after failures Static Resilience : how well the network routes before recovery finishes - captures how quickly recovery algorithms need to work - depends on FRS Evaluation: Fail a fraction of nodes, without recovering any state Metric: % Paths Failed % Failed Paths Does flexibility affect Static Resilience? Tree 9 % Failed Nodes Tree << XOR XOR Hybrid Ring Hypercube Hybrid < Hypercube < Ring Flexibility in Route Selection matters for Static Resilience Analysis of Overlay Path Latency Which is more effective, FNS or FRS? Goal: Minimize end-to-end overlay path latency FNS + FRS Ring - not just the number of hops Both FNS and FRS can reduce latency - Tree has FNS, Hypercube has FRS, Ring & XOR have both CDF FNS Ring FRS Ring Plain Ring Evaluation: Using Internet latency distributions (see paper) Latency (msec) Plain << FRS << FNS FNS+FRS Neighbor Selection is much better than Route Selection 9 Does Geometry affect performance of FNS or FRS? Summary of results CDF FNS Ring FNS XOR FRS Ring FRS Hypercube FRS matters for Static Resilience - Ring has the best resilience Both FNS and FRS reduce Overlay Path Latency But, FNS is far more important than FRS - Ring, Hybrid, Tree and XOR have high FNS Latency (msec) No, performance of FNS/FRS is independent of Geometry A Geometry s support for neighbor selection is crucial Page

13 Conclusions Routing Geometry is a fundamental design choice - Geometry determines flexibility - Flexibility improves resilience and proximity Ring has the greatest flexibility Page