Peer- to- Peer in the Datacenter: Amazon Dynamo

Transcription

1 Peer- to- Peer in the atacenter: mazon ynamo C upload rate u s Last Lecture bits Internet d 4 u 4 Mike reedman COS 461: Computer Networks hp:// 2 upload rates u i download rates d i d 1 d 3 u 1 u 2 u 3 d 2 This Lecture mazon s ig ata Problem Too many (paying) users! Lots of data Performance maers Higher latency = lower conversion rate Scalability: retaining performance when large 3 4 1

2 Tiered Service Structure Horizontal or VerWcal Scalability? Stateless Stateless Stateless ll of the State VerWcal Scaling Horizontal Scaling 5 6 Horizontal Scaling is ChaoWc k = probability a machine fails in given period n = number of machines 1- (1- k) n = probability of any failure in given period or 50K machines, with online Wme of %: 16% of the Wme, data center experiences failures or 100K machines, 30% of the Wme! ynamo Requirements High vailability lways respond quickly, even during failures Replica+on! Incremental Scalability dding nodes should be seamless Comprehensible Conflict ResoluWon High availability in above sense implies conflicts 7 8 2

3 ynamo esign ata ParWWoning and ata ReplicaWon Key- Value Store via HT over data nodes get(k) and put(k, v) QuesWons: ReplicaWon of ata Handling Requests in Replicated System Temporary and Permanent ailures Membership Changes amiliar? Nodes are virtual! Heterogeneity ReplicaWon: Coordinator Node N- 1 successors also Nodes keep preference list C 9 10 Handling Requests etecwng ailures Request coordinator consults replicas How many? orward to N replicas from preference list R or W responses form a read/write quorum ny of top N in pref list can handle req Load balancing & fault tolerance C Purely Local ecision Node may decide independently that has failed In response, requests go further in preference list request hits an unsuspecwng node temporary failure handling occur

4 Handling Temporary ailures Managing Membership is in replica set Needs to receive replica Hinted Handoff: replica contains original node When C comes back forwards the replica back to C X C Peers randomly tell another their known membership history gossiping lso called epidemic algorithm Knowledge spreads like a disease through system reat for ad hoc systems, self- configurawon, etc. oes this make sense in mazon s environment? dd to the replica set! ossip could parwwon the ring Possible Logical ParWWons and choose to join ring at about same Wme: Unaware of one another, may take long Wme to converge to one another SoluWon: Use seed nodes to reconcile membership views: Well- known peers that are contacted frequently Why is ynamo ifferent? So far, looks a lot like normal p2p mazon wants to use this for applicawon data! Lots of potenwal synchronizawon problems Uses versioning to provide eventual consistency

5 Consistency Problems What if a failure occurs? Shopping Cart xample: Object is a history of adds and removes 17 ll adds are important (trying to make money) Client: Put(k, [+1 anana]) Put(k, Z + [+1 anana]) Put(k, Z + [- 1 anana]) xpected ata at Server: [+1 anana] [+1 anana, +1 anana] [+1 anana, +1 anana, - 1 anana] Client: Put(k, [+1 anana]) Put(k, Z + [+1 anana]) Put(k, Z + [- 1 anana]) 18 ata on ynamo: [+1 anana] at Crashes not in first Put s quorum [+1 anana] at [+1 anana, - 1 anana] at Node Comes Online t this point, Node and disagree about object state How is this resolved? Can we even tell a conflict exists? Time is largely a human construct What about Wme- stamping objects? Could authoritawvely say whether object newer or older? ut, all events are not necessarily witnessed If system s nowon of Wme corresponds to real- Wme New object always blasts away older versions ven though those versions may have important updates (as in bananas example). Requires a new nowon of Wme (causal in nature) Causality Objects are causally related if value of one object depends on (or witnessed) the previous Conflicts can be detected when replicas contain causally independent objects for a given key NoWon of Wme which captures causality? nyhow, real- Wme is impossible in any case

6 Versioning Key Idea: very PUT includes a version, indicawng most recently witnessed version of updated object Problem: replicas may have diverged No single authoritawve version number (or clock number) NoWon of Wme must use a par+al ordering of events Vector Clocks very replica has its own logical clock Incremented before it sends a message very message aached with vector version Includes originator s clock Highest seen logical clocks for each replica If M 1 is causally dependent on M 0 : Replica sending M 1 will have seen M 0 Replica will have seen clocks all clocks in M Vector Clocks in ynamo Vector Clocks in anana xample Vector clock per object get() returns obj s vector clock put() has most recent clock Coordinator is originator Serious conflicts are resolved by app / client Client: Put(k, [+1 anana]) Put(k, Z + [+1 anana]) Put(k, Z + [- 1 anana]) ata on ynamo: [+1] v=[(,1)] at Crashes not in first Put s quorum [+1] v=[(,1)] at [+1,- 1] v=[(,2)] at Comes Online [(,1)] and [(,2)] are a conflict! 23 igure 3: Version evolution of an object over tim 24 6

7 ventual Consistency NoSQL Versioning, by itself, does not guarantee consistency If you don t require a majority quorum, you need to periodically check that peers aren t in conflict How oven do you check that events are not in conflict? In ynamo: Nodes consult with one another using a tree hashing (Merkel tree) scheme Quickly idenwfy whether they hold different versions of parwcular objects and enter conflict resoluwon mode NoWce that ventual Consistency and ParWal Orderings do not give you CI! Rise of NoSQL (outside of academia) Memcache Cassandra Redis ig Table Mongo