Why distributed databases suck, and what to do about it. Do you want a database that goes down or one that serves wrong data?"

Transcription

1 Why distributed databases suck, and what to do about it - Regaining consistency Do you want a database that goes down or one that serves wrong data?" 1

2 About the speaker NoSQL team lead at Trifork, Aarhus, Denmark Working with databases since '97 NoSQL since 2008 Danish Shared Medication Record Migrating data from MySQL to Riak Devel Riak clients NoSQL architect on various international projects RuneSkouLarsen 2

3 Agenda Part 1: Working with eventual consistency NoSQL persistence landscape What is consistency Eventual vs. sequential consistence Conflicts and how to handle them CRDT's Consistency models of current OLTP databases Part 2: Stronger consistency in distributed, fault tolerant systems Consensus Delta consistency Dynamic delta Consistency 3

4 Polyglot persistence landscape In-memory Neo4j VoltDB Redis EasyDB MongoDB CouchDB OLTP Riak Cassandra Voldemort CouchBase Analytics Hadoop 4

5 Why distributed databases? Redundancy Availability Scaling Getting closer to your users 5

6 What is Concistency Consistency: All nodes see the same data at the same time Eventual consistency Autonomous consistency Sequential consistency Bureaucratic consistency 6

7 When to be Consistent with what Eventual consistency Support disconnected operations Better to read a stale value than nothing Better to save writes somewhere than nothing Potentially anomalous application behavior Stale reads and conflicting writes Sequential consistency Requires highly available connections Not suitable for certain scenarios: Disconnected clients (e.g. your phone) Apps might prefer potential inconsistency to loss of availability 7

8 Conflicting updates User A User B A B A Asynchronous Synchronization B 8

9 Last Write Wins (LWW) User A User B A t=t0 B t=t1 A t=t0 Asynchronous Synchronization B t=t1 Assign timestamp to all objects Simple but fragile depends on precise synchronization of timers Data is lost 9

10 Google Spanner As a distributed-systems developer, you re taught from I want to say childhood not to trust time. What we did is find a way that we could trust time and understand what it meant to trust time. Andrew Fikes 1

11 Detecting conflicts using Vector Clocks (1) User A User B A vclock=a:1 B vclock=a:1,b:1 A vclock=a:1 Asynchronous Synchronization A B vclock=a:1 vclock=a:1,b:1 Assign vector clock to objects Ancestors are removed descendants remain 1

12 Detecting conflicts using Vector Clocks (2) User A User B A vclock=a:1 B vclock=b:1 A vclock=a:1 Asynchronous Synchronization B vclock=b:1 Spawn siblings when causality chain is broken 1

13 Semantic resolution User A User B CA B A Asynchronous Synchronization B Keep both values as siblings User does the merging The only solution if you need to do intelligent merging or start outside processes. 1

14 Conflict-free Replicated DataTypes User A User B A B AB A Asynchronous Synchronization AB B Datastructure intrinsically merges objects Limited applicability 1

15 Conflict-free Replicated Data Types Convergent (CvRDT) State is replicated Moves towards one value Commutative (CmRDT) Operations to the state are replicated The order of operations is insignificant a*b = b*a 1 CvRDT and CmRDT can emulate eachother

16 CRDT examples: G-set and 2P-Set Tombstone RIP 1

17 CRDT References CRDTs: Consistency without concurrency control 2009 INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE A comprehensive study of Convergent and Commutative Replicated Data Types 2011 INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE Sean Cribbs - Eventually Consistent Data Structures 1

18 Methods for handling conflicts Last Write Wins Easy Data is lost Depends on timestamps Semantic resolution Requires application/user involvement Generic solution Conflict-free Data Types Data structure has built-in convergence Limited ability to model real-world problems 1

19 Consistency models of OLTP databases Last write wins Riak CouchDB/CouchBase Cassandra User resolvable conflicts Riak Voldemort CouchDB/CouchBase (but unreliable) Active anti-entropy Riak (Soon) Hinted handoff with sloppy quorums (highest write-availability) Riak Cassandra Strong consistency (read you own writes + strict quorums) Riak Voldemort Cassandra CouchBase MongoDB Traditional SQL databases (Oracle, MySQL, etc.) 1

20 Consistency ph Atomic Consistent Isolated Durable vs Basically Available Soft state Eventual Consistency Consistency availability 2

21 Consensus Protocol for agreeing on a decision More than half the nodes must be in agreement (n/2+1) Tolerates remaining nodes being down/slow/un-updated. Consistency availability Consensus 2

22 Example: Ensuring idempotence using consensus Communication protocols are unreliable and requests can be resent even when they have already completed. Clients assign requestid. If a request is resent, we should return the first answer instead of processing it again. vnodes serialize writes in Riak. We use Riak. N=3, PW=quorum to ensure strict quorums.(*) (*) Riak has a bug in the P checks, but we have deemed it insignificant to our use. 2

23 Example: Ensuring idempotence using consensus Doctor system Request idempotence Proxy instance Pharmacy system Requests Requests Requests Requests 2

24 Example: Ensuring idempotence using consensus Doctor system reqid=xyz Request idempotence Proxy instance Pharmacy system reqid=xyz Down We tolerate one node down at a time 2 reqid=xyz Asuming n<=nodes: n=3: quorum=2, maxdown=1 n=4: quorum=3, maxdown=1 n=5: quorum=3, maxdown=2 n=6: quorum=4, maxdown=2 n=7: quorum=4, maxdown=3

25 Example: Ensuring idempotence using consensus Doctor system reqid=xyz Request idempotence Proxy instance Pharmacy system reqid=xyz reqid=xyz 2

26 Delta consistency An update will propagate through the system and all replicas will be consistent after a fixed time period δ Easy to understand for customer Consistency availability Consensus Delta consistency 2

27 Example: Delta Consistency with prescription replication We guarentee that prescriptions are replicated from Oracle to Riak in 20 minutes. Max 20 minutes Prescription server Drug medication server Oracle Master Oracle MView Riak Riak Riak 2

28 Dynamic Delta consistency Same as Delta Consistency, but users can monitor directly how far behind we are Define one or more authorities, and track how far behind they are. All responses are added information on updatedness of data for each authority. Useful when delay is normally low (sub-second), but can be high in times of degraded service. Useful for CQRS or temporarily offline systems Pro/Con: Users have to understand what data delay means. Consistency availability 2 Consensus Delta consistency Dynamic Delta consistency

29 Example: Dynamic Delta Consistency using mobile device When beginning a sync, note the time on the authority After completing a sync, store the time of last sync on one or boths sides. Mobile device Riak Sync Riak Relay server Expose updatedness of data. Riak Riak Riak 2

30 Example: Dynamic Delta Consistency using CQRS Eventlog Commands trigger async events Events update views View Expose the oldest waiting event as updated_until on view, or now if no events are in queue. 3

31 Example: Dynamic Delta Consistency using multiple authorities Setup is multiple datacenters everybody replicates with everybody at intervals. full sync When a full sync is done, save the sync data in each data center DC2 Example: DC1 done syncing with DC2 sync started at time t. When a datacenter is internally consistent (no pending handoffs for instance), it can expose the time of sync with the other authorities as updated_until timestamp. DC1 DC3 3

32 Thank you! RuneSkouLarsen 3