Strong Eventual Consistency and CRDTs

Transcription

1 Strong Eventual Consistency and CRDTs Marc Shapiro, INRIA & LIP6 Nuno Preguiça, U. Nova de Lisboa Carlos Baquero, U. Minho Marek Zawirski, INRIA & UPMC

2 Large-scale replicated data structures Large, dynamic graph Incremental, parallel, asynchronous: - updates - processing Replication + eventual consistency + fast + simple Eventual Consistency Conflict-free objects = no synchronisation whatsoever Is this practical? 2

3 Consistency

4 Strong consistency Preclude conflict: Replicas update in same total order Any deterministic object Consensus Serialisation bottleneck way agreement Tolerates < n/2 faults Sequential, linearisable Simultaneous N- Very general Correct Doesn't scale 4

5 Eventual Consistency Conflict! Reconcile Update local + propagate No foreground synch Expose tentative state Eventual, reliable delivery On conflict Arbitrate Roll back Availability ++ Parallelism++ Latency -- Complexity ++ Consensus (in background) Designed for human collaboration Consensus moved to background 5 No (foreground) synchronisation Local updates; Speculate: replicas diverge Reliable broadcast, scalable If no conflicts, merge Otherwise, arbitrate; roll back inconsistent replicas Requires replicas to agree on arbitrations Consensus in background OK for human collaboration; not for high-performance computing

6 Strong Eventual Consistency Available, responsive More parallelism No conflicts Update local No + rollback propagate No synch Expose intermediate state Eventual, reliable delivery No conflict Unique outcome of concurrent updates No consensus: n-1 faults Not universal Fast, responsive Solves the CAP problem 6 No synchronisation Local updates; Speculate: replicas diverge Reliable broadcast, scalable Merge: unique, predictable outcome may not depend on: concurrent updates, order received Solves the CAP problem

7 Definition of EC delivered formally defined No (foreground) synchronisation Local speculative updates: replicas diverge Reconcile conflicts: arbitrate, merge, roll Eventual delivery: An update executed at a correct replica eventually executes at all correct replicas Termination: All update executions terminate Convergence: Correct replicas that have executed the same updates Availability eventually ++ reach Latency -- equivalent state (and stay) Complexity ++ Consensus (in background) 7 No (foreground) synchronisation Local updates Speculate: replicas diverge Reliable broadcast, scalable If no conflicts, merge Otherwise, reconcile; arbitrate, merge roll back inconsistent replicas Requires replicas to agree on arbitrations Consensus in background

8 Definition of SEC Eventual delivery: An update executed at some correct replica eventually executes at all correct replicas Termination: All update executions terminate Strong Convergence: Correct replicas that have executed the same updates have equivalent state No conflicts No rollback No consensus Limited 8 No (foreground) synchronisation Local updates Speculate: replicas diverge Reliable broadcast, scalable If no conflicts, merge Otherwise, reconcile; arbitrate, merge roll back inconsistent replicas Requires replicas to agree on arbitrations Consensus in background

9 SEC Sequential Consistency Consider Set-like object S such that: {true} add(e) {e S} {true} remove(e) {e S} {true} add(e) remove(e) {e S} Satisfies SEC conditions add(e); remove(e') {true} {e, e' S} add(e'); remove(e) Not sequentially consistent OR-Set add wins Conversely, SC is not SEC requires consensus 9 SEC is weaker than linearisability: it allows state changes after the return sequential consistency: see above Conversely, SC is stronger than SEC requires consensus It's a new kind of consistency

10 Conflict-free Replicated Data Types (CRDTs) Intuition: Conflicts are the problem Design data types with no conflicts CRDTs Available, fast Reconcile scalability + consistency Simple sufficient conditions Principled, correct 10

11 Object model & Sufficient conditions

12 Query client s s1 s2.q(b) S s1.q(a) s2 S s3 Local at source replica Client's choice Example: Amazon shopping cart is replicated unspecified client, e.g., Web frontend One or more load-balancer, failures may direct client to different replicas 12

13 State-based replication client s s1 s2 s2.u(b) S S s1.u(a) s1 M s2.m(s1) s2 s1.m(s2) M s2 s3 M s3.m(s2) Local at source s1.u(a), s2.u(b), Compute Update local payload Convergence: Episodically: send si payload On delivery: merge payloads m merge two valid states produce valid state no historical info available Inefficient if payload is large 13 State-based: Local queries, local updates Send full state; on receive, merge Conceptually simple File systems (NFS, Unison, Dynamo)

14 State-based: monotonic semilattice CRDT s s1 s2 s3 s1.u(a) S s2.u(b) S s1 M s2.m(s1) s2 M s3.m(s2) s1.m(s2) M s2 = Least Upper Bound LUB = merge If payload type forms a semi-lattice updates are increasing merge computes Least Upper Bound then replicas converge to LUB of last values Example: Payload = int, merge = max no reference to history 14

15 Operation-based replication s s1 s2 s2.u(b) s1.u(a) S S b b a s1.u(b) D a s2.u(a) D push to all replicas eventually push small updates - more efficient than state-based s3 D s3.u(b) D s3.u(a) At source: Eventually, at all replicas: prepare broadcast to all replicas update local replica 15 Operation-based: Local queries Replicated updates Log, send operations Reconcile non-commutative operations Vector clock: static network Collaborative editing, Bayou, PNUTS

16 Operation-based replication s s1 s2 s1.t(a'); s1.u(a) s2.t(b'); s2.u(b) S S b b a s1.u(b) D a s2.u(a) D push to all replicas eventually push small updates - more efficient than state-based s3 D s3.u(b) D s3.u(a) At source: Eventually, at all replicas: prepare broadcast to all replicas update local replica 16

17 Op-based: commute CRDT s s1 s1.u(a) S b s1.u(b) D a s2.u(a) s2 s2.u(b) S b a D s3 D s3.u(b) D s3.u(a) Delivery order ensures downstream precondition happened-before or weaker If: (Liveness) all replicas execute all operations in delivery order (Safety) concurrent operations all commute Then: replicas converge 17 Not same order at 1 and 2? OK if concurrent f and g commute Commutativity: sufficient necessary? (conjecture)

18 Monotonic semi-lattice commutative Systematic transformation Inefficient Hand-crafted op-based implementation 1. A state-based object can emulate an operation-based object, and vice-versa 2. State-based emulation of an op-based CRDT is a CRDT 3. Operation-based emulation of a state-based CRDT is a CRDT 18

19 Emulating operation-based with state-based RBi reliable broadcast emulated as CvRDT Replica payload at i: Grow-Set Msgs Grow-Set Dlvrd Operations: send(m): Msgs Msgs { m } deliver (Pre): choose m Msgs \ Dlvrd: Pre(m); Dlvrd Dlvrd { m }; m s s' s. Msgs s'. Msgs merge(s,s'): s. Msgs s'. Msgs RB = send, deliver Deliver(Pre) Pre = predicate (pre-cond) merge computes LUB it's a CvRDT 19

20 Comparison State-based: Update merge operation Simple data types State includes preceding updates; no separate historical information Operation-based: Update operation Higher level, more complex More powerful, more constraining Small messages State-based or op-based, as convenient 20 It will often be convenient to design using state-based Convert to deltas or op-based for efficiency at large scale

21 Composition and sharding A combination of independent CRDTs remains a CRDT Very large objects Independent shards Static: hash Statically-Sharded CRDT Each shard is a CRDT Update: single shard No cross-object invariants (Dynamic: requires consensus to rebalance) 21

22 The challenge: What interesting objects can we design with no synchronisation whatsoever?

23 Portfolio of CRDTs Register Last-Writer Wins Multi-Value Set Grow-Only 2P Observed-Remove Map Set of Registers Counter Unlimited Non-negative Graphs Directed Monotonic DAG Edit graph Sequence Edit sequence 23

24 Multi-master counter Increment / decrement Payload: P = [int, int, ], N = [int, int, ] value() = i P[i] i N[i] increment () = P[MyID]++ decrement () = N[MyID]++ merge(s,s') = like vector clock s s' = ([,max(s.p[i],s'.p[i]), ]i, [,max(s.n[i],s'.n[i]), ]i) Positive or negative 24

25 Multi-master counter Increment / decrement Payload: P = [int, int, ], N = [int, int, ] value() = i P[i] i N[i] increment () = P[MyID]++ decrement () = N[MyID]++ merge(s,s') = like vector clock s s' = ([,max(s.p[i],s'.p[i]), ]i, [,max(s.n[i],s'.n[i]), ]i) Positive or negative can't maintain global invariant such as s>0 25

26 Set design alternatives Sequential specification: {true} add(e) {e S} {true} remove(e) {e S} {true} add(e) remove(e) {????} linearisable? error state? last writer wins? add wins? remove wins? linearisable: sequential order equivalent to real-time order Requires consensus 26 Linearisability is too strong for SEC

27 Observed-Remove Set s s1 s2 s3 {} {} {} add(a) S {aα} add(a) S {aβ} {aβ} rmv (a) S {aβ, aα} M {aα} {aβ} Can never remove more tokens than {aα} {aα} exist Op order removed M M M {aβ} {aβ, aα} {aβ, aα} tokens have been previously added Payload: added, removed (element, unique-token) add(e) = A A {(e, α)} Remove: all unique elements observed remove(e) = R R { (e, ) A} lookup(e) = (e, ) A \ R merge (S, S') = (A A', R R') {true} add(e) remove(e) {e S} 27 In case of add+remove, OR-Set gives precedence to add

28 OR-Set Set: solves Dynamo Shopping Cart anomaly Optimisations: No tombstones Operation-based approach Snapshots Sharded 28

29 OR-Set + Snapshot Read consistent snapshot Despite concurrent, incremental updates Unique token = time (vector clock) α = Lamport (process i, counter t) UIDs identify snapshot version Snapshot: vector clock value Retain tombstones until not needed lookup(e, t) = (e, i, t') A : t'>t (e, i, t') R: t'>t 29

30 Graph design alternatives Graph = (V, E) where E V V Sequential specification: {v,v' V} addedge(v,v') { } { (v,v') E} removevertex(v) { } Concurrent: removevertex(v') addedge(v,v') linearisable? last writer wins? addedge wins? removevertex wins? etc. for our Web Search Engine application, removevertex wins Do not check precondition at add/remove 30

31 Directed Graph Payload = OR-Set V, OR-Set E Updates add/remove to V, E addvertex(v), removevertex(v) addedge(v,v'), removeedge(v,v') Do not enforce invariant a priori lookupedge(v,v') = (v,v') E v V v' V removevertex(v') addedge(v,v') removevertex wins 31

32 Graph + shards + snapshots Snapshot see OR-Set Sharding See OR-Set Do not enforce invariant a priori lookupedge(v,v') = (v,v') E v V v' V 32

33 Ongoing work

34 CRDTs for P2P & Cloud Computing ConcoRDanT: ANR Systematic study of conflict-free design space Theory and practice Characterise invariants Library of data types Not universal Conflict-free vs. conflict semantics Move consensus off critical path, non-critical ops 34 Other data types: Trivial: set with add, remove

35 CRDT + dataflow Web site 1 Web site 2 Set Whitelist crawl extract links Graph Graph spam detector Web site 3 Incremental, asynchronous processing Replicate, shard CRDTs near the edge Propagate updates dataflow Throttle according to QoS metrics (freshness, availability, cost, etc.) Scale: sharded Synchronous processing: snapshot, at centre Content DB Map URL to last 2 versions extract words Words Map word to Set of URLs 35

36 More scalability Incremental, asynchronous processing Operation-based: send small updates Dataflow-style notification network Push/throttle according to required QoS Sharding Composability Large-scale transactional processing Snapshot Isolation CRDTs appear ideal 36

37 Thought experiment (1) Web site 1 Map URL to text Content DB extract links Graph Links page rank Query 2 Query 1 Query 3 Web site 2 Web site 3 Key: Input crawl Canonical URL extract words computation Data (replicated) Object type (replicated) Map word to Set of URLs Words ranking indexer query resolver Ranked Map of word to list of URLs 37

38 Thought experiment (2) Links rank content DB, index: decentralised, replicated, close to the network edge Words ranking indexer crawl, extract Links Words Links crawl, extract Words crawl, extract Links Words crawl, extract Links Words 38 Crawl and extract content close to the source of data = partition geographically, replicate Compute page rank on consistent data: snapshot Fast results: replicate index widely near the edge

39 Contributions Strong Eventual Consistency (SEC) A solution to the CAP problem Formal definitions Two sufficient conditions Strong equivalence between the two SEC shown incomparable to sequential consistency CRDTs integer vectors, counters sets graphs 39

40 40