This talk is about. Data consistency in 3D. Geo-replicated database q: Queue c: Counter { q c } Shared database q: Queue c: Counter { q c }

Transcription

1 Data consistency in 3D (It s the invariants, stupid) Marc Shapiro Masoud Saieda Ardekani Gustavo Petri This talk is about Understanding consistency Primitive consistency mechanisms How primitives compose models How models relate / differ What they cost Understanding invariants Some interesting classes of invariants Relating consistency to invariants Which primitives guarantee which invariants Useful intuitions for app. and system designers 2 Shared database q: Queue c: Counter { q c } c.inc() Geo-replicated database q: Queue c: Counter { q c } 5 ms q: Queue c: Counter { q c } c.inc() q.push(e) c.inc() Social, web, e-commerce: shared mutable data Scalability replication consistency issues q.val() c.val() 3 q.push(e) c.inc() Social, web, e-commerce: shared mutable data Scalability replication consistency issues q.val() c.val() q: Queue c: Counter { q c } q3 Queue? q1 = q2? q1 c4? 4

2 Consistency More replicas: Better read availability, responsiveness, performance, etc. More work to keep replicas in sync Consistent = behavior similar to sequential: Satisfies specs: does q behave like a queue? Replicas agree: is q identical everywhere? Objects agree: is q c? Same flow of time? q1.push() before q2.push() Consistency opportunities and costs Availability Parallelism keeps the hardware busy More implem. options, scalable But consistency constrains order of events: Delay delivery Stale reads Waits, synchronisation (mutual wait) Keeping track of order requires metadata Significant! 5 6 Costs illustrated 90% Read-only transactions; Disaster Tolerant Strict Serialisability Termination Latency of Update Transactions (ms) 70% Read-only transactions; Disaster Tolerant T1 client T1 R1 T2 T3 3 R2 R3 T1 T2 T3 Serrano-SI P-Store-SER GMU-US Jessy2pc-NMSI RC Walter-PSI SDUR-SER 4.5 Invariant Invariant Invariant Credit: Masoud Saeida Ardekani 7 8

3 Eventual consistency Strong vs. weak? Predictable Strict Serialisability Low performance Op1 R1 R2 R3 Op2 Snapshot Isolation PRAM Hard to program Eventual Consistency High performance 9 10 Strong vs. weak? Strong vs. weak? Predictable Strict Serialisability Low performance Strict Serializability (PL-SS) Full Serializability (PL-3) Fork Timed serial &,Γ-atomicity Sequential Prefix linearizable Linearizability Regular Eventual linearizability Snapshot Isolation Serialisability Snapshot Isolation (PL-SI) Update Serializability (PL-3U) Forward Consistent View (PL-FCV) Weak fork-lin. Bounded fork-join causal Fork sequential Fork* Causal+ Real-time causal Timed causal Causal models Causal Prefix sequential Safe Per-object models Processor Per-key Per-record sequential timeline & Coherence Synchronized models Weak ordering Release Lazy release Staleness-based models k-atomicity Bounded staleness & Delta k-regular Strong eventual Eventual serializabilit PRAM Monotonic Snapshot Reads (PL-MSR) Repeatable Read (PL-2.99) Cursor Stability (PL-CS) Consistent View (PL-2+) Monotonic View (PL-2L) Fork-join causal Fork-based models Writes-follow-reads Read-your-writes (WFR) (RYW) Session models Per-object causal PRAM (FIFO) Monotonic Writes Monotonic Reads (MW) (MR) Slow memory Scope Entry Location PBS t-visibility PBS k-staleness k-safe Eventual Quiescent PL-2 Weak Hard to program Eventual Consistency High performance PL-1 Transactional Adya 1999 Non-transactional Viotti & Vukolić

4 Three classes of invariant of protocol Gen1 Object value Total order of Mostly orthogonal (but not all combinations make sense.) Gen1 / Total Order Serialisability Strict Serialisability Linearisability PO EQ Relative ordering of State equivalence Visibility Composition 13 EQ / Composition Snapshot Isolation Eventual Consistency Txnl CC 14 Causal Operation Sequential correctness x=4 y= 2 x+y 0 x+y>0 x:=3 x+y>0 y 3 x+y>0 skip x=4 y= 2 x=3 y= 2 x=4 true y= 2 x+y 0 x:=3 y 3 skip generator: read, compute, generate effector effector: compute, write side-effect Sequential execution: precondition invariant each effector individually safe generator: read, compute, generate effector effector: compute, write side-effect Sequential execution: precondition invariant each effector individually safe 15 16

5 Guarantee vs. semantics Data types Guarantee: Class of invariants that is always true Regardless of application code Assuming sequentially correct Application can compensate for absence of guarantee e.g. Inv={ c 0 }, app: c.inc() Register Update: assign with constant Not commutative Absorbing High-level types Counter, ORset, Sequence: commute Stock, Account, Queue: commute Composed data + structural invariants Replicated operation client origin replica replica replica u? u upre upre upre u: state (retval, (state state)) Read one, write all (ROWA) Deferred-update replication (DUR) u! v? v! u! u! Sharded, geo-replicated arbitrary origin DC1 DC2 x1 y1 z1 x2 y2 z2 x=1 y =1 x1>0 z2%2=0 x1:=0 y2+=1 y1+=1 concurrent updates DC3 x2:=0 x=0 y=0 read my writes sharded, parallel 19 20

6 Type EQ invariants A = B x.friendof (y) y.friendof (x) x + y = constant South Boat North = { sheep, dog, wolf } Joint update to two objects Atomicity (all-or-nothing) property of transactions Protocol: single update message Asynchronous EQ: transactional composition Airplane reservation Allocate a seat to me Pay for the flight Two EQ relations: paid = have_seat my $$ + airline $$ = constant Ad-hoc grouping (This txn also needs TO ) EQ/Composition axis EQ/Composition axis Transaction groups : Deliver indivisibly packaged together + same TOE 2-phase commit Snapshot reads: all generators read from same set of maintain versions + same TO, VIS guarantees coordination Linearisability PRAM RC Serialisability Snapshot Isolation Trans. Causal Transaction groups : Deliver indivisibly packaged together + same TOE 2-phase commit Snapshot reads: all generators read from same set of maintain versions + same TO, VIS guarantees coordination 23 24

7 EQ/Composition axis Type PO invariants Linearisability PRAM RC Serialisability Snapshot Isolation Trans. Causal Transaction groups : Deliver indivisibly packaged together + same TOE 2-phase commit Snapshot reads: all generators read from same set of maintain versions + same TO, VIS guarantees coordination employee.manager.salary employee.salary S1; S2; S3 S1 S2 S3 dog S sheep S wolf S Referential integrity inode references disk block ACL (u, p) access (u, p) Demarcation Protocol: 1. increase LHS by c 2. increase RHS by c' c ordered delivery No synchronisation: Available PO: transitive / causal visibility x = 100; y = 100 Inv = { x y } Ex 1: P1: x += 100 P2: if x > y then y += (x y)/2 P3: x y? Transitive visibility vis* vis Ex 2: P1: x += 100; d 100 P2: if d > 0 then y += d/2 P3: x y? Causal visibility (vis; po)* vis 25 PO: transitive / causal visibility x = 100; y = 100 Inv = { x y } Ex 1: P1: x += 100 P2: if x > y then y += (x y)/2 P3: x y? Transitive visibility vis* vis Ex 2: P1: x += 100; d 100 P2: if d > 0 then y += d/2 P3: x y? Causal visibility (vis; po)* vis d! d! 26 client is part of DB 27 28

8 PO/Visibility axis Visibility Which writes visible to reads Transitive closure property Metadata System-wide Sender not delayed writes available Stale data reads available Read My Writes Monotonic Reads Often assumed Buffer 29 Eventual Consistency Not reasonable 30 Transitive, causal vis. Effector: metadata identifies set of predecessor Delay delivery after predecessors Read stale data Graph: unbounded Vector clock: entries 8 bytes! client is Approximate VC: stronger order x part of DB d d! 31 y Total/external causal Total order extends causal order Metadata: 1 single scalar but cost of total order : real-time clock SER NMSI PSI Gentle Rain Linearisable SSER 32

9 Gen1 invariants Gen1: total order Inv = 0 x u! = x x 1 { Inv 1 x} u! { Inv } Predict that Inv will be true after u!: Sequential: weakest precondition Generalises to bounded concurrency Unbounded concurrency: no sufficient precondition Invariant is not stable Limit concurrency: escrow No concurrency: order updates Gen1 Do replicas observe events in the same order? Pick a unique number = unordered Capricious TO Do replicas observe events in the same order? Pick a unique number Do replicas observe events in the same order? Pick a unique number 7 10 No: concurrent Commute converge Stable precondition Invariant 35 Lamport LWW EC Pick a number locally: Gap: will arrive later? Non-monotonic: rollback Monotonic Wait for gap to fill (Lamport 78) Lost updates (LWW) 36

10 Capricious TO Lamport LWW EC Do replicas observe events in the same order? Pick a unique number 10 Pick a number locally: Gap: will arrive later? Non-monotonic: rollback Monotonic Wait for gap to fill (Lamport 78) Lost updates (LWW) 37 7 SMR PSI NMSI Do replicas observe events in the same order? Pick a unique number Gapless: No lost updates Consensus, 2PC to uniquely allocate next free number not available TO generators LIN SER SSER SI Do replicas observe events in the same order? Pick a unique number TO + TO generators separate from same order as 39 40

11 Lin EC Lin Lin SSER SER SSER 43 44

12 Lin Lin SER SSER SER SSER PSI SI PSI Txnl CC 47 Total Order TOG=TOE E Capricious TOE Concurrent Ops Composition Visibility Rollbacks Monotonic Transitive Causal All-or-Nothing + Snapshot SER SSER All-or-Nothing E ectors Single Operation SC LIN All-or-Nothing + Snapshot NMSI PSI SSI All-or-Nothing E ectors Single Operation All-or-Nothing + Snapshot Bayou ÿ All-or-Nothing E ectors ÿ Single Operation LWW ÿ All-or-Nothing + Snapshot Causal HAT ÿ All-or-Nothing E ectors RC ÿ Single Operation EC PRAM CC ÿ 48

13 Summary Distributed, replicated data Improves read availability Parallel updates may violate invariants Guarantee: invariants maintained by system System vs. application cost trade-off Tools needed 3D consistency design space Total order (, generators) Visibility order Transactional Composition Work in progress 49 Creative Commons Attribution-ShareAlike 4.0 Intl. License You are free to: Share copy and redistribute the material in any medium or format Adapt remix, transform, and build upon the material for any purpose, even commercially, under the following terms: Attribution You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. ShareAlike If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. 4 session guarantees causal Monotonic reads r2 r3 w1 Read My Writes r2 50 Client / No rollback: r3 must include w1 Client / RMW: r2 must include w1 w1 Monotonic writes w2 Writes Follow Reads r2 w3 w2! w3! w2! w3! Global / No rollback: r3 must include w1 Global / WR dependence: w3 must follow w1 52