(It s the invariants, stupid)

Transcription

1 Consistency in three dimensions (It s the invariants, stupid) Marc Shapiro Cloud to the edge Social, web, e-commerce: shared mutable data Scalability replication consistency issues Inria & Sorbonne-Universités UPMC-LIP6 Social, web, e-commerce: shared mutable data Scalability replication consistency issues 2 Cloud to the edge Why consistency models Social, web, e-commerce: shared mutable data Scalability replication consistency issues SER, SI, PSI, US, NMSI, EC, More parallelism, less synchronisation Implementation, execution freedom Availability, Performance Anomalies, hard to understand Fundamental trade-off: availability, responsiveness vs. programmability What is right for my application? 3 4

2 Why consistency models Two parts 90% -only transactions; Disaster Tolerant Termination Latency of Transactions (ms) 70% -only transactions; Disaster Tolerant 1. Understanding consistency Organising consistency design space into independent dimensions Consistency and invariants 3 Serrano-SI P-Store-SER GMU-US Jessy2pc-NMSI RC Walter-PSI SDUR-SER Application-specific consistency Minimise synchronisation While maintaining application correctness Credit: Masoud Saeida Ardekani 5 6 Atul Adya s hierarchy Operation More invariants More predictable Strict Serializability (PL-SS) Full Serializability (PL-3) Snapshot Isolation (PL-SI) Serializability (PL-3U) Low performance client origin replica u Forward Consistent View (PL-FCV) other replica Repeatable (PL-2.99) Monotonic Snapshot s (PL-MSR) Consistent View (PL-2+) Hard to program More behaviours Cursor Stability (PL-CS) PL-2 PL-1 Monotonic View (PL-2L) One dimension is not enough High performance u: state (retval, (state state)) Prepare (@origin) ; deliver one, write all (ROWA) Deferred-update replication (DUR) 7 8

3 Application invariants v? Convergence? Safety? South Boat North = { sheep, dog, wolf } carrynorth(s) 1 S 2 carrysouth(s) 1 S 2 S {South, Boat, North} : sheep S wolf S dog S, Multi-master Strong: total, identical state Weak: concurrent, interleaving, no global state Hard to tease invariants out Silent invariants 9 10 Three classes / Gen1 of invariant of protocol Mostly orthogonal (but not all combinations make sense.) Strict Serialisability Gen1 Constrain value of an object Total of operations EQ PO State equivalence between objects Ordering between operations Units Causality 11 Units / EQ Snapshot Isolation Eventual Consistency HAT 12 / PO Causal

4 Gen1 invariants Gen1: ed updates Inv = 0 x = x x 1 { Inv 1 x} { Inv } Predict that Inv will be true after : Sequential: weakest precondition Generalises to bounded concurrency Unbounded concurrency: no sufficient precondition Invariant is not stable Limit concurrency: escrow No concurrency: updates Generic invariants No Lost s Total updates updates Total, Consensus Unavailable Pick a number Will all replicas observe updates in the same total? No: full parallelism Local: lost work Global synchronisation, unavailable () If yes, will reads appear in the same? Ex.: Serialisability, Linearisability Not: EC, Causal Consistency Type EQ invariants EQ: indivisible unit A = B x.friendof (y) y.friendof (x) x + y = constant South Boat North = { sheep, dog, wolf } Joint update to two objects Atomicity (all-or-nothing) property of transactions Protocol: single update message Asynchronous + Unit: group operations ( transaction ) Indivisible, atomic, all-or-nothing Deliver effectors together Snapshot: all reads from same set of effectors Plain: cached reads (repeatable) Consistent: according to Causal/ Total constraints 15 16

5 EQ + Gen: + Type PO invariants + Unit: group operations ( transaction ) Indivisible, atomic, all-or-nothing Deliver effectors together Snapshot: all reads from same set of effectors Plain: cached reads (repeatable) Consistent: according to Causal/ Total constraints employee.salary employee.manager.salary sheep S wolf S dog S Referential integrity inode references disk block Security access (u, p) ACL (u, p) Demarcation Protocol: Decrease LHS / increase RHS, or 1. increase RHS by c 2. increase LHS by c' c causal delivery Causal- delivery Don t show photos to Bob post photo Causal- delivery u v u v u v u v Bob v u Bob u v Bob sees photo access (Bob, photo) ACL (Bob, photo) v observed effects of u at source v to be delivered after u v visible(i) u v u visible(i) 19 v observed effects of u at origin v to be delivered after u if u v, then observing both u and v implies u; ; v Asynchronous: doesn t slow down sender Cost: metadata + transitive closure 20

6 PO: Causal- visblty Total causal + Write- Monotonic s + My Writes Causality Which writes visible to read delivery of writes Transitive closure property Metadata Sender not delayed available Ex.: SI, Causal C. Not: Serialisability + Total updates updates Total, Monotonic s + My Writes + Write- Total causal Total updates updates Total updates updates Lin + Total, EC Monotonic s + My Writes + Write- Total causal + Total, Monotonic s + My Writes + Write- Total causal 23 24

7 Total updates updates SSER Lin SER Total updates updates SSER Lin + Total, Monotonic s + My Writes + Write- Total causal + Total, Monotonic s + My Writes + Write- Total causal SER Total updates updates SSER SER Total updates updates SSER + Total, SI Monotonic s + My Writes + Write- Total causal + Total, PSI SI Monotonic s + My Writes + Write- Total causal 27 28

8 + Total updates updates Total, Monotonic s + My Writes HAT + Write- Total causal 29 Total XXX unit orthogonal causal writes reads Eventual C. N N N Causal+ N Y N Cmtd Y N N CausalTxn Y Y N Total Centrlisd writes N N Y N This table does not take reads into account read = writes SC, Lineariz. N Y Y Y Y Serlzbty Y N Y Y N Serializability Y N Y Y Y PSI Snapshot Isolatn Y Y Y Y N Strict Serlzbty Y Y N 30 Explicit consistency Hybrid, opportunistic consistency model Programme(r) declares application invariants. System ensures that every state transition preserves the invariant. Synchronise only when strictly necessary for invariant Improve performance, availability Assume: causal ing accrue 5% debit(100) Convergent: do replicas that delivered the same updates have the same state? Safe: Are invariants preserved? Sequential: in isolation maintains invariant execution maintains invariant +5 I? I?

9 Specification Verification Problem balance 0 {balance amount } debit!(amount)= {balance amount} Token = {τ,...} Token Token Invariant precondition effector tokens u: state (retval, (state state)) Given the specification of operations: preconditions effects tokens uτ conflict relation on tokens does any interleaving of the operations always preserve a given integrity invariant? CISE Proof Tool Analysis and Co-design Tokens Semantics Invariant CISE OK Counter-example CISE proof obligations: O(n 2 ), no combinatorial explosion Manual or automated Tool based on Z3 SAT solver Discharges verification conditions OK = Safe Tokens Semantics Invariant CISE OK Counter-example Counter-example? Design decision: either Strengthen precondition Weaken effect Weaken invariant or Strengthen synchronisation (tokens) 35 36

10 I? I? Simple example: bank account CISE Proof Obligations 1: Sequential correctness Any single operation maintains the invariant 2: Convergence If concurrent, effectors commute 3: Precondition Stability Every precondition is stable under every operation, if concurrent If satisfied: invariant is guaranteed 37 Operations: credit(amt), debit(amt), accrue() Invariant: balance 0 Start with weak specification Rule 1 strengthen precondition for debit Rule 2 accrue adds Rule 3 debit debit unsafe, fixed with concurrency control 38 I I I I CISE Proof Obligations 1: Sequential correctness Any single operation maintains the invariant 2: Convergence If concurrent, effectors commute 3: Precondition Stability Every precondition is stable under every operation, if concurrent If satisfied: invariant is guaranteed 39 invariant "balance >= 0 { amt 0 } deposit! = {balance += alt} { amt 0 } debit! = {balance -= alt} No counter-example for deposit, accrue. Counter-example for debit: if balance initially zero, debit will make it negative. { amt balance } debit! = {balance -= alt} 40

11 I I invariant "balance >= 0 { amt 0 } deposit! = {balance += alt} { amt 0 } debit! = {balance -= alt} No counter-example for deposit, accrue. Counter-example for debit: if balance initially zero, debit will make it negative. { amt balance } debit! = {balance -= alt} 41 CISE Proof Obligations 1: Sequential correctness Any single operation maintains the invariant 2: Convergence If concurrent, effectors commute 3: Precondition Stability Every precondition is stable under every operation, if concurrent If satisfied: invariant is guaranteed 42 I balance = 1 balance = 1 debitpre {1 1} debit(1) balance 1 balance 1 balance 1 balance = 1? CISE Proof Obligations 1: Sequential correctness Any single operation maintains the invariant 2: Convergence If concurrent, effectors commute 3: Precondition Stability Every precondition is stable under every operation, if concurrent If satisfied: invariant is guaranteed 43 debit(1) debitpre { 1 0 }? CISE Proof Obligations 1: Sequential correctness Any single operation maintains the invariant 2: Convergence If concurrent, effectors commute 3: Precondition Stability Every precondition is stable under every operation, if concurrent If satisfied: invariant is guaranteed Fix: concurrency control 44

12 Advanced example: file system Advanced example: file system Operations: mkdir, rmdir, mv, update, etc. Invariant: Tree Rule 1 precondition on mv May not move node under self Rule 2 Use CRDTs for update update Rule 3 mv mv precondition unstable Operations: mkdir, rmdir, mv, update, etc. Invariant: Tree Rule 1 precondition on mv May not move node under self Rule 2 Use CRDTs for update update Rule 3 mv mv precondition unstable root mvpre { B/.../A} mv /B, /A A root B root Applying the logic A B A B root mv /A, /B mvpre { B/.../A}? CISE Proof Obligations 1: Sequential correctness Any single operation maintains the invariant 2: Convergence If concurrent, effectors commute 3: Precondition Stability Every precondition is stable under every operation, if concurrent If satisfied: invariant is guaranteed Fix: concurrency control 47 A B Only O(n 2 ): no need to consider all possible interleavings We use a tool You can apply the same logic manually 48

13 Deployment Ad-service Data-centers deployed in AWS: 3 Regions (EU, US-EAST/WEST); N app-servers connect to local DBs; Clients submit operations to the app-server in close loop. Compare performance: Causal Consistency Strong Consistency (Writes to single server) Red-Blue Consistency (Causal + Writes to single server) Explicit Consistency (Causal + Reservations) ads; 100% I-offenders 50 Tournament Tournament: latency 82% reads; 4% safe writes; 14% I-offenders Detailed Operations Latency 51 52

14 Summary Summary Not all concurrency leads to invariant violation Explicit consistency + CISE Identify just enough coordination Indigo Enforces explicit consistency Deconstructing consistency CRDTs provide you consistency Mechanisms for consistency Explicit consistency for just right consistency Acknowledgments SyncFree European FP7 project, Carlos Baquero, Valter Balegas, Annette Bieniusa, Russell Brown, Sérgio Duarte, Nuno Preguiça, etc. Masoud Saeida-Ardekani, Marek Zawirski. Alexey Gotsman, Hongseok Yang, Carla Ferreira, Mahsa Najafzadeh 55