What about asynchronous systems? The Game of Paxos. Quorum Systems. The Game of Paxos

Transcription

1 What about asynhronous systems? FLP says onsensus annot be solved... For benign failures, Paos provides net best thing always safe And for Byzantine failures? The Game of Paos Proesses are ompeting to write a value in a write-one register 1. Push the read button and eamine the token that falls into the tray 2. value of the register is stamped on the token! 3. If the token is red and stamped with a value, plae the token in the slot, set the dial to the same value, and push the write button value Read Value dial Write The Game of Paos Quorum Systems Proesses are ompeting to write a value in a write-one register 1. Push the read button and eamine the token that falls into the tray 2. value of the register is stamped on the token! 3. If the token is red and stamped with a value, plae the token in the slot, set the dial to the same value, and push the write button 4. If the token is red and not stamped, plae the token in the slot, set the dial to any value, and push the write button Read Value dial Write Given a set of servers U, U = n a quorum system is a set Q 2 U suh that Q 1,Q 2 Q : Q 1 Q 2 Eah Q in Q is a quorum

2 A R/W Register A R/W Register servers } store at eah server a (v, ts) pair store at eah server a (v, ts) pair Write (, d) Ask servers in some Q for their ts Set ts > ma({ts} any previous ts ) Update some Q with (d, ts ) Semantis A R/W Register *387 A read not onurrent with any write returns the most reently written value store at eah server a (v, ts) pair )79G>3D Safe + a read that overlaps with a write obtains either the old or the new value FA?;5 Write (, d) Read () Ask servers in some Q for their ts Set ts > ma({ts} any previous ts ) Update some Q with (d, ts ) Ask servers in some Q for their (v, ts) Selet most reent (v, ts) Reads and writes are totally ordered so that values returned by reads are the same as if the operations had been performed with no overlapping

3 System Model Universe U of servers, U = n Byzantine faulty servers modeled as a non-empty fail-prone system B 2 U no B B is ontained in another some B B ontains all faulty servers Clients are orret (an be weakened) Point-to-point authentiated and reliable hannels A orret proess q reeives a message from another orret proess p if and only if p sent it Masking Quorum System [Malkhi and Reiter, 1998] A quorum system Q is a masking quorum system for a fail-prone system B if the M-Consisteny Q 1,Q 2 Q B 1,B 2 B :(Q 1 Q 2 ) \ B 1 B 2 M-Availability B B Q Q : B Q = Dissemination Quorum System A masking quorum system for self-verifying data D-Consisteny Q 1,Q 2 Q B B :(Q 1 Q 2 ) B D-Availability B B Q Q : B Q = f-threshold Masking Quorum Systems M-Consisteny Q 1,Q 2 Q : Q 1 Q 2 2f +1 M-Availability Q n f { Q } n +2f +1 Q = Q U : Q = 2 B = {B U : B = f} D-Consisteny Q 1,Q 2 Q : Q 1 Q 2 f +1 D-Availability Q n f { Q } n + f +1 Q = Q U : Q = 2 n n n 4f +1 n 3f +1

4 A safe read/write protool Tradeoffs Write(d) Ask all servers for their urrent timestamp t Wait for answer from Q different servers Set ts > ma( {t} any previous ts) Send (d,ts) to all servers Wait for Q aknowledgments Best known n. self-verifying generi 3f +1 4f +1 Read() Ask all servers for latest value/timestamp pair Wait for answer from Q different servers Selet most reent (v,ts) for whih at least f + 1 answers agree (if any) verifiable self-verifying and generi Tradeoffs Best known n. 3f +1 A Byzantine Renaissane Pratial Byzantine Fault-Tolerane (CL99, CL00) safe in asynhronous systems live under weak synhrony assumptions -Byzantine Paos! fast! PBFT uses MACs instead of publi key ryptography uses proative reovery to tolerate more failures over f window BASE (RCL 01) uses abstration to redue orrelated faults

5 The Setup Crypto System Model Asynhronous system Unreliable hannels Servie Byzantine lients Up to f Byzantine servers N > 3f total servers Publi/Private key pairs MACs Collision-resistant hashes Unbreakable System Goals Always safe Live during periods of synhrony What ould possibly go wrong? The ould be faulty! ould ignore ommands; assign same sequene number to different requests; skip sequene numbers; et Bakups monitor primary s behavior and trigger view hanges to replae faulty primary Bakups ould be faulty! ould inorretly store ommands forwarded by a orret primary use dissemination Byzantine quorum systems [MR98] Faulty replias ould inorretly respond to the lient! The General Idea -bakup + quorum system eeutions are sequenes of views lients send signed ommands to primary of urrent view primary assigns sequene number to lient s ommand primary writes sequene number to the register implemented by the quorum system 67S@76 4K 3>> F:7 E7DH7DE (primary inluded) What ould possibly go wrong? The ould be faulty! ould ignore ommands; assign same sequene number to different requests; skip sequene numbers; et Bakups monitor primary s behavior and trigger view hanges to replae faulty primary Bakups ould be faulty! ould inorretly store ommands forwarded by a orret primary use dissemination Byzantine quorum systems [MR98] Faulty replias ould inorretly respond to the lient! Client waits for f +1 mathing replies before aepting response

6 Me, or your lying eyes? Sets (quorums) of signed messages from distint replias proving that a property of interest holds With quorums of size at least 2f +1 Any two quorums interset in at least one orret replia Always one quorum ontains only non-faulty replias Normal operation How the protool works in the absene of failures - hopefully, the ommon ase View hanges How to depose a faulty primary and elet a new one Garbage olletion Reovery How to make a faulty replia behave orretly again Normal Operation Pre-prepare assigns sequene number to request Prepare ensures fault-tolerant onsistent ordering of requests within views Commit ensures fault-tolerant onsistent ordering of requests aross views i Servie state A message log with all messages sent or reeived An integer representing i s urrent view Bakup 1 Bakup 2 Bakup 3 Client issues request <REQUEST,o,t, > σ

7 Client issues request <REQUEST,o,t, > σ state mahine operation Client issues request <REQUEST,o,t, timestamp > σ Bakup 1 Bakup 1 Bakup 2 Bakup 3 Bakup 2 Bakup 3 Client issues request <REQUEST,o,t, > σ lient id Client issues request <REQUEST,o,t, > σ lient signature Bakup 1 Bakup 1 Bakup 2 Bakup 3 Bakup 2 Bakup 3

8 Pre-prepare Pre-prepare View multiasts <<PRE-PREPARE,v,n,d>, m> σ p multiasts <<PRE-PREPARE,v,n,d>, m> σ p Bakup 1 Bakup 1 Bakup 2 Bakup 3 Bakup 2 Bakup 3 Pre-prepare Sequene number multiasts <<PRE-PREPARE,v,n,d>, m> σ p Pre-prepare lient s request multiasts <<PRE-PREPARE,v,n,d>, m> σ p Bakup 1 Bakup 1 Bakup 2 Bakup 3 Bakup 2 Bakup 3

9 Pre-prepare digest of m multiasts <<PRE-PREPARE,v,n,d>, m> σ p Pre-prepare multiasts <<PRE-PREPARE,v,n,d>, m> σ p Bakup 1 Bakup 1 Bakup 2 Bakup 2 Bakup 3 Bakup 3 Corret bakup i aepts PRE-PREPARE PRE-PREPARE is well formed i is in view v i has not aepted another PRE-PREPARE for v, n with a different d n is between two water-marks L and H (to prevent sequene number ehaustion) Pre-prepare Prepare multiasts <<PRE-PREPARE,v,n,d>, m> σ p Bakup i multiasts <PREPARE,v,n,d,i> σ i Bakup 1 Bakup 1 Bakup 2 Bakup 2 Bakup 3 Eah aepted PRE-PREPARE message is stored in the aepting replia s message log (inluding the s) Bakup 3 Pre-prepare phase Corret replia i aepts PREPARE PREPARE is well formed i is in view v n is between two water-marks L and H

10 Prepare Prepare Bakup i multiasts <PREPARE,v,n,d,i> σ i ensure total order within views Bakup 1 Bakup 2 Bakup 3 Pre-prepare phase Replias that send PREPARE aept seq.# n for m in view v Eah aepted PREPARE message is stored in the aepting replia s message log Prepare Prepare ensure total order within views (m,v,n) The request m A PRE-PREPARE for m in view v with sequene number n 2f PREPARE from different bakups that math the preprepare ensure total order within views (m,v,n) The request m A PRE-PREPARE for m in view v with sequene number n 2f PREPARE from different bakups that math the preprepare (m,v,n) assigning sequene number n to m in view v (m 1,v,n) (m 2,v,n)

11 are not enough Commit replia i multiasts <COMMIT,v,n,d,i> σ i orret replias has agreed on a sequene number for a lient s request leader eleted in a view hange Bakup 1 Bakup 2 Bakup 3 Pre-prepare phase Prepare phase Commit phase Commit ensure total order aross views Reply After eeuting request, replia i replies with <REPLY,v,t,,i,r > σi A replia has a (m,v,n) Bakup 1 (m,v,n) log ontains 2f +1 mathing COMMIT from different replias (inluding itself) Replia eeutes a request after it gets C- with smaller sequene numbers Bakup 2 Bakup 3 Pre-prepare phase Prepare phase Commit phase Reply phase

12 Au armes les bakups! stops aepting messages (but for VIEW-CHANGE & NEW-VIEW) multiasts <VIEW-CHANGE,v+1, P > σi P i A bakup joins mutiny after seeing distint VIEW-CHANGE messages f +1 Mutiny sueeds if new primary ollets a, V indiating support from 2f +1 distint replias (inluding itself) v+1 the new primary The primary elet ˆp (replia v+1 mod N ) V the highest sequene number h of any message V v+1 the new primary The primary elet ˆp (replia v+1 mod N ) V the highest sequene number h of any message V h v+1 the new primary The primary elet ˆp (replia v+1 mod N ) V the highest sequene number h of any message V two sets O and N n,m in V, n h O = O <PRE-PREPARE,v+1,n,m> n h N = N <PRE-PREPARE,v+1,n,null > σ ˆp σ ˆp

13 FA H;7I v+1 the new primary The primary elet p (replia v+1 mod N ) V 7JFD35FE 8DA? H;7I 57DF;S53F7 the highest sequene number h of any message V 8AD I:;5: 5A@F3;@E 3 ' 57DF;S53F7 two sets O and N "8 F:7D7 ;E 3 ' 57DF;S53F7 8AD n,m in V, n h O = O <PRE-PREPARE,v+1,n,m>σp n h &F:7DI;E7 ;8 ' 57DF;S53F7 N = N <PRE-PREPARE,v+1,n,null >σp p multiasts <NEW-VIEW,v+1,V,O, N>σp &@ FA H;7I v+1 the bakup Bakup aepts NEW-VIEW message for v+1 if it is signed properly it ontains in V a valid VIEW-CHANGE messages for v+1 it an verify loally that O is orret (repeating the primary s omputation) Adds all entries in O to its log (so did p!) Multiasts a PREPARE for eah message in O Adds all PREPARE to log and enters new view Why then another BFT protool? Yes Zyzzyva PBFT Yes HQ High ontention? Yes # Replias 5f+1? No Low lateny? No No PBFT Q/U Comple deision tree hampers BFT adoption

14 Simplify, simplify H.D. Thoreau Yes PBFT Yes High ontention? Yes # Replias 5f+1? HQ Simplify, simplify H.D. Thoreau No Low lateny? No BFT? Yes No Zyzzyva PBFT Q/U One protool that mathes or tops its ompetitors in lateny Replia oordination All orret replias eeute the same sequene of ommands throughput ost of repliation How it is done now Voter Command For eah reeived ommand, 5ADD75F D7B>;53E Agree on s position in the sequene Eeute in the agreed upon order Replies to the lient Agreement Eeution

15 How Zyzzyva does it Stability Command Voter A ommand is stable at a replia one its position in the sequene annot hange RSM Safety RSM Liveness Agreement Eeution Corret lients only proess replies to stable ommands All ommands issued by orret lients eventually beome stable and eliit a reply Enforing safety Corret lients only proess replies to stable ommands Corret replias only eeute and reply to ommands that are stable Servie performs an output ommit with eah reply Trust, but Verify Insight output ommit at the lient, not at the servie! Replias eeute and reply to a ommand without knowing whether it is stable trust order provided by primary no epliit replia agreement! that it orresponds to stable ommand if not, lient takes ation to ensure liveness

16 Verifying stability Command History A ommand an beome stable only if a majority of orret replias agree on its position in the sequene a majority of orret replias agrees on s position the set of replies is inompatible, for all possible future eeutions, with a majority of orret replias agreeing on a different ommand holding s urrent position H i,k k ommands eeuted by replia i On reeipt of a ommand from the primary, replia appends to its ommand history the appliation-level response the orresponding ommand history r 1,H 1,k Voter A majority of orret replias agrees on s position (all do!), k, k, k r 2,H 2,k r 3,H 3,k r 4,H 4,k New primary determines k-th ommand by asking n f replias for their H r 1 =...= r 4 H 1,k =...= H 4,k

17 A majority of orret replias agrees on s position (all do!) New primary determines k-th ommand by asking n f replias for their H A majority of orret replias agrees on s position (all do!) New primary determines k-th ommand by asking n f replias for their H orret replias agree A majority of orret replias agrees on s position (all do!) r 1,H 1,k Voter New primary determines s position by asking n f replias for their H It is impossible for a majority of orret replias to agree on a different ommand for s position, k r 2,H 2,k, k r 3,H 3,k, k At least 2f +1 replies math

18 A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H

19 A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H A majority of orret replias agrees on s position New primary determines k-th ommand by asking n f replias for their H

20 orret replias agree A majority of orret replias agrees on s position r i,h i,k Voter CC H 1,k,...,H 4,k New primary determines k-th ommand by asking n f replias for their H, k Not safe! Client sends to all a ontaining 2f +1 mathing histories orret replias agree r 1,H 1,k, k CC Voter aks Client proesses response if it reeives at least 2f +1 aks replias agreed on s position New primary determines k-th ommand by ontating n f replias This set ontains at least one orret Inompatible with a majority baking a different ommand for that position

21 Stability and ommand histories Stability depends on mathing ommand histories Stability is If a ommand with sequene number n is stable, then so is every ommand with sequene number n <n Voter r 1,H 1,k, k r 2,H 2,k, k, k Fewer than 2f +1 replies math Clients retransmits to all replias-hinting primary may be faulty Zyzzyva reap The Case of the Missing Phase Output ommit at the lient, not the servie Replias eeute requests without epliit agreement stable ommand At most 2 phases within a view to make ommand stable Command Pre-prepare Prepare Commit Client proesses response if it reeives at least f +1 mathing replies after ommit phase Voter

22 The Case of the Missing Phase The Case of the Missing Phase Voter Voter Command Command Pre-prepare Unanimity Pre-prepare Prepare Majority it The Case of the Missing Phase replaing the primary Command BFT Voter In PBFT, a replia that suspets primary is faulty goes unilaterally on strike Stops proessing messages in the view Third Commit phase needed for liveness Pre-prepare Prepare Commit Where did the third phase go? Why was it there to begin with?

23 replaing the primary In PBFT, a replia that suspets primary is faulty goes unilaterally on strike Stops proessing messages in the view Third Commit phase needed for liveness In Zyzzyva, the replia goes on Tehnion strike Broadasts I hate the primary and keeps on working Stops when sees enough hate mail to ensure all orret replia will stop as well Etra phase is moved to the unommon ase Faulty lients an t affet safety Faulty lients annot reate inonsistent Clients annot fabriate ommand histories, as they are signed by replias It is impossible to generate a valid ommit any stable request Olly Olly Oen Free! or, faulty lients an t affet liveness Olly Olly Oen Free! or, faulty lients an t affet liveness Faulty lient omits to send CC for Replias ommit histories are unaffeted! Later orret lient who establishes > stable frees as well is