Zyzzyva: Speculative Byzantine Fault Tolerance By Ramakrishna Kotla,* Allen Clement, Edmund Wong, Lorenzo Alvisi, and Mike Dahlin

Transcription

1 : Speulative Byzantine Fault Tolerane By Ramakrishna Kotla,* Allen Clement, Edmund Wong, Lorenzo Alvisi, and Mike Dahlin doi: / Abstrat A longstanding vision in distributed systems is to build reliable systems from unreliable omponents. An entiing formulation of this vision is Byzantine fault-tolerant (BFT) state mahine repliation, in whih a group of servers olletively at as a orret server even if some of the servers misbehave or malfuntion in arbitrary ( Byzantine ) ways. Despite this promise, pratitioners hesitate to deploy BFT systems at least partly beause of the pereption that BFT must impose high overheads. In this artile, we present, a protool that uses speulation to redue the ost of BFT repliation. In, replias reply to a lient s request without first running an expensive three-phase ommit protool to agree on the order to proess requests. Instead, they optimistially adopt the order proposed by a primary server, proess the request, and reply immediately to the lient. If the primary is faulty, replias an beome temporarily inonsistent with one another, but lients detet inonsistenies, help orret replias onverge on a single total ordering of requests, and only rely on responses that are onsistent with this total order. This approah allows to redue repliation overheads to near their theoretial minima and to ahieve throughputs of tens of thousands of requests per seond, making BFT repliation pratial for a broad range of demanding servies. 1. INTRODUCTION Mounting evidene suggests that real systems must ontend not only with simple rashes but also with more omplex failures ranging from hardware data orruption 22 to nondeterministi software errors 25 to seurity breahes. Suh failures an ause even highly engineered servies to beome unavailable or to lose data. For example, a single orrupted bit in a handful of messages reently brought down the Amazon S3 storage servie for several hours, 3 and several well-known e- mail servie providers have oasionally lost ustomer data. 14 Byzantine fault-tolerant (BFT) state mahine repliation is a promising approah to masking many suh failures and onstruting highly reliable and available servies. In BFT repliation, n 3f + 1 servers olletively at as a orret server even if up to f servers misbehave or malfuntion in arbitrary ( Byzantine ) ways. 15,16 Today, three trends make real-world deployment of BFT inreasingly attrative. First, as noted above, there is mounting evidene of non-fail-stop behaviors in real systems, motivating the use of new tehniques to improve robustness. Seond, the growing value of data and the falling osts of hardware make it advantageous for servie providers to trade inreasingly inexpensive hardware for the peae of * This work was mostly done when the author was at the University of Texas at Austin. mind potentially provided by BFT repliation. Third, improvements to the state of the art in BFT algorithms 1,4,6,13,23,26 have narrowed the gap between BFT repliation osts and the osts already being paid for non-bft repliation by many ommerial servies. For example, by default, the Google file system uses three-way repliation of storage, 9 whih is roughly the ost of tolerating one Byzantine failure by using three full replias plus one additional lightweight node to help the replias oordinate their ations. 26 Unfortunately, pratitioners hesitate to deploy BFT systems at least partly beause of the pereption that BFT must impose high overheads. This onern motivates our work, whih seeks to answer a simple question: Can we build a system that tolerates a broad range of faults while meeting the demands of high-performane servies? To answer this question, this artile presents. seeks to make BFT repliation deployable for the widest range of pratial servies by implementing the extremely general abstration of a repliated state mahine at an extremely low ost. The basi idea of BFT state mahine repliation is simple: a lient sends a request to a repliated servie and the servie s distributed agreement protool ensures that orret servers exeute the same requests in the same order. 24 If the servie is deterministi, eah orret replia thus traverses the same series of states and produes the same reply to eah request. The servers send their replies bak to the lient, and the lient aepts a reply that mathes aross a suffiient number of servers. builds on this basi approah, but redues its ost through speulation. As is ommon in existing BFT repliation protools, an eleted primary server proposes an order on lient requests to the other server replias. 4 However, unlike in traditional protools, replias then immediately exeute requests speulatively, without running an expensive agreement protool to definitively establish the order. As a result, if the primary is faulty, orret replias states may diverge, and they may send different responses to a lient. Nonetheless, preserves orretness beause a orret lient detets suh divergene and avoids ating on a reply until the reply and the sequene of preeding requests are stable and guaranteed to eventually be adopted (ZIZ-uh-vuh) is the last word in most ditionaries. Aording to ditionary.om, a zyzzyva is any of various South Amerian weevils of the genus, often destrutive to plants. A previous version of this paper was published in Proeedings of 21st ACM Symposium on Operating Systems Priniples (SOSP), Otober 2007, p november 2008 vol. 51 no. 11 ommuniations of the am 86

2 researh highlights by all orret servers. Thus, appliations at orret lients observe the traditional abstration of a repliated state mahine that exeutes requests in a linearizable 10 order. Essentially, rethinks the syn 19 for BFT. Whereas past BFT systems have pessimistially enfored the ondition that a orret server only emits replies that are stable, reognizes that this ondition is stronger than required. Instead, enfores the weaker ondition: a orret lient only ats on replies that are stable. This hange allows us to move the output ommit from the servers to the lient, whih in the optimized ase allows servers to avoid expensive all-to-all ommuniation that they would otherwise require to ensure the stronger ondition. Leveraging the lient in this way allows us to minimize server overheads and maximize throughputs in the optimized, failure-free ase. As a result, s peak measured throughput of over 86K requests/seond on 3.0 GHz Pentium-IV mahines makes it feasible to utilize BFT repliation in a broad range of demanding servies. Despite this aggressive optimization to the fault-free ase, retains good performane of over 82K requests/seond even when up to f bakup replias rash. In fat, s repliation osts, proessing overheads, and ommuniation latenies approah their theoretial lower bounds. 2. SYSTEM MODEL To maximize fault tolerane, BFT repliation assumes what is essentially an adversarial failure model. Under this model, faulty nodes (servers or lients) may deviate from their intended behavior in arbitrary ways, representing problems suh as hardware faults, software faults, node misonfigurations, or even maliious attaks. This model further assumes a strong adversary that an oordinate faulty nodes to ompromise the repliated servie. Note, however, that our model assumes the adversary annot break ryptographi tehniques like ollision-resistant hashes, enryption, and signatures; we denote a message m signed by prinipal q s publi key as m sq. ensures its safety and liveness properties if at most f replias are faulty, and it assumes a finite lient population, any number of whih may be faulty. It makes little sense to build a system that an tolerate Byzantine replias/servers and lients but that an be orrupted by an unexpetedly slow node or network link, hene we design so that its safety properties hold in any asynhronous distributed system where nodes operate at arbitrary speeds and are onneted by a network that may fail to deliver messages, orrupt them, delay them, or deliver them out of order. Unfortunately, ensuring both safety and liveness for onsensus in an asynhronous distributed system is impossible if any server an rash, 8 let alone if servers an be Byzantine. s liveness, therefore, is ensured only during intervals in whih messages sent to orret nodes are proessed within some arbitrarily large fixed (but potentially unknown) worst-ase delay from when they are sent. This assumption appears easy to meet in pratie if broken links are eventually repaired. implements a BFT servie using state mahine repliation. 16,24 Traditional state mahine repliation tehniques We use the terms replia and server interhangeably. an be applied only to deterministi servies. opes with the nondeterminism present in many real-world appliations suh as file systems and databases using standard tehniques to abstrat the observable appliation state at the replias and to resolve nondeterministi hoies via the agreement stage. 23 If a lient of a servie issues an erroneous or maliious request, s job is to ensure that the request is proessed onsistently at all orret replias; the repliated servie, itself, is responsible for proteting its appliation state from suh erroneous requests. Servies typially limit the damage by authentiating lients and enforing aess ontrol. For example, in a repliated file system, if a lient tries to write a file without appropriate redentials, orret replias ould all proess the request by returning an error ode. 3. AGREEMENT PROTOCOL is a state mahine repliation protool exeuted by 3f + 1 replias and based on three subprotools: (1) agreement, (2) view hange, and (3) hekpoint. The agreement subprotool orders requests for exeution by the replias. Agreement operates within a sequene of views, and in eah view a single replia, designated the primary, is responsible for leading the agreement subprotool. The view hange subprotool oordinates the eletion of a new primary when the urrent primary is faulty or the system is running slowly. The hekpoint subprotool limits the state that must be stored by replias and redues the ost of performing view hanges. For simpliity, this artile fouses on the agreement subprotool. The view hange and exeution subprotools are similar to those used previously. 4,26 Interested readers may refer to Kotla et al. 11 for the full protool. Figure 1 shows the ommuniation pattern for a single instane of s agreement subprotool. In the fast, no-fault ase (Figure 1a), a lient simply sends a request to the primary, the primary forward the request to the replias, and the replias Figure 1: Protool ommuniation pattern for agreement within a view for (a) the fast ase and (b) the two-phase faulty replia ase. The numbers refer to the main steps of the protool in the text. Client Primary Replia 1 Replia 2 Replia 3 Client Primary Replia 1 Replia 2 Replia X 2 3f+1 Appliation Speulative exeution (a) Fast ase f+1 4a 4b 5 Speulative exeution Commit (b) Two-phase ase 2f+1 Appliation 6 87 ommuniations of the am november 2008 vol. 51 no. 11

3 exeute the request and send their responses to the lient. A request ompletes at a lient when the lient has a suffiient number of mathing responses to ensure that all orret replias will eventually exeute the request and all preeding requests in the same order, thus guaranteeing that all orret replias proess the request in the same way, issue the same reply, and transition to the same subsequent system state. To allow a lient to determine when a request ompletes, a lient reeives from replias responses that inlude both an appliation-level reply and the history on whih the reply depends. The history is the sequene of all requests exeuted by a replia prior to and inluding this request. As Figure 1 illustrates, a request ompletes at a lient in one of two ways. First, if the lient reeives 3f + 1 mathing responses (Figure 1a), then the lient onsiders the request omplete and ats on it. Seond, if the lient reeives between 2f + 1 and 3f mathing responses (Figure 1b), then the lient gathers 2f + 1 mathing responses and distributes this ommit ertifiate to the replias. A ommit ertifiate inludes ryptographi proof that 2f + 1 servers agree on a linearizable order for the request and all preeding requests, and suessfully storing a ommit ertifiate to 2f + 1 servers (and thus at least f + 1 orret servers) ensures that no other ordering an muster a quorum of 2f + 1 servers to ontradit this order. Therefore, one 2f + 1 replias aknowledge reeiving a ommit ertifiate, the lient onsiders the request omplete and ats on the orresponding reply. then ensures the following safety ondition: saf If a request with sequene number n and history h n ompletes, then any request that ompletes with a higher sequene number n' n has a history h n' that inludes h n as a prefix. If fewer than 2f + 1 responses math, then to ensure liveness the lient retransmits the request to all replias at inreasing intervals, and replias demand that the primary orders retransmitted requests. If the primary orders requests too slowly or orders requests inonsistently, a replia will suspet that the primary is faulty. If a suffiient number of replias suspet that the primary is faulty, then a view hange ours and a new primary is eleted. thereby ensures the following liveness ondition assuming eventual synhrony : liv Any request issued by a orret lient eventually ompletes. In the rest of this setion, we detail s agreement subprotool by onsidering three ases: (1) the fast ase when all nodes at orretly and no timeouts our, (2) the twophase ase that an our when a nonprimary replia is faulty or some timeouts our, and (3) the view hange ase that an our when the primary is faulty or more serious timeouts our. Table 1 summarizes the labels we give to fields in messages. Most readers will be happier if on their first reading they skip the text marked Additional Pedanti Details. In pratie, eventual synhrony 7 an be ahieved by using exponentially inreasing timeouts. 4 Table 1: Labels given to fields in messages. Label Meaning CC d Client ID Commit ertifiate Digest (ryptographi one-way hash) of lient request message: d = H(m) i, j Server IDs h n m max n n ND o OR POM r t u History through sequene number h n enoded as ryptographi one-way hash: h n = H(h n 1, d) Message ontaining lient request Maximum sequene number aepted by replia Sequene number Seletion of nondeterministi values needed to exeute a request Operation requested by lient Order request message Proof of misbehavior Appliation reply to a lient operation Time stamp assigned to an operation by a lient View number 3.1. Fast ase Figure 1a illustrates the basi flow of messages in the fast ase. We trae these messages through the system to explain the protool, with the numbers in the figure orresponding to the numbers of major steps in the text. As the figure illustrates, the fast ase proeeds in four major steps: 1. Client sends request to the primary. 2. Primary reeives request, assigns sequene number, and forward ordered request to replias. 3. Replia reeives ordered request, speulatively exeutes it, and responds to the lient. 4a. Client reeives 3f + 1 mathing responses and ompletes the request. To ensure orretness, the messages are arefully onstruted to arry suffiient information to link these steps with one another and with past system ations. We now detail the ontents of eah message and desribe the steps eah node takes to proess eah message Message proessing details 1. Client sends request to the primary. A lient requests an operation o be performed by the repliated servie by sending a message request, o, t, s to the replia it believes to be the primary (i.e., the primary for the last response the lient reeived). november 2008 vol. 51 no. 11 ommuniations of the am 88

4 researh highlights Additional Pedanti Details: If the lient guesses the wrong primary, the retransmission mehanisms disussed in step 4 below forward the request to the urrent primary. The lient s time stamp t is inluded to ensure exatly one semantis of exeution of requests Primary reeives request, assigns sequene number, and forwards ordered request to replias. A view s primary has the authority to propose the order in whih the system should exeute requests. It does so by produing order-req messages in response to lient request messages. In partiular, when the primary p reeives message m = request, o, t, s from lient, the primary assigns to the request a sequene number n in the urrent view u and relays a message order-req, u, n, h n, d, ND s, m to the bakup replias where n and u indiate the proposed sequene p number and view number for m, digest d = H(m) is the ryptographi one-way hash of m, h n = H(h n 1, d) is a ryptographi hash summarizing the history, and ND is a set of values for nondeterministi appliation variables (time in file systems, loks in databases, et.) required for exeuting the request. Additional Pedanti Details: The primary only takes the above ations if t > t where t is the highest time stamp previously reeived from. 3. Replia reeives ordered request, speulatively exeutes it, and responds to the lient. When a replia reeives an order-req message, it optimistially assumes that the primary is orret and that other orret replias will reeive the same request with the same proposed order. It therefore speulatively exeutes requests in the order proposed by the primary and produes a speresponse message that it sends to the lient. In partiular, upon reeipt of a message order-req, u, n, h n, d, ND s, m from the primary p, replia i aepts the p ordered request if m is a well-formed request message, d is a orret ryptographi hash of m, u is the urrent view, n = max n + 1 where max n is the largest sequene number in i s history, and h n = H(h n 1, d). Upon aepting the message, i appends the ordered request to its history, exeutes the request using the urrent appliation state to produe a reply r, and sends to a message spe-response, u, n, h n, H(r),, t, i, r, OR where OR = order-req, u, n, h, d, ND. s i n sp Additional Pedanti Details: A replia may only aept and speulatively exeute requests in sequene-number order, but message loss or a faulty primary an introdue holes in the sequene number spae. Replia i disards the order-req message if n max n. If n > max n + 1, then i disards the message, sends a message fill-hole, u, max n + 1, n, i s to the primary, i and starts a timer. Upon reeipt of a message fill-hole, u, k, n, i s from replia i, the primary p sends a order-req, u, n', i h n', d, ND s, m' to i for eah request m that p ordered in k n p n during the urrent view; the primary ignores fill-hole requests from other views. If i reeives the valid order-req messages needed to fill the holes, it anels the timer. Otherwise, the replia broadasts the fill-hole message to all other replias and initiates a view hange when the timer fires. Any replia j that reeives a fill-hole message from i sends the orresponding order-req message, if it has reeived one. If, in the proess of filling in holes in the replia sequene, replia i reeives onfliting order-req messages, then the onfliting messages form a proof of misbehavior (POM) as desribed in protool step 4d. 4a. Client reeives 3f + 1 mathing responses and ompletes the request. In the absene of faults and timeouts, the lient reeives mathing spe-response messages from all 3f + 1 replias. The lient an then onsider the request and its history to be omplete and delivers the reply r to the appliation. 3f + 1 idential replies with idential histories suffie to ensure that a lient an rely on a response. In partiular, 3f + 1 mathing responses means all orret servers have exeuted the request and all preeding requests in the same order, so orret servers an always form a majority to vote to keep this response, even aross view hanges. 11 In partiular, the view hange subprotool exeutes aross 2f + 1 responsive servers, but any group of 2f + 1 servers must inlude at least f + 1 orret servers and at most f faulty servers. Thus, the orret servers are always able to vote to keep this response, inluding both the appliation reply and the history of previous ations. Therefore, upon reeiving 3f + 1 distint messages spe-response, u, n, h n, H(r),, t s, i, r, OR, where i identifies the replia issuing the response, a lient determines i if they math. spe-response messages from distint replias math if they have idential u, n, h n, H(r),, t, OR, and r fields Two-phase ase If the network, primary, or some replias are slow or faulty, the lient may not reeive mathing responses from all 3f + 1 replias. The two-phase ase applies when the lient reeives between 2f + 1 and 3f mathing responses. As Figure 1b illustrates, steps 1 3 our as desribed above, but step 4 is different: 4b. Client reeives between 2f + 1 and 3f mathing responses, assembles a ommit ertifiate, and transmits the ommit ertifiate to the replias. The ommit ertifiate is ryptographi proof that a majority of orret servers agree on the ordering of requests up to and inluding the lient s request. Protool steps 5 and 6 omplete the seond phase of agreement by ensuring that enough servers have this proof. 5. Replia reeives a o m m i t message from a lient ontaining a ommit ertifiate and aknowledges with a l o a l- o m m i t message. 6. Client reeives a l o a l- o m m i t messages from 2f + 1 replias and ompletes the request. Again, the details of message onstrution and proessing 89 ommuniations of the am november 2008 vol. 51 no. 11

5 are designed to allow lients and replias to link the system s ations together into a single linearizable history Message proessing details 4b. Client reeives between 2f + 1 and 3f mathing responses, assembles a ommit ertifiate, and transmits the ommit ertifiate to the replias. A lient sets a timer when it first issues a request. When this timer expires, if has reeived mathing speulative responses from between 2f + 1 and 3f replias, then has a proof that a majority of orret replias agree on the order in whih the request should be proessed. Unfortunately, the replias, themselves, are unaware of this quorum of mathing responses they only know of their loal deision, whih may not be enough to guarantee that the request ompletes in this order. Figure 2 illustrates the problem. A lient reeives 2f + 1 mathing speulative responses indiating that a request req was exeuted as the nth operation in view u. Let these responses ome from f + 1 orret servers C; and f faulty servers F; and assume the remaining f orret servers C' reeived an orderreq message from a faulty primary proposing to exeute a different request req at sequene number n in view u. Suppose a view hange ours at this time. The view hange subprotool must determine what requests were exeuted with what sequene numbers in view u so that the state in view u + 1 is onsistent with the state in view u. Furthermore, sine up to f replias may be faulty, the view hange subprotool must be able to omplete using information from only 2f + 1 replias. Suppose now that the 2f + 1 replias ontributing state to a view hange operation are one orret server from C, f faulty servers from F, and f orret but misled servers from C'. In this ase, only one of the replias initializing the new view is guaranteed to vote to exeute req as operation n in the new view, while as many as 2f replias may vote to exeute req in that position. Thus, the system annot ensure that view u + 1 s state reflets the exeution of req as the operation with sequene number n. Before lient an rely on this response, it must take additional steps to ensure the stability of this response. The lient Figure 2: Example of a problem that ould our if a lient were to rely on just 2f + 1 mathing responses without depositing a ommit ertifiate with the servers. C F C Client Server (orret) Server (orret) Pimary (failed) Server (orret, misled) ORDER-REQ(v, n, h n, H(req),...), req ORDER-REQ(v, n, h n, H(req),...), req ORDER-REQ(v, n, h n, H(req ),...), req Client reeives 2f+1 mathing SPEC-RESP messages SPEC-RESP (v, n, h n,...) SPEC-RESP (v, n, h n,...) View hange "req is operation n" "req is operation n" "req is operation n" therefore sends a message ommit,, CC s where CC is a ommit ertifiate onsisting of a list of 2f + 1 replias, the replia- signed portions of the 2f + 1 mathing spe-response messages from those replias, and the orresponding 2f + 1 replia signatures. Additional Pedanti Details: CC ontains 2f + 1 signatures on the spe-response message and a list of 2f + 1 nodes, but, sine all the responses reeived by from replias are idential, only needs to inlude one replia-signed portion of the spe-response message. Also note that, for effiieny, CC does not inlude the body r of the reply but only the hash H(r). 5. Replia reeives a o m m i t message from a lient ontaining a ommit ertifiate and aknowledges with a l o a l- o m m i t message. When a replia i reeives a message ommit,, CC s ontaining a valid ommit ertifiate CC proving that a request should be exeuted with a speified sequene number and history in the urrent view, the replia first ensures that its loal history is onsistent with the one ertified by CC. If so, replia i (1) stores CC if CC s sequene number exeeds the stored ertifiate s sequene number and (2) sends a message loal-ommit, u, d, h, i, s to. i Additional Pedanti Details: If the loal history simply has holes enompassed by CC s history, then i fills them as desribed in 3. If, however, the two histories ontain different requests for the same sequene number, then i initiates the view hange subprotool. Note that as the view hange protool exeutes, orret replias onverge on a single ommon history, and those replias whose loal state reflet the wrong history (e.g., beause they speulatively exeuted the wrong requests) restore their state from a ryptographially signed distributed hekpoint Client reeives a l o a l- o m m i t messages from 2f + 1 replias and ompletes the request. The lient resends the ommit message until it reeives orresponding loal-ommit messages from 2f + 1 distint replias. The lient then onsiders the request to be omplete and delivers the reply r to the appliation. 2f + 1 loal ommit messages suffie to ensure that a lient an rely on a response. In partiular, at least f + 1 orret servers store a ommit ertifiate for the response, and sine any ommit or view hange requires partiipation by at least 2f + 1 of the 3f + 1 servers, any subsequent ommitted request or view hange inludes information from at least one orret server that holds this ommit ertifiate. Sine the ommit ertifiate inludes 2f + 1 signatures vouhing for the response, even a single orret server an use the ommit ertifiate to onvine all orret servers to aept this response (inluding the appliation reply and the history.) Additional Pedanti Details: When the lient first sends the ommit message to the replias it starts a timer. If this timer november 2008 vol. 51 no. 11 ommuniations of the am 90

6 researh highlights expires before the lient reeives 2f + 1 loal-ommit messages then the lient moves on to protool steps desribed in the next subsetion Client trust At first glane, it may appear imprudent to rely on lients to transmit ommit ertifiates to replias (4b): what if a faulty lient sends an altered ommit ertifiate (threatening safety) or fails to send a ommit ertifiate (imperiling liveness)? Safety is ensured even if lients are faulty beause ommit ertifiates are authentiated by 2f + 1 replias. If a lient alters a ommit ertifiate, orret replias will ignore it. Liveness is ensured for orret lients beause ommit ertifiates are umulative: suessfully storing a ommit ertifiate for request n at 2f + 1 replias ommits those replias to a linearizable total order for all requests up to request n. So, if a faulty lient fails to deposit a ommit ertifiate, that lient may not learn when its request ompletes, and a replia whose state has diverged from its peers may not immediately disover this fat. However, if at any future time a orret lient issues a request, that request (and a linearizable history of earlier requests on whih it depends) will either (1) omplete via 3f + 1 mathing responses (4a), (2) omplete via suessfully storing a ommit ertifiate at 2f + 1 replias (4b 6), or (3) trigger a view hange (4 or 4d below) Timeout and view hange ases Cases 4a and 4b allow a lient s request to omplete with 2f + 1 to 3f + 1 mathing responses. However, if the primary or network is faulty, may not reeive mathing spe-response or loal-ommit messages from even 2f + 1 replias. Cases 4 and 4d therefore ensure that a lient s request either ompletes in the urrent view or that a new view with a new primary is initiated. In partiular, ase 4 is triggered when a lient reeives fewer than 2f + 1 mathing responses and ase 4 ours when a lient reeives responses indiating inonsistent ordering by the primary. 4. Client reeives fewer than 2f + 1 mathing s p e -r e s p o n s e messages and resends its request to all replias, whih forward the request to the primary in order to ensure that the request is assigned a sequene number and eventually exeuted. A lient sets a seond timer when it first issues a request. If the seond timer expires before the request ompletes, the lient suspets that the primary may not be ordering requests as intended, so it resends its request message through the remaining replias so that they an trak the request s progress and, if progress is not satisfatory, initiate a view hange. This ase an be understood by examining the behavior of a nonprimary replia and of the primary. Replia. When nonprimary replia i reeives a message request, o, t, s from lient, then if the request has a higher time stamp than the urrently ahed response for Note that ases 4b and 4 are not exlusive of 4d; a lient may reeive messages that are both suffiient to omplete a request and also a proof a misbehavior against the primary. that lient, i sends a message onfirm-req, u, m, i where s i m = request, o, t, s to the primary p and starts a timer. If the replia aepts an order-req message for this request before the timeout, it proesses the order-req message as desribed above. If the timer fires before the primary orders the request, the replia initiates a view hange. Primary. Upon reeiving the message onfirm-req, u, m, i s from replia i, the primary p heks the lient s time i stamp for the request. If the request is new, p sends a new order-req message using a new sequene number as desribed in step 2. Additional Pedanti Details: If replia i does not reeive the order-req message from the primary, the replia sends the onfirm-req message to all other replias. Upon reeipt of a onfirm-req message from another replia j, replia i sends the orresponding order-req message it reeived from the primary to j; if i did not reeive the request from the lient, i ats as if the request ame from the lient itself. To ensure eventual progress, a replia doubles its urrent timeout in eah new view and resets it to a default value if a view sueeds in exeuting a request. Additionally, to retain exatly one semantis, replias maintain a ahe that stores the reply to eah lient s most reent request. If a replia i reeives a request from a lient and the request mathes or has a lower lient-supplied time stamp than the urrently ahed request for lient, then i simply resends the ahed response to. Similarly, if the primary p reeives an old lient request from replia i, p sends to i the ahed orderreq message for the most reent request from. Furthermore, if replia i has reeived a ommit ertifiate or stable hekpoint for a subsequent request, then the replia sends a loalommit to the lient even if the lient has not transmitted a ommit ertifiate for the retransmitted request. 4d. Client reeives responses indiating inonsistent ordering by the primary and sends a proof of misbehavior to the replias, whih initiate a view hange to oust the faulty primary. If lient reeives a pair of spe-response messages ontaining valid messages OR = order-req, u, n, h n, d, ND s j for the same request (d = H(m) ) in the same view u with differing sequene number n or history h n or ND, then the pair of order-req messages onstitutes a proof of misbehavior 2 (POM) against the primary. Upon reeipt of a POM, sends a message pom, u, POM s to all replias. Upon reeipt of a valid pom message, a replia initiates a view hange and forward the pom message to all other replias. 4. EVALUATION This setion examines the performane of and ompares it with existing approahes. We run our experiments on 3.0 GHz Pentium-4 mahines with the Linux 2.6 kernel. We use MD5 for hashing and UMAC 4 for message authentiation odes (MACs). MD5 is known to be vulnerable, but we use it to make our results omparable with those in the literature. Sine uses fewer MACs per request than any of the ompeting algorithms, our advantages over other 91 ommuniations of the am november 2008 vol. 51 no. 11

7 algorithms would be inreased if we were to use the more seure, but more expensive, SHA-256. For omparison, we run Castro and Liskov s implementation of Pratial Byzantine Fault Tolerane () 4 and Cowling et al. s implementation of hybrid quorum () 6 ; we sale-up s measured throughput for the small request/ response benhmark by 9% to aount for their use of SHA- 1 rather than MD5. We inlude published throughput measurements for Q/U 1 ; we sale Q/U s reported performane up by 7.5% to aount for our use of 3.0 GHz rather than 2.8GHz mahines. We also ompare against the measured performane of an unrepliated server. To stress-test we use the mirobenhmarks devised by Castro and Liskov 4 In the 0/0 benhmark, lients send null requests and reeive null replies. In the 4/0 benhmark, lients send 4KB requests and reeive a null replies. In the 0/4 benhmark, lients send null requests and reeive 4KB replies. In all experiments, we onfigure all BFT systems to tolerate f = 1 faults; we examine performane for other onfigurations elsewhere. 11 In the preeding setions, we desribe a simplified version of the protool. In our extended paper, 12 we detail a number of optimizations, all implemented in the prototype measured here, that (1) redue enryption osts by replaing publi key signatures with MACs, 4 (2) improve throughput by agreeing on the order of bathes of requests, 4 (3) redue the impat of lost messages by ahing out-of-order messages, (4) improve read performane by optimizing read-only requests, 4 redue bandwidth by allowing most replias to send hashes rather than full replies to lients, 4 (5) improve the performane of s two-phase ase by using a ommit optimization in whih replias use a lient hint to initiate and omplete the seond phase to ommit the request before they exeute the request and send the response (with the ommitted history) bak to the lient, and (6) redue overheads by inluding MACs only for a preferred quorum. 6 In the extended paper we also desribe 5, a variation of the protool that requires 5f + 1 agreement replias but that improves performane in the presene of faulty replias by ompleting in three one-way message exhanges as in Figure 1a even when up to f nonprimary replias are faulty. In the following experiments, unless noted otherwise, we use all of the optimizations other than preferred quorums for. 4 does not implement the preferred quorum optimization, but does. 6 We do not use the read-only optimization for and unless we state so expliitly Cost model Our evaluation fouses on three metris that BFT repliation must optimize to be pratial for a broad range of servies: repliation ost, throughput, and lateny. Before we dive into experimental evaluation in the following setions, Table 2 puts our results in perspetive by providing a high-level analyti model of and of several other reent BFT protools. The table also shows lower bounds on BFT state mahine repliation overheads for eah of these dimensions. In the first row of the table body, repliation ost refers to the number of replias required to onstrut a system that tolerates f Byzantine faults. The importane of minimizing this metri for pratial servies is readily apparent. We show two values: replias with appliation state indiates the number of replias that must both partiipate in the oordination protool and also maintain appliation state for exeuting appliation requests. Conversely, total replias indiates the total number of mahines that must partiipate in the protool inluding, for some protools, witness nodes that do not maintain appliation state or exeute appliation requests. This distintion is important beause witness nodes may be simpler or less expensive than nodes that must also exeute requests to run the repliated servie. and (with Yin et al. s optimization for separating agreement and exeution 26 ) meet the repliation ost lower bounds of 2f + 1 appliation replias (so a majority of nodes are orret) 24 and 3f + 1 total replias (so agreement on request order an be reahed). 21 In the next row of the table body, throughput is determined by the proessing overhead per request. Our simple model fouses on CPU intensive ryptographi operations. All of the systems we examine use Castro s MAC authentiator onstrut 4 to avoid using expensive asymmetri ryptography operations. Table 2: Properties of state-of-the-art and optimal Byzantine fault-tolerant (BFT) servie repliation systems tolerating f faults, using MACS for authentiation, 4 assuming preferred quorum optimization, and using a bath size of b. 4 4 Q/U 1 6 State Mahine Repliation Lower Bound Cost Total replias 3f + 1 5f + 1 3f + 1 3f + 1 3f Replias with appliation state 2f f + 1 3f + 1 2f + 1 2f Throughput MAC ops at bottlenek server 2 + (8f + 1)/b 2 + 8f 4 + 4f 2 + 3f/b 2 a Lateny NW one-way latenies on ritial path or 3 b Bold entries denote protools that math known lower bounds or those with the lowest known ost. a It is not lear that this trivial lower bound is ahievable. b The distributed systems literature typially onsiders three one-way latenies to be the lower bound for agreement on lient requests 17 ; two one-way latenies is ahievable if no request ontention is assumed. november 2008 vol. 51 no. 11 ommuniations of the am 92

8 researh highlights The (trivial) lower bound on proessing overhead is for eah server to proess two MAC operations per lient request: one to verify the lient s request and one to authentiate its reply. and approah this bound by using a bathing optimization in whih the primary aumulates a bath of b lient requests and leads agreement on that bath rather than on eah individual request. s speulative exeution allows it to avoid several rounds of allto-all ommuniation among servers, so it requires fewer MAC operations per bath than. In the last row of the table body, lateny ounts the number of one-way message delays from when a lient issues a request until it reeives a reply. In the general ase, agreement requires three message delays, 17 and mathes this bound by having requests go from the lient to the primary to the replias to the lient. Q/U irumvents this bound by optimizing for the ase of no request ontention so that requests go diretly from the lient to the replias to the lient. We hose to retain the extra hop through the primary in beause it failitates bathing, whih we onsider to be an important throughput optimization. The models desribed in this subsetion fous on what we regard as important fators for understanding the performane trade-offs of different algorithms, but they neessarily omit details present in implementations. Also, as is ustomary, 1,4,6,23,26 Table 2 ompares the protools performane during the optimized ase of fault-free, timeout-free exeution. In the rest of this setion we experimentally examine these protools throughput, lateny, and performane during failures Throughput Figure 3 shows the throughput measured for the 0/0 benhmark for, 5, 11, and (saled as noted above). For referene, we also show the peak throughput reported for Q/U 1 in the f = 1 onfiguration, saled to our environment as desribed above. exeutes over 50K requests/seond without bathing, and this number rises to 86K requests/seond when bathing is ativated with 10 requests per bath. As the figure illustrates, enjoys a signifiant throughput advantage over the other protools. It is also worth noting that when bathing is enabled, s throughput is within 35% of the throughput of an unrepliated server that simply replies to lient requests over an authentiated hannel. Furthermore, this gap would fall if the servie being repliated were more demanding than the null servie examined here. Overall, we speulate that s throughput is suffiient to support BFT repliation for a broad range of demanding servies Lateny Figure 4 shows the latenies of, 5,, and for the 0/0, 0/4, and 4/0 workloads with a single lient issuing one request at a time. We examine both the default read/ write requests that use the full protool and read-only requests that an exploit and s read-only optimization. 4 We did not sueed in getting Abd-El-Malek et al. s implementation of Q/U running in our environment. However, beause Table 2 suggests that Q/U may have a lateny advantage over other protools, for omparison we implement an idealized model of Q/U designed to provide an optimisti estimate of Q/U s lateny in our environment. In our idealized implementation, a lient simply generates and sends 4f + 1 MACs with a request, eah replia verifies 4f + 1 MACs (1 to authentiate the lient and 4f + 1 to validate the reported state), eah replia in a preferred quorum 6 generates and sends 4f + 1 MACs (1 to authentiate the reply to the lient and 4f to authentiate the new state) with a reply to the lient, and the lient verifies 4f + 1 MACs. For the read/write 0/0 and 4/0 benhmarks, Q/U does have a modest lateny advantage over as predited by Table 2. For the read-only benhmarks, the situation is reversed with exhibiting modestly lower lateny than Q/U beause s read-only optimization ompletes read-only requests in two message delays (like Q/U) but uses fewer ryptographi operations. Figure 5 shows lateny and throughput as we vary offered load for the 0/0 benhmark. As the figure illustrates, bathing in, 5, and inreases lateny but also inreases Figure 3: Realized throughput for the 0/0 benhmark as the number of lient varies. Q/U throughput is saled from 1. Throughput (Kops/s) (b=10) 5 (b=10) (b=10) 5 Q/U max throughput Figure 4: Lateny for 0/0, 0/4, and 4/0 benhmarks. Time (ms) Number of lients 0 0/0 0/0 (r/o) 0/4 0/4 (r/o) 4/0 4/0 (r/o) 93 ommuniations of the am november 2008 vol. 51 no. 11

9 Figure 5: Lateny versus throughput for the 0/0 benhmark. Q/U throughput is saled from 1. Q/U best lateny is the measured lateny for our idealized model implementation of Q/U under low offered load. Lateny per request (ms) Q/U max throughput 5 (b=10) 5 (b=10) (b=10) Q/U best lateny Throughput (Kops/s) Figure 6: Realized throughput for the 0/0 benhmark as the number of lients varies when f = 1 nonprimary replias fail to respond to requests. Throughput (Kops/s) (b=1) 5 (b=10) (b=10) (b=10) without ommit opt (b=10) 5 (b=1) (b=1) without ommit opt (b=1) Number of lients peak throughput. Adaptively setting the bath size in response to workload harateristis is an avenue for future work. Overall, all of the BFT protools do add servie lateny ompared to an unrepliated server, but is generally ompetitive with the best protools by this metri. We speulate that the additional 120 to 250 miroseonds that requires ompared to an unrepliated server will be a signifiant barrier for only the most demanding servies, and we note that the relative gap would shrink for servies that do more than exeute the null request Performane during failures guarantees orret exeution with any number of faulty lients and up to f faulty replias. However, its performane is optimized for the ase of failure-free operation, and a single faulty replia an fore to exeute the slower two-phase protool. One solution is to buttress s fast one-phase path by employing additional servers uses a total of 5f + 1 servers (2f + 1 full replias and 3f additional witnesses) to allow the system to omplete requests via the fast ommuniation pattern shown in Figure 1a when the lient reeives 4f + 1 (out of 5f + 1) mathing replies. Surprisingly, however, even when running with 3f + 1 replias, remains ompetitive with existing protools even when it falls bak on two-phase operation. In partiular, s ryptographi overhead at the bottlenek replia inreases from 2 + ( (3f + 1)/b) to 3 + ( (5f + 1)/b) operations per request if we simply exeute the two-phase algorithm desribed above.** Furthermore, our implementation also inludes a ommit optimization 12 that redues ryptographi overheads to 2 + ( (5f + 1)/b) ryptographi operations per request by having replias that suspet a faulty primary initiate and omplete the seond phase to ommit the request before they exeute the request and send the response (with ** As noted at the start of this setion we omit the preferred quorums optimization in our experimental evaluations, so the 2 + (3f +1)b MAC operations per request in our measurements is higher than the 2 + 3f/b listed in Table 2. the ommitted history) bak to the lient. Figure 6 ompares throughputs of, 5,, and in the presene of f nonprimary-server failstop failures. We do not inlude a disussion of Q/U in this setion as the throughput numbers of Q/U with failures are not reported, 1 but we would not expet a fail-stop failure by a replia to signifiantly redue the performane shown for Q/U in Figure 3. Also, we do not inlude a line for the unrepliated server ase as the throughput falls to zero when the only server suffers a fail-stop failure. As Figure 6 shows, without the ommit optimization, falling bak on two-phase operation redues s maximum throughput from 86K requests/seond (Figure 3) to 52K requests/seond. Despite this extra overhead, s slow ase performane remains within 13% of s performane, whih is less highly tuned for the failure-free ase and whih suffers no slowdown in this senario. s ommit optimization repairs most of the damage aused by a fail-stop replia, maintaining a throughput of 82 K requests/ seond whih is within 5% of the peak throughput ahieved for the failure-free ase. For systems that an afford extra witness replias, 5 s throughput is not signifiantly affeted by the fail-stop failure of a replia, as expeted. 5. RELATED WORK stands on the shoulders of reent efforts that have dramatially ut the osts and improved the pratiality of BFT repliation. Castro and Liskov s protool 4 devised tehniques to eliminate expensive signatures and potentially fragile timing assumptions, and it demonstrated high throughputs of over 10K requests/seond. This surprising result jump started an arms rae in whih researhers redued repliation osts, 26 and improved performane 1,6,13 of BFT servie repliation. inorporates many of the ideas developed in these protools and folds in the new idea of speulative exeution to onstrut an optimized fast path that signifiantly outperforms existing protools and that has repliation ost, proessing overhead, and lateny that approah the theoretial minima for these metris. november 2008 vol. 51 no. 11 ommuniations of the am 94

10 researh highlights Numerous BFT agreement protools 4,6,13,18,23,26 have used tentative exeution to redue the lateny experiened by lients. This optimization allows replias to exeute a request tentatively as soon as they have olleted the equivalent of a ommit ertifiate for that request. This optimization may appear similar to s support for speulative exeution, but there are two fundamental differenes. First, s speulative exeution allows requests to omplete at a lient after a single phase, without the need to ompute a ommit ertifiate: this redution in lateny is not possible with traditional tentative exeutions. Seond, and more importantly, in traditional BFT systems a replia an exeute a request tentatively only after the replia knows that all previous requests have been ommitted. In, replias ontinue to exeute requests speulatively, without waiting to know that requests with lower sequene numbers have ompleted. This differene is what lets leverage speulation to ahieve not just lower lateny but also higher throughput. Speulator 20 allows lients to speulatively omplete operations at the appliation level and perform lient-level rollbak. A similar approah ould be used in onjuntion with to support lients that want to at on a reply optimistially rather than waiting on the speified set of responses. s fous is on maximizing the peak performane of BFT repliation. Conversely, Clement et al. 5 argue that BFT systems should seek not only to ensure safety but also good performane in the presene of Byzantine faults, and their Aardvark system eliminates fragile optimizations that maximize best-ase performane but that an allow a faulty lient or server to drive the system down expensive exeution paths. 6. CONCLUSION By systematially exploiting speulation, exhibits signifiant performane improvements over existing BFT protools. The throughput overheads and lateny of approah the theoretial lower bounds for any BFT state mahine repliation protool. Looking forward, although we expet ontinued progress in improving the performane (for example, by making additional assumptions about the appliation harateristis) and robustness (in the presene of broader range of failure senarios) of BFT repliation, we believe that demonstrates that BFT overheads should no longer be regarded as a barrier to using BFT repliation for even many highly demanding servies. aknowledgments Aknowledgments This work was supported in part by NSF grants CNS , CNS , and CNS We thank Rodrigo Rodrigues, James Cowling, and Mihael Abd-El-Malek for sharing soure ode for,, and Q/U, respetively. We are grateful for Maurie Herlihy s feedbak on earlier drafts of this artile. 3. Amazon s3 availability event: July 20, s html. 4. Castro, M. and Liskov, B. Pratial Byzantine fault tolerane and proative reovery. ACM Transations on Computer Systems, November Clement, A., Marhetti, M., Wong, E., Alvisi, L., and Dahlin, M. Making Byzantine fault tolerant servies tolerate Byzantine faults. Tehnial Report 08-27, UT Austin Department of Computer Sienes, May Cowling, J., Myers, D., Liskov, B., Rodrigues, R., and Shrira, L. repliation: A hybrid quorum protool for Byzantine fault tolerane. Proeedings of the Symposium on Operating Systems Design and Implementation, November Dwork, C., Lynh, N., and Stokmeyer, L. Consensus in the presene of partial synhrony. Journal of the ACM, Fisher, M., Lynh, N., and Paterson, M. Impossibility of distributed onsensus with one faulty proess. Journal of the ACM, 32(2): , April Ghemawat, S., Gobioff, H., and Leung, S. The Google File System. Proeedings of 19th ACM Symposium on Operating Systems Priniples, Otober Herlihy, M. and Wing, J. Linearizability: A orretness ondition for onurrent objets. ACM Transations on Programming Languages and Systems, 12(3), Kotla, R., Alvisi, L., Dahlin, M., Clement, A., and Wong, E. : Speulative Byzantine fault tolerane. ACM Symposium on Operating Systems Priniples, Kotla, R., Alvisi, L., Dahlin, M., Clement, A., and Wong, E. : Speulative Byzantine fault tolerane. Tehnial Report UTCS-TR-07-40, University of Texas at Austin, Kotla, R. and Dahlin, M. Highthroughput Byzantine fault tolerane. Proeedings of the International Conferene on Dependable Systems and Networks, June Kotla, R., Dahlin, M., and Alvisi, L. SafeStore: A durable and pratial storage system. Proeedings of the USENIX Annual Tehnial Conferene, June Lamport, L., Shostak, R., and Pease, Ramakrishna Kotla (kotla@mirosoft.om) Mirosoft Researh Silion Valley, Mountain View, CA. Allen Clement (alement@s.utexas.edu) Department of Computer Sienes, University of Texas, Austin. Edmund Wong (elwong@s.utexas.edu) Department of Computer Sienes, University of Texas, Austin. M. The Byzantine generals problem. ACM Transations on Programming Languages and Systems, 4(3): , July Lamport, L. Using time instead of timeout for fault-tolerant distributed systems. ACM Transations on Programming Languages and Systems, 6(2): , April lamport, L. Lower bounds for asynhronous onsensus. Proeedings of the FUDICO, pp , June Martin, J.-P. and Alvisi, L. Fast Byzantine onsensus. IEEE TODSC, 3(3): , July Nightingale, E., Veeraraghavan, K., Chen, P., and Flinn, J. Rethink the syn. Proeedings of the Symposium on Operating Systems Design and Implementation, Nightingale, E. B., Chen, P., and Flinn, J. Speulative exeution in a distributed file system. Proeedings of the Symposium on Operating Systems Priniples, Otober Pease, M., Shostak, R., and Lamport, L. Reahing agreement in the presene of faults. Journal of the ACM, 27(2): , April Prabhakaran, V., Bairavasundaram, L., Agrawal, N., Arpai-Dusseau, H. G. A., and Arpai-Dusseau, R. IRON file systems. Proeedings of the Symposium on Operating Systems Priniples, Rodrigues, R., Castro, M., and Liskov, B. BASE: Using abstration to improve fault tolerane. Proeedings of the Symposium on Operating Systems Priniples, Otober Shneider, F. B. Implementing faulttolerant servies using the state mahine approah: A tutorial. ACM Computing Surveys, 22(4): , Yang, J., Sar, C., and Engler, D. Explode: A lightweight, general system for finding serious storage system errors. Proeedings of the Symposium on Operating Systems Design and Implementation, yin, J., Martin, J.-P., Venkataramani, A., Alvisi, L., and Dahlin, M. Separating agreement from exeution for Byzantine fault tolerant servies. Pro ee dings of the Symposium on Operating Systems Priniples, Otober Lorenzo Alvisi (lorenzo@s.utexas.edu) Department of Computer Sienes, University of Texas, Austin. Mike Dahlin (dahlin@s.utexas.edu) Department of Computer Sienes, University of Texas, Austin. Referenes 1. Abd-El-Malek, M., Ganger, G., Goodson, G., Reiter, M., and Wylie, J. Fault-salable Byzantine faulttolerant servies. Proeedings of the Symposium on Operating Systems Priniples, Otober Aiyer, A. S., Alvisi, L., Clement, A., Dahlin, M., Martin, J.-P., and Porth, C. BAR fault tolerane for ooperative servies. Proeedings of the Symposium on Operating Systems Priniples, pp , Otober ACM /08/1100 $ ommuniations of the am november 2008 vol. 51 no. 11