Failures, Elections, and Raft

Size: px
Start display at page:

Download "Failures, Elections, and Raft"

Transcription

1 Failures, Elections, and Raft CS 8 XI Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

2 Distributed Banking SFO add interest based on current balance PVD deposit $000 CS 8 XI Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

3 Synchronous vs. Asynchronous Execution speed synchronous: bounded asynchronous: unbounded Message transmission delays synchronous: bounded asynchronous: unbounded Local clock drift rate: synchronous: bounded asynchronous: unbounded CS 8 XI Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

4 Failures Omission failures something doesn t happen - process crashes - data lost in transmission - etc. Byzantine (arbitrary) failures something bad happens - message is modified - message received twice - any kind of behavior, including malicious Timing failures something takes too long CS 8 XI 4 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

5 Detecting Crashes Synchronous systems timeouts Asynchronous systems? Fail-stop an oracle lets us know CS 8 XI 5 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

6 Consensus Setting: a group of processes, some correct, some faulty Two primitives, propose(v) and decide(v) If each correct process proposes a value: Termination: every correct process eventually decides on a value Agreement: if a correct process decides v, then all processes eventually decide v Integrity: if a correct process decides v, then v was previously proposed by some process CS 8 XI 6 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

7 Consensus Variations on the problem, depending on assumptions Synchronous vs asynchronous Fail-stop vs crash/omission failures vs Byzantine failures Equivalent to reliable, totally- and causallyordered broadcast (Atomic broadcast) CS 8 XI 7 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

8 Byzantine Agreement Problem :Retreat :Attack :Retreat :Retreat ::Retreat ::Attack Will cover in a future lecture ::Retreat ::Attack CS 8 XI 8 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

9 Impossibility of Consensus There is no deterministic protocol that solves consensus in a message-passing asynchronous system in which at most one process may fail by crashing. Solve means to satisfy safety and liveness Famous result from Fisher, Lynch, Paterson (FLP), 985 There is hope, though Introducing (some) synchrony timeouts Introducing randomization Introducing failure detectors CS 8 XI 9 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

10 Oceanstore: Replicated Primary Replica File File Inner ring of servers File File CS 8 XI 0 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

11 State-Machine Replication An approach to dealing with failures by replication A data-structure with deterministic operations replicated among servers State consistent if every server sees the same sequence of operations Multiple rounds of consensus Common algorithms (non-byzantine): Paxos [Lamport], Viewstamped Replication [Oki and Liskov], Zab [Junqueira et al.], Raft [Ongaro and Ousterhout] CS 8 XI Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

12 Raft Proposed by Ongaro and Ousterhout in 04 Four components Leader election Log replication Safety Membership changes Assumes crash failures No dependency on time for safety But depends on time for availability CS 8 XI Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

13 Demo CS 8 XI Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

14 Server States At any given time, each server is either: Leader: handles all client interactions, log replication - At most viable leader at a time Follower: completely passive (issues no RPCs, responds to incoming RPCs) Candidate: used to elect a new leader Normal operation: leader, N- followers CS 8 XI 4 Slides based on Raft slides by John Ousterhout

15 Life as a Leader Client sends command to leader Leader appends command to its log Leader sends AppendEntries RPCs to followers Once new entry committed: Leader passes command to its state machine, returns result to client Leader notifies followers of committed entries in subsequent AppendEntries RPCs Followers pass committed commands to their state machines Crashed/slow followers? Leader retries RPCs until they succeed Performance is optimal in common case: One successful RPC to any majority of servers CS 8 XI 5 Slides based on Raft slides by John Ousterhout

16 Heartbeats and Timeouts Servers start up as followers Followers expect to receive RPCs from leaders or candidates Leaders must send heartbeats (empty AppendEntries RPCs) to maintain authority If electiontimeout elapses with no RPCs: Follower assumes leader has crashed Follower starts new election Timeouts typically ms CS 8 XI 6 Slides based on Raft slides by John Ousterhout

17 Terms Term Term Term Term 4 Term 5 time Elections Split Vote Normal Operation Time divided into terms: Election Normal operation under a single leader At most leader per term Some terms have no leader (failed election) Each server maintains current term value Key role of terms: identify obsolete information CS 8 XI 7 Slides based on Raft slides by John Ousterhout

18 Server State Machine CS 8 XI 8 Slides based on Raft slides by John Ousterhout

19 Election Basics Increment current term Change to Candidate state Vote for self Send RequestVote RPCs to all other servers, retry until either:. Receive votes from majority of servers: - Become leader - Send AppendEntries heartbeats to all other servers. Receive RPC from valid leader: - Return to follower state. No-one wins election (election timeout elapses): - Increment term, start new election CS 8 XI 9 Slides based on Raft slides by John Ousterhout

20 Elections, cont d Safety: allow at most one winner per term Each server gives out only one vote per term (persist on disk) Two different candidates can t accumulate majorities in same term B can t also get majority Servers Voted for candidate A Liveness: some candidate must eventually win Choose election timeouts randomly in [T, T] One server usually times out and wins election before others wake up Works well if T >> broadcast time CS 8 XI 0 Slides based on Raft slides by John Ousterhout

21 term command add Log Structure add cmp cmp ret ret mov mov jmp jmp div shl sub log index leader add add cmp cmp ret mov jmp div shl sub followers add cmp ret mov jmp Log entry = index, term, command Log stored on stable storage (disk); survives crashes Entry committed if known to be stored on majority of servers* Durable, will eventually be executed by state machines div shl committed entries CS 8 XI Slides based on Raft slides by John Ousterhout

22 Log Consistency If log entries on different servers have same index and term: They store the same command The logs are identical in all preceding entries add cmp ret mov jmp div add cmp ret mov jmp 4 sub If a given entry is committed, all preceding entries are also committed CS 8 XI Slides based on Raft slides by John Ousterhout

23 AppendEntries Consistency Check Each AppendEntries RPC contains index, term of entry preceding new ones Follower must contain matching entry; otherwise it rejects request Implements an induction step, ensures coherency leader follower 4 5 add add cmp cmp ret ret mov mov jmp AppendEntries succeeds: matching entry leader follower add add cmp cmp ret ret mov shl jmp AppendEntries fails: mismatch CS 8 XI Slides based on Raft slides by John Ousterhout

24 Leader Changes At beginning of new leader s term: Old leader may have left entries partially replicated No special steps by new leader: just start normal operation Leader s log is the truth Will eventually make follower s logs identical to leader s Multiple crashes can leave many extraneous log entries: log index term s s s s 4 s CS 8 XI 4 Slides based on Raft slides by John Ousterhout

25 Safety Requirement Once a log entry has been applied to a state machine, no other state machine must apply a different value for that log entry Raft safety property: If a leader has decided that a log entry is committed, that entry will be present in the logs of all future leaders This guarantees the safety requirement Leaders never overwrite entries in their logs Only entries in the leader s log can be committed Entries must be committed before applying to state machine Restrictions on commitment Committed Present in future leaders logs Restrictions on leader election CS 8 XI 5 Slides based on Raft slides by John Ousterhout

26 Picking the Best Leader Can t tell which entries are committed! 4 5 committed? unavailable during leader transition During elections, choose candidate with log most likely to contain all committed entries Candidates include log info in RequestVote RPCs (index & term of last log entry) Voting server V denies vote if its log is more complete : (lastterm V > lastterm C ) (lastterm V == lastterm C ) && (lastindex V > lastindex C ) Leader will have most complete log among electing majority CS 8 XI 6 Slides based on Raft slides by John Ousterhout

27 Picking the Best Leader Who can be elected for term 8? S, S, S S4 and S5 cannot log index term s s s s 4 s CS 8 XI 7 Slides based on Raft slides by John Ousterhout

28 Committing Entry from Current Term Case #/: Leader decides entry in current term is committed s Leader for term s s AppendEntries just succeeded s 4 s 5 Can t be elected as leader for term Safe: leader for term must contain entry 4 CS 8 XI 8 Slides based on Raft slides by John Ousterhout

29 Committing Entry from Earlier Term Case #/: Leader is trying to finish committing entry from an earlier term s Leader for term 4 s s AppendEntries just succeeded s 4 s 5 Entry not safely committed: s 5 can be elected as leader for term 5 If elected, it will overwrite entry on s, s, and s! CS 8 XI 9 Slides based on Raft slides by John Ousterhout

30 New Commitment Rules For a leader to decide an entry is committed: Must be stored on a majority of servers s Leader for term 4 At least one new entry from leader s term must also be stored on majority of servers Once entry 4 committed: s 5 cannot be elected leader for term 5 s s s 4 s Entries and 4 both safe Combination of election rules and commitment rules makes Raft safe CS 8 XI 0 Slides based on Raft slides by John Ousterhout

31 log index leader for term 8 Log Inconsistencies Leader changes can result in log inconsistencies: (a) (b) Missing Entries possible followers (c) (d) (e) Extraneous Entries (f) CS 8 XI Slides based on Raft slides by John Ousterhout

32 Repairing Follower Logs New leader must make follower logs consistent with its own Delete extraneous entries Fill in missing entries Leader keeps nextindex for each follower: Index of next log entry to send to that follower Initialized to ( + leader s last index) When AppendEntries consistency check fails, decrement nextindex and try again: nextindex log index leader for term followers (a) (b) 4 CS 8 XI Slides based on Raft slides by John Ousterhout

33 Repairing Logs, cont d When follower overwrites inconsistent entry, it deletes all subsequent entries: nextindex log index leader for term follower (before) follower (after) 4 CS 8 XI Slides based on Raft slides by John Ousterhout

34 Neutralizing Old Leaders Deposed leader may not be dead: Temporarily disconnected from network Other servers elect a new leader Old leader becomes reconnected, attempts to commit log entries Terms used to detect stale leaders (and candidates) Every RPC contains term of sender If sender s term is older, RPC is rejected, sender reverts to follower and updates its term If receiver s term is older, it reverts to follower, updates its term, then processes RPC normally Election updates terms of majority of servers Deposed server cannot commit new log entries CS 8 XI 4 Slides based on Raft slides by John Ousterhout

35 Client Protocol Send commands to leader If leader unknown, contact any server If contacted server not leader, it will redirect to leader Leader does not respond until command has been logged, committed, and executed by leader s state machine If request times out (e.g., leader crash): Client reissues command to some other server Eventually redirected to new leader Retry request with new leader CS 8 XI 5 Slides based on Raft slides by John Ousterhout

36 Client Protocol, cont d What if leader crashes after executing command, but before responding? Must not execute command twice Solution: client embeds a unique id in each command Server includes id in log entry Before accepting command, leader checks its log for entry with that id If id found in log, ignore new command, return response from old command Result: exactly-once semantics as long as client doesn t crash Enforces linearizability (will see in upcoming lecture) CS 8 XI 6 Slides based on Raft slides by John Ousterhout

37 Partitioned Leader A client talking to a partitioned leader could be delayed forever. Solution: leader will step down after a number of rounds of heartbeats with no response from majority CS 8 XI 7 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

38 What if clients can crash? Servers maintain a session for each client Keep track of latest sequence number processed for client, and response Generalizes for multiple outstanding requests Server keeps set of <seq, resp> for client Client includes with request lowest seq with no response Server can discard smaller sequence numbers Must expire sessions All replicas must agree on when to do this Raft uses leader timestamp, committed to log CS 8 XI 8 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

39 Alive clients with expired sessions How to distinguish between client which exited from client which just took too long? Require clients to register with the leader when starting a session RegisterClient RPC Leader returns unique ID to the client Client uses this ID in subsequent request If server receives request for non-existing session Return an error. Current implementation crashes the client, forcing restart CS 8 XI 9 Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved.

40 Configuration Changes Cannot switch directly from one configuration to another: conflicting majorities could arise See the paper for details Server Server Server Server 4 Server 5 C old time C new Majority of C old Majority of C new CS 8 XI 40 Slides based on Raft slides by John Ousterhout

Distributed Systems. Day 11: Replication [Part 3 Raft] To survive failures you need a raft

Distributed Systems. Day 11: Replication [Part 3 Raft] To survive failures you need a raft Distributed Systems Day : Replication [Part Raft] To survive failures you need a raft Consensus Consensus: A majority of nodes agree on a value Variations on the problem, depending on assumptions Synchronous

More information

To do. Consensus and related problems. q Failure. q Raft

To do. Consensus and related problems. q Failure. q Raft Consensus and related problems To do q Failure q Consensus and related problems q Raft Consensus We have seen protocols tailored for individual types of consensus/agreements Which process can enter the

More information

P2 Recitation. Raft: A Consensus Algorithm for Replicated Logs

P2 Recitation. Raft: A Consensus Algorithm for Replicated Logs P2 Recitation Raft: A Consensus Algorithm for Replicated Logs Presented by Zeleena Kearney and Tushar Agarwal Diego Ongaro and John Ousterhout Stanford University Presentation adapted from the original

More information

Recall: Primary-Backup. State machine replication. Extend PB for high availability. Consensus 2. Mechanism: Replicate and separate servers

Recall: Primary-Backup. State machine replication. Extend PB for high availability. Consensus 2. Mechanism: Replicate and separate servers Replicated s, RAFT COS 8: Distributed Systems Lecture 8 Recall: Primary-Backup Mechanism: Replicate and separate servers Goal #: Provide a highly reliable service Goal #: Servers should behave just like

More information

Extend PB for high availability. PB high availability via 2PC. Recall: Primary-Backup. Putting it all together for SMR:

Extend PB for high availability. PB high availability via 2PC. Recall: Primary-Backup. Putting it all together for SMR: Putting it all together for SMR: Two-Phase Commit, Leader Election RAFT COS 8: Distributed Systems Lecture Recall: Primary-Backup Mechanism: Replicate and separate servers Goal #: Provide a highly reliable

More information

Two phase commit protocol. Two phase commit protocol. Recall: Linearizability (Strong Consistency) Consensus

Two phase commit protocol. Two phase commit protocol. Recall: Linearizability (Strong Consistency) Consensus Recall: Linearizability (Strong Consistency) Consensus COS 518: Advanced Computer Systems Lecture 4 Provide behavior of a single copy of object: Read should urn the most recent write Subsequent reads should

More information

Designing for Understandability: the Raft Consensus Algorithm. Diego Ongaro John Ousterhout Stanford University

Designing for Understandability: the Raft Consensus Algorithm. Diego Ongaro John Ousterhout Stanford University Designing for Understandability: the Raft Consensus Algorithm Diego Ongaro John Ousterhout Stanford University Algorithms Should Be Designed For... Correctness? Efficiency? Conciseness? Understandability!

More information

CSE 124: REPLICATED STATE MACHINES. George Porter November 8 and 10, 2017

CSE 124: REPLICATED STATE MACHINES. George Porter November 8 and 10, 2017 CSE 24: REPLICATED STATE MACHINES George Porter November 8 and 0, 207 ATTRIBUTION These slides are released under an Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) Creative Commons

More information

REPLICATED STATE MACHINES

REPLICATED STATE MACHINES 5/6/208 REPLICATED STATE MACHINES George Porter May 6, 208 ATTRIBUTION These slides are released under an Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) Creative Commons license These

More information

In Search of an Understandable Consensus Algorithm

In Search of an Understandable Consensus Algorithm In Search of an Understandable Consensus Algorithm Abstract Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to Paxos, and it is as efficient as Paxos, but its

More information

Byzantine Fault Tolerant Raft

Byzantine Fault Tolerant Raft Abstract Byzantine Fault Tolerant Raft Dennis Wang, Nina Tai, Yicheng An {dwang22, ninatai, yicheng} @stanford.edu https://github.com/g60726/zatt For this project, we modified the original Raft design

More information

Paxos and Raft (Lecture 21, cs262a) Ion Stoica, UC Berkeley November 7, 2016

Paxos and Raft (Lecture 21, cs262a) Ion Stoica, UC Berkeley November 7, 2016 Paxos and Raft (Lecture 21, cs262a) Ion Stoica, UC Berkeley November 7, 2016 Bezos mandate for service-oriented-architecture (~2002) 1. All teams will henceforth expose their data and functionality through

More information

Distributed Consensus Protocols

Distributed Consensus Protocols Distributed Consensus Protocols ABSTRACT In this paper, I compare Paxos, the most popular and influential of distributed consensus protocols, and Raft, a fairly new protocol that is considered to be a

More information

Distributed systems. Lecture 6: distributed transactions, elections, consensus and replication. Malte Schwarzkopf

Distributed systems. Lecture 6: distributed transactions, elections, consensus and replication. Malte Schwarzkopf Distributed systems Lecture 6: distributed transactions, elections, consensus and replication Malte Schwarzkopf Last time Saw how we can build ordered multicast Messages between processes in a group Need

More information

Today: Fault Tolerance

Today: Fault Tolerance Today: Fault Tolerance Agreement in presence of faults Two army problem Byzantine generals problem Reliable communication Distributed commit Two phase commit Three phase commit Paxos Failure recovery Checkpointing

More information

Consensus and related problems

Consensus and related problems Consensus and related problems Today l Consensus l Google s Chubby l Paxos for Chubby Consensus and failures How to make process agree on a value after one or more have proposed what the value should be?

More information

Today: Fault Tolerance. Fault Tolerance

Today: Fault Tolerance. Fault Tolerance Today: Fault Tolerance Agreement in presence of faults Two army problem Byzantine generals problem Reliable communication Distributed commit Two phase commit Three phase commit Paxos Failure recovery Checkpointing

More information

Distributed Systems 11. Consensus. Paul Krzyzanowski

Distributed Systems 11. Consensus. Paul Krzyzanowski Distributed Systems 11. Consensus Paul Krzyzanowski pxk@cs.rutgers.edu 1 Consensus Goal Allow a group of processes to agree on a result All processes must agree on the same value The value must be one

More information

Raft and Paxos Exam Rubric

Raft and Paxos Exam Rubric 1 of 10 03/28/2013 04:27 PM Raft and Paxos Exam Rubric Grading Where points are taken away for incorrect information, every section still has a minimum of 0 points. Raft Exam 1. (4 points, easy) Each figure

More information

Recall our 2PC commit problem. Recall our 2PC commit problem. Doing failover correctly isn t easy. Consensus I. FLP Impossibility, Paxos

Recall our 2PC commit problem. Recall our 2PC commit problem. Doing failover correctly isn t easy. Consensus I. FLP Impossibility, Paxos Consensus I Recall our 2PC commit problem FLP Impossibility, Paxos Client C 1 C à TC: go! COS 418: Distributed Systems Lecture 7 Michael Freedman Bank A B 2 TC à A, B: prepare! 3 A, B à P: yes or no 4

More information

FAULT TOLERANT LEADER ELECTION IN DISTRIBUTED SYSTEMS

FAULT TOLERANT LEADER ELECTION IN DISTRIBUTED SYSTEMS FAULT TOLERANT LEADER ELECTION IN DISTRIBUTED SYSTEMS Marius Rafailescu The Faculty of Automatic Control and Computers, POLITEHNICA University, Bucharest ABSTRACT There are many distributed systems which

More information

Paxos and Replication. Dan Ports, CSEP 552

Paxos and Replication. Dan Ports, CSEP 552 Paxos and Replication Dan Ports, CSEP 552 Today: achieving consensus with Paxos and how to use this to build a replicated system Last week Scaling a web service using front-end caching but what about the

More information

CSE 5306 Distributed Systems. Fault Tolerance

CSE 5306 Distributed Systems. Fault Tolerance CSE 5306 Distributed Systems Fault Tolerance 1 Failure in Distributed Systems Partial failure happens when one component of a distributed system fails often leaves other components unaffected A failure

More information

CS /15/16. Paul Krzyzanowski 1. Question 1. Distributed Systems 2016 Exam 2 Review. Question 3. Question 2. Question 5.

CS /15/16. Paul Krzyzanowski 1. Question 1. Distributed Systems 2016 Exam 2 Review. Question 3. Question 2. Question 5. Question 1 What makes a message unstable? How does an unstable message become stable? Distributed Systems 2016 Exam 2 Review Paul Krzyzanowski Rutgers University Fall 2016 In virtual sychrony, a message

More information

Last time. Distributed systems Lecture 6: Elections, distributed transactions, and replication. DrRobert N. M. Watson

Last time. Distributed systems Lecture 6: Elections, distributed transactions, and replication. DrRobert N. M. Watson Distributed systems Lecture 6: Elections, distributed transactions, and replication DrRobert N. M. Watson 1 Last time Saw how we can build ordered multicast Messages between processes in a group Need to

More information

Transactions. CS 475, Spring 2018 Concurrent & Distributed Systems

Transactions. CS 475, Spring 2018 Concurrent & Distributed Systems Transactions CS 475, Spring 2018 Concurrent & Distributed Systems Review: Transactions boolean transfermoney(person from, Person to, float amount){ if(from.balance >= amount) { from.balance = from.balance

More information

Distributed Commit in Asynchronous Systems

Distributed Commit in Asynchronous Systems Distributed Commit in Asynchronous Systems Minsoo Ryu Department of Computer Science and Engineering 2 Distributed Commit Problem - Either everybody commits a transaction, or nobody - This means consensus!

More information

Agreement and Consensus. SWE 622, Spring 2017 Distributed Software Engineering

Agreement and Consensus. SWE 622, Spring 2017 Distributed Software Engineering Agreement and Consensus SWE 622, Spring 2017 Distributed Software Engineering Today General agreement problems Fault tolerance limitations of 2PC 3PC Paxos + ZooKeeper 2 Midterm Recap 200 GMU SWE 622 Midterm

More information

CSE 5306 Distributed Systems

CSE 5306 Distributed Systems CSE 5306 Distributed Systems Fault Tolerance Jia Rao http://ranger.uta.edu/~jrao/ 1 Failure in Distributed Systems Partial failure Happens when one component of a distributed system fails Often leaves

More information

hraft: An Implementation of Raft in Haskell

hraft: An Implementation of Raft in Haskell hraft: An Implementation of Raft in Haskell Shantanu Joshi joshi4@cs.stanford.edu June 11, 2014 1 Introduction For my Project, I decided to write an implementation of Raft in Haskell. Raft[1] is a consensus

More information

ZooKeeper & Curator. CS 475, Spring 2018 Concurrent & Distributed Systems

ZooKeeper & Curator. CS 475, Spring 2018 Concurrent & Distributed Systems ZooKeeper & Curator CS 475, Spring 2018 Concurrent & Distributed Systems Review: Agreement In distributed systems, we have multiple nodes that need to all agree that some object has some state Examples:

More information

Fault Tolerance. Distributed Software Systems. Definitions

Fault Tolerance. Distributed Software Systems. Definitions Fault Tolerance Distributed Software Systems Definitions Availability: probability the system operates correctly at any given moment Reliability: ability to run correctly for a long interval of time Safety:

More information

Consensus Problem. Pradipta De

Consensus Problem. Pradipta De Consensus Problem Slides are based on the book chapter from Distributed Computing: Principles, Paradigms and Algorithms (Chapter 14) by Kshemkalyani and Singhal Pradipta De pradipta.de@sunykorea.ac.kr

More information

Practical Byzantine Fault Tolerance. Castro and Liskov SOSP 99

Practical Byzantine Fault Tolerance. Castro and Liskov SOSP 99 Practical Byzantine Fault Tolerance Castro and Liskov SOSP 99 Why this paper? Kind of incredible that it s even possible Let alone a practical NFS implementation with it So far we ve only considered fail-stop

More information

Data Consistency and Blockchain. Bei Chun Zhou (BlockChainZ)

Data Consistency and Blockchain. Bei Chun Zhou (BlockChainZ) Data Consistency and Blockchain Bei Chun Zhou (BlockChainZ) beichunz@cn.ibm.com 1 Data Consistency Point-in-time consistency Transaction consistency Application consistency 2 Strong Consistency ACID Atomicity.

More information

Recovering from a Crash. Three-Phase Commit

Recovering from a Crash. Three-Phase Commit Recovering from a Crash If INIT : abort locally and inform coordinator If Ready, contact another process Q and examine Q s state Lecture 18, page 23 Three-Phase Commit Two phase commit: problem if coordinator

More information

Intuitive distributed algorithms. with F#

Intuitive distributed algorithms. with F# Intuitive distributed algorithms with F# Natallia Dzenisenka Alena Hall @nata_dzen @lenadroid A tour of a variety of intuitivedistributed algorithms used in practical distributed systems. and how to prototype

More information

Network Protocols. Sarah Diesburg Operating Systems CS 3430

Network Protocols. Sarah Diesburg Operating Systems CS 3430 Network Protocols Sarah Diesburg Operating Systems CS 3430 Protocol An agreement between two parties as to how information is to be transmitted A network protocol abstracts packets into messages Physical

More information

Distributed Systems (ICE 601) Fault Tolerance

Distributed Systems (ICE 601) Fault Tolerance Distributed Systems (ICE 601) Fault Tolerance Dongman Lee ICU Introduction Failure Model Fault Tolerance Models state machine primary-backup Class Overview Introduction Dependability availability reliability

More information

Introduction to Distributed Systems Seif Haridi

Introduction to Distributed Systems Seif Haridi Introduction to Distributed Systems Seif Haridi haridi@kth.se What is a distributed system? A set of nodes, connected by a network, which appear to its users as a single coherent system p1 p2. pn send

More information

Fault Tolerance. Basic Concepts

Fault Tolerance. Basic Concepts COP 6611 Advanced Operating System Fault Tolerance Chi Zhang czhang@cs.fiu.edu Dependability Includes Availability Run time / total time Basic Concepts Reliability The length of uninterrupted run time

More information

Consensus a classic problem. Consensus, impossibility results and Paxos. Distributed Consensus. Asynchronous networks.

Consensus a classic problem. Consensus, impossibility results and Paxos. Distributed Consensus. Asynchronous networks. Consensus, impossibility results and Paxos Ken Birman Consensus a classic problem Consensus abstraction underlies many distributed systems and protocols N processes They start execution with inputs {0,1}

More information

Distributed Consensus: Making Impossible Possible

Distributed Consensus: Making Impossible Possible Distributed Consensus: Making Impossible Possible QCon London Tuesday 29/3/2016 Heidi Howard PhD Student @ University of Cambridge heidi.howard@cl.cam.ac.uk @heidiann360 What is Consensus? The process

More information

Distributed Operating Systems. Distributed Synchronization

Distributed Operating Systems. Distributed Synchronization 2 Distributed Operating Systems Distributed Synchronization Steve Goddard goddard@cse.unl.edu http://www.cse.unl.edu/~goddard/courses/csce855 1 Synchronization Coordinating processes to achieve common

More information

Coordination 1. To do. Mutual exclusion Election algorithms Next time: Global state. q q q

Coordination 1. To do. Mutual exclusion Election algorithms Next time: Global state. q q q Coordination 1 To do q q q Mutual exclusion Election algorithms Next time: Global state Coordination and agreement in US Congress 1798-2015 Process coordination How can processes coordinate their action?

More information

Consensus, impossibility results and Paxos. Ken Birman

Consensus, impossibility results and Paxos. Ken Birman Consensus, impossibility results and Paxos Ken Birman Consensus a classic problem Consensus abstraction underlies many distributed systems and protocols N processes They start execution with inputs {0,1}

More information

EMPIRICAL STUDY OF UNSTABLE LEADERS IN PAXOS LONG KAI THESIS

EMPIRICAL STUDY OF UNSTABLE LEADERS IN PAXOS LONG KAI THESIS 2013 Long Kai EMPIRICAL STUDY OF UNSTABLE LEADERS IN PAXOS BY LONG KAI THESIS Submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science in the Graduate

More information

WA 2. Paxos and distributed programming Martin Klíma

WA 2. Paxos and distributed programming Martin Klíma Paxos and distributed programming Martin Klíma Spefics of distributed programming Communication by message passing Considerable time to pass the network Non-reliable network Time is not synchronized on

More information

Distributed Consensus: Making Impossible Possible

Distributed Consensus: Making Impossible Possible Distributed Consensus: Making Impossible Possible Heidi Howard PhD Student @ University of Cambridge heidi.howard@cl.cam.ac.uk @heidiann360 hh360.user.srcf.net Sometimes inconsistency is not an option

More information

CprE Fault Tolerance. Dr. Yong Guan. Department of Electrical and Computer Engineering & Information Assurance Center Iowa State University

CprE Fault Tolerance. Dr. Yong Guan. Department of Electrical and Computer Engineering & Information Assurance Center Iowa State University Fault Tolerance Dr. Yong Guan Department of Electrical and Computer Engineering & Information Assurance Center Iowa State University Outline for Today s Talk Basic Concepts Process Resilience Reliable

More information

Distributed Systems. replication Johan Montelius ID2201. Distributed Systems ID2201

Distributed Systems. replication Johan Montelius ID2201. Distributed Systems ID2201 Distributed Systems ID2201 replication Johan Montelius 1 The problem The problem we have: servers might be unavailable The solution: keep duplicates at different servers 2 Building a fault-tolerant service

More information

Three modifications for the Raft consensus algorithm

Three modifications for the Raft consensus algorithm Three modifications for the Raft consensus algorithm 1. Background Henrik Ingo henrik.ingo@openlife.cc August 2015 (v0.2, fixed bugs in algorithm, in Figure 1) In my work at MongoDB, I've been involved

More information

Linearizability CMPT 401. Sequential Consistency. Passive Replication

Linearizability CMPT 401. Sequential Consistency. Passive Replication Linearizability CMPT 401 Thursday, March 31, 2005 The execution of a replicated service (potentially with multiple requests interleaved over multiple servers) is said to be linearizable if: The interleaved

More information

EECS 498 Introduction to Distributed Systems

EECS 498 Introduction to Distributed Systems EECS 498 Introduction to Distributed Systems Fall 2017 Harsha V. Madhyastha Replicated State Machines Logical clocks Primary/ Backup Paxos? 0 1 (N-1)/2 No. of tolerable failures October 11, 2017 EECS 498

More information

Replication in Distributed Systems

Replication in Distributed Systems Replication in Distributed Systems Replication Basics Multiple copies of data kept in different nodes A set of replicas holding copies of a data Nodes can be physically very close or distributed all over

More information

Synchronization. Clock Synchronization

Synchronization. Clock Synchronization Synchronization Clock Synchronization Logical clocks Global state Election algorithms Mutual exclusion Distributed transactions 1 Clock Synchronization Time is counted based on tick Time judged by query

More information

Consensus in Distributed Systems. Jeff Chase Duke University

Consensus in Distributed Systems. Jeff Chase Duke University Consensus in Distributed Systems Jeff Chase Duke University Consensus P 1 P 1 v 1 d 1 Unreliable multicast P 2 P 3 Consensus algorithm P 2 P 3 v 2 Step 1 Propose. v 3 d 2 Step 2 Decide. d 3 Generalizes

More information

Distributed Systems. 09. State Machine Replication & Virtual Synchrony. Paul Krzyzanowski. Rutgers University. Fall Paul Krzyzanowski

Distributed Systems. 09. State Machine Replication & Virtual Synchrony. Paul Krzyzanowski. Rutgers University. Fall Paul Krzyzanowski Distributed Systems 09. State Machine Replication & Virtual Synchrony Paul Krzyzanowski Rutgers University Fall 2016 1 State machine replication 2 State machine replication We want high scalability and

More information

Synchronization Part 2. REK s adaptation of Claypool s adaptation oftanenbaum s Distributed Systems Chapter 5 and Silberschatz Chapter 17

Synchronization Part 2. REK s adaptation of Claypool s adaptation oftanenbaum s Distributed Systems Chapter 5 and Silberschatz Chapter 17 Synchronization Part 2 REK s adaptation of Claypool s adaptation oftanenbaum s Distributed Systems Chapter 5 and Silberschatz Chapter 17 1 Outline Part 2! Clock Synchronization! Clock Synchronization Algorithms!

More information

Distributed Data Management Replication

Distributed Data Management Replication Felix Naumann F-2.03/F-2.04, Campus II Hasso Plattner Institut Distributing Data Motivation Scalability (Elasticity) If data volume, processing, or access exhausts one machine, you might want to spread

More information

A definition. Byzantine Generals Problem. Synchronous, Byzantine world

A definition. Byzantine Generals Problem. Synchronous, Byzantine world The Byzantine Generals Problem Leslie Lamport, Robert Shostak, and Marshall Pease ACM TOPLAS 1982 Practical Byzantine Fault Tolerance Miguel Castro and Barbara Liskov OSDI 1999 A definition Byzantine (www.m-w.com):

More information

SimpleChubby: a simple distributed lock service

SimpleChubby: a simple distributed lock service SimpleChubby: a simple distributed lock service Jing Pu, Mingyu Gao, Hang Qu 1 Introduction We implement a distributed lock service called SimpleChubby similar to the original Google Chubby lock service[1].

More information

Distributed File System

Distributed File System Distributed File System Last Class NFS Design choices Opportunistic locking Local name spaces CS 138 XVIII 1 Copyright 2018 Theophilus Benson, Thomas W. Doeppner. All DFS Basic Info Global name space across

More information

Distributed Algorithms Benoît Garbinato

Distributed Algorithms Benoît Garbinato Distributed Algorithms Benoît Garbinato 1 Distributed systems networks distributed As long as there were no machines, programming was no problem networks distributed at all; when we had a few weak computers,

More information

Causal Consistency and Two-Phase Commit

Causal Consistency and Two-Phase Commit Causal Consistency and Two-Phase Commit CS 240: Computing Systems and Concurrency Lecture 16 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Consistency

More information

MODELS OF DISTRIBUTED SYSTEMS

MODELS OF DISTRIBUTED SYSTEMS Distributed Systems Fö 2/3-1 Distributed Systems Fö 2/3-2 MODELS OF DISTRIBUTED SYSTEMS Basic Elements 1. Architectural Models 2. Interaction Models Resources in a distributed system are shared between

More information

At-most-once Algorithm for Linearizable RPC in Distributed Systems

At-most-once Algorithm for Linearizable RPC in Distributed Systems At-most-once Algorithm for Linearizable RPC in Distributed Systems Seo Jin Park 1 1 Stanford University Abstract In a distributed system, like RAM- Cloud, supporting automatic retry of RPC, it is tricky

More information

Fault Tolerance Part I. CS403/534 Distributed Systems Erkay Savas Sabanci University

Fault Tolerance Part I. CS403/534 Distributed Systems Erkay Savas Sabanci University Fault Tolerance Part I CS403/534 Distributed Systems Erkay Savas Sabanci University 1 Overview Basic concepts Process resilience Reliable client-server communication Reliable group communication Distributed

More information

A Reliable Broadcast System

A Reliable Broadcast System A Reliable Broadcast System Yuchen Dai, Xiayi Huang, Diansan Zhou Department of Computer Sciences and Engineering Santa Clara University December 10 2013 Table of Contents 2 Introduction......3 2.1 Objective...3

More information

Distributed Systems Consensus

Distributed Systems Consensus Distributed Systems Consensus Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) Consensus 1393/6/31 1 / 56 What is the Problem?

More information

PRIMARY-BACKUP REPLICATION

PRIMARY-BACKUP REPLICATION PRIMARY-BACKUP REPLICATION Primary Backup George Porter Nov 14, 2018 ATTRIBUTION These slides are released under an Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) Creative Commons

More information

Exam 2 Review. Fall 2011

Exam 2 Review. Fall 2011 Exam 2 Review Fall 2011 Question 1 What is a drawback of the token ring election algorithm? Bad question! Token ring mutex vs. Ring election! Ring election: multiple concurrent elections message size grows

More information

Clock Synchronization. Synchronization. Clock Synchronization Algorithms. Physical Clock Synchronization. Tanenbaum Chapter 6 plus additional papers

Clock Synchronization. Synchronization. Clock Synchronization Algorithms. Physical Clock Synchronization. Tanenbaum Chapter 6 plus additional papers Clock Synchronization Synchronization Tanenbaum Chapter 6 plus additional papers Fig 6-1. In a distributed system, each machine has its own clock. When this is the case, an event that occurred after another

More information

Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitionin

Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitionin Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitioning (Different data at different sites More space Better

More information

Synchronization. Chapter 5

Synchronization. Chapter 5 Synchronization Chapter 5 Clock Synchronization In a centralized system time is unambiguous. (each computer has its own clock) In a distributed system achieving agreement on time is not trivial. (it is

More information

Recap. CSE 486/586 Distributed Systems Paxos. Paxos. Brief History. Brief History. Brief History C 1

Recap. CSE 486/586 Distributed Systems Paxos. Paxos. Brief History. Brief History. Brief History C 1 Recap Distributed Systems Steve Ko Computer Sciences and Engineering University at Buffalo Facebook photo storage CDN (hot), Haystack (warm), & f4 (very warm) Haystack RAID-6, per stripe: 10 data disks,

More information

Fault Tolerance. Distributed Systems. September 2002

Fault Tolerance. Distributed Systems. September 2002 Fault Tolerance Distributed Systems September 2002 Basics A component provides services to clients. To provide services, the component may require the services from other components a component may depend

More information

Primary-Backup Replication

Primary-Backup Replication Primary-Backup Replication CS 240: Computing Systems and Concurrency Lecture 7 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Simplified Fault Tolerance

More information

Fault Tolerance. Distributed Systems IT332

Fault Tolerance. Distributed Systems IT332 Fault Tolerance Distributed Systems IT332 2 Outline Introduction to fault tolerance Reliable Client Server Communication Distributed commit Failure recovery 3 Failures, Due to What? A system is said to

More information

Practical Byzantine Fault

Practical Byzantine Fault Practical Byzantine Fault Tolerance Practical Byzantine Fault Tolerance Castro and Liskov, OSDI 1999 Nathan Baker, presenting on 23 September 2005 What is a Byzantine fault? Rationale for Byzantine Fault

More information

CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 21: Network Protocols (and 2 Phase Commit)

CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 21: Network Protocols (and 2 Phase Commit) CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring 2003 Lecture 21: Network Protocols (and 2 Phase Commit) 21.0 Main Point Protocol: agreement between two parties as to

More information

MODELS OF DISTRIBUTED SYSTEMS

MODELS OF DISTRIBUTED SYSTEMS Distributed Systems Fö 2/3-1 Distributed Systems Fö 2/3-2 MODELS OF DISTRIBUTED SYSTEMS Basic Elements 1. Architectural Models 2. Interaction Models Resources in a distributed system are shared between

More information

416 practice questions (PQs)

416 practice questions (PQs) 416 practice questions (PQs) 1. Goal: give you some material to study for the final exam and to help you to more actively engage with the material we cover in class. 2. Format: questions that are in scope

More information

Distributed Synchronization. EECS 591 Farnam Jahanian University of Michigan

Distributed Synchronization. EECS 591 Farnam Jahanian University of Michigan Distributed Synchronization EECS 591 Farnam Jahanian University of Michigan Reading List Tanenbaum Chapter 5.1, 5.4 and 5.5 Clock Synchronization Distributed Election Mutual Exclusion Clock Synchronization

More information

PROCESS SYNCHRONIZATION

PROCESS SYNCHRONIZATION DISTRIBUTED COMPUTER SYSTEMS PROCESS SYNCHRONIZATION Dr. Jack Lange Computer Science Department University of Pittsburgh Fall 2015 Process Synchronization Mutual Exclusion Algorithms Permission Based Centralized

More information

Distributed Systems. 10. Consensus: Paxos. Paul Krzyzanowski. Rutgers University. Fall 2017

Distributed Systems. 10. Consensus: Paxos. Paul Krzyzanowski. Rutgers University. Fall 2017 Distributed Systems 10. Consensus: Paxos Paul Krzyzanowski Rutgers University Fall 2017 1 Consensus Goal Allow a group of processes to agree on a result All processes must agree on the same value The value

More information

Today: Fault Tolerance. Replica Management

Today: Fault Tolerance. Replica Management Today: Fault Tolerance Failure models Agreement in presence of faults Two army problem Byzantine generals problem Reliable communication Distributed commit Two phase commit Three phase commit Failure recovery

More information

Distributed Computing. CS439: Principles of Computer Systems November 20, 2017

Distributed Computing. CS439: Principles of Computer Systems November 20, 2017 Distributed Computing CS439: Principles of Computer Systems November 20, 2017 Last Time Network Programming: Sockets End point of communication Identified by (IP address : port number) pair Client-Side

More information

CSE 486/586 Distributed Systems

CSE 486/586 Distributed Systems CSE 486/586 Distributed Systems Mutual Exclusion Steve Ko Computer Sciences and Engineering University at Buffalo CSE 486/586 Recap: Consensus On a synchronous system There s an algorithm that works. On

More information

CS October 2017

CS October 2017 Atomic Transactions Transaction An operation composed of a number of discrete steps. Distributed Systems 11. Distributed Commit Protocols All the steps must be completed for the transaction to be committed.

More information

Practical Byzantine Fault Tolerance. Miguel Castro and Barbara Liskov

Practical Byzantine Fault Tolerance. Miguel Castro and Barbara Liskov Practical Byzantine Fault Tolerance Miguel Castro and Barbara Liskov Outline 1. Introduction to Byzantine Fault Tolerance Problem 2. PBFT Algorithm a. Models and overview b. Three-phase protocol c. View-change

More information

Exam 2 Review. October 29, Paul Krzyzanowski 1

Exam 2 Review. October 29, Paul Krzyzanowski 1 Exam 2 Review October 29, 2015 2013 Paul Krzyzanowski 1 Question 1 Why did Dropbox add notification servers to their architecture? To avoid the overhead of clients polling the servers periodically to check

More information

Assignment 12: Commit Protocols and Replication Solution

Assignment 12: Commit Protocols and Replication Solution Data Modelling and Databases Exercise dates: May 24 / May 25, 2018 Ce Zhang, Gustavo Alonso Last update: June 04, 2018 Spring Semester 2018 Head TA: Ingo Müller Assignment 12: Commit Protocols and Replication

More information

Distributed Systems Principles and Paradigms. Chapter 08: Fault Tolerance

Distributed Systems Principles and Paradigms. Chapter 08: Fault Tolerance Distributed Systems Principles and Paradigms Maarten van Steen VU Amsterdam, Dept. Computer Science Room R4.20, steen@cs.vu.nl Chapter 08: Fault Tolerance Version: December 2, 2010 2 / 65 Contents Chapter

More information

Agreement in Distributed Systems CS 188 Distributed Systems February 19, 2015

Agreement in Distributed Systems CS 188 Distributed Systems February 19, 2015 Agreement in Distributed Systems CS 188 Distributed Systems February 19, 2015 Page 1 Introduction We frequently want to get a set of nodes in a distributed system to agree Commitment protocols and mutual

More information

Comparative Analysis of Big Data Stream Processing Systems

Comparative Analysis of Big Data Stream Processing Systems Comparative Analysis of Big Data Stream Processing Systems Farouk Salem School of Science Thesis submitted for examination for the degree of Master of Science in Technology. Espoo 22 June, 2016 Thesis

More information

Basic vs. Reliable Multicast

Basic vs. Reliable Multicast Basic vs. Reliable Multicast Basic multicast does not consider process crashes. Reliable multicast does. So far, we considered the basic versions of ordered multicasts. What about the reliable versions?

More information

Module 8 Fault Tolerance CS655! 8-1!

Module 8 Fault Tolerance CS655! 8-1! Module 8 Fault Tolerance CS655! 8-1! Module 8 - Fault Tolerance CS655! 8-2! Dependability Reliability! A measure of success with which a system conforms to some authoritative specification of its behavior.!

More information

Concepts. Techniques for masking faults. Failure Masking by Redundancy. CIS 505: Software Systems Lecture Note on Consensus

Concepts. Techniques for masking faults. Failure Masking by Redundancy. CIS 505: Software Systems Lecture Note on Consensus CIS 505: Software Systems Lecture Note on Consensus Insup Lee Department of Computer and Information Science University of Pennsylvania CIS 505, Spring 2007 Concepts Dependability o Availability ready

More information