Distributed Coordination with ZooKeeper - Theory and Practice. Simon Tao EMC Labs of China Oct. 24th, 2015

Size: px
Start display at page:

Download "Distributed Coordination with ZooKeeper - Theory and Practice. Simon Tao EMC Labs of China Oct. 24th, 2015"

Transcription

1

2 Distributed Coordination with ZooKeeper - Theory and Practice Simon Tao EMC Labs of China {simon.tao@emc.com} Oct. 24th, 2015

3 Agenda 1. ZooKeeper Overview 2. Coordination in Spring XD 3. ZooKeeper Under the Hood

4 ZooKeeper Overview

5 What is ZooKeeper? Because Distributed Computing is a Zoo A service for coordinating process of distributed applications Coordination Kernel Enables more specific coordination primitives Shared data store Hierarchical key/value data Distributed service Fault tolerant and highly available High performance

6 Leader election Group membership Coordination Problems Configuration management Others Rendezvous Barriers Queues Two-phased Commit

7 Powered by ZooKeeper

8 Data Store with Sequence arbitration Sequential Mutual Exclusion Ordering guarantee Wait-free Change notification What is Coordination?

9 Hierarchical namespace Znode Data Model Persistent or Ephemeral, can be Sequential Each znode can hold data Non-ephemeral znode can have children Data is read and written in its entirety No partial read or write Not for large size of data Usually < 1MB

10 API For read: exists(path, watch) getdata(path, watch) getchildren(path, watch) For write: create(path, data, flags) setdata(path, data, version) delete(path, version) For consistency: sync(path)

11 Lock: Lock without Herd Effect 1 n = create(l + /lock-, EPHEMERAL SEQUENTIAL) 2 C = getchildren(l, false) 3 if n is lowest znode in C, exit 4 p = znode in C ordered just before n 5 if exists(p, true) wait for watch event 6 goto 2 Unlock: 1 delete(n)

12 Write Lock: Read Lock: Read-write Lock 1 n = create(l + /write-, EPHEMERAL SEQUENTIAL) 2 C = getchildren(l, false) 3 if n is lowest znode in C, exit 4 p = znode in C ordered just before n 5 if exists(p, true) wait for watch event 6 goto 2 1 n = create(l + /read-, EPHEMERAL SEQUENTIAL) 2 C = getchildren(l, false) 3 if no write znodes lower than n in C, exit 4 p = write znode in C ordered just before n 5 if exists(p, true) wait for watch event 6 goto 2

13 Watch Receive timely notifications of changes Without requiring polling Indicate change occurrence, not the change itself One-time trigger Not event stream! Associated with a session

14 Coordination in Spring XD

15 Spring XD Overview A unified, distributed, and extensible service for data ingestion, real time analytics, batch processing, and data export Provides servers and a configuration DSL for immediately starting processing data Runtime Architecture DSL XD Admin XD Container Linear streams UNIX pipes and filters model DAG Composing Modules Named channels

16 Stream Deployment Container Server Listen for module deployment events initiated from the Admin Server via ZooKeeper Deploy assigned modules and Connect module input and output channels to the data bus Data bus Used to transport messages during stream processing

17 Coordination with ZK In Spring XD, ZooKeeper enables: Leader failover, i.e. XD Admin HA Coordinate module deployment and tracking Group membership Task assignment Task status management

18 / xd ZK Hierarchy in Spring XD admins LEGEND admina adminb containers System Node App Persistent Node containera App Ephemeral Node deployments modules requested streams allocated streams streama streama modulea containera modulea status status

19 ZooKeeper Under the Hood

20 ZooKeeper Service Architecture Replicated over an ensemble of servers All servers store a copy of the data in memory One server is elected as leader on service startup Operation processing Clients only connect to a single ZooKeeper server at a time Reads are serviced from any Zookeeper server Writes forwarded to the single leader

21 ZK Service Components Request Processor Transform requests into idempotent transactions Atomic Broadcast Run Zab protocol Transactions are appended to an on-disk log Replicated Database In-memory data tree Client connects to one server Read requests: serviced from local replica Write requests: go through the leader and need majority consensus

22 Session Abstraction of Client s connection to Zookeeper All watches are attached to the session Allow client to switch connecting to a new server Zookeeper Ensure that the data view from new server is at least as recent as the client Session ends on Failures are detected by timeout expiration Being closed explicitly

23 Benefits Reliability Performance Replication Replicated State Machine Goal All replicas end up in the same state Rules All replicas start in the same initial state Every replica apply operations in the same order All operations must be deterministic

24 Atomic broadcast Maintain a single order Guarantee a Total Ordering of all messages Require all processes to deliver all messages in the same order

25 PO atomic broadcast Zookeeper provide stronger order guarantee than regular atomic broadcast Primary-order (PO) atomic broadcast Broadcasted changes are delivered in order they were sent All changes from previous leaders are delivered to an established leader, before new changes are broadcasted

26 What is Consistency? Consistency! Meaning of concurrent reads and writes on shared, possibly replicated state Consistency model Contract between client processes and the data store Consistency model of ZooKeeper Asynchronous-Linearizability (A-Linearizability) Local read

27 A-Linearizability: the L part Linearizable write Linearizable Consistency The Consistency in the CAP sense Strong notion of consistency for concurrent data store Linearizability: A Correctness Condition for Concurrent Objects - MAURICE P. HERLIHY and JEANNETTE M. WING Equivalent to some legal sequential behavior Real-time behavior of operations must be preserved 1. Program Order 2. Operation precedence order (from different processes) Order of overlapping operations is ambiguous W(0) A R(1) A W(0) A W(1) B R(?) B

28 FIFO client order A-Linearizability: the A part Allow multiple outstanding operations with guaranteed FIFO order Increases throughput while still remains linearizable Differ from original definition of linearizability by Herlihy, in which One outstanding operation at a time is allowed for a client No specific order is guaranteed for outstanding operations

29 Local read Read requests are processed locally at each replica In-memory operation, no disk activity or agreement protocol (i.e. Zab) involved Scale linearly for read-dominant workloads May return a stale value Not all coordination tasks require precedence order strictly

30 Sync + Read Want Linearizable Read? Local read may violate linearizability Sync to the rescue Call Sync, followed by a Read Cause the server to apply all pending write Enable read operation to reflect any changes that happened before the sync was issued

31 Zab: A leader-based protocol Atomic updates to an ensemble of servers Three or five being the norm One leader supported by a quorum Quorums guarantee pairwise intersection Atomic Broadcast protocol manages updating replicas Service is available if: A quorum of processes is up 2f +1 servers can tolerate up to f server failures Avoid split-brain Processes from the quorum can exchange messages for sufficiently long Leader totally order state updates and replica to followers via Zab Recover with valid state upon leader switch

32 Broadcast Simplified 2-phase commit Leader Upon receiving requests: Propose transactions Upon receiving ACKs: Collect ACKs Send Commit message once ACKs are received from a quorum of servers Follower Upon receiving transactions: Append to log Acknowledge to leader Upon receiving Commit message: Commit transaction Deliver transaction Follower Follower 1.Propose 1.Propose 2.ACK 3.Commit Leader Request 1.Propose 1.Propose 2.ACK 3.Commit Follower Follower Deliver Deliver

33 Transaction State updates transformed from requests Represented as <v, z> v V, (V is the set of broadcast values) z = <e, c> (zxid: transaction identifiers) e N represents a period of time when a server exercise leadership c N counter that increments for a new transaction Identifier order z1<z2 if either epoch(z1) < epoch(z2) or epoch(z1) = epoch(z2) and counter(z1) counter(z2)

34 Leader election 3 Zab phases Discovery Protocol Summary Collect history of accepted transactions from followers Pick the history with latest epoch and highest zxid as the initial history of the new epoch Synchronization Synchronize the initial history to all the quorum servers Thus bring quorum to consistent state Broadcast Replicate changes to the quorum

35 Fast Leader Election Zab Implementation Elect server with most update-to-date history as leader Optimization that merges leader election and discovery 2 Zab phases Recovery Similar to synchronization phase Broadcast

36 Leader Election Fast Leader Election (FLE) Servers start election in LOOKING state Servers exchange leader election notifications Each server send its current vote in notification to all the others Vote contains server id, and zxid of the last accepted proposal For each received vote: Server records the received votes If it is more up-to-date than the server s current one Take the received vote as the server s new current vote Send the next round of notification If same votes are received from a quorum of serves Switch to either FOLLOWING or LEADING state

37 Election Illustration

38 Election Illustration

39 Election Illustration

40 Election Illustration

41 Election Illustration

42 THANKS!

ZooKeeper. Wait-free coordination for Internet-scale systems

ZooKeeper. Wait-free coordination for Internet-scale systems ZooKeeper Wait-free coordination for Internet-scale systems Patrick Hunt and Mahadev (Yahoo! Grid) Flavio Junqueira and Benjamin Reed (Yahoo! Research) Internet-scale Challenges Lots of servers, users,

More information

ZooKeeper & Curator. CS 475, Spring 2018 Concurrent & Distributed Systems

ZooKeeper & Curator. CS 475, Spring 2018 Concurrent & Distributed Systems ZooKeeper & Curator CS 475, Spring 2018 Concurrent & Distributed Systems Review: Agreement In distributed systems, we have multiple nodes that need to all agree that some object has some state Examples:

More information

Project Midterms: March 22 nd : No Extensions

Project Midterms: March 22 nd : No Extensions Project Midterms: March 22 nd : No Extensions Team Presentations 10 minute presentations by each team member Demo of Gateway System Design What choices did you make for state management, data storage,

More information

Agreement and Consensus. SWE 622, Spring 2017 Distributed Software Engineering

Agreement and Consensus. SWE 622, Spring 2017 Distributed Software Engineering Agreement and Consensus SWE 622, Spring 2017 Distributed Software Engineering Today General agreement problems Fault tolerance limitations of 2PC 3PC Paxos + ZooKeeper 2 Midterm Recap 200 GMU SWE 622 Midterm

More information

Knowns and Unknowns in Distributed Systems

Knowns and Unknowns in Distributed Systems Apache Zookeeper Hunt, P., Konar, M., Junqueira, F.P. and Reed, B., 2010, June. ZooKeeper: Wait-free Coordination for Internet-scale Systems. In USENIX Annual Technical Conference (Vol. 8, p. 9). And other

More information

ZooKeeper Atomic Broadcast (for Project 2) 10/27/2016

ZooKeeper Atomic Broadcast (for Project 2) 10/27/2016 ZooKeeper Atomic Broadcast (for Project 2) 10/27/2016 Apache Hadoop 2002: Internet Archive search director Doug CuFng and UW grad student Mike Carafella set out to build a bemer open- source search engine.

More information

ZooKeeper. Table of contents

ZooKeeper. Table of contents by Table of contents 1 ZooKeeper: A Distributed Coordination Service for Distributed Applications... 2 1.1 Design Goals... 2 1.2 Data model and the hierarchical namespace... 3 1.3 Nodes and ephemeral nodes...

More information

Applications of Paxos Algorithm

Applications of Paxos Algorithm Applications of Paxos Algorithm Gurkan Solmaz COP 6938 - Cloud Computing - Fall 2012 Department of Electrical Engineering and Computer Science University of Central Florida - Orlando, FL Oct 15, 2012 1

More information

ZooKeeper Recipes and Solutions

ZooKeeper Recipes and Solutions by Table of contents 1 A Guide to Creating Higher-level Constructs with ZooKeeper...2 1.1 Important Note About Error Handling... 2 1.2 Out of the Box Applications: Name Service, Configuration, Group Membership...2

More information

ZooKeeper Recipes and Solutions

ZooKeeper Recipes and Solutions by Table of contents 1 A Guide to Creating Higher-level Constructs with ZooKeeper...2 1.1 Out of the Box Applications: Name Service, Configuration, Group Membership... 2 1.2 Barriers... 2 1.3 Queues...

More information

Apache ZooKeeper and orchestration in distributed systems. Andrew Kondratovich

Apache ZooKeeper and orchestration in distributed systems. Andrew Kondratovich Apache ZooKeeper and orchestration in distributed systems Andrew Kondratovich andrew.kondratovich@gmail.com «A distributed system is one in which the failure of a computer you didn't even know existed

More information

Dynamic Reconfiguration of Primary/Backup Clusters

Dynamic Reconfiguration of Primary/Backup Clusters Dynamic Reconfiguration of Primary/Backup Clusters (Apache ZooKeeper) Alex Shraer Yahoo! Research In collaboration with: Benjamin Reed Dahlia Malkhi Flavio Junqueira Yahoo! Research Microsoft Research

More information

ZooKeeper Atomic Broadcast

ZooKeeper Atomic Broadcast ZooKeeper Atomic Broadcast The heart of the ZooKeeper coordination service Benjamin Reed, Flavio Junqueira Yahoo! Research ZooKeeper Service Transforms a request into an idempotent transaction Request

More information

Coordinating distributed systems part II. Marko Vukolić Distributed Systems and Cloud Computing

Coordinating distributed systems part II. Marko Vukolić Distributed Systems and Cloud Computing Coordinating distributed systems part II Marko Vukolić Distributed Systems and Cloud Computing Last Time Coordinating distributed systems part I Zookeeper At the heart of Zookeeper is the ZAB atomic broadcast

More information

Apache Zookeeper. h,p://zookeeper.apache.org

Apache Zookeeper. h,p://zookeeper.apache.org Apache Zookeeper h,p://zookeeper.apache.org What is a Distributed System? A distributed system consists of mulaple computers that communicate through a computer network and interact with each other to

More information

A Distributed System Case Study: Apache Kafka. High throughput messaging for diverse consumers

A Distributed System Case Study: Apache Kafka. High throughput messaging for diverse consumers A Distributed System Case Study: Apache Kafka High throughput messaging for diverse consumers As always, this is not a tutorial Some of the concepts may no longer be part of the current system or implemented

More information

Distributed systems. Lecture 6: distributed transactions, elections, consensus and replication. Malte Schwarzkopf

Distributed systems. Lecture 6: distributed transactions, elections, consensus and replication. Malte Schwarzkopf Distributed systems Lecture 6: distributed transactions, elections, consensus and replication Malte Schwarzkopf Last time Saw how we can build ordered multicast Messages between processes in a group Need

More information

There Is More Consensus in Egalitarian Parliaments

There Is More Consensus in Egalitarian Parliaments There Is More Consensus in Egalitarian Parliaments Iulian Moraru, David Andersen, Michael Kaminsky Carnegie Mellon University Intel Labs Fault tolerance Redundancy State Machine Replication 3 State Machine

More information

Trek: Testable Replicated Key-Value Store

Trek: Testable Replicated Key-Value Store Trek: Testable Replicated Key-Value Store Yen-Ting Liu, Wen-Chien Chen Stanford University Abstract This paper describes the implementation of Trek, a testable, replicated key-value store with ZooKeeper-like

More information

The Google File System

The Google File System October 13, 2010 Based on: S. Ghemawat, H. Gobioff, and S.-T. Leung: The Google file system, in Proceedings ACM SOSP 2003, Lake George, NY, USA, October 2003. 1 Assumptions Interface Architecture Single

More information

Important Lessons. A Distributed Algorithm (2) Today's Lecture - Replication

Important Lessons. A Distributed Algorithm (2) Today's Lecture - Replication Important Lessons Lamport & vector clocks both give a logical timestamps Total ordering vs. causal ordering Other issues in coordinating node activities Exclusive access to resources/data Choosing a single

More information

Replication in Distributed Systems

Replication in Distributed Systems Replication in Distributed Systems Replication Basics Multiple copies of data kept in different nodes A set of replicas holding copies of a data Nodes can be physically very close or distributed all over

More information

Intra-cluster Replication for Apache Kafka. Jun Rao

Intra-cluster Replication for Apache Kafka. Jun Rao Intra-cluster Replication for Apache Kafka Jun Rao About myself Engineer at LinkedIn since 2010 Worked on Apache Kafka and Cassandra Database researcher at IBM Outline Overview of Kafka Kafka architecture

More information

Distributed Computation Models

Distributed Computation Models Distributed Computation Models SWE 622, Spring 2017 Distributed Software Engineering Some slides ack: Jeff Dean HW4 Recap https://b.socrative.com/ Class: SWE622 2 Review Replicating state machines Case

More information

Paxos Replicated State Machines as the Basis of a High- Performance Data Store

Paxos Replicated State Machines as the Basis of a High- Performance Data Store Paxos Replicated State Machines as the Basis of a High- Performance Data Store William J. Bolosky, Dexter Bradshaw, Randolph B. Haagens, Norbert P. Kusters and Peng Li March 30, 2011 Q: How to build a

More information

Assignment 12: Commit Protocols and Replication Solution

Assignment 12: Commit Protocols and Replication Solution Data Modelling and Databases Exercise dates: May 24 / May 25, 2018 Ce Zhang, Gustavo Alonso Last update: June 04, 2018 Spring Semester 2018 Head TA: Ingo Müller Assignment 12: Commit Protocols and Replication

More information

Introduction to riak_ensemble. Joseph Blomstedt Basho Technologies

Introduction to riak_ensemble. Joseph Blomstedt Basho Technologies Introduction to riak_ensemble Joseph Blomstedt (@jtuple) Basho Technologies riak_ensemble Paxos framework for scalable consistent system 2 node node node node node node node node 3 What about state? 4

More information

A simple totally ordered broadcast protocol

A simple totally ordered broadcast protocol A simple totally ordered broadcast protocol Benjamin Reed Yahoo! Research Santa Clara, CA - USA breed@yahoo-inc.com Flavio P. Junqueira Yahoo! Research Barcelona, Catalunya - Spain fpj@yahoo-inc.com ABSTRACT

More information

Comparative Analysis of Big Data Stream Processing Systems

Comparative Analysis of Big Data Stream Processing Systems Comparative Analysis of Big Data Stream Processing Systems Farouk Salem School of Science Thesis submitted for examination for the degree of Master of Science in Technology. Espoo 22 June, 2016 Thesis

More information

Last time. Distributed systems Lecture 6: Elections, distributed transactions, and replication. DrRobert N. M. Watson

Last time. Distributed systems Lecture 6: Elections, distributed transactions, and replication. DrRobert N. M. Watson Distributed systems Lecture 6: Elections, distributed transactions, and replication DrRobert N. M. Watson 1 Last time Saw how we can build ordered multicast Messages between processes in a group Need to

More information

Exercise 12: Commit Protocols and Replication

Exercise 12: Commit Protocols and Replication Data Modelling and Databases (DMDB) ETH Zurich Spring Semester 2017 Systems Group Lecturer(s): Gustavo Alonso, Ce Zhang Date: May 22, 2017 Assistant(s): Claude Barthels, Eleftherios Sidirourgos, Eliza

More information

Failures, Elections, and Raft

Failures, Elections, and Raft Failures, Elections, and Raft CS 8 XI Copyright 06 Thomas W. Doeppner, Rodrigo Fonseca. All rights reserved. Distributed Banking SFO add interest based on current balance PVD deposit $000 CS 8 XI Copyright

More information

Distributed Systems. Characteristics of Distributed Systems. Lecture Notes 1 Basic Concepts. Operating Systems. Anand Tripathi

Distributed Systems. Characteristics of Distributed Systems. Lecture Notes 1 Basic Concepts. Operating Systems. Anand Tripathi 1 Lecture Notes 1 Basic Concepts Anand Tripathi CSci 8980 Operating Systems Anand Tripathi CSci 8980 1 Distributed Systems A set of computers (hosts or nodes) connected through a communication network.

More information

Distributed Systems. Characteristics of Distributed Systems. Characteristics of Distributed Systems. Goals in Distributed System Designs

Distributed Systems. Characteristics of Distributed Systems. Characteristics of Distributed Systems. Goals in Distributed System Designs 1 Anand Tripathi CSci 8980 Operating Systems Lecture Notes 1 Basic Concepts Distributed Systems A set of computers (hosts or nodes) connected through a communication network. Nodes may have different speeds

More information

BookKeeper overview. Table of contents

BookKeeper overview. Table of contents by Table of contents 1...2 1.1 BookKeeper introduction...2 1.2 In slightly more detail...2 1.3 Bookkeeper elements and concepts... 3 1.4 Bookkeeper initial design... 3 1.5 Bookkeeper metadata management...

More information

AGORA: A Dependable High-Performance Coordination Service for Multi-Cores

AGORA: A Dependable High-Performance Coordination Service for Multi-Cores AGORA: A Dependable High-Performance Coordination Service for Multi-Cores Rainer Schiekofer 1, Johannes Behl 2, and Tobias Distler 1 1 Friedrich-Alexander University Erlangen-Nürnberg (FAU) 2 TU Braunschweig

More information

Distributed Systems. replication Johan Montelius ID2201. Distributed Systems ID2201

Distributed Systems. replication Johan Montelius ID2201. Distributed Systems ID2201 Distributed Systems ID2201 replication Johan Montelius 1 The problem The problem we have: servers might be unavailable The solution: keep duplicates at different servers 2 Building a fault-tolerant service

More information

Transactions. CS 475, Spring 2018 Concurrent & Distributed Systems

Transactions. CS 475, Spring 2018 Concurrent & Distributed Systems Transactions CS 475, Spring 2018 Concurrent & Distributed Systems Review: Transactions boolean transfermoney(person from, Person to, float amount){ if(from.balance >= amount) { from.balance = from.balance

More information

SimpleChubby: a simple distributed lock service

SimpleChubby: a simple distributed lock service SimpleChubby: a simple distributed lock service Jing Pu, Mingyu Gao, Hang Qu 1 Introduction We implement a distributed lock service called SimpleChubby similar to the original Google Chubby lock service[1].

More information

Chapter 4: Distributed Systems: Replication and Consistency. Fall 2013 Jussi Kangasharju

Chapter 4: Distributed Systems: Replication and Consistency. Fall 2013 Jussi Kangasharju Chapter 4: Distributed Systems: Replication and Consistency Fall 2013 Jussi Kangasharju Chapter Outline n Replication n Consistency models n Distribution protocols n Consistency protocols 2 Data Replication

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google* 정학수, 최주영 1 Outline Introduction Design Overview System Interactions Master Operation Fault Tolerance and Diagnosis Conclusions

More information

! Design constraints. " Component failures are the norm. " Files are huge by traditional standards. ! POSIX-like

! Design constraints.  Component failures are the norm.  Files are huge by traditional standards. ! POSIX-like Cloud background Google File System! Warehouse scale systems " 10K-100K nodes " 50MW (1 MW = 1,000 houses) " Power efficient! Located near cheap power! Passive cooling! Power Usage Effectiveness = Total

More information

Distributed Systems. 10. Consensus: Paxos. Paul Krzyzanowski. Rutgers University. Fall 2017

Distributed Systems. 10. Consensus: Paxos. Paul Krzyzanowski. Rutgers University. Fall 2017 Distributed Systems 10. Consensus: Paxos Paul Krzyzanowski Rutgers University Fall 2017 1 Consensus Goal Allow a group of processes to agree on a result All processes must agree on the same value The value

More information

Distributed Systems 16. Distributed File Systems II

Distributed Systems 16. Distributed File Systems II Distributed Systems 16. Distributed File Systems II Paul Krzyzanowski pxk@cs.rutgers.edu 1 Review NFS RPC-based access AFS Long-term caching CODA Read/write replication & disconnected operation DFS AFS

More information

Mission Critical Linux

Mission Critical Linux http://www.mclx.com kohari@mclx.com High Availability Middleware For Telecommunications September, 2002 1 Founded in 1999 as an engineering company with financial backing from top name venture capitalist

More information

TAPIR. By Irene Zhang, Naveen Sharma, Adriana Szekeres, Arvind Krishnamurthy, and Dan Ports Presented by Todd Charlton

TAPIR. By Irene Zhang, Naveen Sharma, Adriana Szekeres, Arvind Krishnamurthy, and Dan Ports Presented by Todd Charlton TAPIR By Irene Zhang, Naveen Sharma, Adriana Szekeres, Arvind Krishnamurthy, and Dan Ports Presented by Todd Charlton Outline Problem Space Inconsistent Replication TAPIR Evaluation Conclusion Problem

More information

Distributed Algorithms (PhD course) Consensus SARDAR MUHAMMAD SULAMAN

Distributed Algorithms (PhD course) Consensus SARDAR MUHAMMAD SULAMAN Distributed Algorithms (PhD course) Consensus SARDAR MUHAMMAD SULAMAN Consensus The processes use consensus to agree on a common value out of values they initially propose Reaching consensus is one of

More information

Building Durable Real-time Data Pipeline

Building Durable Real-time Data Pipeline Building Durable Real-time Data Pipeline Apache BookKeeper at Twitter @sijieg Twitter Background Layered Architecture Agenda Design Details Performance Scale @Twitter Q & A Publish-Subscribe Online services

More information

Data Storage Revolution

Data Storage Revolution Data Storage Revolution Relational Databases Object Storage (put/get) Dynamo PNUTS CouchDB MemcacheDB Cassandra Speed Scalability Availability Throughput No Complexity Eventual Consistency Write Request

More information

Intuitive distributed algorithms. with F#

Intuitive distributed algorithms. with F# Intuitive distributed algorithms with F# Natallia Dzenisenka Alena Hall @nata_dzen @lenadroid A tour of a variety of intuitivedistributed algorithms used in practical distributed systems. and how to prototype

More information

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures GFS Overview Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures Interface: non-posix New op: record appends (atomicity matters,

More information

Tail Latency in ZooKeeper and a Simple Reimplementation

Tail Latency in ZooKeeper and a Simple Reimplementation Tail Latency in ZooKeeper and a Simple Reimplementation Michael Graczyk Abstract ZooKeeper [1] is a commonly used service for coordinating distributed applications. ZooKeeper uses leader-based atomic broadcast

More information

Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitionin

Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitionin Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitioning (Different data at different sites More space Better

More information

Abstract. Introduction

Abstract. Introduction Highly Available In-Memory Metadata Filesystem using Viewstamped Replication (https://github.com/pkvijay/metadr) Pradeep Kumar Vijay, Pedro Ulises Cuevas Berrueco Stanford cs244b-distributed Systems Abstract

More information

ZooKeeper Dynamic Reconfiguration

ZooKeeper Dynamic Reconfiguration by Table of contents 1 Overview... 2 2 Changes to Configuration Format...2 2.1 Specifying the client port... 2 2.2 The standaloneenabled flag...3 2.3 Dynamic configuration file...3 2.4 Backward compatibility...

More information

Basic vs. Reliable Multicast

Basic vs. Reliable Multicast Basic vs. Reliable Multicast Basic multicast does not consider process crashes. Reliable multicast does. So far, we considered the basic versions of ordered multicasts. What about the reliable versions?

More information

Cloud-Native Applications. Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0

Cloud-Native Applications. Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cloud-Native Applications Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cloud-Native Characteristics Lean Form a hypothesis, build just enough to validate or disprove it. Learn

More information

CockroachDB on DC/OS. Ben Darnell, CTO, Cockroach Labs

CockroachDB on DC/OS. Ben Darnell, CTO, Cockroach Labs CockroachDB on DC/OS Ben Darnell, CTO, Cockroach Labs Agenda A cloud-native database CockroachDB on DC/OS Why CockroachDB Demo! Cloud-Native Database What is Cloud-Native? Horizontally scalable Individual

More information

Going beyond MapReduce

Going beyond MapReduce Going beyond MapReduce MapReduce provides a simple abstraction to write distributed programs running on large-scale systems on large amounts of data MapReduce is not suitable for everyone MapReduce abstraction

More information

Diskless Data Analytics on Distributed Coordination Systems

Diskless Data Analytics on Distributed Coordination Systems Diskless Data Analytics on Distributed Coordination Systems by Dayal Dilli Bachelor of Engineering, Anna University, 2012 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF

More information

Distributed Systems 11. Consensus. Paul Krzyzanowski

Distributed Systems 11. Consensus. Paul Krzyzanowski Distributed Systems 11. Consensus Paul Krzyzanowski pxk@cs.rutgers.edu 1 Consensus Goal Allow a group of processes to agree on a result All processes must agree on the same value The value must be one

More information

arxiv: v3 [cs.dc] 4 Dec 2018

arxiv: v3 [cs.dc] 4 Dec 2018 RCanopus: Making Canopus Resilient to Failures and Byzantine Faults arxiv:1810.09300v3 [cs.dc] 4 Dec 2018 1 Introduction S. Keshav, W. Golab, B. Wong, S. Rizvi, and S. Gorbunov School of Computer Science,

More information

Large-Scale Key-Value Stores Eventual Consistency Marco Serafini

Large-Scale Key-Value Stores Eventual Consistency Marco Serafini Large-Scale Key-Value Stores Eventual Consistency Marco Serafini COMPSCI 590S Lecture 13 Goals of Key-Value Stores Export simple API put(key, value) get(key) Simpler and faster than a DBMS Less complexity,

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung ACM SIGOPS 2003 {Google Research} Vaibhav Bajpai NDS Seminar 2011 Looking Back time Classics Sun NFS (1985) CMU Andrew FS (1988) Fault

More information

Paxos Made Live. An Engineering Perspective. Authors: Tushar Chandra, Robert Griesemer, Joshua Redstone. Presented By: Dipendra Kumar Jha

Paxos Made Live. An Engineering Perspective. Authors: Tushar Chandra, Robert Griesemer, Joshua Redstone. Presented By: Dipendra Kumar Jha Paxos Made Live An Engineering Perspective Authors: Tushar Chandra, Robert Griesemer, Joshua Redstone Presented By: Dipendra Kumar Jha Consensus Algorithms Consensus: process of agreeing on one result

More information

CA485 Ray Walshe Google File System

CA485 Ray Walshe Google File System Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage

More information

A Brief Introduction to Key-Value Stores

A Brief Introduction to Key-Value Stores A Brief Introduction to Key-Value Stores Jakob Blomer ALICE Offline Week, CERN July 1st 2015 1 / 26 1 A Little Bit on NoSQL, ACID, BASE, and the CAP Theorem 2 Three Examples: Riak, ZooKeeper, RAMCloud

More information

Distributed Systems 8L for Part IB

Distributed Systems 8L for Part IB Distributed Systems 8L for Part IB Handout 3 Dr. Steven Hand 1 Distributed Mutual Exclusion In first part of course, saw need to coordinate concurrent processes / threads In particular considered how to

More information

CHAPTER 4: INTERPROCESS COMMUNICATION AND COORDINATION

CHAPTER 4: INTERPROCESS COMMUNICATION AND COORDINATION CHAPTER 4: INTERPROCESS COMMUNICATION AND COORDINATION Chapter outline Discuss three levels of communication: basic message passing, request/reply and transaction communication based on message passing

More information

Assignment 12: Commit Protocols and Replication

Assignment 12: Commit Protocols and Replication Data Modelling and Databases Exercise dates: May 24 / May 25, 2018 Ce Zhang, Gustavo Alonso Last update: June 04, 2018 Spring Semester 2018 Head TA: Ingo Müller Assignment 12: Commit Protocols and Replication

More information

The Google File System (GFS)

The Google File System (GFS) 1 The Google File System (GFS) CS60002: Distributed Systems Antonio Bruto da Costa Ph.D. Student, Formal Methods Lab, Dept. of Computer Sc. & Engg., Indian Institute of Technology Kharagpur 2 Design constraints

More information

PNUTS and Weighted Voting. Vijay Chidambaram CS 380 D (Feb 8)

PNUTS and Weighted Voting. Vijay Chidambaram CS 380 D (Feb 8) PNUTS and Weighted Voting Vijay Chidambaram CS 380 D (Feb 8) PNUTS Distributed database built by Yahoo Paper describes a production system Goals: Scalability Low latency, predictable latency Must handle

More information

DrRobert N. M. Watson

DrRobert N. M. Watson Distributed systems Lecture 15: Replication, quorums, consistency, CAP, and Amazon/Google case studies DrRobert N. M. Watson 1 Last time General issue of consensus: How to get processes to agree on something

More information

Coupling Thursday, October 21, :23 PM

Coupling Thursday, October 21, :23 PM Coupling Page 1 Coupling Thursday, October 21, 2004 3:23 PM Two kinds of multiple-processor systems Tightly-coupled Can share efficient semaphores. Usually involve some form of shared memory. Loosely-coupled

More information

GFS: The Google File System

GFS: The Google File System GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one

More information

Two phase commit protocol. Two phase commit protocol. Recall: Linearizability (Strong Consistency) Consensus

Two phase commit protocol. Two phase commit protocol. Recall: Linearizability (Strong Consistency) Consensus Recall: Linearizability (Strong Consistency) Consensus COS 518: Advanced Computer Systems Lecture 4 Provide behavior of a single copy of object: Read should urn the most recent write Subsequent reads should

More information

CS 333 Introduction to Operating Systems. Class 3 Threads & Concurrency. Jonathan Walpole Computer Science Portland State University

CS 333 Introduction to Operating Systems. Class 3 Threads & Concurrency. Jonathan Walpole Computer Science Portland State University CS 333 Introduction to Operating Systems Class 3 Threads & Concurrency Jonathan Walpole Computer Science Portland State University 1 The Process Concept 2 The Process Concept Process a program in execution

More information

Recall: Primary-Backup. State machine replication. Extend PB for high availability. Consensus 2. Mechanism: Replicate and separate servers

Recall: Primary-Backup. State machine replication. Extend PB for high availability. Consensus 2. Mechanism: Replicate and separate servers Replicated s, RAFT COS 8: Distributed Systems Lecture 8 Recall: Primary-Backup Mechanism: Replicate and separate servers Goal #: Provide a highly reliable service Goal #: Servers should behave just like

More information

Minuet Rethinking Concurrency Control in Storage Area Networks

Minuet Rethinking Concurrency Control in Storage Area Networks Minuet Rethinking Concurrency Control in Storage Area Networks FAST 09 Andrey Ermolinskiy (U. C. Berkeley) Daekyeong Moon (U. C. Berkeley) Byung-Gon Chun (Intel Research, Berkeley) Scott Shenker (U. C.

More information

Replication and Consistency. Fall 2010 Jussi Kangasharju

Replication and Consistency. Fall 2010 Jussi Kangasharju Replication and Consistency Fall 2010 Jussi Kangasharju Chapter Outline Replication Consistency models Distribution protocols Consistency protocols 2 Data Replication user B user C user A object object

More information

Consensus and related problems

Consensus and related problems Consensus and related problems Today l Consensus l Google s Chubby l Paxos for Chubby Consensus and failures How to make process agree on a value after one or more have proposed what the value should be?

More information

Distributed Systems. Day 13: Distributed Transaction. To Be or Not to Be Distributed.. Transactions

Distributed Systems. Day 13: Distributed Transaction. To Be or Not to Be Distributed.. Transactions Distributed Systems Day 13: Distributed Transaction To Be or Not to Be Distributed.. Transactions Summary Background on Transactions ACID Semantics Distribute Transactions Terminology: Transaction manager,,

More information

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 1: Distributed File Systems GFS (The Google File System) 1 Filesystems

More information

Distributed Systems Question Bank UNIT 1 Chapter 1 1. Define distributed systems. What are the significant issues of the distributed systems?

Distributed Systems Question Bank UNIT 1 Chapter 1 1. Define distributed systems. What are the significant issues of the distributed systems? UNIT 1 Chapter 1 1. Define distributed systems. What are the significant issues of the distributed systems? 2. What are different application domains of distributed systems? Explain. 3. Discuss the different

More information

MENCIUS: BUILDING EFFICIENT

MENCIUS: BUILDING EFFICIENT MENCIUS: BUILDING EFFICIENT STATE MACHINE FOR WANS By: Yanhua Mao Flavio P. Junqueira Keith Marzullo Fabian Fuxa, Chun-Yu Hsiung November 14, 2018 AGENDA 1. Motivation 2. Breakthrough 3. Rules of Mencius

More information

Replication. Feb 10, 2016 CPSC 416

Replication. Feb 10, 2016 CPSC 416 Replication Feb 10, 2016 CPSC 416 How d we get here? Failures & single systems; fault tolerance techniques added redundancy (ECC memory, RAID, etc.) Conceptually, ECC & RAID both put a master in front

More information

CLOUD-SCALE FILE SYSTEMS

CLOUD-SCALE FILE SYSTEMS Data Management in the Cloud CLOUD-SCALE FILE SYSTEMS 92 Google File System (GFS) Designing a file system for the Cloud design assumptions design choices Architecture GFS Master GFS Chunkservers GFS Clients

More information

The Google File System

The Google File System The Google File System By Ghemawat, Gobioff and Leung Outline Overview Assumption Design of GFS System Interactions Master Operations Fault Tolerance Measurements Overview GFS: Scalable distributed file

More information

PushyDB. Jeff Chan, Kenny Lam, Nils Molina, Oliver Song {jeffchan, kennylam, molina,

PushyDB. Jeff Chan, Kenny Lam, Nils Molina, Oliver Song {jeffchan, kennylam, molina, PushyDB Jeff Chan, Kenny Lam, Nils Molina, Oliver Song {jeffchan, kennylam, molina, osong}@mit.edu https://github.com/jeffchan/6.824 1. Abstract PushyDB provides a more fully featured database that exposes

More information

Distributed System. Gang Wu. Spring,2018

Distributed System. Gang Wu. Spring,2018 Distributed System Gang Wu Spring,2018 Lecture7:DFS What is DFS? A method of storing and accessing files base in a client/server architecture. A distributed file system is a client/server-based application

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung December 2003 ACM symposium on Operating systems principles Publisher: ACM Nov. 26, 2008 OUTLINE INTRODUCTION DESIGN OVERVIEW

More information

Enhancing Throughput of

Enhancing Throughput of Enhancing Throughput of NCA 2017 Zhongmiao Li, Peter Van Roy and Paolo Romano Enhancing Throughput of Partially Replicated State Machines via NCA 2017 Zhongmiao Li, Peter Van Roy and Paolo Romano Enhancing

More information

CS /15/16. Paul Krzyzanowski 1. Question 1. Distributed Systems 2016 Exam 2 Review. Question 3. Question 2. Question 5.

CS /15/16. Paul Krzyzanowski 1. Question 1. Distributed Systems 2016 Exam 2 Review. Question 3. Question 2. Question 5. Question 1 What makes a message unstable? How does an unstable message become stable? Distributed Systems 2016 Exam 2 Review Paul Krzyzanowski Rutgers University Fall 2016 In virtual sychrony, a message

More information

Spanner: Google's Globally-Distributed Database. Presented by Maciej Swiech

Spanner: Google's Globally-Distributed Database. Presented by Maciej Swiech Spanner: Google's Globally-Distributed Database Presented by Maciej Swiech What is Spanner? "...Google's scalable, multi-version, globallydistributed, and synchronously replicated database." What is Spanner?

More information

Everything You Need to Know About MySQL Group Replication

Everything You Need to Know About MySQL Group Replication Everything You Need to Know About MySQL Group Replication Luís Soares (luis.soares@oracle.com) Principal Software Engineer, MySQL Replication Lead Copyright 2017, Oracle and/or its affiliates. All rights

More information

Paxos and Raft (Lecture 21, cs262a) Ion Stoica, UC Berkeley November 7, 2016

Paxos and Raft (Lecture 21, cs262a) Ion Stoica, UC Berkeley November 7, 2016 Paxos and Raft (Lecture 21, cs262a) Ion Stoica, UC Berkeley November 7, 2016 Bezos mandate for service-oriented-architecture (~2002) 1. All teams will henceforth expose their data and functionality through

More information

Real-Time Component Software. slide credits: H. Kopetz, P. Puschner

Real-Time Component Software. slide credits: H. Kopetz, P. Puschner Real-Time Component Software slide credits: H. Kopetz, P. Puschner Overview OS services Task Structure Task Interaction Input/Output Error Detection 2 Operating System and Middleware Application Software

More information

locker: distributed locking Knut

locker: distributed locking Knut locker: distributed locking Knut Nesheim @knutin The need Real-time multiplayer game at Wooga Stateful One process per user One process per world The need Only one process per user & world Cannot reconcile

More information

EECS 498 Introduction to Distributed Systems

EECS 498 Introduction to Distributed Systems EECS 498 Introduction to Distributed Systems Fall 2017 Harsha V. Madhyastha Implementing RSMs Logical clock based ordering of requests Cannot serve requests if any one replica is down Primary-backup replication

More information

Exam 2 Review. Fall 2011

Exam 2 Review. Fall 2011 Exam 2 Review Fall 2011 Question 1 What is a drawback of the token ring election algorithm? Bad question! Token ring mutex vs. Ring election! Ring election: multiple concurrent elections message size grows

More information