Spanner: Google's Globally-Distributed Database* Huu-Phuc Vo August 03, 2013
|
|
- Gerard Cunningham
- 6 years ago
- Views:
Transcription
1 Spanner: Google's Globally-Distributed Database* Huu-Phuc Vo August 03, 2013 *OSDI '12, James C. Corbett et al. (26 authors), Jay Lepreau Best Paper Award
2 Outline What is Spanner? Features & Example Structure of Spanner's Implementation Data model Version management Novel TrueTime API External consistency TrueTime API Lock-free read-only transactions Performance 2
3 What is Spanner? What is Spanner? scalable, multi-version database global-distributed database synchronously-replicated database Goal: building rich apps easy at Google scale Key ideas to build transactional storage system replicated globally to manage cross-datacenter replicated data to manage consistency globally 3
4 Features Features externally consistent reads & writes non-blocking reads in the past lock-free read-only transactions atomic schema changes SQL-like query language Applications support Google's advertising backend replace a sharded MySQL database 4
5 Key ideas Relational data model with SQL & general purpose transactions External consistency transactions can be ordered by their commit time commit times correspond to real world notions of time Paxos-based replication with number of replicas and distance to replicas controllable Data partitioned across thousands of servers 5
6 Example: Global Social Network Spanner maintains single system looks like single database 6
7 Example: Global Social Network sharded data US Brazil Spain Russia 7
8 Example: Global Social Network shard data into in many ways across datacenters User posts User posts US User posts User posts Brazil User User posts posts Spain User posts Russia 8
9 Example: Global Social Network replicate & store across datacenters User posts User posts San Francisco Seattle Arizona User posts User posts Sao Paulo Santiago US Brazil User User posts posts London Paris Berlin Madrid User posts Moscow Berlin Spain Russia 9
10 Structure of Spanner's Implementation Spanserver software stack Directories are the unit of data movement between Paxos groups 10
11 Spanserver software stack Transaction manager (TM): support distributed transactions Participant leader: implemented by using TM Lock table: implement concurrency control Leader: a replica that is selected among replicas Paxos state machine: support data replication Tablet: instances of a data structure Colossus: distributed file system (Google File System's successor) 11
12 Structure of Spanner's Implementation Spanner deployment = universe Zones set of zones = set of locations of dist. data can be more than 1 zone in a datacenter Servers in Spanner universe universe master: a console to show all zones for debug placement driver: handles automated movement of data across zones on timescale of minutes & communicates with spanserver location proxy: used by clients to locate spanservers to serve their data 12
13 Directory: unit of data placement move data between Paxos groups Purpose of moving: put frequently accessed dir into same group move to group that close to its accessors load balancing Directories moving while client operations are ongoing move 50MB directory in a few seconds 13
14 Data Model A data model based on schematized semi-relational tables with popularity of Megastore Query language with popularity of Dremel General-purpose Xacts experienced the lack with BigTable Not purely relational (rows have names) DB must be partitioned into hierarchies 14
15 Version Management Timestamp: to order all writes in a transaction (trans.) If T2 starts after commit of T1 then commit TS (T2) must greater commit TS (T1) Version management write transactions use strict 2-Phase Locking (2PL) Strict 2PL: trans. requests lock at any time before action & releases locks when commits each trans. T is assigned a timestamp (TS) s data written by T is timestamped with s 15
16 Concurrency Control Spanner uses multi-paxos each spanserver runs a single instance of Paxos for each "tablet" of data it stores tablet uses Paxos to agree on a "group leader" all writes will be done by the leader to replicas never concurrent outstanding writes to the same object within a replica group as long as the leader is stable only one round of communication is needed between leader and replicas to con7rm writes 16
17 External Consistency External consistency transactions can be ordered by their commit time commit times correspond to real world notions of time Synchronizing Snapshots timestamp order = commit order ensure consistent clock values by using GPS + atomic clocks on some nodes network time sync. protocols (exchange & adjust time accordingly) small differences between clocks define TrueTime to solve 17
18 External Consistency Example: (node = spanserver) suppose two nodes N1/N2 running T1 & N3/N4 running T2 N1/N2 running 30 seconds ahead N3/N4 T1 commits at time T=0 on node N1/N2 (time T=-30 on N3/N4) T2 commit time earlier than T1 to external observer If T1 & T2 wrote data D, recovered replica may apply T1 18
19 TrueTime TrueTime API methods: estimates accuracy of node's clock Global clock time with bounded uncertainty Ɛ: instantaneous error (half of interval width) 19
20 TrueTime Example How to ensure T1's commits time before T2's? ensure that T2 doesn't commit until after t1.latest t1.latest < t2.earliest: need to know each other's trans. instead, if ensure that T1 holds its locks until t1.latest, then T1 commits before T2 commits 20
21 TrueTime Architecture GPS & atomic-clock timemasters cross-check each other timeslave polls masters to reduce vulnerability from other masters Compute reference [earliest, latest] = now ± Ɛ 21
22 Commit & 2-Phase Commit 22
23 Commit & 2-Phase Commit 23
24 Commit & 2-Phase Commit 24
25 Paxos-based replication Spanner's Paxos implementation used timed (10 second) leader leases to make leadership long lived able to tolerate the failure/disconnection of nodes/data centers Idea ensure that all writes & reads go to a "quorum" of nodes Quorum: a simple majority if we have N nodes, read & write to 6oor(N/2) + 1 nodes In reality, send write & read request to all nodes, but only wait for a quorum of them to complete basics of quorum-based replication last time 25
26 Performance Micro-benchmarks for Spanner a single spanserver per zone enough load to saturate spanserver CPU all data is served out of memory to measure overhead of Spanner stack get about 2,500 transactional writes/s per CPU 15,000 transactional reads/s per CPU 26
27 Summary Spanner blended & developed semi-relational interface, transactions & SQL-based query language concepts of scalability, auto segmentation, failure resistance, data replication consistency & wide distribution TrueTime another key functionality of Spanner provides a functionality based on accurate time synchronization in distributed system by expressing inaccuracy of time more specifically in time API 27
28 Thank you 28
Spanner : Google's Globally-Distributed Database. James Sedgwick and Kayhan Dursun
Spanner : Google's Globally-Distributed Database James Sedgwick and Kayhan Dursun Spanner - A multi-version, globally-distributed, synchronously-replicated database - First system to - Distribute data
More informationSpanner: Google s Globally-Distributed Database. Wilson Hsieh representing a host of authors OSDI 2012
Spanner: Google s Globally-Distributed Database Wilson Hsieh representing a host of authors OSDI 2012 What is Spanner? Distributed multiversion database General-purpose transactions (ACID) SQL query language
More informationSpanner: Google's Globally-Distributed Database. Presented by Maciej Swiech
Spanner: Google's Globally-Distributed Database Presented by Maciej Swiech What is Spanner? "...Google's scalable, multi-version, globallydistributed, and synchronously replicated database." What is Spanner?
More informationCorbett et al., Spanner: Google s Globally-Distributed Database
Corbett et al., : Google s Globally-Distributed Database MIMUW 2017-01-11 ACID transactions ACID transactions SQL queries ACID transactions SQL queries Semi-relational data model ACID transactions SQL
More informationGoogle Spanner - A Globally Distributed,
Google Spanner - A Globally Distributed, Synchronously-Replicated Database System James C. Corbett, et. al. Feb 14, 2013. Presented By Alexander Chow For CS 742 Motivation Eventually-consistent sometimes
More informationDistributed Systems. 19. Spanner. Paul Krzyzanowski. Rutgers University. Fall 2017
Distributed Systems 19. Spanner Paul Krzyzanowski Rutgers University Fall 2017 November 20, 2017 2014-2017 Paul Krzyzanowski 1 Spanner (Google s successor to Bigtable sort of) 2 Spanner Take Bigtable and
More informationSpanner: Google s Globally- Distributed Database
Today s Reminders Spanner: Google s Globally-Distributed Database Discuss Project Ideas with Phil & Kevin Phil s Office Hours: After class today Sign up for a slot: 11-12:30 or 3-4:20 this Friday Phil
More informationSpanner: Google s Globally- Distributed Database
Spanner: Google s Globally- Distributed Database Google, Inc. OSDI 2012 Presented by: Karen Ouyang Problem Statement Distributed data system with high availability Support external consistency! Key Ideas
More informationMegastore: Providing Scalable, Highly Available Storage for Interactive Services & Spanner: Google s Globally- Distributed Database.
Megastore: Providing Scalable, Highly Available Storage for Interactive Services & Spanner: Google s Globally- Distributed Database. Presented by Kewei Li The Problem db nosql complex legacy tuning expensive
More informationDistributed Data Management. Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig
Distributed Data Management Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de Sommerfest! Distributed Data Management Christoph Lofi IfIS TU
More informationNewSQL Databases. The reference Big Data stack
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica NewSQL Databases Corso di Sistemi e Architetture per Big Data A.A. 2017/18 Valeria Cardellini The reference
More informationBeyond TrueTime: Using AugmentedTime for Improving Spanner
Beyond TrueTime: Using AugmentedTime for Improving Spanner Murat Demirbas University at Buffalo, SUNY demirbas@cse.buffalo.edu Sandeep Kulkarni Michigan State University, sandeep@cse.msu.edu Spanner [1]
More information7680: Distributed Systems
Cristina Nita-Rotaru 7680: Distributed Systems BigTable. Hbase.Spanner. 1: BigTable Acknowledgement } Slides based on material from course at UMichigan, U Washington, and the authors of BigTable and Spanner.
More informationCloud Computing. DB Special Topics Lecture (10/5/2012) Kyle Hale Maciej Swiech
Cloud Computing DB Special Topics Lecture (10/5/2012) Kyle Hale Maciej Swiech Managing servers isn t for everyone What are some prohibitive issues? (we touched on these last time) Cost (initial/operational)
More informationIntegrity in Distributed Databases
Integrity in Distributed Databases Andreas Farella Free University of Bozen-Bolzano Table of Contents 1 Introduction................................................... 3 2 Different aspects of integrity.....................................
More informationDistributed Systems. GFS / HDFS / Spanner
15-440 Distributed Systems GFS / HDFS / Spanner Agenda Google File System (GFS) Hadoop Distributed File System (HDFS) Distributed File Systems Replication Spanner Distributed Database System Paxos Replication
More informationConcurrency Control II and Distributed Transactions
Concurrency Control II and Distributed Transactions CS 240: Computing Systems and Concurrency Lecture 18 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material.
More informationHow do we build TiDB. a Distributed, Consistent, Scalable, SQL Database
How do we build TiDB a Distributed, Consistent, Scalable, SQL Database About me LiuQi ( 刘奇 ) JD / WandouLabs / PingCAP Co-founder / CEO of PingCAP Open-source hacker / Infrastructure software engineer
More informationBuilding Consistent Transactions with Inconsistent Replication
Building Consistent Transactions with Inconsistent Replication Irene Zhang, Naveen Kr. Sharma, Adriana Szekeres, Arvind Krishnamurthy, Dan R. K. Ports University of Washington Distributed storage systems
More informationDistributed Transactions
Distributed Transactions CS6450: Distributed Systems Lecture 17 Ryan Stutsman Material taken/derived from Princeton COS-418 materials created by Michael Freedman and Kyle Jamieson at Princeton University.
More informationMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES
MINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department of Computer Science University of California at Santa Barbara Joint work with: Amr El Abbadi, Hatem Mahmoud, Faisal
More informationMulti-version concurrency control
MVCC and Distributed Txns (Spanner) 2P & CC = strict serialization Provides semantics as if only one transaction was running on DB at time, in serial order + Real-time guarantees CS 518: Advanced Computer
More informationDistributed Systems. Fall 2017 Exam 3 Review. Paul Krzyzanowski. Rutgers University. Fall 2017
Distributed Systems Fall 2017 Exam 3 Review Paul Krzyzanowski Rutgers University Fall 2017 December 11, 2017 CS 417 2017 Paul Krzyzanowski 1 Question 1 The core task of the user s map function within a
More informationDrRobert N. M. Watson
Distributed systems Lecture 15: Replication, quorums, consistency, CAP, and Amazon/Google case studies DrRobert N. M. Watson 1 Last time General issue of consensus: How to get processes to agree on something
More informationCSE 444: Database Internals. Lectures 26 NoSQL: Extensible Record Stores
CSE 444: Database Internals Lectures 26 NoSQL: Extensible Record Stores CSE 444 - Spring 2014 1 References Scalable SQL and NoSQL Data Stores, Rick Cattell, SIGMOD Record, December 2010 (Vol. 39, No. 4)
More informationReferences. What is Bigtable? Bigtable Data Model. Outline. Key Features. CSE 444: Database Internals
References CSE 444: Database Internals Scalable SQL and NoSQL Data Stores, Rick Cattell, SIGMOD Record, December 2010 (Vol 39, No 4) Lectures 26 NoSQL: Extensible Record Stores Bigtable: A Distributed
More informationExtreme Computing. NoSQL.
Extreme Computing NoSQL PREVIOUSLY: BATCH Query most/all data Results Eventually NOW: ON DEMAND Single Data Points Latency Matters One problem, three ideas We want to keep track of mutable state in a scalable
More informationApplications of Paxos Algorithm
Applications of Paxos Algorithm Gurkan Solmaz COP 6938 - Cloud Computing - Fall 2012 Department of Electrical Engineering and Computer Science University of Central Florida - Orlando, FL Oct 15, 2012 1
More informationCMU SCS CMU SCS Who: What: When: Where: Why: CMU SCS
Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB s C. Faloutsos A. Pavlo Lecture#23: Distributed Database Systems (R&G ch. 22) Administrivia Final Exam Who: You What: R&G Chapters 15-22
More informationBigtable. Presenter: Yijun Hou, Yixiao Peng
Bigtable Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber Google, Inc. OSDI 06 Presenter: Yijun Hou, Yixiao Peng
More informationHow we build TiDB. Max Liu PingCAP Amsterdam, Netherlands October 5, 2016
How we build TiDB Max Liu PingCAP Amsterdam, Netherlands October 5, 2016 About me Infrastructure engineer / CEO of PingCAP Working on open source projects: TiDB: https://github.com/pingcap/tidb TiKV: https://github.com/pingcap/tikv
More informationMulti-version concurrency control
Spanner Storage insights 2P & CC = strict serialization Provides semantics as if only one transaction was running on DB at time, in serial order + Real-time guarantees CS 518: Advanced Computer Systems
More informationDistributed KIDS Labs 1
Distributed Databases @ KIDS Labs 1 Distributed Database System A distributed database system consists of loosely coupled sites that share no physical component Appears to user as a single system Database
More informationBigtable: A Distributed Storage System for Structured Data By Fay Chang, et al. OSDI Presented by Xiang Gao
Bigtable: A Distributed Storage System for Structured Data By Fay Chang, et al. OSDI 2006 Presented by Xiang Gao 2014-11-05 Outline Motivation Data Model APIs Building Blocks Implementation Refinement
More informationDistributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
More informationOutline. Spanner Mo/va/on. Tom Anderson
Spanner Mo/va/on Tom Anderson Outline Last week: Chubby: coordina/on service BigTable: scalable storage of structured data GFS: large- scale storage for bulk data Today/Friday: Lessons from GFS/BigTable
More informationTiDB: NewSQL over HBase.
TiDB: NewSQL over HBase liuqi@pingcap.com https://github.com/pingcap/tidb weibo: @goroutine Agenda HBase introduction TiDB features Internals of TiDB over HBase Features of HBase Linear and modular scalability.
More informationGhislain Fourny. Big Data 5. Wide column stores
Ghislain Fourny Big Data 5. Wide column stores Data Technology Stack User interfaces Querying Data stores Indexing Processing Validation Data models Syntax Encoding Storage 2 Where we are User interfaces
More informationBuilding Consistent Transactions with Inconsistent Replication
DB Reading Group Fall 2015 slides by Dana Van Aken Building Consistent Transactions with Inconsistent Replication Irene Zhang, Naveen Kr. Sharma, Adriana Szekeres, Arvind Krishnamurthy, Dan R. K. Ports
More information10. Replication. Motivation
10. Replication Page 1 10. Replication Motivation Reliable and high-performance computation on a single instance of a data object is prone to failure. Replicate data to overcome single points of failure
More informationEECS 498 Introduction to Distributed Systems
EECS 498 Introduction to Distributed Systems Fall 2017 Harsha V. Madhyastha Dynamo Recap Consistent hashing 1-hop DHT enabled by gossip Execution of reads and writes Coordinated by first available successor
More informationToday s Papers. Google Chubby. Distributed Consensus. EECS 262a Advanced Topics in Computer Systems Lecture 24. Paxos/Megastore November 24 th, 2014
EECS 262a Advanced Topics in Computer Systems Lecture 24 Paxos/Megastore November 24 th, 2014 John Kubiatowicz Electrical Engineering and Computer Sciences University of California, Berkeley Today s Papers
More informationDISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems INTRODUCTION. Transparency: Flexibility: Slide 1. Slide 3.
CHALLENGES Transparency: Slide 1 DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems ➀ Introduction ➁ NFS (Network File System) ➂ AFS (Andrew File System) & Coda ➃ GFS (Google File System)
More informationIntroduction Data Model API Building Blocks SSTable Implementation Tablet Location Tablet Assingment Tablet Serving Compactions Refinements
Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber Google, Inc. M. Burak ÖZTÜRK 1 Introduction Data Model API Building
More informationChapter 24 NOSQL Databases and Big Data Storage Systems
Chapter 24 NOSQL Databases and Big Data Storage Systems - Large amounts of data such as social media, Web links, user profiles, marketing and sales, posts and tweets, road maps, spatial data, email - NOSQL
More informationCISC 7610 Lecture 2b The beginnings of NoSQL
CISC 7610 Lecture 2b The beginnings of NoSQL Topics: Big Data Google s infrastructure Hadoop: open google infrastructure Scaling through sharding CAP theorem Amazon s Dynamo 5 V s of big data Everyone
More informationDistributed System. Gang Wu. Spring,2018
Distributed System Gang Wu Spring,2018 Lecture7:DFS What is DFS? A method of storing and accessing files base in a client/server architecture. A distributed file system is a client/server-based application
More informationPerformance and Forgiveness. June 23, 2008 Margo Seltzer Harvard University School of Engineering and Applied Sciences
Performance and Forgiveness June 23, 2008 Margo Seltzer Harvard University School of Engineering and Applied Sciences Margo Seltzer Architect Outline A consistency primer Techniques and costs of consistency
More informationGoal of the presentation is to give an introduction of NoSQL databases, why they are there.
1 Goal of the presentation is to give an introduction of NoSQL databases, why they are there. We want to present "Why?" first to explain the need of something like "NoSQL" and then in "What?" we go in
More information10. Replication. CSEP 545 Transaction Processing Philip A. Bernstein Sameh Elnikety. Copyright 2012 Philip A. Bernstein
10. Replication CSEP 545 Transaction Processing Philip A. Bernstein Sameh Elnikety Copyright 2012 Philip A. Bernstein 1 Outline 1. Introduction 2. Primary-Copy Replication 3. Multi-Master Replication 4.
More informationPaxos Replicated State Machines as the Basis of a High- Performance Data Store
Paxos Replicated State Machines as the Basis of a High- Performance Data Store William J. Bolosky, Dexter Bradshaw, Randolph B. Haagens, Norbert P. Kusters and Peng Li March 30, 2011 Q: How to build a
More informationFrom Google File System to Omega: a Decade of Advancement in Big Data Management at Google
2015 IEEE First International Conference on Big Data Computing Service and Applications From Google File System to Omega: a Decade of Advancement in Big Data Management at Google Jade Yang Department of
More informationDistributed PostgreSQL with YugaByte DB
Distributed PostgreSQL with YugaByte DB Karthik Ranganathan PostgresConf Silicon Valley Oct 16, 2018 1 CHECKOUT THIS REPO: github.com/yugabyte/yb-sql-workshop 2 About Us Founders Kannan Muthukkaruppan,
More informationApp Engine: Datastore Introduction
App Engine: Datastore Introduction Part 1 Another very useful course: https://www.udacity.com/course/developing-scalableapps-in-java--ud859 1 Topics cover in this lesson What is Datastore? Datastore and
More informationMDCC MULTI DATA CENTER CONSISTENCY. amplab. Tim Kraska, Gene Pang, Michael Franklin, Samuel Madden, Alan Fekete
MDCC MULTI DATA CENTER CONSISTENCY Tim Kraska, Gene Pang, Michael Franklin, Samuel Madden, Alan Fekete gpang@cs.berkeley.edu amplab MOTIVATION 2 3 June 2, 200: Rackspace power outage of approximately 0
More informationBigtable: A Distributed Storage System for Structured Data by Google SUNNIE CHUNG CIS 612
Bigtable: A Distributed Storage System for Structured Data by Google SUNNIE CHUNG CIS 612 Google Bigtable 2 A distributed storage system for managing structured data that is designed to scale to a very
More informationCS November 2017
Bigtable Highly available distributed storage Distributed Systems 18. Bigtable Built with semi-structured data in mind URLs: content, metadata, links, anchors, page rank User data: preferences, account
More informationTAPIR. By Irene Zhang, Naveen Sharma, Adriana Szekeres, Arvind Krishnamurthy, and Dan Ports Presented by Todd Charlton
TAPIR By Irene Zhang, Naveen Sharma, Adriana Szekeres, Arvind Krishnamurthy, and Dan Ports Presented by Todd Charlton Outline Problem Space Inconsistent Replication TAPIR Evaluation Conclusion Problem
More informationCS /29/17. Paul Krzyzanowski 1. Fall 2016: Question 2. Distributed Systems. Fall 2016: Question 2 (cont.) Fall 2016: Question 3
Fall 2016: Question 2 You have access to a file of class enrollment lists. Each line contains {course_number, student_id}. Distributed Systems 2017 Pre-exam 3 review Selected questions from past exams
More informationThere Is More Consensus in Egalitarian Parliaments
There Is More Consensus in Egalitarian Parliaments Iulian Moraru, David Andersen, Michael Kaminsky Carnegie Mellon University Intel Labs Fault tolerance Redundancy State Machine Replication 3 State Machine
More informationPushyDB. Jeff Chan, Kenny Lam, Nils Molina, Oliver Song {jeffchan, kennylam, molina,
PushyDB Jeff Chan, Kenny Lam, Nils Molina, Oliver Song {jeffchan, kennylam, molina, osong}@mit.edu https://github.com/jeffchan/6.824 1. Abstract PushyDB provides a more fully featured database that exposes
More informationData-Intensive Distributed Computing
Data-Intensive Distributed Computing CS 451/651 (Fall 2018) Part 7: Mutable State (2/2) November 13, 2018 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These slides are
More informationCS 655 Advanced Topics in Distributed Systems
Presented by : Walid Budgaga CS 655 Advanced Topics in Distributed Systems Computer Science Department Colorado State University 1 Outline Problem Solution Approaches Comparison Conclusion 2 Problem 3
More informationSQL, NoSQL, MongoDB. CSE-291 (Cloud Computing) Fall 2016 Gregory Kesden
SQL, NoSQL, MongoDB CSE-291 (Cloud Computing) Fall 2016 Gregory Kesden SQL Databases Really better called Relational Databases Key construct is the Relation, a.k.a. the table Rows represent records Columns
More informationOutline. INF3190:Distributed Systems - Examples. Last week: Definitions Transparencies Challenges&pitfalls Architecturalstyles
INF3190:Distributed Systems - Examples Thomas Plagemann & Roman Vitenberg Outline Last week: Definitions Transparencies Challenges&pitfalls Architecturalstyles Today: Examples Googel File System (Thomas)
More informationShen PingCAP 2017
Shen Li @ PingCAP About me Shen Li ( 申砾 ) Tech Lead of TiDB, VP of Engineering Netease / 360 / PingCAP Infrastructure software engineer WHY DO WE NEED A NEW DATABASE? Brief History Standalone RDBMS NoSQL
More informationSCALARIS. Irina Calciu Alex Gillmor
SCALARIS Irina Calciu Alex Gillmor RoadMap Motivation Overview Architecture Features Implementation Benchmarks API Users Demo Conclusion Motivation (NoSQL) "One size doesn't fit all" Stonebraker Reinefeld
More informationA Brief Introduction of TiDB. Dongxu (Edward) Huang CTO, PingCAP
A Brief Introduction of TiDB Dongxu (Edward) Huang CTO, PingCAP About me Dongxu (Edward) Huang, Cofounder & CTO of PingCAP PingCAP, based in Beijing, China. Infrastructure software engineer, open source
More informationCS555: Distributed Systems [Fall 2017] Dept. Of Computer Science, Colorado State University
CS 555: DISTRIBUTED SYSTEMS [DYNAMO & GOOGLE FILE SYSTEM] Frequently asked questions from the previous class survey What s the typical size of an inconsistency window in most production settings? Dynamo?
More informationCloud Scale Storage Systems. Yunhao Zhang & Matthew Gharrity
Cloud Scale Storage Systems Yunhao Zhang & Matthew Gharrity Two Beautiful Papers Google File System SIGOPS Hall of Fame! pioneer of large-scale storage system Spanner OSDI 12 Best Paper Award! Big Table
More informationMySQL Group Replication. Bogdan Kecman MySQL Principal Technical Engineer
MySQL Group Replication Bogdan Kecman MySQL Principal Technical Engineer Bogdan.Kecman@oracle.com 1 Safe Harbor Statement The following is intended to outline our general product direction. It is intended
More informationBigtable: A Distributed Storage System for Structured Data. Andrew Hon, Phyllis Lau, Justin Ng
Bigtable: A Distributed Storage System for Structured Data Andrew Hon, Phyllis Lau, Justin Ng What is Bigtable? - A storage system for managing structured data - Used in 60+ Google services - Motivation:
More informationEverything You Need to Know About MySQL Group Replication
Everything You Need to Know About MySQL Group Replication Luís Soares (luis.soares@oracle.com) Principal Software Engineer, MySQL Replication Lead Copyright 2017, Oracle and/or its affiliates. All rights
More informationAdvances in Data Management - NoSQL, NewSQL and Big Data A.Poulovassilis
Advances in Data Management - NoSQL, NewSQL and Big Data A.Poulovassilis 1 NoSQL So-called NoSQL systems offer reduced functionalities compared to traditional Relational DBMSs, with the aim of achieving
More informationCISC 7610 Lecture 5 Distributed multimedia databases. Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL
CISC 7610 Lecture 5 Distributed multimedia databases Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL Motivation YouTube receives 400 hours of video per minute That is 200M hours
More informationDon t Give Up on Serializability Just Yet. Neha Narula
Don t Give Up on Serializability Just Yet Neha Narula Don t Give Up on Serializability Just Yet A journey into serializable systems Neha Narula MIT CSAIL GOTO Chicago May 2015 2 @neha PhD candidate at
More informationBigTable. Chubby. BigTable. Chubby. Why Chubby? How to do consensus as a service
BigTable BigTable Doug Woos and Tom Anderson In the early 2000s, Google had way more than anybody else did Traditional bases couldn t scale Want something better than a filesystem () BigTable optimized
More information2/27/2019 Week 6-B Sangmi Lee Pallickara
2/27/2019 - Spring 2019 Week 6-B-1 CS535 BIG DATA FAQs Participation scores will be collected separately Sign-up page is up PART A. BIG DATA TECHNOLOGY 5. SCALABLE DISTRIBUTED FILE SYSTEMS: GOOGLE FILE
More informationDatabase Architectures
Database Architectures CPS352: Database Systems Simon Miner Gordon College Last Revised: 4/15/15 Agenda Check-in Parallelism and Distributed Databases Technology Research Project Introduction to NoSQL
More informationThe Chubby Lock Service for Loosely-coupled Distributed systems
The Chubby Lock Service for Loosely-coupled Distributed systems Author: Mike Burrows Presenter: Ke Nian University of Waterloo, Waterloo, Canada October 22, 2014 Agenda Distributed Consensus Problem 3
More informationConsistency in Distributed Systems
Consistency in Distributed Systems Recall the fundamental DS properties DS may be large in scale and widely distributed 1. concurrent execution of components 2. independent failure modes 3. transmission
More informationAtomicity. Bailu Ding. Oct 18, Bailu Ding Atomicity Oct 18, / 38
Atomicity Bailu Ding Oct 18, 2012 Bailu Ding Atomicity Oct 18, 2012 1 / 38 Outline 1 Introduction 2 State Machine 3 Sinfonia 4 Dangers of Replication Bailu Ding Atomicity Oct 18, 2012 2 / 38 Introduction
More informationJanus. Consolidating Concurrency Control and Consensus for Commits under Conflicts. Shuai Mu, Lamont Nelson, Wyatt Lloyd, Jinyang Li
Janus Consolidating Concurrency Control and Consensus for Commits under Conflicts Shuai Mu, Lamont Nelson, Wyatt Lloyd, Jinyang Li New York University, University of Southern California State of the Art
More informationBigTable. CSE-291 (Cloud Computing) Fall 2016
BigTable CSE-291 (Cloud Computing) Fall 2016 Data Model Sparse, distributed persistent, multi-dimensional sorted map Indexed by a row key, column key, and timestamp Values are uninterpreted arrays of bytes
More informationCA485 Ray Walshe NoSQL
NoSQL BASE vs ACID Summary Traditional relational database management systems (RDBMS) do not scale because they adhere to ACID. A strong movement within cloud computing is to utilize non-traditional data
More informationIntroduction to MySQL InnoDB Cluster
1 / 148 2 / 148 3 / 148 Introduction to MySQL InnoDB Cluster MySQL High Availability made easy Percona Live Europe - Dublin 2017 Frédéric Descamps - MySQL Community Manager - Oracle 4 / 148 Safe Harbor
More informationDistributed Systems Exam 3 Review. Paul Krzyzanowski. Rutgers University. Fall 2016
Distributed Systems 2015 Exam 3 Review Paul Krzyzanowski Rutgers University Fall 2016 2013-2016 Paul Krzyzanowski 1 2015 Question 1 What information does each node need to store for a three-dimensional
More informationZooKeeper. Table of contents
by Table of contents 1 ZooKeeper: A Distributed Coordination Service for Distributed Applications... 2 1.1 Design Goals... 2 1.2 Data model and the hierarchical namespace... 3 1.3 Nodes and ephemeral nodes...
More informationProseminar Distributed Systems Summer Semester Paxos algorithm. Stefan Resmerita
Proseminar Distributed Systems Summer Semester 2016 Paxos algorithm stefan.resmerita@cs.uni-salzburg.at The Paxos algorithm Family of protocols for reaching consensus among distributed agents Agents may
More informationLessons Learned While Building Infrastructure Software at Google
Lessons Learned While Building Infrastructure Software at Google Jeff Dean jeff@google.com Google Circa 1997 (google.stanford.edu) Corkboards (1999) Google Data Center (2000) Google Data Center (2000)
More informationBigTable: A Distributed Storage System for Structured Data
BigTable: A Distributed Storage System for Structured Data Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) BigTable 1393/7/26
More information! Design constraints. " Component failures are the norm. " Files are huge by traditional standards. ! POSIX-like
Cloud background Google File System! Warehouse scale systems " 10K-100K nodes " 50MW (1 MW = 1,000 houses) " Power efficient! Located near cheap power! Passive cooling! Power Usage Effectiveness = Total
More informationMySQL High Availability Solutions. Alex Poritskiy Percona
MySQL High Availability Solutions Alex Poritskiy Percona The Five 9s of Availability Clustering & Geographical Redundancy Clustering Technologies Replication Technologies Well-Managed disasters power failures
More informationExam 2 Review. October 29, Paul Krzyzanowski 1
Exam 2 Review October 29, 2015 2013 Paul Krzyzanowski 1 Question 1 Why did Dropbox add notification servers to their architecture? To avoid the overhead of clients polling the servers periodically to check
More informationMigrating Oracle Databases To Cassandra
BY UMAIR MANSOOB Why Cassandra Lower Cost of ownership makes it #1 choice for Big Data OLTP Applications. Unlike Oracle, Cassandra can store structured, semi-structured, and unstructured data. Cassandra
More informationCPS 512 midterm exam #1, 10/7/2016
CPS 512 midterm exam #1, 10/7/2016 Your name please: NetID: Answer all questions. Please attempt to confine your answers to the boxes provided. If you don t know the answer to a question, then just say
More informationThe Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung December 2003 ACM symposium on Operating systems principles Publisher: ACM Nov. 26, 2008 OUTLINE INTRODUCTION DESIGN OVERVIEW
More informationCSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 14 Distributed Transactions
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 14 Distributed Transactions Transactions Main issues: Concurrency control Recovery from failures 2 Distributed Transactions
More informationReplication in Distributed Systems
Replication in Distributed Systems Replication Basics Multiple copies of data kept in different nodes A set of replicas holding copies of a data Nodes can be physically very close or distributed all over
More informationLow-Latency Multi-Datacenter Databases using Replicated Commit
Low-Latency Multi-Datacenter Databases using Replicated Commit Hatem Mahmoud, Faisal Nawab, Alexander Pucher, Divyakant Agrawal, Amr El Abbadi UCSB Presented by Ashutosh Dhekne Main Contributions Reduce
More information