Consistency and Scalability
|
|
|
- Osborn Ethan Moore
- 5 years ago
- Views:
Transcription
1 COMP 150-IDS: Internet Scale Distributed Systems (Spring 2015) Consistency and Scalability Noah Mendelsohn Tufts University Web: Copyright 2015 Noah Mendelsohn
2 What you should get from today s session You will explore challenges relating to maintaining data consistency in a computing system You will learn about techniques used to make storage systems more reliable You will learn about transactions and their implementation using logs You will learn about the CAP theorem and why scaling and consistency tend not to come together 2
3 A note about scope The challenges & principles we cover today reappear at every level of system design CPU Instruction set and memory Parallel programming languages Single machine databases Distributed applications and databases Today we will focus mainly on larger scale systems 3
4 Why Worry About Consistency? 4
5 Duplicate information in computing systems Why complicated things? Mirrored disks for reliability Parallel processing higher throughput Geographic distribution reduces network delay (one each in Europe, Asia, US) Higher availability if network crashes, each partition may still have a copy Inter-dependent data Bank account records have total for each account Bank record keeps total for all accounts Memory Hierarchies CPU Caches, file system caches, Web proxies, etc. If we allow updates, then maintaining consistency is tricky 5
6 Simple Examples: Parallel Disk Systems 6
7 Mirrored disks Everything written twice Better performance on reads (slower on writes) X Logical disk X X Mirrored Implementation 7
8 Duplicate data and crash recovery After a crash, data survives X Logical disk X Crash! X Mirrored Implementation 8
9 Mirrored disks Replacement drive can be reconstructed in the background X Logical disk X X Mirrored Implementation 9
10 REVIEW: How is the disk used in Unix / Linux? Application Filesystem Block Device Driver Sector In-memory Block Cache Raw Device Driver Sector Access by cylinder/track/sector Unix Kernel Direct read/write of filesystem blocks (hides sector size and device geometry) Files/Dirs security, etc Buffered block r/w: hides timing
11 We can use mirrored disks with Unix Application Abstraction: The mirrored disk provides the same service as a single disk just faster and more reliable! Filesystem Block Device Driver Sector In-memory Block Cache MIRRORED Device Driver Mirrored Implementation Unix Kernel Files/Dirs security, etc Buffered block r/w: hides timing
12 Atomicity and update synchronziation Question: when is the update committed? X Mirrored writes DO NOT happen at quite the same time Logical disk X X Mirrored Implementation 12
13 RAID Reliable Arrays of Inexpensive Disks X Logical disk X X X RAID Implementation 13
14 RAID Reliable Arrays of Inexpensive Disks X Y Logical disk X Y X XOR(X,Y) RAID Implementation 14
15 RAID Reliable Arrays of Inexpensive Disks X Y Z Logical disk Much less space overhead than mirroring but typically slower X Y X Z XOR(X,Y,Z) RAID Implementation 15
16 RAID Reliable Arrays of Inexpensive Disks X Y Z Logical disk If any disk is lost you can reconstruct from information on the others! X Y Crash! X Z XOR(X,Y,Z) RAID Implementation 16
17 Why Consistency is Hard 17
18 Synchronization problem Some code to add money to my account NA =Access Noah s Bank account Bal = NA.Balance; NewBalance = Bal + $1000 NA.Balance.Write NewBalance Let s run code for two deposits in parallel Some code to add money to my account NA =Access Noah s Bank account Bal = NA.Balance; NewBalance = Bal + $1000 NA.Balance.Write NewBalance Can you see the problem? There s a risk that both copies will pick up X before either updates. If that happens, I only get $1000 not $2000! 18
19 Solution - locking Only one transaction or thread can hold the lock at a time Some code to add money to my account Lock Noah s Bank Account NA =Access Noah s Bank account Bal = NA.Balance; NewBalance = Bal + $1000 NA.Balance.Write NewBalance Unlock Noah s Bank Account Now the two copies can t run at once on the same account but if each locks a different bank account they can. 19
20 Consistency and Crash Recovery Some code to transfer money NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; This gets lost during crash Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Balance.Write Nbal YA.Balance.Write Ybal Can you see the problem? If the system crashes just after writing my balance, the bank loses $1000 (it s still in your account too) 20
21 Transactions 21
22 Transactions: automated consistency & crash recovery! Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Balance.Write Nbal YA.Balance.Write Ybal END_TRANSACTION The system guarantees that either everything in the transaction happens, or nothing and it guarantees more! 22
23 ACID Properties of a Transaction Atomicity Everything happens or nothing Consistency If the database has rules they are obeyed at transaction end (e.g. balance must be < $1,000,000) Isolation Any two parallel transactions act as if serial Most transaction systems do the locking automatically! Durability Once committed, never lost That seems almost magic how can we achieve all this? 23
24 How to implement transactions - logging The key idea: a shared log records information needed to undo any change made by any transaction When a transaction commits: All data is written to the main data store A commit record is written to the log. This is the atomic point at which the transaction happens After a crash, the log is replayed For any transactions that did not commit, the undo operations are performed After the crash, only commited operations have happened! When combined with transaction driven locking, we can automatically support ACID properties with almost no application code complexity This is all built into SQL databases like Oracle, Postgres, DB2, and SQL Server Logging and transaction processing are two of the most important and beautiful data processing technologies 24
25 Logging in Action Noah.Bal = $100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Balance.Write Nbal YA.Balance.Write Ybal END_TRANSACTION 25
26 Logging in Action Noah.Bal = $100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Balance.Write Nbal YA.Balance.Write Ybal END_TRANSACTION Begin Trans 1 Log 26
27 Logging in Action Noah.Bal = $1100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Balance.Write Nbal YA.Balance.Write Ybal END_TRANSACTION Begin Trans 1 Old Noah Bal = $100 Log 27
28 Logging in Action Noah.Bal = $1100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Write Nbal YA.Balance.Write Ybal END_TRANSACTION Begin Trans 1 Old Noah Bal = $100 Old Your Bal = $1300 Log 28
29 Logging in Action Noah.Bal = $1100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Write Nbal YA.Write Ybal END_TRANSACTION Begin Trans 1 Old Noah Bal = $100 Old Your Bal = $1300 Commit Tr 1 Log 29
30 Logging in Action Noah.Bal = $1100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Write Nbal YA.Write Ybal END_TRANSACTION What if we crash while the data is inconsistent? Begin Trans 1 Old Noah Bal = $100 Old Your Bal = $1300 Commit Tr 1 Log 30
31 Logging in Action Noah.Bal = $100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Balance.Write Nbal YA.Balance.Write Ybal END_TRANSACTION 31
32 Logging in Action Noah.Bal = $100 Your.Bal = $1300 Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Write Nbal YA.Write Ybal END_TRANSACTION Begin Trans 1 Log 32
33 Logging in Action Noah.Bal = $1100 Your.Bal = $1300 Crash! Some code to transfer money BEGIN_TRANSACTION NA =Access Noah s Bank account YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Write Nbal YA.Write Ybal END_TRANSACTION Begin Trans 1 Old Noah Bal = $100 Log 33
34 Recovery! When system restarts, data is inconsistent Noah.Bal = $1100 Your.Bal = $1300 but we can play the log to restore consistency! Begin Trans 1 Old Noah Bal = $100 Log 34
35 Recovery! Noah.Bal = $1100 Your.Bal = $1300 We notice that Transaction 1 never committed, so we apply all of its undo entries Begin Trans 1 Old Noah Bal = $100 Log 35
36 Recovery! Noah.Bal = $1100 $100 Your.Bal = $1300 We notice that Transaction 1 never committed, so we apply all of its undo entries Begin Trans 1 Old Noah Bal = $100 Log 36
37 Logging keeping consistency Full Disclosure after crashes This explanation is highly simplified but the spirit is exactly right. The key idea: a shared log records information on how to undo any change to the main data Examples of things not covered: When a transaction commits: All data is written to the main data store Some databases use redo vs. undo logging or log both old and A commit record is written to the log. This is the atomic point at which the transaction new values happens Transactions can abort (a ROLLBACK record is logged instead of COMMIT) For any transactions that did not commit, the undo operations are performed Useful if programmer wants to give up After the crash, only commited operations have happened! The system can abort a transaction if there is an error The system can abort a transaction if locking has caused properties deadlock with almost no application code complexity The same logs, if carefully designed, can be used to help with and backup, SQL Server recovery from disk drive failure, and synchronization of distributed systems. After a crash, the log is replayed When combined with locking, we can automatically support ACID This is all built into SQL databases like Oracle, Postgres, DB2, Logging and transaction processing are two of the most important and beautiful data processing technologies 37
38 Atomicity and hardware Important: transactions are committed by an atomic hardware write to the log Before the commit is written, the transaction has not happened After it s written all of its work is committed It all happens at once: atomically Principle: Almost any computing activity that is to be done atomically must be achieved in a single atomic hardware operation! Store, Test_and_set or compare_and_swap CPU instructions Write a disk block When designing systems that require consistency, start by studying what your hardware can do atomically 38
39 Consistency in Distributed Systems 39
40 Problem In a distributed system, we want to do work in lots of places To get consistency, we need to do an atomic update to the system state Challenge: can we get consistency in a distributed system? 40
41 Can we get distributed consensus and consistency? Yes! (but with some limitations) First we need to think about how distributed systems fail individual nodes can fail what if the network partitions? In general, implementing transactions or other consistency guarantees in distributed systems is hard! 41
42 Network Partition This network is fully connected 42
43 Network Partition All computers are still up! Updates in one partition can t be sent to the other. If these links break the network is partitioned 43
44 Questions about failures in distributed systems Can we support replicated data and maintain consistency? Can we run distributed transactions in which work (updating accounts) is spread through the network and achieve consistency? How can we do crash recovery? How do we continue running when the network partitions? 44
45 Voting: a simple approach to replicated data Copies of the same data can be kept at any or all nodes but when reading you must use the value stored at a majority of nodes! 45
46 Network Partition All computers are still up! Updates in one partition can t be sent to the other. During partition, only one group of nodes can be a majority the other can t proceed! 46
47 The Famous CAP Theorem 47
48 The Cap Theorem When designing a system with distributed data you would like to have: Consistency: everyone agrees on the data Availability: nobody ever has to stop processing Partition tolerance: keep going even when the network partitions The CAP theorem says: you can have any two simultaneously, but not all three! If your network can partition, then either some nodes will have to stop working (no availability) or data may become inconsistent (other partition doesn t see the updates) 48
49 Network Partition With the voting algorithm, only the orange partition can do work. The CAP theorem explains why we can never build a system that does better, unless we are willing to sacrifice consistency. 49
50 Distributed Transactions 50
51 Distributed transactions: the challenge What if our computation is distributed? We still want ACID properties Atomicity Consistency Isolation Durability Per the CAP theorem: let s ignore partition for now Amazingly, there are ways to do this: Isolation and Consistency: distributed lock managers Atomicity and Durability: Distributed Two Phase Commit (DTPC) 51
52 Distributed two phase commit Allows a single transaction to be spread across multiple nodes Logging is done at each node as for traditional transactions Special protocol ensures atomic commit of distributed work One of the great achievements of 20 th century distributed computing research 52
53 Distributed Two Phase Commit Node 1 logic BEGIN_DISTRIBUTED_TRANSACTION NA =Access Noah s Bank account NBal = NA.Balance; Nbal += $1000; NA.Balance.Write Nbal COMMIT Node 2 Logic JOIN_DISTRIBUTED_TRANSACTION YA =Access Your Bank account Ybal = YA.Balance; Ybal -= $1000; YA.Balance.Write Ybal Noah.Bal = $100 Your.Bal = $1300 Begin Trans 1 Node 1 Log Join Trans 1 Node 2 Log 53
54 Distributed Two Phase Commit Node 1 logic BEGIN_DISTRIBUTED_TRANSACTION NA =Access Noah s Bank account NBal = NA.Balance; Nbal += $1000; NA.Balance.Write Nbal COMMIT Node 2 Logic JOIN_DISTRIBUTED_TRANSACTION YA =Access Your Bank account Ybal = YA.Balance; Ybal -= $1000; YA.Balance.Write Ybal Noah.Bal = $1100 Your.Bal = $300 Begin Trans 1 Old Noah Balance = $100 Node 1 Log Join Trans 1 Node 2 Log Old YourBalance = $
55 Distributed Two Phase Commit Node 1 logic BEGIN_DISTRIBUTED_TRANSACTION NA =Access Noah s Bank account NBal = NA.Balance; Nbal += $1000; NA.Balance.Write Nbal COMMIT Node 2 Logic JOIN_DISTRIBUTED_TRANSACTION YA =Access Your Bank account Ybal = YA.Balance; Ybal -= $1000; YA.Balance.Write Ybal Noah.Bal = $1100 Your.Bal = $300 Begin Trans 1 Old Noah Balance = $100 Prepared Node 1 Log Join Trans 1 Old YourBalance = $1300 Prepared Node 2 Log 55
56 Distributed Two Phase Commit Node 1 logic Prepared means: if you ask Node 2 Logic BEGIN_DISTRIBUTED_TRANSACTION me later to commit or abort JOIN_DISTRIBUTED_TRANSACTION NA =Access Noah s Bank account I will be able to do either! YA =Access Your Bank account NBal = NA.Balance; Ybal = YA.Balance; Nbal += $1000; Ybal -= $1000; NA.Balance.Write Nbal YA.Balance.Write Ybal COMMIT Noah.Bal = $1100 Your.Bal = $300 Begin Trans 1 Old Noah Balance = $100 Prepared Node 1 Log Join Trans 1 Old YourBalance = $1300 Prepared Node 2 Log 56
57 Distributed Two Phase Commit Node 1 logic BEGIN_DISTRIBUTED_TRANSACTION NA =Access Noah s Bank account NBal = NA.Balance; Nbal += $1000; NA.Balance.Write Nbal COMMIT Node 2 Logic JOIN_DISTRIBUTED_TRANSACTION YA =Access Your Bank account Ybal = YA.Balance; Ybal -= $1000; YA.Balance.Write Ybal Noah.Bal = $1100 Your.Bal = $300 Begin Trans 1 Old Noah Balance = $100 Prepared Commit Node 1 Log Join Trans 1 Old YourBalance = $1300 Prepared Commit Node 2 Log 57
58 What happens if there is a crash? If a node goes down before the commit, the master node writes an abort record and tells other nodes to abort When any node comes up after a crash or after partition, it checks with master what has happened to any prepared transactions Because prepared means it can go either way, that node can either record a commit or execute a rollback using data from the log We can see the CAP theorem in action again: the algorithm stalls while the network is partitioned 58
59 Does Everyone use Distributed 2 Phase Commit? In the late 1990s everyone thought DTPC would be the key to distributed data In practice, systems like Amazon can t stop in case of network partition or master node crashe Today: Massive but non-critical data stores do not even attempt perfect consistency: once in awhile your Amazon shopping cart may lose things you ve parked there Critical transactions (e.g. when you place your order and charge your credit card) are often recorded in less scalable but fully consistent (usually relational) databases 59
60 Summary 60
61 Summary Keeping data consistent is important Techniques like ACID transactions implemented with logs have been spectacularly successful Consistency and scalability tend not to come together Atomicity in software tends to require reduction to a single atomic operation in hardware The CAP theorem says we can t have Consistency, Availability and Parition tolerance Techniques like Voting and Distributed Two Phase Commit can achieve distributed consistency at the cost of availability Many modern systems sacrifice consistency to achieve availability at massive scale 61
Topics. File Buffer Cache for Performance. What to Cache? COS 318: Operating Systems. File Performance and Reliability
Topics COS 318: Operating Systems File Performance and Reliability File buffer cache Disk failure and recovery tools Consistent updates Transactions and logging 2 File Buffer Cache for Performance What
Outline. Failure Types
Outline Database Tuning Nikolaus Augsten University of Salzburg Department of Computer Science Database Group 1 Unit 10 WS 2013/2014 Adapted from Database Tuning by Dennis Shasha and Philippe Bonnet. Nikolaus
Overview of Transaction Management
Overview of Transaction Management Chapter 16 Comp 521 Files and Databases Fall 2010 1 Database Transactions A transaction is the DBMS s abstract view of a user program: a sequence of database commands;
Database Management System
Database Management System Lecture 10 Recovery * Some materials adapted from R. Ramakrishnan, J. Gehrke and Shawn Bowers Basic Database Architecture Database Management System 2 Recovery Which ACID properties
Database Management System Prof. D. Janakiram Department of Computer Science & Engineering Indian Institute of Technology, Madras Lecture No.
Database Management System Prof. D. Janakiram Department of Computer Science & Engineering Indian Institute of Technology, Madras Lecture No. # 18 Transaction Processing and Database Manager In the previous
Transactions. Kathleen Durant PhD Northeastern University CS3200 Lesson 9
Transactions Kathleen Durant PhD Northeastern University CS3200 Lesson 9 1 Outline for the day The definition of a transaction Benefits provided What they look like in SQL Scheduling Transactions Serializability
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons)
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons) Transactions - Definition A transaction is a sequence of data operations with the following properties: * A Atomic All
CHAPTER 3 RECOVERY & CONCURRENCY ADVANCED DATABASE SYSTEMS. Assist. Prof. Dr. Volkan TUNALI
CHAPTER 3 RECOVERY & CONCURRENCY ADVANCED DATABASE SYSTEMS Assist. Prof. Dr. Volkan TUNALI PART 1 2 RECOVERY Topics 3 Introduction Transactions Transaction Log System Recovery Media Recovery Introduction
Introduction. Storage Failure Recovery Logging Undo Logging Redo Logging ARIES
Introduction Storage Failure Recovery Logging Undo Logging Redo Logging ARIES Volatile storage Main memory Cache memory Nonvolatile storage Stable storage Online (e.g. hard disk, solid state disk) Transaction
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons)
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons) Goal A Distributed Transaction We want a transaction that involves multiple nodes Review of transactions and their properties
Consistency. CS 475, Spring 2018 Concurrent & Distributed Systems
Consistency CS 475, Spring 2018 Concurrent & Distributed Systems Review: 2PC, Timeouts when Coordinator crashes What if the bank doesn t hear back from coordinator? If bank voted no, it s OK to abort If
DHANALAKSHMI COLLEGE OF ENGINEERING, CHENNAI
DHANALAKSHMI COLLEGE OF ENGINEERING, CHENNAI Department of Computer Science and Engineering CS6302- DATABASE MANAGEMENT SYSTEMS Anna University 2 & 16 Mark Questions & Answers Year / Semester: II / III
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons)
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons) Goal A Distributed Transaction We want a transaction that involves multiple nodes Review of transactions and their properties
CS October 2017
Atomic Transactions Transaction An operation composed of a number of discrete steps. Distributed Systems 11. Distributed Commit Protocols All the steps must be completed for the transaction to be committed.
Performance and Forgiveness. June 23, 2008 Margo Seltzer Harvard University School of Engineering and Applied Sciences
Performance and Forgiveness June 23, 2008 Margo Seltzer Harvard University School of Engineering and Applied Sciences Margo Seltzer Architect Outline A consistency primer Techniques and costs of consistency
The transaction. Defining properties of transactions. Failures in complex systems propagate. Concurrency Control, Locking, and Recovery
Failures in complex systems propagate Concurrency Control, Locking, and Recovery COS 418: Distributed Systems Lecture 17 Say one bit in a DRAM fails: flips a bit in a kernel memory write causes a kernel
In This Lecture. Transactions and Recovery. Transactions. Transactions. Isolation and Durability. Atomicity and Consistency. Transactions Recovery
In This Lecture Database Systems Lecture 15 Natasha Alechina Transactions Recovery System and Media s Concurrency Concurrency problems For more information Connolly and Begg chapter 20 Ullmanand Widom8.6
Synchronization Part 2. REK s adaptation of Claypool s adaptation oftanenbaum s Distributed Systems Chapter 5 and Silberschatz Chapter 17
Synchronization Part 2 REK s adaptation of Claypool s adaptation oftanenbaum s Distributed Systems Chapter 5 and Silberschatz Chapter 17 1 Outline Part 2! Clock Synchronization! Clock Synchronization Algorithms!
XI. Transactions CS Computer App in Business: Databases. Lecture Topics
XI. Lecture Topics Properties of Failures and Concurrency in SQL Implementation of Degrees of Isolation CS338 1 Problems Caused by Failures Accounts(, CId, BranchId, Balance) update Accounts set Balance
T ransaction Management 4/23/2018 1
T ransaction Management 4/23/2018 1 Air-line Reservation 10 available seats vs 15 travel agents. How do you design a robust and fair reservation system? Do not enough resources Fair policy to every body
Distributed systems. Lecture 6: distributed transactions, elections, consensus and replication. Malte Schwarzkopf
Distributed systems Lecture 6: distributed transactions, elections, consensus and replication Malte Schwarzkopf Last time Saw how we can build ordered multicast Messages between processes in a group Need
Final Exam Solutions
Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.824 Spring 2004 Final Exam Solutions The average score was 84, the standard deviation was 6. 1 I Short
CSE 444: Database Internals. Section 9: 2-Phase Commit and Replication
CSE 444: Database Internals Section 9: 2-Phase Commit and Replication 1 Today 2-Phase Commit Replication 2 Two-Phase Commit Protocol (2PC) One coordinator and many subordinates Phase 1: Prepare Phase 2:
Transaction Management. Pearson Education Limited 1995, 2005
Chapter 20 Transaction Management 1 Chapter 20 - Objectives Function and importance of transactions. Properties of transactions. Concurrency Control Deadlock and how it can be resolved. Granularity of
Databases: transaction processing
Databases: transaction processing P.A.Rounce Room 6.18 p.rounce@cs.ucl.ac.uk 1 ACID Database operation is processing of a set of transactions Required features of a database transaction should be Atomicity
CS /15/16. Paul Krzyzanowski 1. Question 1. Distributed Systems 2016 Exam 2 Review. Question 3. Question 2. Question 5.
Question 1 What makes a message unstable? How does an unstable message become stable? Distributed Systems 2016 Exam 2 Review Paul Krzyzanowski Rutgers University Fall 2016 In virtual sychrony, a message
CS122 Lecture 15 Winter Term,
CS122 Lecture 15 Winter Term, 2017-2018 2 Transaction Processing Last time, introduced transaction processing ACID properties: Atomicity, consistency, isolation, durability Began talking about implementing
Transactions. CS 475, Spring 2018 Concurrent & Distributed Systems
Transactions CS 475, Spring 2018 Concurrent & Distributed Systems Review: Transactions boolean transfermoney(person from, Person to, float amount){ if(from.balance >= amount) { from.balance = from.balance
Transactions and ACID
Transactions and ACID Kevin Swingler Contents Recap of ACID transactions in RDBMSs Transactions and ACID in MongoDB 1 Concurrency Databases are almost always accessed by multiple users concurrently A user
DATABASE TRANSACTIONS. CS121: Relational Databases Fall 2017 Lecture 25
DATABASE TRANSACTIONS CS121: Relational Databases Fall 2017 Lecture 25 Database Transactions 2 Many situations where a sequence of database operations must be treated as a single unit A combination of
TRANSACTION PROCESSING MONITOR OVERVIEW OF TPM FOR DISTRIBUTED TRANSACTION PROCESSING
TPM Transaction Processing TPM Monitor TRANSACTION PROCESSING MONITOR OVERVIEW OF TPM FOR DISTRIBUTED TRANSACTION PROCESSING Peter R. Egli 1/9 Contents 1. What are Transaction Processing Monitors?. Properties
CMU SCS CMU SCS Who: What: When: Where: Why: CMU SCS
Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB s C. Faloutsos A. Pavlo Lecture#23: Distributed Database Systems (R&G ch. 22) Administrivia Final Exam Who: You What: R&G Chapters 15-22
Ext3/4 file systems. Don Porter CSE 506
Ext3/4 file systems Don Porter CSE 506 Logical Diagram Binary Formats Memory Allocators System Calls Threads User Today s Lecture Kernel RCU File System Networking Sync Memory Management Device Drivers
Database Technology. Topic 8: Introduction to Transaction Processing
Topic 8: Introduction to Transaction Processing Olaf Hartig olaf.hartig@liu.se Motivation A DB is a shared resource accessed by many users and processes concurrently Not managing concurrent access to a
SCALABLE CONSISTENCY AND TRANSACTION MODELS
Data Management in the Cloud SCALABLE CONSISTENCY AND TRANSACTION MODELS 69 Brewer s Conjecture Three properties that are desirable and expected from realworld shared-data systems C: data consistency A:
CSE 344 MARCH 5 TH TRANSACTIONS
CSE 344 MARCH 5 TH TRANSACTIONS ADMINISTRIVIA OQ6 Out 6 questions Due next Wednesday, 11:00pm HW7 Shortened Parts 1 and 2 -- other material candidates for short answer, go over in section Course evaluations
PRIMARY-BACKUP REPLICATION
PRIMARY-BACKUP REPLICATION Primary Backup George Porter Nov 14, 2018 ATTRIBUTION These slides are released under an Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) Creative Commons
Replication. Feb 10, 2016 CPSC 416
Replication Feb 10, 2016 CPSC 416 How d we get here? Failures & single systems; fault tolerance techniques added redundancy (ECC memory, RAID, etc.) Conceptually, ECC & RAID both put a master in front
CSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 14 Distributed Transactions
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 14 Distributed Transactions Transactions Main issues: Concurrency control Recovery from failures 2 Distributed Transactions
Outline. Purpose of this paper. Purpose of this paper. Transaction Review. Outline. Aries: A Transaction Recovery Method
Outline Aries: A Transaction Recovery Method Presented by Haoran Song Discussion by Hoyt Purpose of this paper Computer system is crashed as easily as other devices. Disk burned Software Errors Fires or
CSC 261/461 Database Systems Lecture 20. Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101
CSC 261/461 Database Systems Lecture 20 Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101 Announcements Project 1 Milestone 3: Due tonight Project 2 Part 2 (Optional): Due on: 04/08 Project 3
Data Modeling and Databases Ch 14: Data Replication. Gustavo Alonso, Ce Zhang Systems Group Department of Computer Science ETH Zürich
Data Modeling and Databases Ch 14: Data Replication Gustavo Alonso, Ce Zhang Systems Group Department of Computer Science ETH Zürich Database Replication What is database replication The advantages of
Chapter 14: Recovery System
Chapter 14: Recovery System Chapter 14: Recovery System Failure Classification Storage Structure Recovery and Atomicity Log-Based Recovery Remote Backup Systems Failure Classification Transaction failure
Lecture X: Transactions
Lecture X: Transactions CMPT 401 Summer 2007 Dr. Alexandra Fedorova Transactions A transaction is a collection of actions logically belonging together To the outside world, a transaction must appear as
Lectures 8 & 9. Lectures 7 & 8: Transactions
Lectures 8 & 9 Lectures 7 & 8: Transactions Lectures 7 & 8 Goals for this pair of lectures Transactions are a programming abstraction that enables the DBMS to handle recoveryand concurrency for users.
Weak Levels of Consistency
Weak Levels of Consistency - Some applications are willing to live with weak levels of consistency, allowing schedules that are not serialisable E.g. a read-only transaction that wants to get an approximate
Introduction to Data Management CSE 344
Introduction to Data Management CSE 344 Unit 7: Transactions Schedules Implementation Two-phase Locking (3 lectures) 1 Class Overview Unit 1: Intro Unit 2: Relational Data Models and Query Languages Unit
NOTES W2006 CPS610 DBMS II. Prof. Anastase Mastoras. Ryerson University
NOTES W2006 CPS610 DBMS II Prof. Anastase Mastoras Ryerson University Recovery Transaction: - a logical unit of work. (text). It is a collection of operations that performs a single logical function in
CS5460: Operating Systems Lecture 20: File System Reliability
CS5460: Operating Systems Lecture 20: File System Reliability File System Optimizations Modern Historic Technique Disk buffer cache Aggregated disk I/O Prefetching Disk head scheduling Disk interleaving
Transaction Management & Concurrency Control. CS 377: Database Systems
Transaction Management & Concurrency Control CS 377: Database Systems Review: Database Properties Scalability Concurrency Data storage, indexing & query optimization Today & next class Persistency Security
It also performs many parallelization operations like, data loading and query processing.
Introduction to Parallel Databases Companies need to handle huge amount of data with high data transfer rate. The client server and centralized system is not much efficient. The need to improve the efficiency
Database Management Systems
Database Management Systems Distributed Databases Doug Shook What does it mean to be distributed? Multiple nodes connected by a network Data on the nodes is logically related The nodes do not need to be
ATOMIC COMMITMENT Or: How to Implement Distributed Transactions in Sharded Databases
ATOMIC COMMITMENT Or: How to Implement Distributed Transactions in Sharded Databases We talked about transactions and how to implement them in a single-node database. We ll now start looking into how to
ARIES (& Logging) April 2-4, 2018
ARIES (& Logging) April 2-4, 2018 1 What does it mean for a transaction to be committed? 2 If commit returns successfully, the transaction is recorded completely (atomicity) left the database in a stable
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons)
) Intel)(TX)memory):) Transac'onal) Synchroniza'on) Extensions)(TSX))) Transac'ons) Transactions - Definition A transaction is a sequence of data operations with the following properties: * A Atomic All
Causal Consistency and Two-Phase Commit
Causal Consistency and Two-Phase Commit CS 240: Computing Systems and Concurrency Lecture 16 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Consistency
Example: Transfer Euro 50 from A to B
TRANSACTIONS Example: Transfer Euro 50 from A to B 1. Read balance of A from DB into Variable a: read(a,a); 2. Subtract 50.- Euro from the balance: a:= a 50; 3. Write new balance back into DB: write(a,a);
COURSE 1. Database Management Systems
COURSE 1 Database Management Systems Assessment / Other Details Final grade 50% - laboratory activity / practical test 50% - written exam Course details (bibliography, course slides, seminars, lab descriptions
Transactions. A Banking Example
Transactions A transaction is specified by a client as a sequence of operations on objects to be performed as an indivisible unit by the servers managing those objects Goal is to ensure that all objects
Synchronization. Chapter 5
Synchronization Chapter 5 Clock Synchronization In a centralized system time is unambiguous. (each computer has its own clock) In a distributed system achieving agreement on time is not trivial. (it is
Database Systems. Announcement
Database Systems ( 料 ) December 27/28, 2006 Lecture 13 Merry Christmas & New Year 1 Announcement Assignment #5 is finally out on the course homepage. It is due next Thur. 2 1 Overview of Transaction Management
Database Architectures
Database Architectures CPS352: Database Systems Simon Miner Gordon College Last Revised: 4/15/15 Agenda Check-in Parallelism and Distributed Databases Technology Research Project Introduction to NoSQL
Lecture 18: Reliable Storage
CS 422/522 Design & Implementation of Operating Systems Lecture 18: Reliable Storage Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of
Distributed System. Gang Wu. Spring,2018
Distributed System Gang Wu Spring,2018 Lecture4:Failure& Fault-tolerant Failure is the defining difference between distributed and local programming, so you have to design distributed systems with the
Defining properties of transactions
Transactions: ACID, Concurrency control (2P, OCC) Intro to distributed txns The transaction Definition: A unit of work: May consist of multiple data accesses or updates Must commit or abort as a single
6.830 Lecture Recovery 10/30/2017
6.830 Lecture 14 -- Recovery 10/30/2017 Have been talking about transactions Transactions -- what do they do? Awesomely powerful abstraction -- programmer can run arbitrary mixture of commands that read
File Systems: Consistency Issues
File Systems: Consistency Issues File systems maintain many data structures Free list/bit vector Directories File headers and inode structures res Data blocks File Systems: Consistency Issues All data
Distributed Systems
15-440 Distributed Systems 11 - Fault Tolerance, Logging and Recovery Tuesday, Oct 2 nd, 2018 Logistics Updates P1 Part A checkpoint Part A due: Saturday 10/6 (6-week drop deadline 10/8) *Please WORK hard
Administration Naive DBMS CMPT 454 Topics. John Edgar 2
Administration Naive DBMS CMPT 454 Topics John Edgar 2 http://www.cs.sfu.ca/coursecentral/454/johnwill/ John Edgar 4 Assignments 25% Midterm exam in class 20% Final exam 55% John Edgar 5 A database stores
Fault tolerance with transactions: past, present and future. Dr Mark Little Technical Development Manager, Red Hat
Fault tolerance with transactions: past, present and future Dr Mark Little Technical Development Manager, Overview Fault tolerance Transaction fundamentals What is a transaction? ACID properties Distributed
Recoverability. Kathleen Durant PhD CS3200
Recoverability Kathleen Durant PhD CS3200 1 Recovery Manager Recovery manager ensures the ACID principles of atomicity and durability Atomicity: either all actions in a transaction are done or none are
Distributed Systems COMP 212. Revision 2 Othon Michail
Distributed Systems COMP 212 Revision 2 Othon Michail Synchronisation 2/55 How would Lamport s algorithm synchronise the clocks in the following scenario? 3/55 How would Lamport s algorithm synchronise
Distributed Systems. Day 13: Distributed Transaction. To Be or Not to Be Distributed.. Transactions
Distributed Systems Day 13: Distributed Transaction To Be or Not to Be Distributed.. Transactions Summary Background on Transactions ACID Semantics Distribute Transactions Terminology: Transaction manager,,
Actions are never left partially executed. Actions leave the DB in a consistent state. Actions are not affected by other concurrent actions
Transaction Management Recovery (1) Review: ACID Properties Atomicity Actions are never left partially executed Consistency Actions leave the DB in a consistent state Isolation Actions are not affected
Intro to Transactions
Reading Material CompSci 516 Database Systems Lecture 14 Intro to Transactions [RG] Chapter 16.1-16.3, 16.4.1 17.1-17.4 17.5.1, 17.5.3 Instructor: Sudeepa Roy Acknowledgement: The following slides have
Transactions. 1. Transactions. Goals for this lecture. Today s Lecture
Goals for this lecture Transactions Transactions are a programming abstraction that enables the DBMS to handle recovery and concurrency for users. Application: Transactions are critical for users Even
6.830 Lecture Recovery 10/30/2017
6.830 Lecture 14 -- Recovery 10/30/2017 Have been talking about transactions Transactions -- what do they do? Awesomely powerful abstraction -- programmer can run arbitrary mixture of commands that read
CSE 530A ACID. Washington University Fall 2013
CSE 530A ACID Washington University Fall 2013 Concurrency Enterprise-scale DBMSs are designed to host multiple databases and handle multiple concurrent connections Transactions are designed to enable Data
TWO-PHASE COMMIT ATTRIBUTION 5/11/2018. George Porter May 9 and 11, 2018
TWO-PHASE COMMIT George Porter May 9 and 11, 2018 ATTRIBUTION These slides are released under an Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) Creative Commons license These slides
Chapter 22. Transaction Management
Chapter 22 Transaction Management 1 Transaction Support Transaction Action, or series of actions, carried out by user or application, which reads or updates contents of database. Logical unit of work on
CSE380 - Operating Systems. Communicating with Devices
CSE380 - Operating Systems Notes for Lecture 15-11/4/04 Matt Blaze (some examples by Insup Lee) Communicating with Devices Modern architectures support convenient communication with devices memory mapped
CONCURRENCY CONTROL, TRANSACTIONS, LOCKING, AND RECOVERY
CONCURRENCY CONTROL, TRANSACTIONS, LOCKING, AND RECOVERY George Porter May 18, 2018 ATTRIBUTION These slides are released under an Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) Creative
Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitionin
Parallel Data Types of Parallelism Replication (Multiple copies of the same data) Better throughput for read-only computations Data safety Partitioning (Different data at different sites More space Better
RECOVERY CHAPTER 21,23 (6/E) CHAPTER 17,19 (5/E)
RECOVERY CHAPTER 21,23 (6/E) CHAPTER 17,19 (5/E) 2 LECTURE OUTLINE Failures Recoverable schedules Transaction logs Recovery procedure 3 PURPOSE OF DATABASE RECOVERY To bring the database into the most
Last time. Distributed systems Lecture 6: Elections, distributed transactions, and replication. DrRobert N. M. Watson
Distributed systems Lecture 6: Elections, distributed transactions, and replication DrRobert N. M. Watson 1 Last time Saw how we can build ordered multicast Messages between processes in a group Need to
TRANSACTION PROCESSING PROPERTIES OF A TRANSACTION TRANSACTION PROCESSING PROPERTIES OF A TRANSACTION 4/3/2014
TRANSACTION PROCESSING SYSTEMS IMPLEMENTATION TECHNIQUES TRANSACTION PROCESSING DATABASE RECOVERY DATABASE SECURITY CONCURRENCY CONTROL Def: A Transaction is a program unit ( deletion, creation, updating
TRANSACTION PROPERTIES
Transaction Is any action that reads from and/or writes to a database. A transaction may consist of a simple SELECT statement to generate a list of table contents; it may consist of series of INSERT statements
GFS: The Google File System
GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one
Parallel DBs. April 25, 2017
Parallel DBs April 25, 2017 1 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 Node 2 2 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 with
GFS: The Google File System. Dr. Yingwu Zhu
GFS: The Google File System Dr. Yingwu Zhu Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one big CPU More storage, CPU required than one PC can
ò Very reliable, best-of-breed traditional file system design ò Much like the JOS file system you are building now
Ext2 review Very reliable, best-of-breed traditional file system design Ext3/4 file systems Don Porter CSE 506 Much like the JOS file system you are building now Fixed location super blocks A few direct
Database Recovery. Dr. Bassam Hammo
Database Recovery Dr. Bassam Hammo 1 Transaction Concept A transaction is a unit of execution Either committed or aborted. After a transaction, the db must be consistent. Consistent No violation of any
CMPT 354: Database System I. Lecture 11. Transaction Management
CMPT 354: Database System I Lecture 11. Transaction Management 1 Why this lecture DB application developer What if crash occurs, power goes out, etc? Single user à Multiple users 2 Outline Transaction
Introduction to Databases, Fall 2005 IT University of Copenhagen. Lecture 10: Transaction processing. November 14, Lecturer: Rasmus Pagh
Introduction to Databases, Fall 2005 IT University of Copenhagen Lecture 10: Transaction processing November 14, 2005 Lecturer: Rasmus Pagh Today s lecture Part I: Transaction processing Serializability
Transactions. ACID Properties of Transactions. Atomicity - all or nothing property - Fully performed or not at all
Transactions - An action, or series of actions, carried out by a single user or application program, which reads or updates the contents of the database - Logical unit of work on the database - Usually
User Perspective. Module III: System Perspective. Module III: Topics Covered. Module III Overview of Storage Structures, QP, and TM
Module III Overview of Storage Structures, QP, and TM Sharma Chakravarthy UT Arlington sharma@cse.uta.edu http://www2.uta.edu/sharma base Management Systems: Sharma Chakravarthy Module I Requirements analysis
Introduction to Data Management. Lecture #18 (Transactions)
Introduction to Data Management Lecture #18 (Transactions) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Announcements v Project info: Part
CSC 261/461 Database Systems Lecture 21 and 22. Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101
CSC 261/461 Database Systems Lecture 21 and 22 Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101 Announcements Project 3 (MongoDB): Due on: 04/12 Work on Term Project and Project 1 The last (mini)
Database Technology. Topic 11: Database Recovery
Topic 11: Database Recovery Olaf Hartig olaf.hartig@liu.se Types of Failures Database may become unavailable for use due to: Transaction failures e.g., incorrect input, deadlock, incorrect synchronization
Transaction Management: Concurrency Control, part 2
Transaction Management: Concurrency Control, part 2 CS634 Class 16 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Locking for B+ Trees Naïve solution Ignore tree structure,