MDCC MULTI DATA CENTER CONSISTENCY. amplab. Tim Kraska, Gene Pang, Michael Franklin, Samuel Madden, Alan Fekete

Transcription

1 MDCC MULTI DATA CENTER CONSISTENCY Tim Kraska, Gene Pang, Michael Franklin, Samuel Madden, Alan Fekete amplab

2 MOTIVATION 2

3 3

4 June 2, 200: Rackspace power outage of approximately 0 minutes {FAILURE}

5 June 2, 200: Rackspace power outage of approximately 0 minutes {FAILURE} April 21, 2011: AWS East outage of over 2 hours, with data loss {FAILURE} 5

6 June 2, 200: Rackspace power outage of approximately 0 minutes UNRELIABLE DATA CENTERS {FAILURE} April 21, 2011: AWS East outage of over 2 hours, with data loss {FAILURE} 6

7 SOLUTION: GEO-REPLICATION 7

8 HIGH NETWORK LATENCY 8

9 THE ENVIRONMENT UNRELIABLE DATA CENTERS + HIGH NETWORK LATENCY

10 OUR CHALLENGE FAST AND RELIABLE DATABASE UNRELIABLE DATA CENTERS + HIGH NETWORK LATENCY 10

11 MDCC MULTI DATA CENTER CONSISTENCY new distributed commit protocol STRONG CONSISTENCY ACROSS DATA CENTERS LOW LATENCY NO DATA LOSS 11

12 EXISTING SOLUTIONS Consistency Transactional Unit Commit Latency Data Loss Possible? Amazon Dynamo Eventual None 1 round trip Not possible Yahoo PNUTS Timeline per key Single key 1 round trip Possible COPS Causality Multi-record 1 round trip Possible MySQL (async) Serializable Static partition 1 round trip Possible Google Megastore Serializable Static partition 2 round trips Not possible Google Spanner Snapshot Isolation Partition 2 round trips Not possible Walter Parallel Snapshot Isolation Multi-record 2 round trips Not possible 12

13 EXISTING SOLUTIONS + MDCC Consistency Transactional Unit Commit Latency Data Loss Possible? Amazon Dynamo Eventual None 1 round trip Not possible Yahoo PNUTS Timeline per key Single key 1 round trip Possible COPS Causality Multi-record 1 round trip Possible MySQL (async) Serializable Static partition 1 round trip Possible Google Megastore Serializable Static partition 2 round trips Not possible Google Spanner Snapshot Isolation Partition 2 round trips Not possible Walter MDCC Parallel Snapshot Isolation Read-committed without lost updates Multi-record 2 round trips Not possible Multi-record 1* round trip Not possible 13

14 WHY MDCC WORKS OPTIMIZES FOR TWO KEY OBSERVATIONS ON WORKLOADS 1

15 WHY MDCC WORKS CONFLICTS ARE RARE everyone updates their own data 15

16 WHY MDCC WORKS CONFLICTS ARE RARE everyone updates their own data CONFLICTS COMMUTE order of concurrent updates isn t important 16

17 MDCC OUTLINE TWO-PHASE COMMIT: BACKGROUND MDCC DISTRIBUTED TRANSACTIONS OPTIMIZATION: CONFLICTS ARE RARE OPTIMIZATION: CONFLICTS COMMUTE EVALUATION 17

18 TWO-PHASE COMMIT: BACKGROUND 18

19 TWO-PHASE COMMIT (2PC) STANDARD METHOD FOR DISTRIBUTED TRANSACTIONS PREPARE PHASE + COMMIT PHASE SYSTEMS TYPICALLY USE 2PC BETWEEN PARTITIONS 1

20 AN EXAMPLE TRANSACTION buy 1 book-a ( in stock) buy 3 book-b ( in stock) 20

21 TWO-PHASE COMMIT book-a coordinator book-b 21

22 TWO-PHASE COMMIT book-a coordinator book-b coordinator initiates transaction commit by sending PREPARE to all s 22

23 TWO-PHASE COMMIT book-a coordinator book-b s vote yes or no for the commit 23

24 TWO-PHASE COMMIT book-a coordinator book-b coordinator makes final decision (commit/abort) can commit only when ALL votes are yes 2

25 TWO-PHASE COMMIT book-a coordinator all s must respond for commit book-b coordinator makes final decision (commit/abort) can commit only when ALL votes are yes 25

26 TWO-PHASE COMMIT book-a coordinator final decision by coordinator book-b coordinator makes final decision (commit/abort) can commit only when ALL votes are yes 26

27 TWO-PHASE COMMIT transaction committed book-a coordinator book-b coordinator stores final decision 27

28 TWO-PHASE COMMIT transaction committed book-a coordinator decision stored only coordinator book-b coordinator stores final decision 28

29 TWO-PHASE COMMIT transaction committed book-a coordinator coordinator failure could block protocol book-b coordinator stores final decision 2

30 TWO-PHASE COMMIT book-a coordinator 6 6 book-b coordinator informs s with COMMIT phase 6 30

31 MDCC DISTRIBUTED TRANSACTIONS 31

32 MDCC TRANSACTIONS book-a coordinator book-b 32

33 MDCC TRANSACTIONS leader-a book-a coordinator leader-b book-b separate Paxos instance per record 33

34 MDCC TRANSACTIONS leader-a book-a coordinator leader-b book-b each executes Paxos replicated state machine 3

35 MDCC TRANSACTIONS 3 leader-a book-a coordinator 6 leader-b book-b coordinator proposes options (potential updates) 35

36 MDCC TRANSACTIONS coordinator leader-a leader-b book-a book-b s validate the options 36

37 coordinator leader-a leader-b MDCC TRANSACTIONS book-a book-b check with all permutations of pending options

38 coordinator leader-a leader-b MDCC TRANSACTIONS book-a book-b check: write-write conflicts, integrity constraints

39 coordinator leader-a leader-b MDCC TRANSACTIONS book-a book-b s tag options as accepted or rejected

40 MDCC TRANSACTIONS 0 coordinator book-a book-b leader-a leader-b Paxos learners learn the result

41 MDCC TRANSACTIONS transaction committed coordinator leader-a leader-b book-a book-b commits when all options are learned

42 MDCC TRANSACTIONS 2 coordinator book-a book-b leader-a leader-b coordinator cannot change transaction outcome transaction committed

43 transaction committed coordinator leader-a leader-b MDCC TRANSACTIONS book-a book-b once learned, options are never unlearned

44 transaction committed coordinator leader-a leader-b MDCC TRANSACTIONS book-a book-b notify s to execute option, to make visible

45 coordinator leader-a leader-b MDCC TRANSACTIONS book-a book-b option visibility can bypass the coordinator

46 MDCC TRANSACTIONS 3 3 leader-a 3 3 book-a 3 3 coordinator leader-b book-b options enable read-committed isolation

47 MDCC VS. TWO-PHASE COMMIT MDCC makes distributed commit decision stores distributed commit decision can tolerate failures/delays TWO-PHASE COMMIT makes commit decision on coordinator stores commit decision on coordinator failures/delays can block protocol 7

48 OPTIMIZATION: CONFLICTS ARE RARE 8

49 MDCC: FAST PAXOS FIRST TO USE FAST PAXOS FOR TRANSACTIONS POSSIBLE TO BYPASS THE LEADER CAN REDUCE NUMBER OF MESSAGE ROUND-TRIPS, WHEN CONFLICTS ARE RARE

50 FAST PAXOS CLASSIC PAXOS clients propose through leader FAST PAXOS clients propose directly to s leader leader 50

51 THE FAST IN FAST PAXOS BYPASS THE LEADER CONSENSUS IN 1 ROUND-TRIP 51

52 THE SLOW IN FAST PAXOS CONCURRENT UPDATES CAN PREVENT CONSENSUS LEADER MUST STEP IN TO RESOLVE CONFLICTS 2 ADDITIONAL MESSAGE ROUNDS TO RESOLVE CONSENSUS MAY TAKE 3 ROUND-TRIPS 52

53 LARGER QUORUMS IN FAST PAXOS CLASSIC PAXOS N/2 responses needed FAST PAXOS 2N/3 + 1 responses needed leader leader 53

54 MDCC: FAST PAXOS IMPLICATIONS TRANSACTIONS CAN BYPASS THE LEADERS TRANSACTIONS CAN COMMIT IN 1 ROUND-TRIP, WHEN CONFLICTS ARE RARE 5

55 OPTIMIZATION: CONFLICTS COMMUTE 55

56 COMMUTATIVE UPDATES commutative updates do not need to be totally ordered 56

57 COMMUTATIVE UPDATES commutative updates do not need to be totally ordered

58 COMMUTATIVE UPDATES commutative updates do not need to be totally ordered =

59 COMMUTATIVE UPDATES commutative updates do not need to be totally ordered = =

60 MDCC: GENERALIZED PAXOS GENERALIZATION OF FAST PAXOS INSTANCES AGREE ON COMMAND STRUCTURES (C-STRUCTS) C-STRUCTS ARE SEQUENCES OF COMMANDS, NOT JUST A SINGLE COMMAND 60

61 MDCC: COMMUTATIVE UPDATES perfect for c-structs of Generalized Paxos can use Fast Paxos (fast commits) without worrying about the order! however, order matters with integrity constraints 61

62 EXAMPLE OF COMMUTATIVE UPDATES online bookstore: books in stock integrity constraint: stock 0 5 replicas Fast Paxos quorum size of 62

63 COMMUTATIVE UPDATES 63

64 COMMUTATIVE UPDATES 6

65 COMMUTATIVE UPDATES 1 book sold! 65

66 COMMUTATIVE UPDATES 1 book sold! 66

67 COMMUTATIVE UPDATES 2 books sold! 67

68 COMMUTATIVE UPDATES 2 books sold! 68

69 COMMUTATIVE UPDATES 3 books sold! 6

70 COMMUTATIVE UPDATES 3 books sold! 70

71 COMMUTATIVE UPDATES books sold! 71

72 COMMUTATIVE UPDATES books sold! 72

73 COMMUTATIVE UPDATES 5 books sold! 73

74 COMMUTATIVE UPDATES 5 books sold! VIOLATED INTEGRITY CONSTRAINTS! 7

75 SOLUTION: QUORUM DEMARCATION adjust the constraint (stock 0) to keep replicas consistent without coordination commits require quorums, so coordination can piggyback on quorum messages quorum demarcation: stock N Q N value 75

76 COMMUTATIVE UPDATES stock N Q N value =

77 COMMUTATIVE UPDATES 3 books sold! stock N Q N value =

78 COMMUTATIVE UPDATES 3 books sold! N Q stock value = 0.8 N 78

79 MDCC: COMMUTATIVE UPDATES FAST COMMITS EVEN WHEN CONFLICTS OCCUR, WHEN ORDER IS NOT IMPORTANT MAINTAIN CONSISTENCY AND FAST COMMITS, WITH QUORUM DEMARCATION 7

80 EVALUATION 80

81 MDCC EXPERIMENTAL SETUP implemented on SCADS, a distributed key-value store fully replicated to 5 Amazon EC2 regions (California, Virginia, Ireland, Singapore, Tokyo) TPC-W benchmark for general performance micro benchmark for protocol investigation 81

82 TPC-W BENCHMARK 82

83 WRITE RESPONSE TIME CDF Percentage of Transactions QW-3 QW- MDCC 2PC Megastore* Write Transaction Response Times, log-scale (ms) 83

84 MICRO BENCHMARK 8

85 VARIOUS MDCC CONFIGURATIONS MDCC Fast Multi Supports Multi-Paxos Supports Fast Paxos Optimizes Commutative Updates 85

86 WRITE RESPONSE TIME CDF Percentage of Transactions Write Transaction Response Times (ms) MDCC Fast Multi 2PC 86

87 VARYING CONFLICT RATE Commits/Aborts (in thousands) PC Multi Fast MDCC 2PC Multi Fast MDCC 2PC Multi Fast MDCC 2PC Multi Fast MDCC Commits Aborts 2PC Multi Fast MDCC 2PC Multi Fast MDCC 0% 50% 20% 10% Hotspot Size 5% 2% 87

88 CONCLUSION 88

89 MDCC - NEW COMMIT PROTOCOL CONFLICTS ARE RARE -OR- CONFLICTS COMMUTE STRONG CONSISTENCY ACROSS DATA CENTERS 1* ROUND-TRIP DURABLE COMMITS COMPARABLE PERFORMANCE TO EVENTUALLY CONSISTENT PROTOCOLS amplab THANK YOU! GENE PANG - GPANG@CS.BERKELEY.EDU 8

90 0