Database Replication in Tashkent. CSEP 545 Transaction Processing Sameh Elnikety

Transcription

1 Database Replication in Tashkent CSEP 545 Transaction Processing Sameh Elnikety

2 Replication for Performance Expensive Limited scalability

3 DB Replication is Challenging Single database system Large, persistent state Transactions Complex software Replication challenges Maintain consistency Middleware replication 3

4 Background Standalone Replica 1 DBMS 4

5 Background Replica 1 Load Balancer Replica Replica 3 5

6 Read Tx Replica 1 Read tx does not change DB state T Load Balancer Replica Replica 3 6

7 Update Tx 1/ Replica 1 Update tx changes DB state T Load Balancer Replica ws Replica 3 7

8 Update Tx 1/ Apply (or commit) T everywhere Replica 1 ws Update tx changes DB state T Load Balancer Replica ws ws Example: T1: { set x = 1 } Replica 3 ws 8

9 Update Tx / Replica 1 Update tx changes DB state T T Load Balancer Replica ws Ordering Replica 3 ws 9

10 Update Tx / Commit updates in order Replica 1 ws ws Update tx changes DB state Load Balancer Replica T ws ws Ordering ws ws Replica 3 T ws ws Example: T1: { set x = 1 } T: { set x = 7 } 10

11 Sub-linear Scalability Wall Replica 1 ws ws Load Balancer Replica T ws ws Ordering ws ws Replica 4 Replica 3 T ws ws 11

12 This Talk General scaling techniques Address fundamental bottlenecks Synergistic, implemented in middleware Evaluated experimentally 1

13 TPS Super-linear Scalability X X X 7 X 1 X Single Base United MALB UF

14 Big Picture: Let s Oversimplify R U Standalone DBMS reading update logging 14

15 Big Picture: Let s Oversimplify R U Standalone DBMS reading update logging N.R N.U R U (N-1).ws Replica 1/N (traditional) reading update logging 15

16 Big Picture: Let s Oversimplify R U Standalone DBMS reading update logging N.R N.U R U (N-1).ws Replica 1/N (traditional) reading update logging N.R N.U R* U* (N-1).ws* Replica 1/N (optimized) reading update logging 16

17 Big Picture: Let s Oversimplify R U Standalone DBMS reading update logging N.R N.U R U (N-1).ws Replica 1/N (traditional) reading update logging N.R N.U R* U* (N-1).ws* Replica 1/N (optimized) reading update logging MALB Update Filtering Uniting O & D 17

18 Key Points 1. Commit updates in order Perform serial synchronous disk writes Unite ordering and durability. Load balancing Optimize for equal load: memory contention MALB: optimize for in-memory execution 3. Update propagation Propagate updates everywhere Update filtering: propagate to where needed 18

19 Roadmap Replica Tx 1 A Load Balancer Replica Ordering Load balancing Update propagation Replica 3 Commit updates in order 19

20 Key Idea Traditionally: Commit ordering and durability are separated Key idea: Unite commit ordering and durability 0

21 All Replicas Must Agree Tx A Tx B Replica 1 durability Replica durability All replicas agree on which update tx commit their commit order Total order Determined by middleware Followed by each replica Replica 3 durability 1

22 Order Outside DBMS Tx A Replica 1 durability Tx A Tx B Replica durability Tx B Ordering Replica 3 durability

23 Order Outside DBMS Tx A Replica 1 durability A B A B Tx A Tx B Tx B Replica Ordering durability A B A B A B Replica 3 durability A B A B 3

24 Enforce External Commit Order Ordering A B Replica 3 DBMS Tx A Tx B Proxy SQL interface Task A Task B durability 4

25 Enforce External Commit Order Ordering A B Replica 3 DBMS Tx A Tx B Proxy SQL interface Task A Task B durability B A 5

26 Enforce External Commit Order Ordering A B Replica 3 DBMS Tx A Tx B Proxy SQL interface Task A Task B durability B A Cannot commit A & B concurrently! 6

27 Enforce Order = Serial Commit Ordering A B Replica 3 DBMS Tx A Tx B Proxy SQL interface Task A Task B durability A 7

28 Enforce Order = Serial Commit Ordering A B Replica 3 DBMS Tx A Tx B Proxy SQL interface Task A Task B durability A B 8

29 Ack A Ack B Ack C Commit Serialization is Slow Ordering A B C Commit order A B C Proxy DBMS Commit A Commit B Commit C CPU CPU CPU durability Durability A Durability A B Durability A B C 9

30 Ack A Ack B Ack C Commit Serialization is Slow Ordering A B C Commit order A B C Proxy Problem: Durability DBMS & ordering separated serial disk writes Commit A Commit B Commit C CPU CPU CPU durability Durability A Durability A B Durability A B C 30

31 Ack A Ack B Ack C Unite D. & O. in Middleware Ordering A B C durability Durability A B C Commit order A B C Proxy DBMS Commit A Commit B Commit C CPU CPU CPU durability OFF 31

32 Ack A Ack B Ack C Unite D. & O. in Middleware Ordering A B C durability Durability A B C Commit order A B C Proxy Solution: Move durability to MW DBMS Durability & ordering in middleware group commit Commit A Commit B Commit C CPU CPU CPU durability OFF 3

33 Implementation: Uniting D & O in MW Middleware logs tx effects Durability of update tx Guaranteed in middleware Turn durability off at database Middleware performs durability & ordering United group commit fast Database commits update tx serially Commit = quick main memory operation 33

34 TPS Uniting Improves Throughput Metric Throughput Workload TPC-W Ordering (50% updates) System Linux cluster PostgreSQL 16 replicas Serializable exec X 7 X 1 X Single Base United TPC-W

35 Roadmap Replica Tx 1 A Load Balancer Replica Ordering Load balancing Update propagation Replica 3 Commit updates in order 35

36 Key Idea Equal load on replicas Replica 1 Mem Disk Load Balancer Replica Mem Disk 36

37 Key Idea Equal load on replicas Load Balancer Replica 1 Mem Disk Replica Mem MALB: (Memory-Aware Load Balancing) Disk Optimize for in-memory execution 37

38 How Does MALB Work? Database 1 3 Workload A 1 B 3 Memory Mem 38

39 Read Data From Disk A B 1 3 Replica 1 Mem A, B, A, B Least Loaded Disk Replica Mem 1 3 Disk

40 Read Data From Disk A B A, B, A, B 1 3 Least Loaded Slow Replica 1 Mem 1 3 Disk Replica 1 3 Slow Mem 1 3 Disk

41 Data Fits in Memory A B 1 3 Replica 1 Mem A, B, A, B MALB Disk Replica Mem 1 3 Disk

42 Data Fits in Memory A B A, B, A, B 1 3 MALB Memory info? Many tx and replicas? Fast Fast Replica 1 Mem 1 Disk Replica Mem Disk

43 Estimate Tx Memory Needs Exploit tx execution plan Which tables & indices are accessed Their access pattern Linear scan, direct access Metadata from database Sizes of tables and indices 43

44 Objective Grouping Transactions Construct tx groups that fit together in memory Bin packing Item: tx memory needs Bin: memory of replica Heuristic: Best Fit Decreasing Allocate replicas to tx groups Adjust for group loads 44

45 MALB in Action A B C D E F MALB 45

46 MALB in Action Memory needs for A, B, C, D, E, F A B C D E F MALB 46

47 MALB in Action Memory needs for A, B, C, D, E, F Group A A B C D E F MALB Group B C Group D E F 47

48 MALB in Action Memory needs for A, B, C, D, E, F Group A Replica A Disk A B C D E F MALB Group B C Replica B C Disk Group D E F Replica D E F Disk 48

49 Objective Optimize for in-memory execution Method MALB Summary Estimate tx memory needs Construct tx groups Allocate replicas to tx groups 49

50 Experimental Evaluation Implementation No change in consistency Still middleware Compare United: efficient baseline system MALB: exploits working set information Same environment Linux cluster running PostgreSQL Workload: TPC-W Ordering (50% update txs) 50

51 TPS MALB Doubles Throughput % 5 X TPC-W Ordering 16 replicas X 7 X 1 X Single Base United MALB UF 51

52 TPS Read I/O, normalized MALB Doubles Throughput % 5 X X 7 X 1 X Single Base United MALB UF United MALB 5

53 DB Size Big Gains with MALB Big 1% 75% 18% 45% 105% 48% Small 9% 0% 4% Mem Small Big Size

54 DB Size Big Big Gains with MALB Run from disk 1% 75% 18% 45% 105% 48% Small 9% 0% 4% Run from memory Mem Small Big Size

55 Roadmap Replica Tx 1 A Load Balancer Replica Ordering Load balancing Update propagation Replica 3 Commit updates in order 55

56 Key Idea Traditional: Propagate updates everywhere Update Filtering: Propagate updates to where they are needed 56

57 Update Filtering Example A B 1 3 Replica 1 Mem A, B, A, B MALB UF Disk Replica Mem 1 3 Disk

58 Update Filtering Example A B 1 3 Group A Replica 1 Mem 1 A, B, A, B MALB UF Disk Replica 1 3 Mem 3 Group B Disk

59 Update Filtering Example A B A, B, A, B 1 3 MALB UF Group A Update table 1 Replica 1 Mem 1 Disk 1 Replica 3 Mem 3 Group B Disk

60 Update Filtering Example A B A, B, A, B 1 3 MALB UF Group A Update table 1 Replica 1 Mem 1 Disk 1 Replica 3 Mem 3 Group B Disk

61 Update Filtering Example A B A, B, A, B 1 3 MALB UF Group A Update table 1 Replica 1 Mem 1 Disk 1 Replica 3 Mem 3 Group B Disk

62 Update Filtering Example A B A, B, A, B 1 3 MALB UF Group A Update table 1 Update table 3 Replica 1 Mem 1 Disk 1 Replica Mem 3 3 Group B Disk 1 3 6

63 Update Filtering Example A B A, B, A, B 1 3 MALB UF Group A Update table 1 Update table 3 Replica 1 Mem 1 Disk 1 Replica Mem 3 3 Group B Disk

64 Update Filtering Example A B A, B, A, B 1 3 MALB UF Group A Update table 1 Update table 3 Replica 1 Mem 1 Disk 1 Replica Mem 3 3 Group B Disk

65 Update Filtering in Action UF 65

66 Update Filtering in Action Update to red table UF 66

67 Update Filtering in Action Update to red table UF Update to green table 67

68 Update Filtering in Action Update to red table UF Update to green table 68

69 Update Filtering in Action Update to red table UF Update to green table 69

70 TPS MALB+UF Triples Throughput % 5 X 37 X TPC-W Ordering 16 replicas X 7 X 1 X Single Base United MALB UF

71 TPS Prop. Updates MALB+UF Triples Throughput 10 49% 37 X X X 7 X 1 X Single Base United MALB UF MALB UF

72 Ratio MALB+UF / MALB Filtering Opportunities MALB MALB+UF 0 MALB MALB+UF 5% Browsing Mix 50% Ordering Mix Updates 7

73 Conclusions 1. Commit updates in order Perform serial synchronous disk writes Unite ordering and durability. Load balancing Optimize for equal load: memory contention MALB: optimize for in-memory execution 3. Update propagation Propagate updates everywhere Update filtering: propagate to where needed 73