The End of a Myth: Distributed Transactions Can Scale

Transcription

1 The End of a Myth: Distributed Transactions Can Scale Erfan Zamanian, Carsten Binnig, Tim Harris, and Tim Kraska Rafael Ketsetsides

2 Why distributed transactions? 2

3 Cheaper for the same processing power 3

4 The problem with distributed transactions

5 Adding more machines doesn t increase performance Throughput (M trxs/sec) Clustered with TCP/IP Cluster Size 5

6 Adding more machines doesn t increase performance Throughput (M trxs/sec) Clustered with TCP/IP Cluster Size 6

7 Why can t they scale? 7

8 Background: TCP/IP 8

9 TCP/IP is computationally expensive Complex design unusual event client/receiver path server/sender path (Step 2 of the 3-way-handshake) SYN/SYN+ACK ACK/- CONNECT/ SYN (Step 1 of the 3-way-handshake) CLOSE/- SYN RECEIVED SYN/SYN+ACK (simultaneous open) SEND/SYN SYN SENT (Start) CLOSED LISTEN/- LISTEN CLOSE/- RST/- Data exchange occurs ESTABLISHED SYN+ACK/ACK (Step 3 of the 3-way-handshake) CLOSE/FIN CLOSE/FIN FIN/ACK Active open Passive open FIN WAIT 1 FIN/ACK CLOSING CLOSE WAIT FIN+ACK/ACK ACK/- CLOSE/FIN FIN WAIT 2 FIN/ACK TIME WAIT LAST ACK Timeout (Go back to start) CLOSED 9

10 TCP/IP is computationally expensive Fixed window size causes linear overhead Client Server 10

11 How do we replace TCP/IP? 11

12 Background: RDMA 12

13 Remote Direct Memory Access (RDMA) Memory Memory CPU RDMA CPU 13

14 RDMA is great! Low latency High throughput Supported by InfiniBand 14

15 RDMA is hard to use One-sided communication: Receiver is not notified of connection 15

16 RDMA is hard to use One-sided communication: Receiver is not notified of connection So far, most solutions that use RDMA, only do so for part of the design 16

17 We need a system redesign 17

18 Main design Memory Servers hash table hash table hash table hash table *Actually, the hash table might point to a different memory server hash function thread 3 thread 2 thread 1... thread 3 thread 2 thread 1 thread n... thread 3 thread 2 thread 1 thread n... thread n timestamp thread timestamp thread timestamp thread Compute Servers 18

19 Data entries set when entry is moved to overflow region set when entry may be deleted set when entry is being committed Thread-Id 29 Bits Commit-Timest. 32 Bits Deleted Moved 1 Bit 1 Bit Locked 1 Bit 19

20 Old-version Buffers e v16 v17 v15 v14 Data- Buffer v18 v19 v13 v12 w head tail moved v17 v18 v19 v12 v13 v14 v15 v16 20

21 Data entries set when entry is moved to overflow region set when entry may be deleted set when entry is being committed Thread-Id 29 Bits Commit-Timest. 32 Bits Deleted Moved 1 Bit 1 Bit Locked 1 Bit Current Version Old-Version Buffers v15 v14 v16 Header- Buffer v13 v12 v17 Header v20 v18 v19 Overf ow Region Data v20 Copy-on-update Continuous Move unwanted entries are lazily GCed in continuous chunks v15 v14 v16 v13 writenext Data- Buffer v12 v17 writenext v18 v19 v17 v18 v19 v12 v13 v14 v15 v16 moved head tail 21

22 Timestamp vector (Read timestamp) Main design Read before fetching data Each cell is the commit timestamp of a thread Stored in a single Memory server Optimizations Fetched by a dedicated thread in each Compute server (big ts reader) Threads in a Compute server might share commit timestamp (compression) 22

23 Further notes Compute and Memory servers might coexist, taking advantage of locality Timestamp vectors may be partitioned Secondary indexes (B+-trees, hash tables) 23

24 Is it good enough? 24

25 Linear scalability Throughput (M trxs/sec) Classic (2-sided) NAM-DB w/o locality NAM-DB w locality Cluster Size 25

26 Linear scalability Throughput (M trxs/sec) Classic (2-sided) NAM-DB w/o locality NAM-DB w locality Cluster Size 26

27 Experiments 27

28 Setup TPC-C benchmark 2011-released InfiniBand (FDR) Two clusters, with 8 and 57 machines 28

29 Experiment 1: System scalability Throughput (M trxs/sec) Classic (2-sided) NAM-DB w/o locality NAM-DB w locality Cluster Size Latency (us) Classic (2-sided) NAM-DB w/o locality NAM-DB w locality Latency (us) Timestamps Read RS Lock and verify WS Install WS + index Cluster Size (a) Latency (b) Breakdown for NAM-DB Figure 5: Latency and Breakdown Cluster Size

30 Experiment 2: Scalability of the Oracle Timestamp ops/sec 160 M 140 M 120 M 100 M 80 M 60 M 40 M 20 M 0 Classic (global counter) NAM-DB (no opt.) NAM-DB + compression NAM-DB + bg ts reader NAM-DB + both opt # Clients 30

31 Experiment 3: Effect of Locality Throughput (M trxs/sec) NAM-DB w/o locality NAM-DB w locality Probability of distribution Latency (us) NAM-DB w/o locality NAM-DB w locality Probability of distribution (b) Latency Figure 6: Efect of Locality 31 executed all memory accesses using RDMA. When running w/ locality, we directly accessed the local memory if possible.

32 Experiment 4: Effect of Contention uniform low skew medium skew high skew very high skew Cluster Size Abort Rate uniform (almost zero; not visible) low skew medium skew high skew very high skew Cluster Size (b) Abort Rate Figure 8: Efect of Contention F f32 RDMA QPs many unreliable datagrams using

33 Experiment 5: Scalability of RDMA Queue Pairs M operations/s byte READs 64-byte READs 256-byte READs # Queue Pairs 33

34 Gaps in the logic 34

35 Future work 35

36 Future work Optimize for OLAP Reliably emulate large clusters and perform experiments Analyze performance, and optimize constants Explore collocation methods Explore secondary indexes 36

37 37 Thank you!

38 Media sources E. Zamanian et al. The end of a myth: Distributed transactions can scale C. Binnig et al. The end of slow networks: It s time for a redesign 38