TokuDB vs RocksDB. What to choose between two write-optimized DB engines supported by Percona. George O. Lorch III Vlad Lesin

Transcription

1 TokuDB vs RocksDB What to choose between two write-optimized DB engines supported by Percona George O. Lorch III Vlad Lesin

2 What to compare? Amplification Write amplification Read amplification Space amplification Features set 2

3 How to compare? Theoretical review Fractal trees vs LSM Implementation What and how is persistently stored? What and how is cached? Read/write/space tradeoff options Benchmarks 3

4 Fractal tree: B-tree B B B B logb(n/b) 4

5 Fractal tree: B-tree with node buffers B k children buffer logk(n/b) message 5

6 Fractal tree: messages pushdown 1 6

7 Fractal tree: messages pushdown 1 7 2

8 Fractal tree: messages pushdown

9 Fractal tree: messages pushdown

10 Fractal tree: messages pushdown

11 Fractal tree: messages pushdown

12 Fractal tree: amplification for both point and range Write amplification Worst case: O(k*logk(N/B)) Skewed writes: O(logk(N/B)) Read amplification Cold cache: O(logk(N/B)) Warm cache: 1 where k - fanout, B - block size, N - data size. 12

13 LSM: Log Structured Merge tree A collection of sorted runs of data Each run contains kv-pairs with keys sorted A run can be managed as a single file or a collection of smaller files Binary search 13

14 LSM: example 14 Consist of collection of runs Many versions of a row can exist in different runs Binary search in each run Compaction to maintain good read performance

15 LSM: leveled L1 L2 L3 15 Each level is limited by size For example if 10^(i-1)< sizeof(li) <= 10^i then L1 is between 1MB and 10MB L2 is between 10MB and 100MB L3 is between 100MB and 1Gb... When the size of certain level reaches it s limit the level is merged to the next level

16 LSM: compaction example L1 L2 L3 L1 L2 L3 16 L1, L2 are merged to L3 Merge sort algorithm Updates

17 LSM: write amplification Θ(logk(N/B)) levels where k - growth factor, N - data size, B - block size. On average each block is remerged back at the same level k/2 times? L0: B L1: nothing L0: nothing L1: B L0: B L1: B L0: nothing L1: BB... L0: nothing L1: B k-times The first B bytes is remerged back in L1 k-1 times, the second B bytes are remerged back k-1 times, and so on, the last B bytes are remerged back 0 times. The average number of remerged bytes is sum(number of remerges for kb bytes)/k. The sum is arithmetic progression and can be counted as (k^2)/2, so the average number of remerges is k/2. 17 The total write amplification is Θ(klogk(N/B))

18 LSM: read amplification For range queries and cold cache summarizing binary search read amplification for all runs: O(((log(N/B))^2)/log(k)) For range queries and warm cache the first several runs are cached so range read will cost several IOs For point queries and bloom filters 1 IO in most cases. 18

19 FT vs LSM: resume Asymptotically: FT and LSM are the same for write amplification FT is better than LSM for range reads on cold cache, but the same on warm cache FT is the same as LSM for point reads: one or several IO s per read 19

20 TokuDB vs MyRocks: implementation How data is stored on disk? What and how data is cached? 20

21 TokuDB vs MyRocks: storage files TokuDB MyRocks Log files + block files The only FT in each file File per index WAL files + SST files All tables and indexes are in the same space(s) ColumnFamilies 21

22 TokuDB: block file Each block is a tree node The blocks can have different size Two sets of blocks, the sets can intersect Header 0 LSN 0 BTT 0 info Header 1 LSN 1 BTT 1 info Offset 0 Offset 4K 22 Block #42 Block #7 Offset #7 Size #7 There can be used and free blocks The blocks can be compressed Blocks translation table Fragmentation BTT 1 BTT 1 offset Block BTT 0 #177 BTT 0 offset

23 MyRocks: block-based sst file format kv-pairs are ordered and partitioned in data blocks Meta-index contains references to meta blocks Index block contains one entry per data block which contains it s the last key Bloom filter per block or per file Compression dictionary for the whole sst file Data block 1 23 Data block 2... Data block n Meta block 1 filter Meta block 2 stats Meta block 3 comp. dict.... Meta block k Meta index block Index block Footer

24 TokuDB vs MyRocks: persistent storage TokuDB RocksDB Two sets of blocks, the sets can be intersected, but on heavy random updates the intersection is small, the free blocks are reallocated One file per index(drop index is just file removing) 24 Big blocks compressed There are no gaps like in TokuDB, but on heavy random updates there can be several values for the same key in different levels (which are merge on compaction) All indexes in the same namespace(s) Data blocks are compressed, compressed dictionary is common

25 TokuDB: caching Blocks are cached in cache table Leaf nodes can be read partially All modifications are in memory Dirty pages are flushed during checkpoint or eviction Checkpoint is periodical mark all nodes as pending If client modifies pending node it s cloned Checkpoint can lead to extra io and memory usage Clock eviction algorithm In-memory cache table Cache file Block file block block... Disk block... block block... dirty block 25...

26 TokuDB: checkpoint performance 26

27 TokuDB: checkpoints and clone storm effect 27

28 TokuDB: checkpoints and clone storm effect 28

29 MyRocks: caching 29 Block Cache Sharded LRU Clock Indexes and block file bloom filters are caches separately by default, they don t count in memory allocated for BlockCache and can be big memory contributors. MemTables can be considered as write buffers, the more space for MemTables, the less write amplification, several kinds of memtable are supported Memory MemTables BloomFilters Indexes BloomFilters BlockCache Disk WAL SST

30 TokuDB vs MyRocks: benchmarking Different data structures to store data persistently Both engines have settings for read/write/space tradeoff Mysql infrastructure influence Different set of features 30

31 TokuDB vs MyRocks: benchmarking Hard to compare because each engine has settings for read-write tradeoff tuning TokuDB: block size, fanout, etc... MyRocks: blocks size, base level size, etc 31

32 MySQL Transaction Coordinator issues 32 > 1 transactional storage engine installed requires a TC for every transaction. Binlog is considered a transactional storage engine when enabled. MySQL provides two functioning coordinators, TC_BINLOG and TC_MMAP. When binary logging is enabled TC_BINLOG is always used, otherwise TC_MMAP is used when > 1 transactional engine installed. TC_MMAP is not optimized nor has as good of a group commit implementation as TC_BINLOG. When TC_MMAP is used, dramatic increase in fsyncs. Considering options to best improve this in Percona Server: Add session variable to tell if TC is needed for next transaction and return SQL error if transaction tries to engage > 1 transactional storage engine. Remove TC_MMAP and enhance TC_BINLOG to allow use without actual binary logs, i.e. keep TC logic but disable binlog logic.

33 TokuDB vs MyRocks: benchmarking Table: CREATE TABLE sbtest1 ( id INTEGER NOT NULL, k INTEGER DEFAULT '0' NOT NULL, c CHAR(120) DEFAULT '' NOT NULL, pad CHAR(60) DEFAULT '' NOT NULL, PRIMARY KEY (id)) ENGINE =... 33

34 TokuDB vs MyRocks: benchmarking Point reads: SELECT id, k, c FROM sbtest1 WHERE id IN (?) - 10 random points Range reads: SELECT count(id) FROM sbtest1 id BETWEEN? AND? OR, number of ranges 10, interval Point updates: UPDATE sbtest1 SET c=? WHERE id=? Range updates: UPDATE sbtest1 SET c=? WHERE id BETWEEN? AND?, interval=100 recovery logs are not synchronized on each commit 34

35 TokuDB vs MyRocks: benchmarking, first case NVME, ~400G table size, 10G - DB cache, 10G - fs cache 48 HW threads, 40 sysbench threads TokuDB: block size 4M, fanout 16 MyRocks: block size 8k, first level size 512M 35 5 levels for 400G table size

36 TokuDB vs MyRocks: TPS first case 36

37 TokuDB vs MyRocks: benchmarking, second case NVME, ~100G DB size, 4G - DB cache, 4G - fs cache 48 HW threads, 40 sysbench threads TokuDB: block size 1M, fanout 128 MyRocks: block size 8k, first level size 32M 37 5 levels - the same amount as for 400G table size

38 TokuDB vs MyRocks: TPS second case 38

39 TokuDB vs MyRocks: TPS summary Updates cause reads, TokuDB fanout affects read performance The less TokuDB block size, the more effective caching, the less IO s TokuDB range reads looks suspicious MyRocks - as we preserved the same amount of levels and the same cache/persistence storage balance, the TPS is almost the same as on previous test 39

40 MyRocks: range reads io and cpu 40

41 TokuDB: range reads io and cpu 41

42 TokuDB: range reads performance PFS is coming soon... 42

43 TokuDB vs MyRocks: write amplification counting /proc/diskstats shows blocks written Sysbench shows the amount of executed transactions WA = K * (blocks written / amount of executed transactions) K =( (block size) / (row size) + binlog writes) Count relative values, let MyRocks WA is 100%, K is diminished 43

44 TokuDB vs MyRocks: write amplification 44

45 TokuDB vs MyRocks: write amplification 45

46 TokuDB vs MyRocks: what about several tables NVME, ~100G DB size, 4G - DB cache, 4G - fs cache 48 HW threads, 40 sysbench threads TokuDB: block size 1M, fanout 128 MyRocks: block size 8k, first level size 32M 20 tables 46

47 TokuDB vs MyRocks: TPS, 20 tables 47

48 TokuDB vs MyRocks: benchmarking, third case NVME, ~200G compressed table size, 10G - DB cache, 10G - fs cache 48 HW threads, 40 sysbench threads TokuDB: block size 1M, fanout 128 MyRocks: block size 8k, first level size 512M 48 5 levels

49 MyRocks: TPS of range updates 49

50 MyRocks: io of range updates 50

51 TokuDB: TPS of range updates 51

52 TokuDB: io of range updates 52

53 TokuDB vs MyRocks: benchmarking, space For all the benchmarks: The initial space TokuDB and MyRocks tables is approximately the same TokuDB space doubles after heavy updates MyRocks space doubles too but after compaction the allocated space decreases to the initial size + ~10%. 53

54 TokuDB vs MyRocks: benchmarking, what to try more? Play with smaller TokuDB block sizes Play with different amount of levels, block sizes etc. in Rocks Test it on rotated disks Test for timed series load Test for reads after updates/deletes Test for inserts Test for mixed load Test for secondary indexes performance 54

55 TokuDB vs MyRocks: features TokuDB MyRocks Clustered indexes Different settings for MemTable Online index ColumnFamilies Gap locks Per-level compression settings 55

56 DATABASE PERFORMANCE MATTERS