Compression in Open Source Databases. Peter Zaitsev April 20, 2016

Transcription

1 Compression in Open Source Databases Peter Zaitsev April 20, 2016

2 About the Talk 2 A bit of the History Approaches to Data Compression What some of the popular systems implement 2

3 Lets Define The Term 3 Compression - Any Technique to make data size smaller 3

4 A bit of History 4 Early Computers were too slow to compress data in Software Hardware Compression (i.e. Tape) Compression first appears for non performance critical data 4

5 We did not need it much for space 5 5

6 Welcome to the modern age 6 Data Growth outpaces HDD improvements Powerful CPUs Flash Cloud Data we store now 6

7 Exponential Data Size Growth 7 7

8 Powerful CPUs 8 High Performance Multiple Cores 8

9 Compression Libraries Performance Numbers for Xeon E Ghz (Single Thread) 9

10 Compression Ratios Slower Compression generally means Better Compression 10

11 11 LZ4 Rocks!

12 Flash (Solid State) 1 2 Disk space is more costly than for HDDs Write Endurance is expensive Want to write less data Decent at handling fragmentation 12

13 Cloud 1 3 Pay for Space Pay for IOPS More limited Storage Performance Network Performance may be limited 13

14 Data we store in Databases 1 4 Modern Data Compresses Well! Text JSON XML Time Series Data Log Files 14

15 1 5 COMPRESSION BASICS Introduction into ways of making your data smaller

16 Lossy and Lossless 1 6 Database generally use Lossless Compression Lossy compression done on the application level 16

17 Some ways of getting data smaller 1 7 Layout Optimizations Encoding Dictionary Compression Block Compression 17

18 Layout Optimizations 1 8 Column Store versus Row Store Hybrid Formats Variable Block Sizes 18

19 Encoding 1 9 Depends on Data Type and Domain Delta Encoding, Run Length Encoding (RLE) Can be faster than read of uncompressed data UTF8 (strings) and VLQ (Integers) Index Prefix Compression 19

20 Dictionary Compression 2 0 Replacing frequent values with Dictionary Pointers Kind of like STL String ENUM type in MySQL 20

21 Block Compression 2 1 Compress block of data so it is smaller for storage Finding Patterns in Data and Efficiently encoding them Many Algorithms Exist: Snappy, Zlib, LZ4, LZMA 21

22 Block Compression Details 2 2 Compression rate highly depends on data Compression rate depends on block size Speed depends on block size and data 22

23 Block Size impacts compression ratio 2 3 Larger Blocks mean better compression 23

24 Block Size Impacts Compression Speed Impact so compression performance varies 24

25 There is no one size fits all 2 5 Typically Compression Algorithm can be selected Often with additional settings 25

26 2 6 WHERE and HOW Where do we compress data and how do we do that

27 Where to Compress Data 2 7 In Memory? In the Database Data Store? As Part of File System? Storage Hardware? Application? 27

28 Compression in Memory 2 8 Reduce amount of memory needed for same working set Reduce IO for Fixed amount of Memory Typically in-memory Performance Hit Encoding/Dictionary Compression are good fit 28

29 Database Data Store 2 9 Reduce Database Size on Disk Works with all file systems and storage With OS cache can be used as In- Memory compression variant Dealing with fragmentation is common issue 29

30 Compression on File System Level 3 0 Works with all Databases/Storage Engines Performance Impact can be significant Logical Space on disk is not reduced ZFS 30

31 Compression on Storage Hardware 3 1 Hardware Dependent Does not reduce space on disk Can result in Performance Gains rather than free space (SSD) Can become a choke point 31

32 By Application 3 2 No Database Support needed Reduce Database Load and Network Traffic Application may know more about data More Complexity Give up many DBMS features (search, index) 32

33 3 3 Design Considerations What makes database systems do well with compression

34 The Goal 3 4 Minimize Negative Impact for User Operations (Reads and Writes) 34

35 Design Principles 3 5 Fast Decompression Compression in Background Parallel Compression/Decompression Reduce need of Re- Compression on Update 35

36 Choosing Block Size 3 6 Large Blocks Small Blocks Most efficient for compression Bulky Read Writes Fastest to Decompress Best for point lookups 36

37 3 7 Implementation Examples What Database systems Really do with Compression

38 MySQL Packed MyISAM 3 8 Compress table offline with myisampack Table Becomes Read Only Variety of compression methods are used Only data is compressed, not indexes Note MyISAM support index prefix compression for all indexes 38

39 MySQL Archive Storage Engine 3 9 Does not support indexes Essentially file of rows with sequential access Uses zlib compression 39

40 Innodb Table Compression 4 0 Available Since MySQL 5.1 Pages compressed using zlib Compressed page target (1K, 4K, 8K) has to be set Both Compressed and Uncompressed pages can be cached in Buffer Pool Per Page log of changes to avoid recompression Externally Stored BLOBs are compressed as single entity 40

41 Innodb Transparent Page Compression 4 1 Available in MySQL 5.7 Zib and LZ4 Compression Compresses pages as they are written to disk Free space on the page is given back using hole punching Originally designed to work with FusionIO NVMFS Can cause problems for current filesystem due to very high hole number 41

42 42 Disk usage (Linkbench data set by Sunny Bains) 4 2

43 Performance on Fast SSD (FusionIO NVMFS)

44 Results on Slower SSD (Intel 730*2, EXT4)

45 Fractal Trees Compression 4 5 Available as Storage Engine for MySQL and MongoDB Can use many compression libraries Tunable Compression Block Size Reduce Re-Compression due to message buffering 45

46 Can get a lot of compression

47 MongoDB WiredTiger Storage Engine 4 7 Engine Has many compression settings Indexes are using Index Prefix Compression Data Pages can be compressed using zlib,lz4 or Snappy 47

48 Compression Size (results by Asya Kamsky)

49 Compression in RocksDB 4 9 RocksDB LSM Based Storage Engine for MongoDB and MySQL LSM works very well with compression Supports, zlib, lz4, bzip2 compression Can use different compression methods for different Levels in LSM 49

50 Compression results from Mike Kania

51 PostgreSQL 5 1 Uses compression by default with TOAST 2KB (default) or longer Strings, BlOBs Unlike Innodb External Storage is not required for Compression Recommended to use File system compression i.e. ZFS if Compression is Desired 51

52 Summary 5 2 Compression is Important in Modern Age Consider it for your system Many different techniques are used to make data smaller by databases Compression support is rapidly changing and improving

53 More Information Check out Article on Percona s Blog Yann Collet Blog is great Matt Mahoney Compression Web site 53

54 Thank you!