Compression in Open Source Databases. Peter Zaitsev CEO, Percona Percona Technical Webinars January 27 th, 2016

Transcription

1 Compression in Open Source Databases Peter Zaitsev CEO, Percona Percona Technical Webinars January 27 th, 2016

2 About the Talk 2 A bit of the History Approaches to Data Compression What some of the popular systems implement

3 Lets Define The Term 3 Compression - Any Technique to make data size smaller

4 A bit of History 4 Early Computers were too slow to compress data in SoMware Hardware Compression (ie Tape) Compression first appears for non performance criqcal data

5 We did not need it much for space 5

6 Welcome to the modern age 6 Data Growth outpaces HDD improvements Powerful CPUs Flash Cloud Data we store now

7 ExponenCal Data Size Growth 7

8 Powerful CPUs 8 High Performance MulQple Cores

9 Compression and Decompression Performance 9 From hzps://github.com/inikep/lzbench Compressor Compress Decompress RaCo memcpy 8368 MB/sec 8406 MB/sec 100% Brotli level 2 66 MB/sec 207 MB/sec 45% Lz4 487 MB/sec 2452 MB/sec 62% Lz4fast level MB/sec 3112 MB/sec 74% Snappy 326 MB/sec 1147 MB/s 62% Lzma Level 2 10 MB/s 37 MB/s 39% Zlib level 1 39 MB/s 201 MB/s 49% Zstd level MB/s 537 MB/s 49%

10 Flash (Solid State) 10 Disk space is more costly than for HDDs Write Endurance is expensive Want to write less data Decent at handling fragmentaqon

11 Cloud 11 Pay for Space Pay for IOPS More limited Storage Performance Network Performance may be limited

12 Data we store in Databases 12 Modern Data Compresses Well! Text JSON XML Time Series Data Log Files

13 13 IntroducQon into a ways of making your data smaller COMPRESSION BASICS

14 Lossy and Lossless 14 Database generally use Lossless Compression Lossy compression done on the applicaqon level

15 Some ways of geqng data smaller 15 Layout OpQmizaQons Encoding DicQonary Compression Block Compression

16 Layout OpCmizaCons 16 Column Store versus Row Store Hybrid Formats Variable Block Sizes

17 Encoding 17 Depends on Data Type and Domain Delta Encoding, Run Length Encoding (RLE) Can be faster than read of uncompressed data UTF8 (strings) and VLQ (Integers) Index Prefix Compression

18 DicConary Compression 18 Replacing frequent values with DicQonary Pointers Kind of like STL String ENUM type in MySQL

19 Block Compression 19 Compress block of data so it is smaller for storage Finding PaZerns in Data and Efficiently encoding them Many Algorithms Exist: Snappy, Zlib, LZ4, LZMA

20 Block Compression Details 20 Compression rate highly depends on data Compression rate depends on block size Speed depends on block size and data

21 Block Size Dependence (by Leif Walsh) 21

22 There is no one size fits all 22 Typically Compression Algorithm can be selected OMen with addiqonal seongs

23 23 Where do we compress data and how do we do that WHERE AND HOW

24 Where to Compress Data 24 In Memory? In the Database Data Store? As Part of File System? Storage Hardware? ApplicaQon?

25 Compression in Memory 25 Reduce amount of memory needed for same working set Typically in- Memory Performance Hit Reduce IO for Fixed amount of Memory Encoding/ DicQonary Compression are good fit

26 Database Data Store 26 Reduce Database Size on Disk Works with all file systems and storage With OS cache can be used as In- Memory compression variant Dealing with fragmentaqon is common issue

27 Compression on File System Level 27 Works with all Databases/ Storage Engines Performance Impact can be significant Logical Space on disk is not reduced ZFS

28 Compression on Storage Hardware 28 Hardware Dependent Does not reduce space on disk Can result in Performance Gains rather than free space (SSD) Can become a choke point

29 By ApplicaCon 29 No Database Support needed Reduce Database Load and Network Traffic ApplicaQon may know more about data More Complexity Give up many DBMS features (search, index)

30 30 What makes database system to do well with compression DESIGN CONSIDERATIONS

31 The Goal 31 Minimize NegaQve Impact for User OperaQons (Reads and Writes)

32 Design Principles 32 Fast Decompression Compression in Background Parallel Compression/ Decompression Reduce need of Re- Compression on Update

33 Choosing Block Size 33 Large Blocks Most efficient for compression Bulky Read Writes Small Blocks Fastest to Decompress Best for point lookups

34 34 What Database systems Really do with Compression IMPLEMENTATION EXAMPLES

35 MySQL Packed MyISAM 35 Compress table offline with myisampack Table Becomes Read Only Variety of compression methods are used Only data is compressed, not indexes Note MyISAM support index prefix compression for all indexes

36 MySQL Archive Storage Engine 36 Does not support indexes EssenQally file of rows with sequenqal access Uses zlib compression

37 Innodb Table Compression 37 Available Since MySQL 5.1 Pages compressed using zlib Compressed page target (1K, 4K, 8K) has to be set Both Compressed and Uncompressed pages can be cached in Buffer Pool Per Page log of changes to avoid recompression Extenrally Stored BLOBs are compressed as single enqty

38 Innodb Transparent Page Compression 38 Available in MySQL 5.7 Zib and LZ4 Compression Compresses pages as they are wrizen to disk Free space on the page is given back using hole punching Originally designed to work with FusionIO NVMFS Can cause problems for current filesystem due to very high hole number

39 Disk usage (Linkbench data set by Sunny Bains) 39

40 Performance on Fast SSD (FusionIO NVMFS) 40

41 Results on Slower SSD (Intel 730*2, EXT4) 41

42 Fractal Trees Compression 42 Available as Storage Engine for MySQL and MongoDB Can use many compression libraries Tunable Compression Block Size Reduce Re- Compression due to message buffering

43 Can get a lot of compression 43

44 MongoDB WiredTiger Storage Engine 44 Engine Has many compression seongs Indexes are using Index Prefix Compression Data Pages can be compressed using zlib,lz4 or Snappy

45 Compression Size (results by Asya Kamsky) 45

46 Compression in RocksDB 46 RocksDB LSM Based Storage Engine for MongoDB and MySQL LSM works very well with compression Supports, zlib, lz4, bzip2 compression Can use different compression methods for different Levels in LSM

47 Compression results from Mike Kania 47

48 PostgreSQL 48 Uses compression by default with TOAST 2KB (default) or longer Strings, BlOBs Unlike Innodb External Storage is not required for Compression Recommended to use File system compression ie ZFS if Compression is Desired

49 Summary 49 Compression is Important in Modern Age Consider it for your system Many different techniques are used to make data smaller by databases Compression support is rapidly changing and improving

50 Percona Live Data Performance Conference April in Santa Clara, CA at the Santa Clara ConvenQon Center Register with code WebinarPL to receive 15% off at registraqon MySQL, NoSQL, Data in the Cloud

51 Thank You! Peter Zaitsev hzps:// 51