MIGRATORY COMPRESSION Coarse-grained Data Reordering to Improve Compressibility

Transcription

1 MIGRATORY COMPRESSION Coarse-grained Data Reordering to Improve Compressibility Xing Lin *, Guanlin Lu, Fred Douglis, Philip Shilane, Grant Wallace * University of Utah EMC Corporation Data Protection and Availability Division 1

2 Overview Compression: finds redundancy among strings within a certain distance (window size) Problem: window sizes are small, and similarity across a larger distance will not be identified Migratory compression (MC): coarse-grained reorganization to group similar blocks to improve compressibility A generic pre-processing stage for standard compressors In many cases, improve both compressibility and throughput Effective for improving compression for archival storage 2

3 Background Compression Factor (CF) = ratio of original / compressed Throughput = original size / (de)compression time Compressor MAX Window size Techniques gzip 64 KB LZ; Huffman coding bzip2 9 KB Run-length encoding; Burrows-Wheeler Transform; Huffman coding 7z 1 GB LZ; Markov chain-based range coder 3

4 Background Compression Factor (CF) = ratio of original / compressed Throughput = original size / (de)compression time Compressor MAX Window size Techniques gzip 64 KB LZ; Huffman coding bzip2 9 KB Run-length encoding; Burrows-Wheeler Transform; Huffman coding 7z 1 GB LZ; Markov chain-based range coder 4

5 Background Compression Factor (CF) = ratio of original / compressed Throughput = original size / (de)compression time Compressor MAX Window size Techniques gzip 64 KB LZ; Huffman coding bzip2 9 KB Run-length encoding; Burrows- Wheeler Transform; Huffman coding 7z 1 GB LZ; Markov chain-based range coder Compression Tput (MB/s) Example Throughput vs. CF 14 gzip bzip z Compression Factor (X) The larger the window, the better the compression but slower

6 Background Compression Factor (CF) = ratio of original / compressed Throughput = original size / (de)compression time Compressor MAX Window size Techniques gzip 64 KB LZ; Huffman coding bzip2 9 KB Run-length encoding; Burrows- Wheeler Transform; Huffman coding 7z 1 GB LZ; Markov chain-based range coder rzip (from rsync): intra-file deduplication; bzip2 the remainder Compression Tput (MB/s) Example Throughput vs. CF gzip bzip Compression Factor (X) rzip 7z The larger the window, the better the compression but slower 6

7 Motivation Compress a single, large file Traditional compressors are unable to exploit redundancy across a large range of data (e.g., many GB) gz window A B A C (GBs) A B compress 7z window A transform A A B B C compress Standard compressor mzip 7

8 Motivation Better compression for long-term retention Data migration from backup to archive tier Observation: similar blocks could be scattered across compression regions (CRs) in the backup tier CR1 CR2 A B C A B C CR: unit of data that is compressed together (e.g. 128 KB) Backup tier 8

9 Motivation Better compression for long-term retention Data migration from backup to archive tier Observation: similar blocks could be scattered across compression regions (CRs) in the backup tier CR1 A B C Migrate CR1 A B C CR2 A B C CR2 A B C Backup tier Current practice Archive tier 9

10 Motivation Better compression for long-term retention Data migration from backup to archive tier Observation: similar blocks could be scattered across compression regions (CRs) in the backup tier Proposal: fill each CR with similar data CR1 A B C CR1 A A B B Migrate CR2 A B C CR2 C C Backup tier Archive tier Migratory compression 1

11 Recap Migratory compression Idea: improve compression by grouping similar blocks Use cases mzip: compress a single, large file Data migration for archival storage Considerations How to identify similar blocks? How to reorganize blocks efficiently? Other tradeoffs (refer to the paper) 11

12 mzip Workflow File Segmentation Similarity detector Reorganizer Reorganized file Compressor Segmentation: partition into blocks, calculate similarity features Similarity detector: group by content and identify duplicate and similar blocks; output migrate and restore recipe Reorganizer: rearrange the input file Reorganized file may exist only as a pipe into compressor Compressed file 12

13 mzip Workflow File Compressed file Segmentation Decompressor Similarity detector Reorganized file Reorganizer Reorganized file Extract restore recipe Compressor Restorer Compressed file File 13

14 mzip Example migrate recipe Block ID A Similarity detector File B C A D B restore recipe 14

15 mzip Example migrate recipe Block ID Reorganize A A File B C A D B A B B C D Reorganized File Restore restore recipe 1

16 Similarity Detection Similarity feature (based on [Broder 1997]) A strong hash for duplication detection Features: weak hashes provide hints about similarity among blocks Two blocks are similar when they share values for some weak hashes Mature technique REBL: Redundancy Elimination at Block Level [Kulkarni et al. 24] WAN Optimized Replication [Shilane et al. 212] 16

17 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level recipe A B C A D B input Block-level 17

18 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level recipe B C A D B input Block-level A 18

19 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level recipe B C D B input Block-level A A 19

20 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level recipe C D B input Block-level A A B 2

21 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level recipe C D input Block-level A A B B 21

22 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level recipe D input Block-level A A B B C 22

23 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level recipe input Block-level A A B B C D 23

24 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level Random IOs Fine for memory and SSD recipe input Block-level A A B B C D 24

25 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level Random IOs Fine for memory and SSD Multipass (HDD) 1st scan recipe A B C A D B input Block-level Multipass Buffer 2

26 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level Random IOs Fine for memory and SSD Multipass (HDD) 1st scan recipe C D B input Block-level Multipass A A B Buffer 26

27 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level Random IOs Fine for memory and SSD Multipass (HDD) 2nd scan recipe C D B Buffer input Block-level Multipass 27

28 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level Random IOs Fine for memory and SSD Multipass (HDD) 2nd scan recipe input Block-level Multipass B C D Buffer 28

29 Data Reorganization Re-arrange the input file, according to a recipe Reorganizer & Restorer Options Block-level Random IOs Fine for memory and SSD Multipass (HDD) Convert random IOs into sequential scans 2nd scan recipe input Block-level Multipass B C D Buffer 29

30 Evaluation mzip: How much can compression be improved? What is the complexity (runtime overhead)? How does SSD or HDD affect data reorganization performance? For HDD, does multi-pass help? How does MC perform, compared with delta compression? (refer to our paper) What are the configuration options for MC? (refer to our paper) Data migration for archival storage Compression and performance 3

31 Datasets Type Name Size (GB) Dedupe Factor (X) Compression Factor (standalone - baseline) gzip bzip2 7z rzip Workstation backup server backup VM image WS1 EXCHANGE2 Ubuntu- VM * Only one dataset for each type is shown here. Refer to our paper for others. 31

32 Compression Factor vs. Compression Throughput Workstation1 MC improves both CF and compression throughput ü Deduplication ü Re-organization Compression Tput (MB/s) gzip(mc) gzip rzip bzip2 7z Compression Factor (X) gzip gz(mc) bzip2 bz(mc) 7z 7z(mc) rzip rz(mc) 32

33 Compression Factor vs. Compression Throughput Exchange2 MC improves CF but slightly reduces compression throughput Compression Tput (MB/s) gzip bzip2 7z rzip gz(mc) bz(mc) 7z(mc) rz(mc) Compression Factor (X) 33

34 Maximal Compression Compressors can be run at various compression levels and window sizes, making a tradeoff between compression and runtime Workstation1 Compression Tput (MB/s) z-DEF(MC) 7z-MAX(MC) 7z-DEF 7z-MAX Compression Factor (X) MC benefits maximal compression Results for other compressors have same trends 34

35 Compression Factor Breakdown Compresion Factor (X) reorg gzip dedup MC improves compression to varying extents beyond dedup and gzip EX2 WS1 UBUN Dataset 3

36 Data Reorganization SSD: reasonably good HDD: multipass helps Configurations mem: input and output stored in tmpfs SSD/HDD: used to store input; output to HDD; 8 GB mem Compression Tput (MB/s) EX1 WS1 UBUN UBUN: 7 G Multipass window: 4.8 G (2 scans) mem ssd-block hdd-multipass hdd-block 36

37 Data Migration for Archival Storage DataDomain Filesystem (DDFS) Backup tier: LZ (fast) Archive tier: recompresses with GZ (2-44% better CF) With MC, Identify and sort by cluster sizes of similar blocks Migrate in 3 stages: top third largest clusters, middle third, bottom third (including distinct blocks) Datasets WORKSTATIONS: many backups of several workstations Exchange[123]: many backups of 3 exchange servers 37

38 Effect of MC on Archival Migration Compression Factor (X) gzip mc_total top1/3 mid1/3 end1/3 ü MC improves CF [44% (EX3), 17% (WS)] Top 2/3 compresses very well WS EX1 EX2 EX3 38

39 Archival Migration Costs Migration Runtime Exchange1: 3X longer Read Performance As data is scattered in file system, read performance suffers [Lillibridge 213] entire EXCHANGE1: 1.3 X longer final backup: 7X longer (1.24 X longer if just the top third of similar clusters are reorganized) Memory overheads 39

40 Related Work Improving traditional compressors LZMA algorithm with extra-large window (7z) rzip rolling hash to deduplicate with a large window Burrows-Wheeler Transform to reorder data (bzip2) Delta compression (MC is slightly better in CF and throughput) Similar to MC when limited to a single file Bookkeeping complexities in a file system Similarity detection WAN Optimized Replication: delta-compress similar chunks for replication REBL: Redundancy Elimination at Block Level (Never applied practically at scale, or in self-contained file) 4

41 Conclusions Migratory Compression preprocesses data to make it more compressible Identify and cluster similar data mzip Improves existing compressors, in both compressibility and frequently runtime Redraw the performance-compression curve! Archival storage MC reduces $/GB further Throughput CF 41

42 Thank you! Questions? 42