Sorting Improves Bitmap Indexes

Transcription

1 Joint work (presented at BDA 08 and DOLAP 08) with Daniel Lemire and Kamel Aouiche, UQAM. December 4, 2008

2 Database Indexes Databases use precomputed indexes (auxiliary data structures) to speed processing.

3 Database Indexes Databases use precomputed indexes (auxiliary data structures) to speed processing. An index costs memory, can hurt update speed.

4 Database Indexes Databases use precomputed indexes (auxiliary data structures) to speed processing. An index costs memory, can hurt update speed. Who makes the tradeoff for a given database? system itself? database designer/administrator?

5 Database Indexes Databases use precomputed indexes (auxiliary data structures) to speed processing. An index costs memory, can hurt update speed. Who makes the tradeoff for a given database? system itself? database designer/administrator? Improving indexes is practically important.

6 Bitmap indexes SELECT * FROM T WHERE x=a AND y=b; Above, compute {r r is the row id of a row where x = a} {r r is the row id of a row where y = b}

7 Bitmap indexes SELECT * FROM T WHERE x=a AND y=b; Bitmap indexes have a long history. (1972 at IBM.) Above, compute {r r is the row id of a row where x = a} {r r is the row id of a row where y = b}

8 Bitmap indexes SELECT * FROM T WHERE x=a AND y=b; Bitmap indexes have a long history. (1972 at IBM.) Long history with DW & OLAP. (Sybase IQ since mid 1990s). Above, compute {r r is the row id of a row where x = a} {r r is the row id of a row where y = b}

9 Bitmap indexes SELECT * FROM T WHERE x=a AND y=b; Bitmap indexes have a long history. (1972 at IBM.) Long history with DW & OLAP. (Sybase IQ since mid 1990s). Main competition: B-trees. Above, compute {r r is the row id of a row where x = a} {r r is the row id of a row where y = b}

10 Bitmaps and fast AND/OR operations Computing the union of two sets of integers between 1 and 64 (eg row ids, trivial table)... E.g., {1, 5, 8} {1, 3, 5}?

11 Bitmaps and fast AND/OR operations Computing the union of two sets of integers between 1 and 64 (eg row ids, trivial table)... E.g., {1, 5, 8} {1, 3, 5}? Can be done in one operation by a CPU: BitwiseOR( , )

12 Bitmaps and fast AND/OR operations Computing the union of two sets of integers between 1 and 64 (eg row ids, trivial table)... E.g., {1, 5, 8} {1, 3, 5}? Can be done in one operation by a CPU: BitwiseOR( , ) Extend to sets from 1..N using N/64 operations. To compute [a 0,..., a N 1 ] [b 0, b 1,..., b N 1 ] : a 0,..., a 63 BitwiseOR b 0,..., b 63 ; a 64,..., a 127 BitwiseOR b 64,..., b 127 ; a 128,..., a 192 BitwiseOR b 128,..., b 192 ;...

13 Bitmap compression column x n index bitmaps x= x=2 0 0 L x= A column with n rows and L distinct values nl bits

14 Bitmap compression column x n index bitmaps x= x=2 0 0 L x= A column with n rows and L distinct values nl bits E.g., n = 10 6, L = Gbits

15 Bitmap compression column x n index bitmaps x= x=2 0 0 L x= A column with n rows and L distinct values nl bits E.g., n = 10 6, L = Gbits Uncompressed bitmaps are often impractical

16 Bitmap compression column x n index bitmaps x= x=2 0 0 L x= A column with n rows and L distinct values nl bits E.g., n = 10 6, L = Gbits Uncompressed bitmaps are often impractical Moreover, bitmaps often contain long streams of zeroes...

17 Bitmap compression column x n index bitmaps x= x=2 0 0 L x= A column with n rows and L distinct values nl bits E.g., n = 10 6, L = Gbits Uncompressed bitmaps are often impractical Moreover, bitmaps often contain long streams of zeroes... Logical operations over these zeroes is a waste of CPU cycles.

18 How to compress bitmaps? Must handle long streams of zeroes efficiently Run-length encoding? (RLE)

19 How to compress bitmaps? Must handle long streams of zeroes efficiently Run-length encoding? (RLE) Bitmap: a run of 0s, a run of 1s, a run of 0s, a run of 1s,...

20 How to compress bitmaps? Must handle long streams of zeroes efficiently Run-length encoding? (RLE) Bitmap: a run of 0s, a run of 1s, a run of 0s, a run of 1s,... So just encode the run lengths, e.g., , 5, 3, 1,1,3

21 Byte/Word-aligned RLE RLE variants can focus on runs that align with machine-word boundaries.

22 Byte/Word-aligned RLE RLE variants can focus on runs that align with machine-word boundaries. Trade compression for speed.

23 Byte/Word-aligned RLE RLE variants can focus on runs that align with machine-word boundaries. Trade compression for speed. Variants: BBC (byte aligned), WAH

24 Byte/Word-aligned RLE RLE variants can focus on runs that align with machine-word boundaries. Trade compression for speed. Variants: BBC (byte aligned), WAH Our EWAH extends Wu et al. s word-aligned hybrid.

25 Byte/Word-aligned RLE RLE variants can focus on runs that align with machine-word boundaries. Trade compression for speed. Variants: BBC (byte aligned), WAH Our EWAH extends Wu et al. s word-aligned hybrid dirty word, run of 2 clean 0 words, dirty word...

26 Computational and storage bounds n number of rows, c number of 1s per row;

27 Computational and storage bounds n number of rows, c number of 1s per row; Total size of bitmaps is in O(nc);

28 Computational and storage bounds n number of rows, c number of 1s per row; Total size of bitmaps is in O(nc); Bounds do not depend on the number of bitmaps. (Assuming O(n) bitmaps).

29 Computational and storage bounds n number of rows, c number of 1s per row; Total size of bitmaps is in O(nc); Bounds do not depend on the number of bitmaps. (Assuming O(n) bitmaps). Implementation scales to millions of bitmaps.

30 Improving compression by sorting the table RLE, BBC, WAH, EWAH are order-sensitive: they compress sorted tables better;

31 Improving compression by sorting the table RLE, BBC, WAH, EWAH are order-sensitive: they compress sorted tables better; But finding the best row ordering is NP-hard.

32 Improving compression by sorting the table RLE, BBC, WAH, EWAH are order-sensitive: they compress sorted tables better; But finding the best row ordering is NP-hard. Lexicographic row sorting is fast, even for very large tables.

33 Improving compression by sorting the table RLE, BBC, WAH, EWAH are order-sensitive: they compress sorted tables better; But finding the best row ordering is NP-hard. Lexicographic row sorting is fast, even for very large tables. easy: sort is a Unix staple.

34 Improving compression by sorting the table RLE, BBC, WAH, EWAH are order-sensitive: they compress sorted tables better; But finding the best row ordering is NP-hard. Lexicographic row sorting is fast, even for very large tables. easy: sort is a Unix staple. Substantial index-size reductions (often 2.5 times)

35 Improving compression via k-of-n encoding With L bitmaps, you can represent L values by mapping each value to one bitmap; 1-of-N value cat dog dish fish cow cat pony

36 Improving compression via k-of-n encoding 1-of-N of-N With L bitmaps, you can represent L values by mapping each value to one bitmap; Alternatively, ( you can represent L ) 2 = L(L 1)/2 values by mapping each value to a pair of bitmaps;

37 Improving compression via k-of-n encoding 1-of-N of-N With L bitmaps, you can represent L values by mapping each value to one bitmap; Alternatively, ( you can represent L ) 2 = L(L 1)/2 values by mapping each value to a pair of bitmaps; More generally, you can represent ( L k) values by mapping each value to a k-tuple of bitmaps;

38 Improving compression via k-of-n encoding 1-of-N of-N With L bitmaps, you can represent L values by mapping each value to one bitmap; Alternatively, ( you can represent L ) 2 = L(L 1)/2 values by mapping each value to a pair of bitmaps; More generally, you can represent ( L k) values by mapping each value to a k-tuple of bitmaps; At query time, you need to load k bitmaps in a look-up for one value;

39 Improving compression via k-of-n encoding 1-of-N of-N With L bitmaps, you can represent L values by mapping each value to one bitmap; Alternatively, ( you can represent L ) 2 = L(L 1)/2 values by mapping each value to a pair of bitmaps; More generally, you can represent ( L k) values by mapping each value to a k-tuple of bitmaps; At query time, you need to load k bitmaps in a look-up for one value; You trade query-time performance for fewer bitmaps;

40 Improving compression via k-of-n encoding 1-of-N of-N With L bitmaps, you can represent L values by mapping each value to one bitmap; Alternatively, ( you can represent L ) 2 = L(L 1)/2 values by mapping each value to a pair of bitmaps; More generally, you can represent ( L k) values by mapping each value to a k-tuple of bitmaps; At query time, you need to load k bitmaps in a look-up for one value; You trade query-time performance for fewer bitmaps; Often, fewer bitmaps translates into a smaller index, created faster.

41 Encode then sort? Or vice versa? Two different conceptual approaches: 1 Encode attributes in table, obtaining an uncompressed index paint maker red ford blue honda green ford paint maker red ford blue honda green ford......

42 Encode then sort? Or vice versa? Two different conceptual approaches: 1 Encode attributes in table, obtaining an uncompressed index Sort the index rows paint maker red ford blue honda green ford paint maker red ford blue honda green ford

43 Encode then sort? Or vice versa? Two different conceptual approaches: 1 Encode attributes in table, obtaining an uncompressed index Sort the index rows Compress each column paint maker red ford blue honda green ford paint maker red ford blue honda green ford

44 Encode then sort? Or vice versa? Two different conceptual approaches: 1 Encode attributes in table, obtaining an uncompressed index Sort the index rows Compress each column 2 Sort the table rows paint maker red ford blue honda green ford paint maker red ford blue honda green ford paint maker blue honda green ford red ford

45 Encode then sort? Or vice versa? Two different conceptual approaches: 1 Encode attributes in table, obtaining an uncompressed index Sort the index rows Compress each column 2 Sort the table rows Encode attributes in table, build compressed index on-the-fly. paint maker red ford blue honda green ford paint maker red ford blue honda green ford paint maker blue honda green ford red ford

46 Gray-code order Lex. order Gray-code Gray-code (GC) order is an alternative to lexicographical order (defined only for bit arrays);

47 Gray-code order Lex. order Gray-code Gray-code (GC) order is an alternative to lexicographical order (defined only for bit arrays); May improve compression more than lex. sort (k > 1);

48 Gray-code order Lex. order Gray-code Gray-code (GC) order is an alternative to lexicographical order (defined only for bit arrays); May improve compression more than lex. sort (k > 1); [Pinar et al., 2005] process an uncompressed bitmap index.

49 Gray-code order Lex. order Gray-code Gray-code (GC) order is an alternative to lexicographical order (defined only for bit arrays); May improve compression more than lex. sort (k > 1); [Pinar et al., 2005] process an uncompressed bitmap index. Slow, if uncompressed index does not fit in RAM.

50 Gray-code order Lex. order Gray-code Gray-code (GC) order is an alternative to lexicographical order (defined only for bit arrays); May improve compression more than lex. sort (k > 1); [Pinar et al., 2005] process an uncompressed bitmap index. Slow, if uncompressed index does not fit in RAM. GC order is not supported by DBMSes or Unix utilities.

51 Gray-code sorting, cheaply Size improvement is small (usually < 4%), but it s essentially free: 1 What Pinar et al. do: expensive GC sort after encoding eg: [Tax, Cat, Girl, Cat] sort([1100, 0110, 1001, 0110]);

52 Gray-code sorting, cheaply Size improvement is small (usually < 4%), but it s essentially free: 1 What Pinar et al. do: expensive GC sort after encoding eg: [Tax, Cat, Girl, Cat] sort([1100, 0110, 1001, 0110]); 2 Instead, sort the table lexicographically comparing values alphabetically or by frequency (easy); eg: [Tax, Cat, Girl, Cat] [Cat, Cat, Girl, Tax]

53 Gray-code sorting, cheaply Size improvement is small (usually < 4%), but it s essentially free: 1 What Pinar et al. do: expensive GC sort after encoding eg: [Tax, Cat, Girl, Cat] sort([1100, 0110, 1001, 0110]); 2 Instead, sort the table lexicographically comparing values alphabetically or by frequency (easy); eg: [Tax, Cat, Girl, Cat] [Cat, Cat, Girl, Tax] 3 Map ordered values to k-tuples of bitmaps ordered as Gray codes: Cat: 0011, Dog: 0110, Girl: 0101, Tax: 1100; Lex ascending sequence: Cat, Dog, Girl, Tax. GC ascending sequence: 0011, 0110, 0101, 1100 for codes

54 Gray-code sorting, cheaply Size improvement is small (usually < 4%), but it s essentially free: 1 What Pinar et al. do: expensive GC sort after encoding eg: [Tax, Cat, Girl, Cat] sort([1100, 0110, 1001, 0110]); 2 Instead, sort the table lexicographically comparing values alphabetically or by frequency (easy); eg: [Tax, Cat, Girl, Cat] [Cat, Cat, Girl, Tax] 3 Map ordered values to k-tuples of bitmaps ordered as Gray codes: Cat: 0011, Dog: 0110, Girl: 0101, Tax: 1100; Lex ascending sequence: Cat, Dog, Girl, Tax. GC ascending sequence: 0011, 0110, 0101, 1100 for codes eg: [Cat, Cat, Girl, Tax] [0011, 0011, 0101, 1100] (generates a GC-sorted result without expensive GC sorting).

55 Gray-code sorting, cheaply Size improvement is small (usually < 4%), but it s essentially free: 1 What Pinar et al. do: expensive GC sort after encoding eg: [Tax, Cat, Girl, Cat] sort([1100, 0110, 1001, 0110]); 2 Instead, sort the table lexicographically comparing values alphabetically or by frequency (easy); eg: [Tax, Cat, Girl, Cat] [Cat, Cat, Girl, Tax] 3 Map ordered values to k-tuples of bitmaps ordered as Gray codes: Cat: 0011, Dog: 0110, Girl: 0101, Tax: 1100; Lex ascending sequence: Cat, Dog, Girl, Tax. GC ascending sequence: 0011, 0110, 0101, 1100 for codes eg: [Cat, Cat, Girl, Tax] [0011, 0011, 0101, 1100] (generates a GC-sorted result without expensive GC sorting). 4 Easily extended for > 1 columns.

56 Gray-code sorting, cheaply Size improvement is small (usually < 4%), but it s essentially free: 1 What Pinar et al. do: expensive GC sort after encoding eg: [Tax, Cat, Girl, Cat] sort([1100, 0110, 1001, 0110]); 2 Instead, sort the table lexicographically comparing values alphabetically or by frequency (easy); eg: [Tax, Cat, Girl, Cat] [Cat, Cat, Girl, Tax] 3 Map ordered values to k-tuples of bitmaps ordered as Gray codes: Cat: 0011, Dog: 0110, Girl: 0101, Tax: 1100; Lex ascending sequence: Cat, Dog, Girl, Tax. GC ascending sequence: 0011, 0110, 0101, 1100 for codes eg: [Cat, Cat, Girl, Tax] [0011, 0011, 0101, 1100] (generates a GC-sorted result without expensive GC sorting). 4 Easily extended for > 1 columns. In our tests, this is as good as a Gray-code bitmap index sort [Pinar et al., 2005], but technically much easier.

57 Test data sets Previous studies used data sets where the uncompressed index would fit in RAM.

58 Test data sets Previous studies used data sets where the uncompressed index would fit in RAM. Do their results apply to more realistic data sets?

59 Test data sets Previous studies used data sets where the uncompressed index would fit in RAM. Do their results apply to more realistic data sets? Our tests: Mix of real and synthetic data, up to 877 M rows, 22 GB, 4 M attribute values. using 4 10 columns/dimensions

60 When sorting, column order matters The first column(s) gain more from the sort (column 1 is primary sort key);

61 When sorting, column order matters The first column(s) gain more from the sort (column 1 is primary sort key); Its bitmaps (first 11 in example) are compressed well, compared to a randomsort. (Red above green) Compression on TWEED-4d 1-C/N Gray Random-sort rang des bitmaps

62 When sorting, column order matters The first column(s) gain more from the sort (column 1 is primary sort key); Its bitmaps (first 11 in example) are compressed well, compared to a randomsort. (Red above green) Least important column s bitmaps (43 49) don t gain much (red vs green) Compression on TWEED-4d 1-C/N rang des bitmaps Gray Random-sort

63 When sorting, column order matters Conceptually, we may wish to reorder columns, eg swap columns 1 & 3.

64 When sorting, column order matters Conceptually, we may wish to reorder columns, eg swap columns 1 & 3. Column order is crucial (to successful sorting).

65 When sorting, column order matters Conceptually, we may wish to reorder columns, eg swap columns 1 & 3. Column order is crucial (to successful sorting). Finding the best ordering quickly remains open. index size Netflix: 24 column orderings 5.5e+08 5e e+08 4e e+08 3e e+08 2e e+08 1e k=1 k= column permutation

66 Progress toward choosing column order Paper models gain of putting a given column first. Idea: order columns greedily (by max gain).

67 Progress toward choosing column order Paper models gain of putting a given column first. Idea: order columns greedily (by max gain). Experimentally, this approach is not promising: the best orderings don t seem to depend on gain.

68 Progress toward choosing column order Paper models gain of putting a given column first. Idea: order columns greedily (by max gain). Experimentally, this approach is not promising: the best orderings don t seem to depend on gain. Factors: skews of columns number of distinct values k density of column s bitmaps

69 What usually works for dimension ordering?: k=1 For 1-of-N bitmaps, a density-based approach was okay:

70 What usually works for dimension ordering?: k=1 For 1-of-N bitmaps, a density-based approach was okay: Ordering rule, k = 1 : sparse but not too sparse Order columns by decreasing ( 1 min, 1 1/n ) i, where n i 4w 1 n i the number of distinct values in column i, w the word size distinct values in column

71 What usually works for dimension ordering?: k=1 For 1-of-N bitmaps, a density-based approach was okay: Ordering rule, k = 1 : sparse but not too sparse Order columns by decreasing ( 1 min, 1 1/n ) i, where n i 4w 1 n i the number of distinct values in column i, w the word size distinct values in column See 30 40% size reduction, merely knowing dimension sizes (n i ).

72 What usually works for dimension ordering?: k > 1 Density formula (n i k n i ) recommends poorly when k > 1. Our experiments on synthetic data give some guidance:

73 What usually works for dimension ordering?: k > 1 Density formula (n i k n i ) recommends poorly when k > 1. Our experiments on synthetic data give some guidance: When k > 1, order columns by 1 descending skew 2 descending size (And do the reverse when k = 1.)

74 What usually works for dimension ordering?: k > 1 Density formula (n i k n i ) recommends poorly when k > 1. Our experiments on synthetic data give some guidance: When k > 1, order columns by 1 descending skew 2 descending size (And do the reverse when k = 1.) Open issues, k > 1 1 How do we balance skew & size factors?

75 What usually works for dimension ordering?: k > 1 Density formula (n i k n i ) recommends poorly when k > 1. Our experiments on synthetic data give some guidance: When k > 1, order columns by 1 descending skew 2 descending size (And do the reverse when k = 1.) Open issues, k > 1 1 How do we balance skew & size factors? 2 What other properties of the histograms are needed?

76 Bitmap-by-bitmap reordering One might instead make the index, reorder its columns, then apply GC sort [Canahuate et al., 2006].

77 Bitmap-by-bitmap reordering One might instead make the index, reorder its columns, then apply GC sort [Canahuate et al., 2006]. Our best implementation of this is 100 times slower, cannot handle larger data sets.

78 Bitmap-by-bitmap reordering One might instead make the index, reorder its columns, then apply GC sort [Canahuate et al., 2006]. Our best implementation of this is 100 times slower, cannot handle larger data sets. We tried several bitmap orders on DBGEN and Census. Out of 8 cases, only one gained, and only by 3%.

79 Bitmap-by-bitmap reordering One might instead make the index, reorder its columns, then apply GC sort [Canahuate et al., 2006]. Our best implementation of this is 100 times slower, cannot handle larger data sets. We tried several bitmap orders on DBGEN and Census. Out of 8 cases, only one gained, and only by 3%. Canahaute suggests ordering does not matter much, but we see factor-of-2 differences (??)

80 Bitmap-by-bitmap reordering One might instead make the index, reorder its columns, then apply GC sort [Canahuate et al., 2006]. Our best implementation of this is 100 times slower, cannot handle larger data sets. We tried several bitmap orders on DBGEN and Census. Out of 8 cases, only one gained, and only by 3%. Canahaute suggests ordering does not matter much, but we see factor-of-2 differences (??) Seems sufficient (and much faster) to work with groups of bitmaps (reorder attributes, not bitmaps)

81 Index size versus block-wise sorting Netflix taille de l index (Mo) k=1 k=2 k=3 k= # de blocs Instead of fully sorting the table, we sorted it block-wise;

82 Index size versus block-wise sorting Netflix taille de l index (Mo) k=1 k=2 k=3 k= # de blocs Instead of fully sorting the table, we sorted it block-wise; Fewer blocks means a more complete sort;

83 Index size versus block-wise sorting Netflix taille de l index (Mo) k=1 k=2 k=3 k= # de blocs Instead of fully sorting the table, we sorted it block-wise; Fewer blocks means a more complete sort; Larger k means smaller index (in this case);

84 Index size versus block-wise sorting Netflix taille de l index (Mo) k=1 k=2 k=3 k= # de blocs Instead of fully sorting the table, we sorted it block-wise; Fewer blocks means a more complete sort; Larger k means smaller index (in this case); Index size diminishes drastically with sorting.

85 Future directions Need better mathematical modelling of bitmap compressed size in sorted tables;

86 Future directions Need better mathematical modelling of bitmap compressed size in sorted tables; Study the effect of word length (16, 32, 64, 128 bits);

87 Future directions Need better mathematical modelling of bitmap compressed size in sorted tables; Study the effect of word length (16, 32, 64, 128 bits); Investigate Long run Gray code (discussed by Knuth).

88 Questions??

89 Canahuate, G., Ferhatosmanoglu, H., and Pinar, A. (2006). Improving bitmap index compression by data reorganization. http: //hpcrd.lbl.gov/~apinar/papers/tkde06.pdf (checked ). Chan, C. Y. and Ioannidis, Y. E. (1999). An efficient bitmap encoding scheme for selection queries. In SIGMOD 99, pages Pinar, A., Tao, T., and Ferhatosmanoglu, H. (2005). Compressing bitmap indices by data reorganization. In ICDE 05, pages Wu, K., Otoo, E. J., and Shoshani, A. (2006). Optimizing bitmap indices with efficient compression. ACM Transactions on Database Systems, 31(1):1 38.

90 Avoiding column order altogether Frequent component approach, but did not work well, plus slower.