Column Store Internals

Transcription

1 Column Store Internals Sebastian Meine SQL Stylist with sqlity.net Outline Outline Column Store Storage Aggregates Batch Processing History 1

2 History First mention of idea to cluster column groups into separate files 1975 [J. A. Hoffer, D. G. Severance] 1985 [G. P. Copeland and S. Khoshafian] First suggestion of fully decomposed storage First commercial columnar database: 1996 Sybase IQ First general-purpose DBMS to fully integrate columnar storage and processing: 2012 SQL Server 2012 Source: [Larson et al] HoBT HoBT H A[1] B[1] C[1] H A[6] B[6] C[6] H A[2] B[2] C[2] H A[11] B[11] C[11] H A[7] B[7] C[7] H A[3] Data B[3] Page C[3] H A[96] B[96] C[96] H A[12] B[12] C[12] H A[8] B[8] C[8] H A[4] B[4] C[4] H A[97] B[97] C[97] H A[13] B[13] C[13] H A[9] B[9] C[9] H A[5] B[5] C[5] H A[98] B[98] C[98] H A[14] B[14] C[14] H A[10] B[10] C[10] Row Offset Array H A[99] B[99] C[99] H A[15] B[15] C[15] Row Offset Array H A[100] B[100] C[100] Row Offset Array Row Offset Array Column Store 2

3 Column Store Page A[1] A[2] A[3] A[4] A[1] A[2] A[3] A[4] A[1] A[2] A[3] A[4] A[85] A[86] A[87] A[88] A[5] A[6] B[1] A[5] A[6] A[7] B[2] A[7] A[8] B[3] B[4] B[1] B[2] A[8] B[3] B[4] B[1] A[5] A[6] A[7] B[2] A[8] B[3] B[4] Header A[89] A[90] B[85] A[9] A[9] A[10] B[5] A[10] A[11] B[6] A[91] C[1] B[86] A[92] B[87] B[88] B[5] A[11] A[12] B[6] B[7] C[29] C[2] B[8] C[3] C[4] B[5] A[9] A[10] A[11] A[12] B[6] B[7] C[1] C[30] B[8] C[31] C[32] A[93] A[94] B[89] A[95] A[12] B[90] B[7] C[85] C[2] A[96] B[91] B[8] C[86] C[3] B[92] C[87] C[4] C[88] A[13] A[13] A[14] B[9] A[14] A[15] B[10] C[5] B[9] A[15] A[16] B[10] B[11] C[33] C[6] B[12] C[7] C[8] A[13] A[14] B[9] A[15] A[16] B[10] B[11] C[5] C[6] B[12] C[7] C[8] A[97] A[98] B[93] A[99] A[16] B[94] B[11] C[89] C[6] A[100] B[95] B[12] C[90] C[7] B[96] C[91] C[8] C[92] A[17] A[17] A[18] B[13] A[18] A[19] B[13] B[14] C[9] A[19] A[20] B[14] B[15] C[37] C[10] B[16] C[11] C[12] A[17] A[18] B[13] A[19] A[20] B[14] B[15] C[9] C[10] B[16] C[11] C[12] B[97] A[20] B[98] B[15] C[93] C[10] B[99] B[16] C[94] C[11] B[100] C[95] C[12] C[96] A[21] A[21] A[22] B[17] A[22] A[23] B[17] B[18] C[13] A[23] A[24] B[18] B[19] C[41] C[14] B[20] C[15] C[16] A[21] A[22] B[17] A[23] A[24] B[18] B[19] C[13] C[14] B[20] C[15] C[16] A[24] B[19] C[97] C[14] B[20] C[98] C[15] C[99] C[16] C[100] A[25] A[25] A[26] B[21] A[26] A[27] B[21] B[22] C[17] A[27] A[28] B[22] B[23] C[45] C[18] B[24] C[19] C[20] A[25] A[26] B[21] A[27] A[28] B[22] B[23] C[17] C[18] B[24] C[19] C[20] A[28] B[23] C[18] B[24] C[19] C[20] B[25] B[25] B[26] C[21] B[26] B[27] C[49] C[22] B[27] B[28] C[22] C[23] C[24] B[25] B[26] C[21] B[27] B[28] C[22] C[23] C[24] B[28] C[23] C[24] C[25] C[53] C[26] C[26] C[27] C[27] C[28] C[25] C[26] C[27] C[28] C[28] Forms of Storage Forms of Storage NSM DSM PAX N-ary Storage Model Decomposition Storage Model Partition Attributes Across (Ailamaki & DeWitt, 2001) xvelocity 3

4 xvelocity xvelocity In-Memory Analytics Engine SQL Server Analysis Services PowerPivot xvelocity Memory-Optimized Columnstore Index SQL Server Database Engine xvelocity Memory-Optimized Columnstore Index xvelocity Memory-Optimized Columnstore Index Not an in-memory construct Columns stored independently Uses VertiPaq compression Requires Enterprise Edition Not an Index 4

5 Not an Index No Order No Key Not a bitmap index Segment Segment ~ 1 million rows Each column Aligned between columns Base Table Order preserved Stored in one continuous BLOB Independently compressed Compression 5

6 Compression VertiPaq Proprietary Not Documented Dictionary Encoding Several Algorithms Run Length Encoding Huffman Encoding Lempel- Ziv-Welch Partitioning Partitioning Fully supported Must be aligned to base table Must include partition column Allows for trickle load Redirect: BLOBs 6

7 Demo Redirect: BLOBs Separate Allocation Unit Pages Values Modified B+Tree per Value Structure Demo Structure Columns BLOBs Segments Dictionaries 7

8 Dictionaries 8/8/2012 Stored in Separate BLOB Per Partition and Column Primary Per multiple Segments Secondary or Shared Not shared between Columns or Partitions Creation Creation Memory requirements (4.2 #Cols + 68) DOP + 34 #StringCols MBs Might cause Msg 701,802, 8657 or 8658 Rows per segment N threads -> N smaller segments Parallelism only for > 10 6 rows Cache 8

9 Cache New cache design Ensures contiguous storage of segments in memory Cached on a segment basis Can handle free memory < index size RBAR RBAR Row By Agonizing Row? 2007 Jeff Moden Relational Operator 9

10 Iterator Iterator 8/8/2012 Relational Operator (RelOp) Clustered Index Scan open getrow close 2011 sqlity.net llc, all rights reserved. Relational Operator Relational Operator (RelOp) Columnstore Index Scan open getrow getbatch close 2011 sqlity.net llc, all rights reserved. Batch Processing 10

11 Batch Processing Only on Columnstore Data ~1000 Rows Independent Column-Vectors Data stays Compressed Never Serial Batch-Advantage Batch-Advantage Loop unrolling Memory prefetching Branch prediction Reduced cache misses Reduced TLB misses Batchables 11

12 Demo Batchables Scan Filter Inner hash join Batch hash table build Local hash (partial) aggregation Apollo Apollo Enterprise Edition Only xvelocity m.o.c.i. Vector-based query execution Segment Elimination 12

13 Demo Segment Elimination Column-Segment stores Actual Min Value Actual Max Value Filter out entire Segments Column Filter Bitmap Filter Limitations Limitations No updates (Partition switching possible) Cannot be a clustered index Restricted set of batch mode operators Restricted join operations No filtered columnstore index No computed columns Not supported on [indexed] views Only one per table Max 1024 columns Cannot include sparse columns Cannot enforce primary key or unique constraint Cannot be "ALTER INDEX"ed Cannot "INCLUDE" columns No sort order No seek! No page or row compression, no vardecimal data format No replication, change tracking, CDC (because read only?) Only 21 of 36 Data Types Situation will be improved in future versions 13

14 Best Practices: DOs Include all columns Favor star-joins, aggregations and grouping Put CS index on large tables (Fact & Dim) Prefer small data types Best Practices: DON Ts Large (mostly) unique string value columns UNION ALL of table with and table without columnstore Avoid filters and joins on string columns Avoid OUTER JOIN and NOT IN Literature 14

15 Session 8/8/2012 Literature [Larsen et al] Columnar Storage in SQL Server 2012 (2012, IEEE) Per-Ake Larson, Eric N. Hanson, Susan L. Price [Abadi et al] Column-Stores vs. Row-Stores: How different are they really? (2008, SIGMOD) Daniel L. Abadi, Samuel R. Madden, Nabil Hachem SQL Server Columnstore Index FAQ (microsoft.com) SQL Server Columnstore Performance Tuning (microsoft.com) [Campbell] The coming in-memory database tipping point (2012, blogs.technet.com) David Campbell Perform Scalar Aggregates and still get the Benefit of Batch Processing (microsoft.com) Work Around Performance Issues for Columnstores Related to Strings (microsoft.com) Ensuring Your Data is Sorted or Nearly Sorted by Date to Benefit from Date Range Elimination (microsoft.com) Columnstore Indexes (msdn.microsoft.com) [Rusanu] Inside the SQL Server 2012 Columnstore Index (2012, rusanu.com) Rusanu Consulting llc Multi-Dimensional Clustering to Maximize the Benefit of Segment Elimination (microsoft.com) References References Sebastian Meine SQL Stylist with sqlity.net Materials empty 15