Columnstore in real life

Transcription

1 Columnstore in real life

2 Enrique Catalá Bañuls Computer Engineer Microsoft Data Platform MVP Mentor at SolidQ Tuning and

3 Agenda What is real-time operational analytics Rowstore VS Columnstore Data loading Tricks Administration

4 Real-time operational analytics Environment where coexist mixed OLAP/OLTP solutions Low user impact of BI users querying operational data

5 Traditional analytic solution BI analysts Key aspects Full stack Licensing (price) Latency of data available to consume IIS Server

6 Real-time operational analytics OLTP workload IIS Server This does nt replace a full BI Solution SSAS with precalculated data is the fastest way If you need multiple data sources it will doesn t help you either

7 Real use case: SolidQ s QueryAnalytics number of events today (october s 6th) Billions of rows Inserting data from servers with > events/sec multithread Brutal compression Avg 15bytes per row Data instantly available for querying No locking, seriously

8 Columnstore ROWSTORE Optional compression It returns all index columns B-Tree structure COLUMNSTORE Totally different storage (not B-Tree) Compression by default It returns only referenced columns Header and data

9 Rowstore vs columnstore Imagen de Niko Neugebauer

10 Rowstore vs columnstore GetNext() Send data to CPU Implemented by all operators Row Mode GetNext() row by row invocations OLTP typical Batch Mode GetNext() ~900 rows by 900 rows 10x-100x improvement It requires columnstore SQL2019 introduced batch mode for Rowstore!!!

11 Data loading Load without preorder( put an ORDER BY!) batch_size > direct to rowgroup < direct to delta store (bad) Always use bulk load without TABLOCK It s always minimal logged Its multithread enhanced DO NOT USE TABLOCK

12 Demo Storage Data loading Real-time operational analytics

13 Data loading Rowgroups an be from until Remember use bulk insert. INSERT INTO <table-name> VALUES (<set of values>) always will do a delta insert store (bad) Rows to Bulk Load Rows Added to the Compressed Rowgroup Rows Added to the Delta Rowgroup 102, , , ,000 1,048,577 1,048,576 Rowgroup size: 145,000 Rowgroup size: 1,048, ,252,152 2,252,152 0 Rowgroup sizes: 1,048,576, 1,048,576, 155,000.

14 AlwaysOn and columnstore OLTP data loading AlwaysOn Availability Group Analytic data load Key aspects Use different servers depending on which data are you consuming BI solutions will prefer columnstore

15 Optimum candidates for NCCIs +1M rows With lots of aggregation clauses With lots of table scans With lots of columns Minimum amount analytics Lots of casuistic Hard to find a covered index

16 Delete bitmap Truco: Non-clustered columnstore index (NCCI) B-tree index Updatable since SQL 2016 Any other index needed Mantained as any other index No application changes Can be filtered Delta rowgroups Non-clustered columnstore index

17 Delete bitmap Trick: Compression delay OLTP workload Control how much time to stay in delta store -Max = (7 days) -Util for HOT/WARM (time based) < 6 hours HOT (delta store) B-tree index HOT Delta rowgroups Non-clustered columnstore index

18 Demo Is it worth to upgrade to SQL2016+? Enterprise vs Standard?

19 Maintenance Reorganize Rebuild Phisical delete only if we delete >10% It compress everithing It combines one or two compressed rowgroups until find rows WITH (COMPRESS_ALL_ROW_GROUPS) It locks the whole table 2017 with online NCCI Best option when suffering fragmentation (deletes)

20 Feature comparison Columnstore Index Feature SQL Server 2012 (11.x) SQL Server 2014 (12.x) SQL Server 2016 (13.x) SQL Server 2017 (14.x) SQL Database Batch execution for multi-threaded queries yes yes yes yes yes yes Batch execution for single-threaded queries yes yes yes yes Archival compression option yes yes yes yes yes Snapshot isolation and read-committed snapshot isolation yes yes yes yes Specify columnstore index when creating a table yes yes yes yes Always On supports columnstore indexes yes yes yes yes yes yes SQL Data Warehouse Most features requires compatibility level 130 applied at database level Always On readable secondary supports read-only nonclustered columnstore index yes yes yes yes yes yes Always On readable secondary supports updateable columnstore indexes yes yes Read-only nonclustered columnstore index on heap or B-tree yes yes yes 1 yes 1 yes 1 yes 1 Updateable nonclustered columnstore index on heap or B-tree yes yes yes yes Additional B-tree indexes allowed on a heap or B-tree that has a nonclustered columnstore index yes yes yes yes yes yes Updateable clustered columnstore index yes yes yes yes yes B-tree index on a clustered columnstore index yes yes yes yes Columnstore index on a memory-optimized table yes yes yes yes Nonclustered columnstore index definition supports using a filtered condition yes yes yes yes Compression delay option for columnstore indexes in CREATE TABLE and ALTER TABLE yes yes yes yes Columnstore index can have a non-persisted computed column yes 1 To create a read-only nonclustered columnstore index, store the index on a read-only filegroup.

21 Summary Load properly Minimum 1M rows Affected by modifications NO LOCK OLTP+OLAP best aproach

22 Preguntas