Columnstore Technology Improvements in SQL Server Presented by Niko Neugebauer Moderated by Nagaraj Venkatesan

Transcription

1 Columnstore Technology Improvements in SQL Server 2016 Presented by Niko Neugebauer Moderated by Nagaraj Venkatesan

2 Thank You microsoft.com hortonworks.com aws.amazon.com red-gate.com Empower users with new insights through familiar tools while balancing the need for IT to monitor and manage user created content. Deliver access to all data types across structured and unstructured sources. Hortonworks develops, distributes and supports the only 100% open source distribution of Apache Hadoop explicitly architected, built and tested for enterprise grade deployments. It is the only Hadoop-based platform available on both Linux and Windows. Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale Microsoft SQL Server databases in the cloud. Redgate makes ingeniously simple tools for Microsoft technology professionals working with SQL Server,.NET, Visual Studio, Azure, TFS. Trusted by 91% of the Fortune

3 JOIN PASS PASS is a not-for-profit organization which offers year-round learning opportunities to data professionals Membership is free, join today at Access to online training and content Join Local Chapters and Virtual Chapters Enjoy discounted event rates Get advance notice of member exclusives 3

4 Niko Neugebauer Microsoft Data Platform Professional OH22 ( Data Platform MVP Founder of a couple of Portuguese PASS Chapters (SQLPort, BITuga, Porto.Data) Creator of CISL Open Source Columnstore Indexes Scripts Library Webcaravela

5 Our Columnstore Agenda Today Operational Analytics Data Warehousing High Availability Batch Mode Data Loading Improvements Performance Improvements Change Tracking Maintenance Improvements (Row Group Merging) Monitoring Improvements (DMV, Extended Events, Performance Counters) Configuration Improvement 5 5

6 Traditional Operational Analytics Architecture BI analysts Key issues Complex implementation Requires two servers (capital expenditures and operational expenditures) Data latency in analytics More businesses demand; requires real-time analytics IIS Server 6 6

7 Minimizing Data Latency for Analytics BI analysts IIS Server Benefits No data latency No ETL No separate data warehouse Challenges Analytics queries are resource intensive and can cause blocking Minimizing impact on operational workloads Sub-optimal execution of analytics on relational schema 7 7

8 Operational Analytics IIS Server BI analysts What is operational analytics and what does it mean to you? Operational analytics with disk-based tables Operational analytics with In-Memory OLTP 8 8

9 Traditional Operational Analytics Problems: Costs $ Integration Problems (data types, constraints, network problems, etc) Delay for getting the actual data 9 9

10 Modern Operational Analytics note: Nothing substitutes analytics queries performance, possible using schemas customized (Star/Snowflake) and/or pre-aggregated cubes 10 10

11 DWH & Operational Analytics Unimportant Messages Important Messages Need Data Now Storm, Kafka OLTP Analytics Process Later Hadoop & co Data Warehousing 11

12 Operational Analytics in SQL Server 2016 Disk-Based You are looking to improve the speed of the current OLTP System Analytics You do not have a lot of RAM in your System and can t upgrade it now. You want to go faster You have enough RAM InMemory 12 12

13 Operational Analytics in SQL Server 2016: 13 13

14 Clustered Columnstore in SQL Server

15 Operational Analytics Architecture in

16 Operational Analytics Query Processing 16

17 Demo Time!

18 Filtered Columnstore You can actually create a filtered Nonclustered Columnstore Index, specifying the area for the Hot & for the Warm Data: 18

19 Demo Time!

20 In-Memory Operational Analytics 20

21 Demo Time!

22 In-Memory Operational Analytics Detail: sys.sp_memory_optimized_cs_migration compresses data from InMemory OLTP Table into In-Memory Columnstore 22

23 Data Warehousing

24 Clustered Columnstore (Data WareHousing) Nonclustered Indexes Primary & Foreign Keys Nonclustered Secondary Rowstore Locking 24

25 Further Columnstore Improvements Data Loading Improvements High Availability Batch Mode Performance Improvements Change Tracking Maintenance Improvements Monitorting Improvements (DMV, Extended Events, Performance Counters) 25

26 Data Loading Improvements Parallel Data Loading finally enabled SIMD support Delta-Stores are not Page-Compressed!!! Under NCCI Delta-Stores are automatically increasing their max size with each iteration (1,2,4,8,16,32 Million Rows) 26

27 Data Loading Landing Improvement Compression Delay alter index PK_NCCi_test_inline on dbo.ncci_test_inline set (COMPRESSION_DELAY = 60 Minutes); 27

28 SSIS 2016 Improvements AutoAdjustBufferSize - True Warning: It s not enough, you will need to set up the maximum number of rows per buffer, otherwise your performance will be extremely slow! DefaultBufferMaxRows =

29 Parallel Data Loading 29

30 High Availability Readable Secondaries for Availability Groups through Snapshot & Read Committed Snapshot Isolation Levels support 30

31 Batch Mode Improvements Batch Mode support for 1 core execution plan operators Batch Mode support for the Sort operator Batch Mode support for the Multiple Distinct Count operations 31

32 Batch Mode for Window functions 32

33 Batch Performance Improvements: Simple Aggregate Predicate Pushdown String Predicate Pushdown for Index Scan operator in Batch Mode 33

34 String Predicate Pushdown 34

35 Change Tracking in SQL Server 2016 Columnstore Change Tracking Change Data Capture Temporal (2016) Clustered no no yes Nonclustered yes yes yes InMemory no no yes 35

36 Maintenance Improvements: Better Index Reorganize (removes deleted rows, less memory pressure) 36

37 Row Group Merging Merging Types Self-Merging (this is when Row Group is being recompressed eliminating obsolete row versions that are found in Deleted Bitmap) Intergroup Merging (when the sum of active rows between multiple Row Groups is less then , then their informations can be merged into a new single Row Group) Qualifying Criterias Being a compressed Row Group (Delta-Stores & Tail Row Groups are not qualified) A Row Group has to have at least 10% of its data being marked as obsolete in Deleted Bitmap (for InMemory this threshold is set on 90% actually). A Row Group should not being trimmed because of the dictionary pressure

38 New DMVs: sys.dm_column_store_object_pool sys.dm_db_column_store_row_group_physical_stats sys.dm_db_column_store_row_group_operational_stats sys.internal_partitions 38

39 Configuration Improvement (Large Pages) Trace Flag 834 should be supported in SQL Server