Přehled novinek v SQL Server 2016 Martin Rys, BI Competency Leader martin.rys@adastragrp.com https://www.linkedin.com/in/martinrys 20.4.2016 1
BI Competency development 2
Trends, modern data warehousing 3
Current DWH/BI challenges Business Intelligence / Data Delivery Reports Dashboards Semantic Models Business Analytics Excel Business Analytics Big Data New data sources & types, increasing data volumes Internet of Things (IoT) Sensors & devices & computers Cloud / hybrid models Integration of on-premise and cloud services Enterprise DWH SMP and MPP ODS Operational reporting Information Management Data Transformation / OLAP Data Workflow Orchestration Data Management Relational / Structured data Right information to all users, in relevant tools & format, at right time Fast data exploration, Agile BI Operational / Near-real time data processing and analytics Advanced analytics Self-service and Mobile BI BICC New scope of work, roles, skills, tools 4
Current DWH/BI Business Intelligence / Data Delivery Reports Dashboards Semantic Models OLAP Enterprise DWH SMP and MPP ODS Operational reporting Information Management Data Transformation / Data Workflow Orchestration Data Management Relational / Structured data 5
Modern data warehousing Big Data Platform Business Intelligence / Data Delivery Reports Dashboards Semantic Models New data sources & types, increasing data volumes Hadoop, NoSQL DWH augmentation Move from DWH centered architecture to heterogeneous architecture OLAP Enterprise DWH SMP and MPP ODS Operational reporting Big Data Platform Data Lake Hadoop, NoSQL Information Management Data Transformation / Data Workflow Orchestration Data Management Relational / Structured data Big data Unstructured data Streaming 6
Modern data warehousing Business Intelligence / Data Delivery Reports Dashboards Semantic Models Event Internet of Things (IoT) Sensors & devices & computers Field Gateway, IoT Network Hot and Cold Path IoT Hubs Complex Event (CEP) OLAP Notifications Enterprise DWH SMP and MPP ODS Operational reporting Big Data Platform Data Lake Hadoop, NoSQL BI / Application Integration Complex Event Data Transformation / Information Management Data Workflow Orchestration Data Management Event Ingestion IoT Network Field Gateway Relational / Structured data Big data Unstructured data Streaming 7
Modern data warehousing Business Intelligence / Data Delivery Reports Dashboards Semantic Models OLAP In-memory Columnar Event Notifications Operational / Near-real time data processing and analytics In-memory technologies In-memory OLTP In-memory DW In-memory Analytics In-memory Event processing Enterprise DWH SMP and MPP In-memory technologies ODS Operational reporting In-memory technologies Big Data Platform Data Lake Hadoop, NoSQL BI / Application Integration Complex Event Data Transformation / Information Management Data Workflow Orchestration Data Management Event Ingestion IoT Network Field Gateway Relational / Structured data Big data Unstructured data Streaming 8
Modern data warehousing Advanced analytics R Business Intelligence / Data Delivery Reports Dashboards In-database data mining Self-service advanced analytics Semantic Models Advanced Analytics Perceptual / cognitive intelligence Event Perceptual / cognitive intelligence OLAP In-memory Columnar Machine Learning In-database Data Mining, R Recognition of human interaction and intent Notifications Text mining, Natural language processing, Sentiment analysis Image analytics Enterprise DWH SMP and MPP In-memory technologies ODS Operational reporting In-memory technologies Big Data Platform Data Lake Hadoop, NoSQL BI / Application Integration Complex Event Speech processing Event Ingestion Information Management Data Transformation / Data Workflow Orchestration Data Management IoT Network Field Gateway Relational / Structured data Big data Unstructured data Streaming 9
Modern data warehousing Reports Semantic Models OLAP In-memory Columnar Enterprise DWH SMP and MPP In-memory technologies Business Intelligence / Data Delivery Dashboards and visualizations ODS Operational reporting In-memory technologies Mobile BI Advanced Analytics Perceptual / cognitive intelligence Machine Learning In-database Data Mining, R Information Management Self-service BI Recognition of human interaction and intent Big Data Platform Data Lake Hadoop, NoSQL Real-time Dashboards Event Notifications BI / Application Integration Complex Event Event Ingestion Self-service BI Heterogeneous data connectivity and transformation Semantic layer Data visualization, interactive dashboards Report delivery, publishing Mobile BI Optimized dashboards for mobile devices Support for Windows, ios and Android Real-time Dashboards Event monitoring Data Transformation / Data Workflow Orchestration Data Management IoT Network Field Gateway Relational / Structured data Big data Unstructured data Streaming 10
Modern data warehousing BICC New scope of work Reports Business Intelligence / Data Delivery Dashboards and visualizations Mobile BI Self-service BI Real-time Dashboards New roles New skills New tools Semantic Models OLAP In-memory Columnar Advanced Analytics Perceptual / cognitive intelligence Machine Learning In-database Data Mining, R Recognition of human interaction and intent Event Notifications Enterprise DWH SMP and MPP In-memory technologies ODS Operational reporting In-memory technologies Big Data Platform Data Lake Hadoop, NoSQL BI / Application Integration Complex Event Data Transformation / Information Management Data Workflow Orchestration Data Management Event Ingestion IoT Network Field Gateway Relational / Structured data Big data Unstructured data Streaming 11
SQL Server 2016 Overview 12
SQL Server 2016 End-to-End DW & Big Data Platform, Driving Analytics on any Data SharePoint Power Pivot Gallery, Power View Excel + Power BI add-ins Query, Pivot, View, Map Power BI Desktop Power BI Portal Power BI Mobile App Excel Data Mining Azure ML Analytics Platform System (APS) 14
Integration with Hadoop using Polybase PolyBase provides a scalable, T-SQL compatible query processing framework for combining data from SQL Server and Hadoop / Azure blob storage SharePoint Power Pivot Gallery, Power View Excel + Power BI add-ins Query, Pivot, View, Map Power BI Desktop Power BI Portal Power BI Mobile App Excel Data Mining Azure ML Analytics Platform System (APS) 15
Possible Hadoop utilization and integration with DWH Reporting Analysis Data Mining Self-service BI Semantic models Operational systems Data Marts DWH Data Stage ETL External data 5 5 1 2 Relational Data 4 3 (1) Enterprise DWH consolidates data from Operational systems (2) Very large / unstructured / new data and events are stored in Hadoop Data processing, exploration, analytics Complex Event (3) After data processing and analysis in Hadoop aggregated information are provided to DWH (4) Large / cold / historical data are moved to Hadoop, Data archiving (5) Data stored in both DWH/BI and Hadoop are available for BI tools User friendly BI tools (including MS Excel) can be used for analysis of data stored in Hadoop. 16
PolyBase View in SQL Server 2016 PolyBase allows you to use Transact-SQL (T-SQL) statements to access data stored in Hadoop or Azure Blob Storage and query it in an ad-hoc fashion. Query Result Execute T-SQL queries against relational data in SQL Server and semi-structured data in HDFS and/or Azure Leverage existing T-SQL skills and BI tools to gain insights from different data stores Expand the reach of SQL Server to Hadoop(HDFS) Optimized for data warehousing workloads and intended for analytical query scenarios PolyBase is also implemented in: Analytics Platform System Azure SQL Data Warehouse 17
In-memory technologies Faster transactions, Faster Queries, Faster Insights SharePoint Power Pivot Gallery, Power View Excel + Power BI add-ins Query, Pivot, View, Map Power BI Desktop Power BI Portal Power BI Mobile App Excel Data Mining Azure ML Analytics Platform System (APS) 18
In-memory technologies overview Memory-optimized OLTP engine / Hekaton In-memory Columnstore engine / xvelocity Row store Objects in-memory (memory-optimized tables, natively compiled stored procedures) Integrated into SQL Server Tables fully transactional, durable and can be accessed using standard T-SQL Columnar store Data stored in columnar format / database Integrated into SQL Server C1 C2 C3 C4 C5 Data stored as columns Data stored as rows Ideal for OLTP Efficient operation on small set of rows Can significantly improve OLTP database application performance (short-running transactions) Workloads where there are many inserts and updates, and business logic in stored procedures Ideal for DW workload Achieves breakthrough performance for analytic queries Improved compression More data fits in memory Optimized for CPU utilization Reduced I/O: Fetch only columns needed 19
In-memory technologies overview Faster Transactions Faster Queries Faster Insights In-memory OLTP In-memory DW In-memory Analytics Memory-optimized OLTP engine / Hekaton Usage OLTP: Highly frequent Insert/Updates, stored procedure that does heavy calculations Operational analytics: queries that aggregate data for analysis Columnstore engine / xvelocity Columnstore index For storing and querying large tables Clustered columnstore index Nonclustered columnstore index Usage Data Warehousing Operational analytics Columnstore engine / xvelocity In-memory analytics engine Provides BI semantic layer Usage SSAS Tabular / Power Pivot for SharePoint Power Pivot (Excel / Power BI Desktop) Note: SSD Bufferpool Extension can also be used to improve OLTP performance Up to 30x faster transaction processing Up to 100x faster queries / stored procedures Improves query performance and data compression by up to 10x Greater performance and data compression 20
Advanced Analytics Enterprise-class big data analytics platform for R SharePoint Power Pivot Gallery, Power View Excel + Power BI add-ins Query, Pivot, View, Map Power BI Desktop Power BI Portal Power BI Mobile App Excel Data Mining Azure ML Analytics Platform System (APS) 21
Community Commercial Revolution R Open Revolution R Enterprise R Open R Server Authoring tool: R Tools for Visual Studio Python / Node.js for Visual Studio Azure ML SDK 22
Advanced Analytics landscape Azure Machine Learning 23
Business Intelligence Deliver right information to all users, in a way that is highly relevant and useful to each, at right time SharePoint Power Pivot Gallery, Power View Excel Data Mining Azure ML Analytics Platform System (APS) 24
Roadmap Powerful, familiar BI tools for everyone Put the power of data in the hands of every business and person on the planet Harmonizing report types Paginated reports built with SQL Server Data Tools or SQL Server Report Builder. Interactive reports built with Power BI Desktop. Mobile reports are based on Datazen technology. Analytical reports and charts created with Excel. Reporting Services is on-premises solution for BI report delivery Publish Power BI Desktop Reports on-premises Unified Mobile BI experience Symmetry across on-premises and cloud 25
BI Competency development 26
Thank you! Restrictions for public release and use: This document can comprise confidential information. As such it may not, without Adastra s prior consent, be copied or transferred. Important: All brands and names of products given in this documentation are or can be registered trademarks of their owners. 2016 Adastra, all rights reserved. ADASTRA CZECH REPUBLIC Adastra, s.r.o. Karolinská 654/2, 186 00 Praha 8 Tel.: +420 271 733 303 infocz@adastragrp.com www.adastra.cz ADASTRA GROUP North America 8500 Leslie St. Markham, Ontario, L3T 7M8 Tel: +1 905 881 7946 info@adastragrp.com 27