SQL Server Evolution. New innovations. George Walters. Sr. Technical Solutions Professional, Data Platform Microsoft

Size: px

Start display at page:

Download "SQL Server Evolution. New innovations. George Walters. Sr. Technical Solutions Professional, Data Platform Microsoft"

Kristin Park
5 years ago
Views:

1 SQL Server Evolution New innovations George Walters Sr. Technical Solutions Professional, Data Platform Microsoft on twitter

2 Microsoft vision for a new era United platform for the modern service provider CUSTOMER DATACENTER Enterprise-grade Global reach, scale, and security to meet business demands Hybrid cloud MICROSOFT AZURE CONSISTENT PLATFORM SERVICE PROVIDER Consistent platform across multiple environments and clouds People-focused Expands technical skill sets to the cloud for new innovation

3 Key trends Expect to have a highly profitable business unit for productizing and commercializing their data by 2020 SQL Server: Most utilized database in the world Gartner 2016

Microsoft is a leader for Magic Quadrant for Operational Database Management Systems 1 Furthest in vision and ability to execute Magic Quadrant for Business Intelligence and Analytics Platforms 2

Database Management Systems, by Donald Feinberg, Merv Adrian, Nick Heudecker, Adam Ronthal, October 2015 [2] *Gartner Magic Quadrant for Business Intelligence and Analytics Platforms, by Josh

4 Microsoft is a leader for Magic Quadrant for Operational Database Management Systems 1 Furthest in vision and ability to execute Magic Quadrant for Business Intelligence and Analytics Platforms 2 Furthest in vision; leader 9 years running Magic Quadrant for Data Warehouse Database Management Solutions 3 A leader for the fifth consecutive year [1] *Gartner Magic Quadrant for Operational Database Management Systems, by Donald Feinberg, Merv Adrian, Nick Heudecker, Adam Ronthal, October 2015 [2] *Gartner Magic Quadrant for Business Intelligence and Analytics Platforms, by Josh Parenteau, Rita L. Sallam, Cindi Howson, Joao Tapadinhas, Kurt Schlegel, Thomas W. Oestreich, February 4, 2016 [3] *Gartner Magic Quadrant for Data Warehouse and Data Management Solutions for Analytics, by Roxane Edjlali and Mark Beyer, February 25, 2016 This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Microsoft. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

5 SQL Server 2016: Everything is built-in built-in built-in built-in built-in built-in Costs reflect EA pricing. $2, SQL Server SQL Server SQL Server #1 #2 #3 Oracle is #5 $480 $120 Microsoft Tableau Oracle SQL Server Oracle MySQL SAP HANA TPC-H non-clustered 10TB Self-service BI per user At massive scale In-memory across all workloads Consistent experience from on-premises to cloud The above graphics were published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Microsoft. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. National Institute of Standards and Technology Comprehensive Vulnerability Database update 2/2016 TPC-H non-clustered results as of 04/06/15, 5/04/15, 4/15/14 and 11/25/13, respectively.

6 The evolution of Microsoft SQL Server XML KPIs SQL Server 2000 Management Studio Mirroring SQL Server 2005 Compression Policy-Based Mgmt Programmability SQL Server 2008 PowerPivot SharePoint Integration Master Data Services SQL Server 2008 R2 AlwaysOn ColumnStore Index Data Quality Services Power View Cloud Connectivity SQL Server 2012 In-Memory Across Workloads Performance & Scale Hybrid Cloud Optimized HDInsight Cloud BI SQL Server 2014 Enhanced Always On In-Memory OLTP Stretch DB Temporal Tables Enhanced Backup to Cloud Polybase Real-Time Operational Analytics Row-Level Security Query Store R Services Always Encrypted Mobile BI SQL Server 2016

7 Updated: Lifecycle Start: Mainstream Support Ends: Current Service Pack SP2: 07/12/2016 Lifecycle Start: Mainstream Support Ends: Current Service Pack SP3: 12/01/ Q 1 Q 2 Q 3 Q 4 Lifecycle Start Date: Q 1 Q 2 Q 3 Q 4 SP2 Lifecycle Start Date: Q 1 Q 2 Q 3 Q 4 Lifecycle Start Date: SP1 Lifecycle Start Date: Q 1 Q 2 Q 3 Q 4 RTM/SP0 Support Ended: SP2 Lifecycle Start Date: Q 1 Q 2 Q 3 Q 4 SP1 Support Ends: SP3 Lifecycle Start Date: Q 1 Q 2 Q 3 Q Q 1 Q 2 Q 3 Q 4 Mainstream Support Ends: Q 1 Q 2 Q 3 Q Q 1 Q 2 Q 3 Q 4 Mainstream Support Ends: Q 1 Q 2 Q 3 Q 4 Extended Support Ends: Extended Support Ends: Lifecycle Start: Mainstream Support Ends: Current Service Pack SP3: SP1 Lifecycle Start Date: SP3 Lifecycle Start Date: RTM/SP0 Support Ended: SP1 Support Mainstream Support Ended: Ended: SP4 Lifecycle Start Date: SP2 Extended Support Requires Current Service Pack Support Ends: Extended Support Ends: Lifecycle Start: Mainstream Support Ended: Current Service Pack SP4: Lifecycle Start: Mainstream Support Ended: Current Service Pack SP4: SP1 Support Ended: Mainstream Support Ended: SP3 Support Ended: SP2 Support Ended: Mainstream Support Ended: Extended Support Requires Current Service Pack SP3 Extended Support Requires Current Service Pack Support Ends: Today Extended Support Ended: Extended Support Ends: For Official Lifecycle Information go to: Future dates are preliminary and subject to change without notice. In the event of an inconsistency between the information here and the information in Microsoft s web-based lifecycle content, the information in Microsoft s web-based lifecycle content is authoritative.

8 How we develop SQL SQL Server and APS Azure SQL Virtual Machines Azure SQL Database Azure SQL Data Warehouse Cloud-first but not cloud-only Use SQL Database to improve core SQL Server features and cadence Many interesting and compelling on-premises cloud scenarios

9 Consistency and integration A consistent experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS On-premises, private cloud, and public cloud SQL Server local, VMs, and SQL Database Common development, management, and identity tools including Active Directory, Visual Studio, Hyper-V, and System Center Scalability, availability, security, identity, backup and restore, and replication Many data sources Reporting, integration, processing, and analytics All supported in the hybrid cloud

Client App No changes to communication stack, parameter passing, result set generation 10-30x more efficient

Algebrizer, Optimizer Proc/Plan cache for ad-hoc T- SQL and SPs Interpreter for TSQL, query plans, expressions

Log latency remains In-Memory OLTP Engine for Memory_optimized Tables & Indexes Query Interop SQL Server.

11 Client App No changes to communication stack, parameter passing, result set generation 10-30x more efficient Natively Compiled SPs and Schema TDS Handler and Session Management In-Memory OLTP Compiler Parser, Catalog, Algebrizer, Optimizer Proc/Plan cache for ad-hoc T- SQL and SPs Interpreter for TSQL, query plans, expressions Access Methods Key Existing SQL Component In-Memory OLTP Component Generated.dll Reduced log bandwidth & contention. Log latency remains In-Memory OLTP Engine for Memory_optimized Tables & Indexes Query Interop SQL Server.exe Buffer Pool for Tables & Indexes Checkpoints are background sequential IO Memory-optimized Table Filegroup Transaction Log Data Filegroup

12 In-memory In the Data Warehouse Data Stored Row-Wise: Heaps, b-trees, Key-Value By using SQL Server 2012 In-Memory ColumnStore, we were able to extract about 100 million records in 2 or 3 seconds versus the 30 minutes required previously. - Atsuo Nakajima Asst Director, Bank of Nagoya Columnstore Index Representation C 1 C 2 C 3 C 4 C 5 C6 In-Memory ColumnStore Both memory and disk Built-in to core RDBMS engine Customer Benefits: x faster Reduced design effort Work on customers existing hardware Easy upgrade; Easy deployment 12

13 Processing an Example

14 Row Groups

15 Segments

16 Compress Each Segment Some Compress More than Others

17 Segment Elimination

18 Segment Elimination

19 In-memory enhancements Operational analytics & enhanced performance In-memory SQL Server data ColumnStore warehouse 2-24 hrs ETL Real-time Fraud detected fraud detection In-memory SQL Server OLTP

20 Operational Analytics with columnstore index B-tree index Key points Create an updateable NCCI for analytics queries Drop all other indexes that were created for analytics No application changes Nonclustered columnstore index (NCCI) Delta row groups Columnstore index is maintained just like any other index Query optimizer will choose columnstore index where needed Performance

21 Always Encrypted Help protect data at rest and in motion, on-premises & cloud Apps Trusted SQL Server SELECT Name FROM Patients WHERE SELECT Name FROM Patients Query Client side Enhanced ADO.NET Column Master Key Result Set Name Jim Gray Result Set Name Jim Gray dbo.patients Name Jane Doe ciphertext SSN 1x7fg655se2e Country USA Jim Gray 0x7ff654ae6d USA John Smith 0y8fj754ea2c USA Column Encryption Key

22 Performance Security Availability Scalability In-Memory OLTP enhancements Greater T-SQL surface area, terabytes of memory supported, and higher number of parallel CPUs Operational Analytics Insights on operational data; works with In-Memory OLTP and disk-based OLTP Query Store Monitored, optimized query plans Always Encrypted Sensitive data remains encrypted at all times, with ability to query Dynamic Data Masking Real-time obfuscation of data to prevent unauthorized access Row-Level Security Fine-grained access control for table rows Other enhancements Audit success/failure of database operations Basic Availability Groups With SQL Server 2016 Standard Edition Enhanced Always On Distributed Availability Groups, automatic replica seeding, distributed transactions, automatic failover, load balancing, manageability Backup enhancements Managed backup to Azure, Database Recovery Advisor Windows Server Support Support of new features in Windows Server 2016 Live migration Faster live migration, live migration for non-clustered VMs Scalability enhancements Hardware acceleration for TDE, parallelized decryption, TempDB optimization, and more Temporal Tables Query data as points in time and recover from accidental data changes and application errors TDE support for storage of In-Memory OLTP tables Enhanced auditing for OLTP with ability to track history of record changes

23 PolyBase Query relational and non-relational data with T-SQL Quote: T-SQL query ************************ ********************** ********************* SQL Server Hadoop ********************** *********************** $ Name DOB State Jim Gray 11/13/58 WA Ann Smith 04/29/76 ME

24 Built-in advanced analytics In-database analytics at massive scale Example Solutions Sales forecasting Warehouse efficiency Predictive maintenance Credit risk protection Extensibility R Integration R? New R scripts Analytic Library T-SQL Interface Relational Data Built-in to SQL Server Data Scientist Interact directly with data Data Developer/DBA Manage data and analytics together Microsoft Azure Marketplace

25 Rich visualizations on mobile devices Business insights through rich visualizations on any device. Native apps for Windows, ios and Android

26 SQL Server 2016 BI at GA CREATE MANAGE CONSUME Datazen Publisher SQL Server Report Builder SQL Server Reporting Services Native mode Datazen Server now a component of SSRS Datazen Windows App Datazen Phone App Datazen Android App Datazen ios App Report Manager (paginated and mobile reports) Report Designer in SQL Server Data Tools SQL Server Reporting Services SharePoint integrated mode SSRS reports rendered as PDF Pin paginated report elements to Power BI dashboards SharePoint Web

27 CREATE MANAGE CONSUME Microsoft Excel Power BI Desktop Power BI Power BI Desktop Microsoft Excel Mobile reports Paginated reports Power BI Web Portal Windows App Android App Datazen Publisher** ios App SQL Server Report Builder Report Designer in SQL Server Data Tools SQL Server BI Reporting Services Power BI Desktop Mobile reports Paginated reports Report Server Web portal (all BI report types) SharePoint Web

28 Deeper insights across data Store your data make it accessible Analyze data Share insights PolyBase Insights from data across SQL Server and Hadoop with the simplicity of T-SQL Data warehousing Modern data warehouse, SMP and MPP SQL Server solutions Data formats and types Import/export JSON, XML, spatial, FILESTREAM, RBS Integration Services enhancements Interoperability with Azure Data Factory, Azure Storage Connector, and HDInsight New and enhanced data connectors: HDFS, JSON, Oracle/Teradata, and more Enhanced MDS Excel add-in 15x faster, moregranular security roles, archival options for transaction logs, and reuse entities across models Real-time operational analytics Running of analytical workloads concurrently with OLTP workloads through columnstore indexes Analysis Services enhancements Many-to-many support with bi-directional cross filtering, improved performance Advanced analytics with R Integrated R Services, predictive analytics, big data analytics Mobile BI Mobile reports on SSRS with Mobile Report Publisher Reporting Services Modern web browser support (HTML5, CSS3) Power BI Connection of Power BI to on-premises SSAS cubes with Power BI Analysis Services Connector

29 Analysis Services Architectural Overview CLIENT TOOLS BI SEMANTIC MODEL Queries Data model Business logic Data access Use SQL Server Data Tools for Visual Studio to create BI semantic models DATA SOURCES

30 Hyperscale cloud Hyperscale features Simplicity Consistency Stretch Database Stretch operational tables in a secure manner into Azure for cost-effective historic data availability. Works with Always Encrypted and Row-Level Security. High availability Distributed availability groups add flexibility to HA/DR. Add Azure Replica Wizard makes it easy to implement. Enhanced backup to Azure Enhanced backup includes faster restore times and 50% reduction in storage. Supports larger DBs with block blobs and custom backup schedule with local staging. Easy migration of on-premises SQL Server Simple point-and-click migration to Azure Suite of advisors for upgrading to SQL Server 2016 SQL Server 2016 Upgrade Advisor in the adoption of new SQL Server features Common development, management, and identity tools Including Active Directory, Visual Studio, Hyper-V, and System Center Consistent experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS

31 Stretch SQL Server into Azure Stretch warm and cold tables to Azure with remote query processing Microsoft Azure Jim Gray ox7ff654ae6d 3/18/2005 Order history Stretch to cloud Name SSN Date Customer data Jane Doe cm61ba906fd 2/28/2005 Jim Gray ox7ff654ae6d Product data 3/18/2005 John Smith i2y36cg776rg 4/10/2005 Order History Bill Brown nx290pldo90l 4/27/2005 Sue Daniels ypo85ba616rj 5/12/2005 Sarah Jones bns51ra806fd 5/22/2005 Jake Marks mci12hh906fj 6/07/2005 Query App

32 SQL Server, Azure VMs, SQL Database On-premises Hybrid cloud Options: Shared lower cost Virtual Infrastructure as a service SQL Platform as a service SQL SQL Server in Azure VM Virtualized machines Software as a service SQL Database Virtualized databases SQL Server on physical machines SQL Server in on-premises VMs (private cloud) SQL Server in Azure VMs (public cloud) Physical SQL Server Private Cloud Virtualized machines + appliances SQL Database (public cloud) Dedicated higher cost SQL SQL Server Physical machines (raw iron) Off-premises Higher administration Lower administration Consistency

33 SQL Database Service Tiers

34 SQL DB V12

35 P11 B S0 S1 S2 S3 P1 P2 P4 P6

36 Max per-database burst level

37 Azure SQL Database Getting Started Azure SQL Database documentation GitHub Code Samples Elastic Database Pools

38 Enterprise grade data warehousing On-premises, cloud or hybrid SMP Data marts SQL Server MPP Petabytescale Petabytescale Analytics Platform System + SMP Data marts SQL Server in Azure VM MPP Azure SQL Data Warehouse Scale to MPP on-premises or in the cloud Simple T-SQL to manage structured and unstructured data ½ the cost of Exadata Consistent experience

39 Cloud On-Premises Data Warehouse vision & portfolio Comprehensive. Connected. Choice. Relational Beyond relational Microsoft SQL Server Microsoft Analytics SQL Server 2014 Hortonworks HDP SQL Server 2016 Analytics Platform System Fast Track for SQL Server Analytics Platform System Azure Data Factory Federated Query Azure Machine Learning PowerBI Microsoft Azure SQL Server for DW in Azure VM Azure SQL Database Azure SQL Data Warehouse Microsoft Azure Azure HDInsight Azure Data Lake

40 Analytics Platform System Microsoft s big data appliance MPP SQL Server PolyBase Hadoop Pre-built HW appliance Massively Parallel Processing (MPP) to scale to 6 PBs In-memory columnstore for 100x speed improvement Dedicated region for Hadoop PolyBase enables joining relational & non-relational data Available from HP, Dell and Quanta AU3/AU4

41 Introducing Azure SQL Data Warehouse Separate storage & compute Elastic scale Integration with Power BI, Azure Data Factory, and Machine Learning services Scale-out relational data warehouse

42 Azure SQL Data Warehouse Elastic scale & performance Scales to petabytes of data with MPP processing Resize compute nodes <1 minute Faster time to insight than other SMP offerings Designed for on-demand workload End-to-end platform built for the cloud Integrated with Azure platform and other Microsoft services Enables hybrid solutions Built on SQL Server experience & technology Market leading price/ performance Simple compute & storage billing Pay for what you need High performance without rewriting applications Low cost for latent data Infrastructure, management and support provided

43 Data warehouse in Azure VMs

44 Power of integration Hadoop SQL Azure SQL Data Warehouse Azure SQL Database App Service Intelligent App Azure Machine Learning Power BI

45 Azure Data Lake Batch, real-time and interactive analytics made easy Analytics service Unstructured ` Azure Data Lake YARN WebHDFS Store Semi-Structured Managed clusters (HDInsight) Structured Principles Maximize return on accessible data Reduce time to value Reduce time to insight Approach Productivity day one (Developers, Scientists, Analysts) Open (Yarn, HDFS), designed for the cloud All data available for analysis Leverages existing skills, use SQL, Spark, Hive, Storm, Hbase Dynamically scales to meet your business objectives Managed and supported with an enterprise grade SLA

46 SQL Server 2016

processing Modeling Machine learning The Microsoft data

47 Applications Reports Dashboards Natural language query Mobile Orchestration Information management Complex event processing Modeling Machine learning The Microsoft data platform Relational Non-relational NoSQL Streaming Internal & external

48 Thank Resources:

SQL Server Everything built-in

SQL Server Everything built-in 2016 Everything built-in 2016: Everything built-in built-in built-in built-in built-in built-in $2,230 80 70 60 50 43 69 49 40 30 20 10 0 34 6 0 1 29 4 22 20 15 5 0 0 2010 2011 2012 2013 2014 2015 18 3