One is the Loneliest Number: Scaling out your Data Warehouse

Size: px

Start display at page:

Download "One is the Loneliest Number: Scaling out your Data Warehouse"

Valerie Cook
6 years ago
Views:

1 One is the Loneliest Number: Scaling out your Data Warehouse Greg Galloway SQL Saturday Dallas #396 BI Edition Page 1

2 Agenda Common data warehouse pain points Analytics Platform System (APS) overview Analytics Platform System performance demos Modern hybrid data warehouse demos Azure SQL Data Warehouse preview SQL Saturday Dallas #396 BI Edition Page 2

3 Missed SLAs Slow Reports Huge Data Sources Ignored Yesterday is Irrelevant Dinky Dev Server Data volumes grow and your data warehouse isn t done loading until after 10am Reports are so slow that users take a coffee break while they run Your data warehouse architect is scared of your biggest data sources like clickstream website traffic logs Yesterday s data is good for many subject areas but users beg for near real-time for some subject areas like inventory and that s hard in a data warehouse Your dev server is a bottleneck among several developers and is underpowered compared to production it is hard to be productive Business Data Model Your data model doesn t reflect the business and doesn t let you answer the questions you need to answer SQL Saturday Dallas #396 BI Edition Page 3

4 PAIN POINT Premise #1: APS Can Grow With You Missed SLAs Scale up has limits APS can scale performance linearly by scaling out With linear scale out, which is more cost effective? Having your team optimize performance? Or throwing hardware at the problem? SQL Saturday Dallas #396 BI Edition Page 4

APS Appliance Hardware Overview Add capacity Add

capacity of several terabytes Add capacity up to 6

Hardware is balanced Start small and grow Internal

disks Largest warehouse Maximum racks 56Gbps 14 TB

5 APS Appliance Hardware Overview Add capacity Add capacity Add capacity Start small with a warehouse capacity of several terabytes Add capacity up to 6 PB of user data (assumes 5x compression) 53 TB 6 PB Hardware is balanced Start small and grow Internal network speed Maximum RAM Maximum cores Maximum disks Largest warehouse Maximum racks 56Gbps 14 TB 896 1,960 6 PB 7 SQL Saturday Dallas #396 BI Edition Page 5

6 PAIN POINT Premise #2: APS = Fast Queries Slow Reports Scan your 1TB fact table in 20 seconds! What s the magic of APS? SQL Saturday Dallas #396 BI Edition Page 6

7 Distribution vs. Replication SQL Saturday Dallas #396 BI Edition Page 7

8 SQL Saturday Dallas #396 BI Edition Page 8

Premise #3: Polybase & semi-structured data PAIN POINT Huge Data Sources Ignored Select Result set Polybase lets you write T-SQL queries joining data warehouse tables to Hadoop or Azure Blob Storage

9 Premise #3: Polybase & semi-structured data PAIN POINT Huge Data Sources Ignored Select Result set Polybase lets you write T-SQL queries joining data warehouse tables to Hadoop or Azure Blob Storage Microsoft Azure HDInsight (Azure Blob Storage) Hortonworks (Windows, Linux), Cloudera Analytics Platform System PolyBase Microsoft HDInsight Polybase lets you write data to Hadoop or Azure Blob Storage with CREATE EXTERNAL TABLE AS SELECT (CETAS) SQL Saturday Dallas #396 BI Edition Page 9

10 SQL Saturday Dallas #396 BI Edition Page 10

Premise #4: Polybase & Stream Analytics PAIN POINT Yesterday is Irrelevant Select Analytics Platform System Result set Azure Stream Analytics can summarize real-time sensor data from blob storage and

11 Premise #4: Polybase & Stream Analytics PAIN POINT Yesterday is Irrelevant Select Analytics Platform System Result set Azure Stream Analytics can summarize real-time sensor data from blob storage and write to blob storage for Polybase Azure Stream Analytics uses a SQL syntax. Example login hack attempt detection: Azure Blob Storage Real-Time Summaries Sensor Data PolyBase SELECT System.Timestamp as Time, IP FROM WebLogs TIMESTAMP BY EventTime WHERE UrlPath='/FailedLogin.aspx' GROUP BY IP, SlidingWindow(second, 60) HAVING count(*)>=60 SQL Saturday Dallas #396 BI Edition Page 11

12 Premise #5: Dev/Test in Azure SQL DW PAIN POINT Dinky Dev Server Azure SQL Data Warehouse is APS as a service in the cloud Scale compute in seconds or pause SQL Saturday Dallas #396 BI Edition Page 14

13 Ziosk Case Study Changing the Game The Restaurant Experience With Data Jeff Johnston Time: 12:45 PM - 1:45 PM Room: MPR 1200 SQL Saturday Dallas #396 BI Edition Page 15

14 Conclusion APS can grow with you APS provides fast queries APS can integrate with semi-structured and real-time streaming data with Polybase to create a Modern Hybrid Data Warehouse Give it a try in the cloud soon with Azure SQL DW SQL Saturday Dallas #396 BI Edition Page 17

15 Greg Galloway SQL Saturday Dallas #396 BI Edition Page 18

Modern Data Warehouse The New Approach to Azure BI

Modern Data Warehouse The New Approach to Azure BI History On-Premise SQL Server Big Data Solutions Technical Barriers Modern Analytics Platform On-Premise SQL Server Big Data Solutions Modern Analytics