Gerhard Brueckl. Deep-dive into Polybase

Size: px

Start display at page:

Download "Gerhard Brueckl. Deep-dive into Polybase"

Christian Hopkins
5 years ago
Views:

1 Gerhard Brueckl Deep-dive into Polybase

2 Sponsors Many thanks to our sponsors, without whom such an event would not be possible.

3 Sponsors Many thanks to our sponsors, without whom such an event would not be possible.

4 "Das super nerdige Solisyon Quiz" Track 1 13:40 14:10

5 Deep-dive into Polybase

6 About Me Gerhard Brückl From Austria Working with Microsoft Data Platform since 2006 Mainly focused on Analytics and Reporting Big Data / MPP Microsoft blog.gbrueckl.at gerhard@gbrueckl.at

7 Deep-dive into Polybase Big Data Polybase Introduction Setup Scalability Use-Cases

9 Machine Generated New kinds of Data Big Data The way we generate data is changing ERP CRM Sales Customer Product Date Amount Cust1 ProdA Oct Cust2 ProdB Oct CustN ProdZ Oct-01 Sensors Logs Social Media Digital Media

10 Big Data The way we store data is changing HDD/SSD Raid HDD/SSD Array SAN / NAS Multi-Node Cluster

11 Big Data The way we analyze/use data is changing Machine Learning Advanced Analytics Static List Reports Interactive / AdHoc Forecasting

12 Big Data and the classic DWH Combining the old data with the new data Reporting Analysis SQL Server Reporting Services Data Warehouse SQL Server RDBMS Data Integration / ETL SAP ERP CRM Transactional Data External Systems SQL Server Integration Services

13 Big Data and the classic DWH Combining the old data with the new data Reporting Analysis Advanced Analytics SSRS / Power BI Data Warehouse SQL Server 2016 Data Integration / ETL SSIS / Polybase SAP ERP CRM Transactional Data External Systems Sensors Web Logs Social Digital Media Media Machine generated Semi-structured Data * * Requires further processing

15 What is Polybase Polybase is a Technology which allows us to access data which resides in a distributed file system (like HDFS) in a traditional way using regular SQL commands. MSDN Analytical Platform System (APS, former PDW) Separate Storage and Compute Hot / Cold Storage Scalability

16 SQL Server 2016 Polybase Engine PolyBase DMS

17 SQL Server Scale-Out Group Head Node Polybase Engine Polybase DMS Polybase DMS Polybase DMS Polybase DMS

18 Scalability Head Node Polybase Engine PolyBase Engine PolyBase Engine PolyBase Engine Polybase DMS Polybase DMS Polybase DMS Polybase DMS

19 Supported Sources File Systems Hadoop Distributed File System (HDFS) Hortonworks Data Platform (HDP) Cloudera (CDH) Azure Blog Store (Azure Data Lake Store) File Formats Delimited Text (CSV, TSV, ) ORC RC Parquet

21 Setting Up Polybase Requires JAVA Development Kit Use a Domain Account

22 PolyBase Internal Databases

23 Configure Hadoop Connectivity Different Sources require different settings --Configure Hadoop connectivity = 'hadoop = { 0-7 } [;] RECONFIGURE; -- REBOOT Instance Value Description 0 Disable Hadoop connectivity 1 HDP 1.3 on Windows Azure Blob Store 2 HDP 1.3 on Linux 3 CDH 4.3 on Linux 4 HDP 2.0 on Windows Azure Blob Store 5 HDP 2.0 on Linux 6 CHD 5.1+ on Linux 7 (default) HDP 2.1+ on Linux HDP 2.1+ on Windows Azure Blob Store

24 New RDBMS Objects SQL Query Table External Table View Credential External Data Source External File Format

25 External Objects

26 MasterKey and Credential MasterKey Encrypt sensitive Information MSDN -- Create a Master Key using my own password. CREATE MASTER KEY ENCRYPTION BY PASSWORD='Pass@word1!'; Credential Access to DataSource -- Create database Credential CREATE DATABASE SCOPED CREDENTIAL CRED_AzureStorage WITH IDENTITY = 'gbdomaindata', --> Storage Account SECRET='<AccessKey>'; --> AccessKey

27 External Data Sources Reference to external Storage Service Type Azure Blob Store Hadoop / HDFS Location Hortonworks Cloudera MSDN -- Create the External Data Source using Credential CREATE EXTERNAL DATA SOURCE AzureStorage WITH ( TYPE = HADOOP, LOCATION = 'wasbs://<container>@<name>.blob.core.windows.net', CREDENTIAL = CRED_AzureStorage );

28 External File Format Definition of File Format Format Type Delimited Text ORC RCFile Parquet Data Compression Format Options MSDN -- Create External File Format CREATE EXTERNAL FILE FORMAT CSV_Quoted WITH ( FORMAT_TYPE = DELIMITEDTEXT, FORMAT_OPTIONS ( FIELD_TERMINATOR = ',', STRING_DELIMITER = '"', USE_TYPE_DEFAULT = FALSE ) ); SerDe

29 External Tables Definition of external data Columns Data Type External Data Source Location External File Format Properties Reject Settings MSDN -- Create the External Table CREATE EXTERNAL TABLE [azure].[dimcurrency] ( CurrencyKey int NOT NULL, CurrencyAlternateKey nchar(3) NOT NULL, CurrencyName nvarchar(50) NOT NULL ) WITH (LOCATION='/AdventureWorksDW2012/DimCurrency', DATA_SOURCE = AzureStorage, FILE_FORMAT = CSV_Quoted, REJECT_TYPE = VALUE, REJECT_VALUE = 0, REJECTED_ROW_LOCATION_VALUE ='/Errors' );

30 Working with External Tables SELECT or INSERT Export: CETAS Full SQL Syntax Joins Group By Aggregation SELECT dim.[productgroup], SUM(ext.[Revenue]) AS TotalRevenue FROM [dwh].[dimproduct] dim INNER JOIN [ext].[sales] ext ON dim.[productkey] = ext.[productkey] WHERE ext.[quantity] > 10 GROUP BY dim.[productgroup] HAVING SUM(ext.[Revenue]) > 100

32 Polybase as SQL Interface SQL Server 2016 External Table External Table S Q L

33 Polybase for Non-Persisted Staging SQL Server 2016 Stage DWH External Table External Table

34 Polybase for Archiving SQL Server 2016 Stage DWH ETL Table External Table ETL

35 Polybase for Staging Archive SQL Server 2016 Stage DWH ETL External Table Table

38 Processing of a Query CREATE Temp-Table Only required columns All Nodes Add Extended Properties All Nodes UPDATE STATISTICS All Nodes

39 Processing of a Query Run regular SQL Query on Temp-Table Return intermediate result to HeadNode DROP Temp-Table All Nodes All Nodes All Nodes

40 Processing of a Query Combine results of all Nodes Head Node Head Node Processing Return results to Client Head Node Head Node

41 Transactions and Sessions Temp-Tables have a unique name: [tempdb].[dbo].[qtable_d4cd3ddd8c9642dfbf62ccd36cdbf57b_d] One Temp-Table for each Worker (8) Transaction Level is changed for external queries: SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;

42 Predicate Push-Down HDFS only! Resource Manager link set up Starts a MapReduce Job Data Volume Statistics on Table Forced in Query OPTION(FORCE EXTERNALPUSHDOWN); OPTION(DISABLE EXTERNALPUSHDOWN);

43 Predicates HDFS only! Push-able Select subset of columns Arithmetic Operators Comparison Operators Logical Operators Unary Operators Non-Push-able Joins Complex Calculations Group By * Aggregations * * Partially Push-able

44 Query Distribution! Head Node PolyBase Engine SQL Server 2016 PolyBase DMS PolyBase DMS PolyBase DMS PolyBase DMS Namenode (HDFS) 8 [External] Workers per Node Hadoop Cluster FS FS FS FS

45 File Layout One Big File vs. Many Small Files Compressed vs. Uncompressed Files File Format Align with Number of Readers!!! No Partitioning yet!!!

46 File Format CSV UTF-8 and UTF-16 only! No support for Header Rows No Linebreaks in Text Proper Date-Formats [Date] = yyyy-mm-dd [Time] = hh:mm:ss [DateTime] = {Date} {Time} Only one format for all Date*-datatypes!

Azure SQL Data Warehouse APS in the cloud Pause/Stop Compute Persisted Storage Always 60 Distributions for Storage Flexible number of ComputeNodes DWU Number of:

47 Azure SQL Data Warehouse APS in the cloud Pause/Stop Compute Persisted Storage Always 60 Distributions for Storage Flexible number of ComputeNodes DWU Number of: ,000 1,200 1,500 2,000 Compute Nodes Readers Writers

48 Many thanks to all volunteers! Rafael Dabrowski Alexander Klein Volker Bachmann Ben Kettner Tobias Blödt Dirk Hondong Christian Gräfe Cornelia Matthesius Gabi Münster Dominik Petri Kai Michael Poppe Kai Gerlach Björn Peters Henrik Schütze Christa Kurschat Klaus Betzing Nadine Witthöft Tanja Salwiczek

49 SQLSaturday #772 - Munich

50 PASS Deutschland e.v. For further information about future events, visit our PASS Deutschland e.v. booth in the exhibitor area.

51 Sponsors Many thanks to our sponsors, without whom such an event would not be possible.

52 Sponsors Many thanks to our sponsors, without whom such an event would not be possible.

Felix Möller. TFS / VSTS for Continuous Integration / Continuous Delivery for SSIS and database projects in a Data Warehouse environment

Felix Möller. TFS / VSTS for Continuous Integration / Continuous Delivery for SSIS and database projects in a Data Warehouse environment Felix Möller TFS / VSTS for Continuous Integration / Continuous Delivery for SSIS and database projects in a Data Warehouse environment Sponsors Many thanks to our sponsors, without whom such an event