Introduction to U-SQL & Data Lake Alex Whittles

Transcription

1 Introduction to U-SQL & Data Lake Alex Whittles PurpleFrogSystems.com

2 Alex Whittles SQL Relay Committee SQLRelay.co.uk SQL Bits Committee SQLBits.com Birmingham SQL UG SQLMidlands.com Birmingham Azure UG AzureBirmingham.uk MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD Run BI Consultancy Business Intelligence Consultancy Data Modelling Data Warehousing OLAP Cubes ETL Systems Reporting Systems Managed Service

3 Alex Whittles SQL Relay Committee SQLRelay.co.uk SQL Bits Committee SQLBits.com Birmingham SQL UG SQLMidlands.com Birmingham Azure UG AzureBirmingham.uk MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD Run BI Consultancy Business Intelligence Consultancy Data Modelling Data Warehousing OLAP Cubes ETL Systems Reporting Systems Managed Service

4 Our Partners

5 Agenda

6 Cortana Intelligence Suite

7 What is Azure Data Lake? The Microsoft version

8 What is Azure Data Lake? U-SQL Data Lake Analytics Processing Engine Node Node Node Node Node Node Node Node Node Node Node Data Lake Store Node Data Storage (HDFS) Structured Semi-Structured Unstructured

9 Big Data is like teenage sex Everyone talks about it, Nobody really knows how to do it, Everybody thinks everyone else is doing it, So everyone claims they are doing it. Dan Ariely, Duke University

10 Big Data is like teenage sex Everyone talks about it, Nobody really knows how to do it, Everybody thinks everyone else is doing it, So everyone claims they are doing it. Dan Ariely, Duke University

11 An Analytics tool Data Sources Data Lake Store Data Lake Analytics (U-SQL) Insight

12 Or Azure s ETL Tool? (or ETLT/ECTLT?) Data Sources Data Lake Store Data Lake Analytics (U-SQL) Azure SQL Data Warehouse Power BI, etc.

13 Upload Data Into Azure Data Lake Store Azure Portal UI * 2 GB Limit Visual Studio Manual

14 Upload Data Into Azure Data Lake Store Automatic Azure Data Factory { "name": "DataLakeUploads", "properties": { "activities": [ {"type": "Copy"}, "inputs": [ {"name": "OnPremFileServer"} ], "outputs": [ {"name": "DataLakeStore"} //etc... Power Shell Import-AzureRmDataLakeStoreItem ` -AccountName $datalakestorename ` -Path "C:\Temp\SampleData.txt" -Destination $myrootdir\samples\data.txt.net SDK var parameters = new UploadParameters( srcfilepath, destfilepath, accname, isoverwrite: force ); var frontend = new DataLakeStoreFrontEndAdapter( accname, adlsfilesstemclient ); var uploader = new DataLakeStoreUploader( parameters, frontend ); uploader.execute();

15 Introduction to U-SQL What Is It? Where Is It? When Do I Use It? How Do I Use It? Should I Learn It?

16 What Is U-SQL Hybrid of T-SQL and C# Created by Michael #USQL

17 U-SQL - Like T-SQL Based on T-SQL SELECT DISTINCT SUM() FROM GROUP BY ORDER BY Windowing Functions ROW_NUMBER() etc.

18 U-SQL - Like T-SQL, but with C# BUT. Uses.Net Data Types bool byte char decimal double float int long sbyte short uint ulong ushort string Embeds C# functions Access C# Assemblies

19 U-SQL - Like T-SQL, but with C# T-SQL CASE WHEN Gender IN ( M, F ) THEN Gender ELSE - END U-SQL (Gender=="M" Gender=="F")? Gender : "-" AS Gender IF THEN ELSE

20 U-SQL - Like T-SQL, but with C#

21 Why U? Story 1 Unifies T-SQL and C# Unifies structured and unstructured data Story 2 Dives into data like a U-boat Story 3 The next letter after T-SQL

22 Portal.Azure.com Basic but it works Improving How Do I Use U-SQL? Visual Studio 2013, 2015 or 2017 Microsoft Azure SDK for.net Microsoft.Azure.DataLakeToolsForVS2015.msi

23 Demo

24 Triggering Azure Data Lake Analytics Jobs Azure Portal UI Visual Studio Manual

25 Triggering Azure Data Lake Analytics Jobs Azure Data Factory { "name": "DataLakeProcessing", "properties": { "activities": [ {"type": "DataLakeAnalyticsU-SQL"}, Power Shell Submit-AzureRmDataLakeAnalyticsJob ` -Name "ProcessData" ` -AccountName $datalakeanalyticsname ` ScriptPath $usqlscript Automatic.Net SDK var script = File.ReadAllText(scriptPath); var jobid = Guid.NewGuid(); var properties = new USqlJobProperties(script); var parameters = new JobInformation(jobName, JobType.USql, properties, priority: 1, degreeofparallelism: 1, jobid: jobid); var jobinfo = _adlajobclient.job.create(_adlaaccountname, jobid, parameters); return jobid;

26 Points to Note Case Sensitive Schema on read. It s unstructured data Read from Data Lake SQL DW (via Polybase) Power BI (native source) Azure Analysis Services Data Factory (native source/destination) SSIS (Azure Feature Pack) WebHDFS (everything else)

27 U-SQL / Data Lake Summary Data Lake Bulk Fast Storage Long term archiving, with analytics/querying Structured / Unstructured Data Orchestrate via Data Factory U-SQL / Hive / Spark / Etc HDFS U-SQL Hybrid of T-SQL and C# Visual Studio Plugin Big Data Analytics Data Transformation The T in Azure s ETL? (or ETLT / ECTLT?)

28 Thank You