Azure Data Lake Store

Transcription

1 Azure Data Lake Store Analytics 101 Kenneth M. Nielsen Data Solution Architect, MIcrosoft

2 Our Sponsors

3 About me Kenneth M. Nielsen Worked with SQL Server since 1999 Data Solution Architect at Linkedin.com/in/KennethMNielsen

4 Agenda Azure Data Lake Store Azure Data Lake Analytics Azure Data Lake Analytics Using Visual Studio Azure Data Lake Analytics Using PowerShell Q & A

5 Traditional business analytics process 1. Start with end-user requirements to identify desired reports and analysis 2. Define corresponding database schema and queries 3. Identify the required data sources 4. Create a Extract-Transform-Load (ETL) pipeline to extract required data (curation) and transform it to target schema ( schema-on-write ) 5. Create reports. Analyze data Dedicated ETL tools (e.g. SSIS) Relational ETL pipeline Queries LOB Applications Defined schema Results All data not immediately required is discarded or archived 6

6 New big data thinking: All data has value All data has potential value Data hoarding No defined schema stored in native format Schema is imposed and transformations are done at query time (schema-on-read). Apps and users interpret the data as they see fit Iterate Gather data from all sources Store indefinitely Analyze See results 7

7 Azure Data Lake Store A hyper scale repository for big data analytics workloads No limits to SCALE Store ANY DATA in its native format HADOOP FILE SYSTEM (HDFS) for the cloud ENTERPRISE READY access control, encryption at rest Optimized for analytic workload PERFORMANCE

8 Azure Data Lake Store Any Data Unstructured Semi-structured Structured

9 Azure Data Lake Store

10 Azure Data Lake Store HDFS for the cloud New filesystem build from the ground up, based on HADOOP file system Integrates with HDInsight, Hortonworks and Cloudera Supports Files and Folder objects and operations

11 Azure Data Lake Store Unlimited storage Files sizes can be from Gigabytes to Petabytes No limits to scale

12 Azure Data Lake Store Security Integrates with Azure Active Directory Audit logs for all operations* Server side Encryption* ACL on files and folders* Enterprise ready security when in GA

13 Azure Data Lake Analytics A elastic analytics service built on Apache YARN that processes all data, at any size No limits to SCALE Includes U-SQL, a language that unifies the benefits of SQL with the expressive power of C# Optimized to work with ADL STORE FEDERATED QUERY across Azure data sources ENTERPRISE READY Role based access control & Auditing Pay PER JOB & Scale PER JOB

14 U-SQL A new language for Big Data Familiar syntax to millions of SQL &.NET developers Unifies declarative nature of SQL with the imperative power of C# Unifies structured, semi-structured and unstructured data Distributed query support over all data

15 Language Overview U-SQL Fundamentals All the familiar SQL clauses SELECT FROM WHERE GROUP BY JOIN OVER Operate on unstructured and structured data Relational metadata objects.net integration and extensibility U-SQL expressions are full C# expressions Reuse.NET code in your own assemblies Use C# to define your own: Types Functions Joins Aggregators I/O (Extractors, Outputters)

16 U-SQL Capabilities

17 U-SQL Distributed Query Azure Data Lake Store Azure Storage Blobs Azure SQL Database Azure SQL Data Warehouse Azure SQL DB in Azure VM READ READ READ READ READ WRITE WRITE WRITE WRITE WRITE

18 U-SQL Distributed Query

19 U-SQL Distributed Query Read the input, write it directly to output (just a simple = EXTRACT OrderId int, Customer string, Date DateTime, Amount float FROM "/input/orders.txt" USING Extractors.Tsv(); TO "/output/orders_copy.txt" USING Outputters.Tsv(); Rowset Apply Schema on read From a file in a Data Lake Easy delimited text handling Write out

20 Azure Data Lake Pattern Data Engineer ADL Storage Visual Studio ADL ADL Storage ADL Analytics Upload Dataset Get Data From CSV Where CAQS Files are stored, but would load into ADLS directly if ingesting from scratch Data Scientist Data Analyst AML Experiment Power BI Desktop

21 Execution with Requested Parallelism Requested Parallelism = 1 (reserve enough to do 1 vertex at a time) Requested Parallelism = 4 (reserve enough to do 4 vertices at a time)

22 Stage Details 252 Pieces of work AVG Vertex execution time 4.3 Billion rows Data Read & Written

23 ADLAUs Azure Data Lake Analytics Unit Parallelism N = N ADLAUs 1 ADLAU ~= A VM with 2 cores and 6 GB of memory

24 Visual Studio DATA LAKE ANALYTICS

25 Azure Data Lake Visual Studio Available project types

26 Azure Data Lake Visual Studio Fully integrates to Solution Explorer

27 Azure Data Lake Visual Studio Monitor and manage jobs Browse and manage storage Browse U-SQL catalog

28 CREATING U-SQL

29 Creating U-SQL IntelliSense Supported

30 Creating U-SQL Code behind enhance your code

31

32 Installing Azure PowerShell PowerShell Gallery Recommended approach PowerShell 5.0 supports PowerShell Gallery Windows 10 ships with PowerShell 5.0 Web Platform Installation (WebPI)

33 Installing from the PowerShell Gallery Launch Windows PowerShell ISE as Administrator Install-Module AzureRM Install-AzureRM

34 Finding the ADL cmdlets Option 1 Get-Command -Module AzureRM.DataLakeStore Get-Command -Module AzureRM.DataLakeAnalytics Option 2 Get-Command *DataLake*

35 Logging in to Azure $subname = Login-AzureRmAccount SubscriptionName $subname

36 ADLS: Listing files in a store $adls = sqlsatmunich Get-AzureRmDataLakeStoreChildItem -Account $adls -Path /

37 ADLS: Upload and download $adls = sqlsatmunich Import-AzureRmDataLakeStoreItem -Account $adls -Path d:\somefile.txt -Destination /somefile.txt Export-AzureRmDataLakeStoreItem -Account $adls -Path /somefile.txt -Destination d:\somefile_copy.txt

38 ADLA: List and submit jobs $adla = sqlsatmunich Get-AzureRmDataLakeAnalyticsJob -Account $adla Submit-AzureRmDataLakeAnalyticsJob -Account $adla -Script # U-SQL text -Name myjob Submit-AzureRmDataLakeAnalyticsJob -Account $adla -ScriptPath D:\test.script -Name myjob

39 ADL Store (ADLS) feature set Account Management Create new account List accounts Update account properties Delete account Transferring Data Upload into store from local disk Download from store to local disk Files and Folders List contents of folder Create Move Delete Does file exist Security Get ACLs Update ACLs Get Owner Set Owner File Content Set file content Append file content Get file content Merge files

40 ADL Analytics (ADLA) feature set Account Management Create new account List accounts Update account properties Delete account Data Sources Add a data source List data sources Update data source Delete data source Compute List jobs Submit job Cancel job Catalog Items List items in U-SQL catalog Update item Catalog Secrets Create catalog secret List catalog secrets Delete catalog secrets

41

42 Additional capabilities and resources Tools: Blogs and community page: Documentation and articles and slides: GITHUB Get startet Github ADL forums and feedback

43 QUESTIONS

44 Save the Dates! PASS Camp to 09. December 2016 SQL Konferenz to 16. February 2017

45 How did you like it? Please give feedback to the event: to me as a speaker: