WPC010 Introduction to Azure Data Lake. Andrea Uggetti Microsoft Francesco Diaz Insight

Transcription

1 WPC010 Introduction to Azure Data Lake P R E S E N T A Andrea Uggetti Microsoft Francesco Diaz Insight

2 Agenda Data Lake concepts Introduction to Azure Data Lake DEMO Q/A

3 Data Lake concepts

4 Data Lake definition A data lake is a method of storing data within a system or repository, in its natural format, that facilitates the collocation of data in various schemata and structural forms, usually object blobs or files. info@wpc2016.it

5 Data Staging Areas

6 But... "The main challenge is not creating a data lake, but taking advantage of the opportunities it presents." info@wpc2016.it

7 Part of Cortana Intelligence Suite

8 Azure Data Lake Service that enable developers, data scientists and analysts to store data of any size, shape and speed. Perform analysis and processing across platforms and languages

9 Azure Data Lake Store

10 Data Lake Store A highly scalable, distributed, parallel file system in the cloud specifically designed to work with multiple analytic frameworks Devices ADL Analytics ADL Store HDInsight R Spark Machine Learning

11 Data Lake Store key concept Store any data in its native format Without schema and transformation Unstructured Semi-Structured and Structured Hadoop file system (HDFS) for the cloud Built on Yarn/WebHDFS Expose WebHDFS via compatible Rest API Support for file/folder objects and operations Integrate with HDInsight, Hortonworks, Cloudera Accessible to all HDFS-compliant project (Spark, Storm, R, )

12 Enterprise Grade High Available: automate replicate data No limits to Scale Unlimited account sizes, From bytes to petabytes Data Lake Store key concept Optimized for analytic workload Large throughput Optimized for parallel computation Automatically optimizes for any throughput

13 Auditing Audit logs for all operations Stored in text format (JSON) Logs accessible via U-SQL scripts Data Lake Store Manage and data security Access Control POSIX-compliant Access Control List (ACL) Built-in groups (Owner, Contributor, Reader) Read, Write, Execute permission ACL on files and folders Integrated with Azure Active Directory IP Filter Encryption at rest Transparent server-side encryption Azure-managed (Azure Key Vault) Customer Managed Keys

14 Data Lake Store Ingress Data can be ingested into Azure Data Lake Store from a variety of sources SQL Apache Flume Azure SQL DB Server logs Built-in copy service Azure SQL DW Table Storage Azure Table Azure Data Factory Apache Sqoop ADL Store.NET SDK JavaScript CLI Azure portal Azure PowerShell Azure Storage blobs On-premises databases Azure Event Hub Custom programs

15 Data Lake Store Egress Data can be exported from Azure Data Lake Store into numerous targets/sinks SQL Apache Flume Azure SQL DB Server logs Azure SQL DW Azure Data Factory Table Storage Azure Table Apache Sqoop ADL Store.NET SDK JavaScript CLI Azure portal Azure PowerShell On-premises databases Custom programs

16 Copying data with Data Lake Sources Sinks Azure Blob Azure SQL Database Azure DocumentDB SQL Server Oracle database DB2 database Sybase database Azure Table Azure SQL Data Warehouse Azure Data Lake Store File system MySQL database Teradata database PostgreSQL database Azure Blob Azure Table Azure SQL Database Azure SQL Data Warehouse Azure DocumentDB Azure Data Lake Store SQL Server

17 Azure Data Factory Copy Activity ADL tool ADLCopy Copying data with Data Lake Open Source Tools Apache Sqoop Runs on HDInsight clusters Transfer data between relational databases and ADLS DistCp Runs on HDInsight clusters Transfer data between Amazon S3, Azure Blob Storage and ADLS

18 Different ingestion way depending by on the source of the data Ad Hoc Data Azure Portal; Azure PowerShell; Azure Cross-Platform CLI; Using Data Lake Tools for Visual Studio; Azure Data Factory, AdlCopy tool, DistCp running on HDInsight Cluster Streamed Data Azure Stream Analytics; Azure HDInsight Storm; EventProcessorHost Relational Data Azure Data Factory; Apache Sqoop Copying data with Data Lake Web server log data Azure Cross-Platform CLI; Powershell; Data Lake Store.NET SDK; Data Factory On-Premises files Data Factory; Powershell+Azure Cross-Platform CLI; Data Lake Store.NET SDK

19 Copying data with Data Lake ADLCopy Download Syntax AdlCopy /Source <Blob or Data Lake Store source> /Dest <Data Lake Store destination> /SourceKey <Key for Blob account> /Account <Data Lake Analytics account> /Unit <Number of Analytics units> /Pattern Account: ADL Analytic account, used to copy Gb/Tb data via Analytic Unit: Data Lake Analytics Units used by AdlCopy, mandatory if use /Account Pattern: RegEx pattern that indicate which blobs/files to copy

20 DEMO Azure Data Lake Store Copy file Security overviewsecurity overview

21 Data Lake Analytics: U-SQL

22 What is U-SQL U-SQL is a query language specifically designed for big data type of workloads. It is the combination of the SQL-like declarative language with the extensibility and programmability provided by C# info@wpc2016.it

23 How U-SQL has born Based on Microsoft experience on SCOPE ( + (T- SQL, ANSI SQL, Hive, C#) The need: analyze massive data sets The Map-Reduce model limitations «The programmer provides a map function that performs grouping and a reduce function that performs aggregation. The underlying run-time system achieves parallelism by partitioning the data and processing different partitions concurrently using multiple machines. However, this model has its own set of limitations. Users are forced to map their applications to the mapreduce model in order to achieve parallelism. For some applications this mapping is very unnatural.» info@wpc2016.it

24 SCOPE and COSMOS ( How U-SQL has born

25 Data Lake Analytics Why U-SQL Integrate strengths of SQL and Big Data programming Declarativity, optimizable and parallelizability of SQL Extensibility, expressiveness and familiarity of C# U-SQL makes it easy for Unstructured and Structured data processing Declarative SQL and customer imperative code (C#) Local and remote queries Increase productivity and agility Using familiar tools (Visual Studio) Leverage your SQL/.NET skills

26 Data Lake Analytics Usage scenarios Achieve the same programming experience in batch or interactive Schematizing unstructured data (Load-Extract-Transform-Store) for analysis Prepare data for other users (LETS & Share) As unstructured data As structured data Large-scale custom processing with custom code Supplement big data with high-value data from where it lives

27 U-SQL Queries: General pattern Read Process Store Azure Storage Blobs EXTRACT OUTPUT Azure Storage Blobs Azure SQL DB SELECT RowSet SELECT FROM WHERE RowSet Azure Data Lake EXTRACT SELECT OUTPUT INSERT

28 Anatomy of a U-SQL query 10 log records by Duration (End time minus Start time). Sort rows in descending order of Duration. Rowset: Conceptually is like an intermediate table REFERENCE ASSEMBLY = EXTRACT UserID string, Start DateTime, End DatetTime, Region string, SitesVisited string, PagesVisited string FROM "swebhdfs://logs/weblogrecords.txt" USING = SELECT UserID, (End.Subtract(Start)).TotalSeconds AS Duration ORDER BY Duration DESC FETCH 10; TO "swebhdfs://logs/results/top10.txt" USING Outputter.Tsv(); U-SQL types are the same as C# types The structure (schema) is first imposed when the data is first extracted/read from the file (schema-on-read) Input is read from this file in ADL Custom function to read from input file C# Expression Output is stored in this file in ADL Built-in function that writes the output in TSV format

29 Logical Expression Tree - U-SQL does not assign the rowset to a variable - The variable is the name for the expression that is being assigned to - U-SQL is not executing each statement sequentially - It composes bigger and bigger statements until there is a big expression tree that can be optimized and executed (lambda expression composition) - Execution in parallel, no intermediate state - Intermediate investigation using declarative framework (e.g. Write to a file) or external workflows info@wpc2016.it

30 U-SQL data types Category Types Numeric Text Complex Temporal Other byte, byte? sbyte, sbyte? int, int? uint, unint? long, long? decimal, decimal? char, char? string MAP<K> ARRAY<K,T> DateTime, DateTime? bool, bool? Guid, Guid? Byte[] short, short? ushort, ushort? ulong, unlong? float, float? double, double? Note: Nullable types have to be declared with a question mark? 30

31 Reading files with custom formats Use built-in extractors to read CSV and TSV files, or create custom extractors for different formats 1 Implement IExtractor Interface 2 Upload and Register Assembly 3 Reference the Assembly and Use using Microsoft.SCOPE.Interfaces; public WebLogExtractor:IExtractor { public override IEnumerable<IRow> Extract( ) { } } CREATE ASSEMBLY WebLogExtAsm /WebLogExtAsm.dll" WITH PERMISSION_SET = RESTRICTED; CREATE EXTRACTOR WebLogExtractor EXTERNAL NAME WebLogExtractor; REFERENCE ASSEMBLY WebLogExtAsm; //now just use it like a built-in extractor SELECT * swebhdfs://logs/webrecords.txt USING WebLogExtractor(); 31

32 WebLogExtractor: Implementation Custom extractor for WebLogRecords.txt file with this format using System.Collections.Generic; using System.IO; using System.Text.RegularExpressions; namespace Demo { [SqlUserDefinedExtractor] public class MyExtractor : IExtractor { public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output) { string line; Regex ItemRegex = new Regex(@"\<(.*?)\>", RegexOptions.Compiled); string[] tokens = { "UserId", "Start", "End", "Region", "SitesVisited", "PagesVisited" }; var reader = new StreamReader(input.BaseStream); while ((line = reader.readline())!= null) { int i = 0; foreach (Match ItemMatch in ItemRegex.Matches(line)){ output.set(tokens[i], ItemMatch.Groups[1]); i++; } yield return output.asreadonly(); } } } } 32

33 Tables CREATE TABLE CREATE TABLE T (col1 int, col2 string, col3 SQL.MAP<string,string>, INDEX idx CLUSTERED (col1 ASC) DISTRIBUTED BY HASH (driver_id) ); Structured Data Built-in Data types only (no UDTs) Clustered Index (needs to be specified): row-oriented Fine-grained distribution (needs to be specified): HASH, DIRECT HASH, RANGE, ROUND ROBIN CREATE TABLE AS SELECT CREATE TABLE T (INDEX idx CLUSTERED ) AS SELECT ; CREATE TABLE T (INDEX idx CLUSTERED ) AS EXTRACT ; CREATE TABLE T (INDEX idx CLUSTERED ) AS mytvf(default); Infer the schema from the query Still requires index and partitioning ALTER TABLE ADD/DROP COLUMN ALTER TABLE T ADD COLUMN eventname string; ALTER TABLE T DROP COLUMN col3; ALTER TABLE T ADD COLUMN result string, clientid string, payload int?; ALTER TABLE T DROP COLUMN clientid, result; Meta-data only operation Existing rows will get Non-nullable types: C# data type default value (e.g., int will be 0) Nullable types: null

34 Query 4 Improve performance with TABLEs Improve performance of Query1! CREATE TABLE LogRecordsTable(UserId int, Start DateTime, End Datetime, Region string INDEX idx CLUSTERED (Region ASC) PARTITIONED BY HASH (Region)); WebLogRecords.txt Populate the table Select only required fields Azure Data Lake Run query directly against the table Top10.Tsv INSERT INTO LogRecordsTable SELECT UserId, Start, End, = SELECT UserID, (End.Subtract(Start)).TotalSeconds AS Duration FROM LogRecordsTable ORDER BY Duration DESC FETCH 10; TO adls://logs/results/top10.tsv USING Outputters.Tsv(); 34

35 U-SQL extensibility Extend U-SQL with C#/.NET in Visual Studio Built-in operators, function, aggregates User-defined operators (UDOs) User-defined functions (UDFs) User-defined aggregates (UDAGGs) C# expressions (in SELECT statements)

36 Federated queries: Query data where it lives Easily query data in multiple Azure data stores without moving it to a single store Benefits Avoid moving large amounts of data across the network between stores Single view of data irrespective of physical location Azure Storage Blobs Minimize data proliferation issues caused by maintaining multiple copies Single query language for all data Each data store maintains its own sovereignty Design choices based on the need U-SQL Query Azure Data Lake Analytics Query Result Azure SQL in VMs Azure SQL DB

37 What can you do in the Azure Portal? Create a new Big Data Analytics account Author U-SQL scripts Submit U-SQL jobs Cancel running jobs Provision users who can submit jobs Visualize usage stats (compute hours) Visualize job management chart

38 Code behind support

39 DEMO Trasform text files Load transformed text files into table Aggregate functions Produce an output table Job execution graph Job execution progress playback

40 Session Recap Data Lake concepts Introduction to Azure Data Lake DEMO Q/A

41 Additional resources Blogs and community page The bible:

42 Q/A

43 GRAZIE!

44 Contatti OverNet Education OverNet Education Tel