Diving into your Azure Data Lake with U-SQL. Helge Rege

Size: px

Start display at page:

Download "Diving into your Azure Data Lake with U-SQL. Helge Rege"

Chastity Carr
6 years ago
Views:

1 Diving into your Azure Data Lake with U-SQL Helge Rege

2 THANKS to all Sponsors! EVENT SPONSORS EXPO SPONSORS EXPO LIGHT SPONSORS

3 Azure Data Lake has three components

Data Lake Store Hafslund s Data Lake Store is divided by subsidiary, and then organized with Input folder for input formats Staging folder for processed data Reference for

Data Lake Store directly Auditing is performed with built in functions Data is encrypted in transit and in storage Data Lake Store is a high capacity storage for all types of data

4 Data Lake Store Hafslund s Data Lake Store is divided by subsidiary, and then organized with Input folder for input formats Staging folder for processed data Reference for reference data Sandbox for sandboxing and experimentation Access is limited by Access Control Lists (ACL) in Active Directory Only analysts and super users that access data in the Data Lake Store directly Auditing is performed with built in functions Data is encrypted in transit and in storage Data Lake Store is a high capacity storage for all types of data We ingest data into the Data Lake Store without changing the format Processed data is written into the Data Lake Store for storage and analytics Parts of the Data Lake Store is a sandbox

Data Lake Analytics Data Lake Analytics is the primary data transformation method for Hafslund Market, and business logic should be implemented in Data Lake Analytics Data is

run as batch, with a given number of Analytics Units.

5 Data Lake Analytics Data Lake Analytics is the primary data transformation method for Hafslund Market, and business logic should be implemented in Data Lake Analytics Data is typically read from input or staging folders/tables of Data Lake Store and written into staging files and tables Scheduling of Data Lake Analytics is handled by Data Factory Jobs are run as batch, with a given number of Analytics Units. Cost and time are considered when setting the number of units Data Lake Analytics Data Lake Analytics is a highly scalable analytics service for transforming data. Data is transformed with U- SQL scripts, that unifies SQL and C# The job service provides flexibility for cost/value considerations, and scalable performance.

6 U-SQL basics: All U-SQL scripts follow the same basic structure: Extract data from one or more source files/tables by EXTRACT statement Transform data Output data to files or = EXTRACT date string, time string, author string, tweet string FROM "/HafslundStrom/Sandbox/SQLNexus/Input/TwitterInput.csv" USING = SELECT author, COUNT(*) AS tweetcount GROUP BY author; TO "/HafslundStrom/Sandbox/SQLNexus/Output/TwitterDemo.csv" ORDER BY tweetcount DESC USING Outputters.Csv(outputHeader:true);

7 Demo: U-SQL basics

8 Data Factory Data Factory is used for scheduling and monitoring of pipelines. These pipelines typically: Copy data from source into Data Lake Transform data with Data Lake Analytics Copy data to user friendly storage Data Factory is an orchestration service for scheduling and monitoring of pipelines. A pipeline is a grouping of activities Activities consumes and produces data sets A data set is a table or file, stored in a linked service

Service) Access Discover Data Factory runs official data flows and operational pipeline Shared logic and data must be

9 Hafslunds combination of Azure components Data Factory Orchestration Data Catalog Document Principles Data Source Main operational pipeline Adhoc Sources Data Lake Shared Data & Logic Use Data Lake Sandbox Analyst s playground Data Marts (SQL Service) Access Discover Data Factory runs official data flows and operational pipeline Shared logic and data must be deployed in DataLake and not remain in Sandbox End Users main entry point to all data is via Data Catalog Data Lake Storage & Analytics

10 Demo: Copy Wizard with Data Factory and U-SQL for products

11 A typical Data Lake + Data Factory pipeline

12 But what if we had more data? 3.5M Contracts, with a start and end date 1M Customers 600k Meterpoints Challenge: Count the number of Contracts for the last two years, identifying: New customers (customers with first day of contract, and no contract the day before) Contract or product changes Churn (customers with last day of contract, and no contract the day before) Aggregate by zip code, product, customer segment ++, returning a total of 35M rows, from a dataset with 335M+ rows

13 Demo: ContractCount

14 U-SQL extensions Remember: Data is transformed with U-SQL scripts, that unifies SQL and C# U-SQL has a limited syntax with datasets and operations, but this can be extended with C# expressions This can be simple in-line expressions or Assemblies Microsoft.Analytics.Samples.Formats for reading XML or JSON (find it at usql.io) or Code-behind Demo! Inline Examples: Time functions: DateTime.UtcNow.AddHours(2) AS CopenhagenTime Conversions: Int32.Parse(SiteID?? "-1") AS SiteID If: DateFrom > DateTo? DateTo : DateFrom AS DateTest

15 Demo: Extending U-SQL

16 Some advice

17 Don t write use copy/paste Use U-SQL rather than C# extensions when possible

18 Be deliberate when setting parallelism

19 Have fun, dive in, and experiment!

20 Please evaluate all sessions! QR / LINK on posters and in program

21 Meet me at the Community Zone After this session, you can speak with me in the Community Zone WE MIGHT Discuss additional questions Review parts of my session in more detail Network Take selfies

22 Thank you! Helge Rege

Azure Data Lake Store

Azure Data Lake Store Analytics 101 Kenneth M. Nielsen Data Solution Architect, MIcrosoft Our Sponsors About me Kenneth M. Nielsen Worked with SQL Server since 1999 Data Solution Architect at Microsoft