Processing Big Data. with AZURE DATA LAKE ANALYTICS. Sean Forgatch - Senior Consultant. 6/23/ TALAVANT. All Rights Reserved.

Size: px

Start display at page:

Milton Hamilton
5 years ago
Views:

Processing Big Data with AZURE DATA LAKE ANALYTICS Sean Forgatch

3 About Me Sean Forgatch Milwaukee, WI originally West Frankfort, IL Consulting Data and Analytics Management Healthcare, Insurance, SaaS Industry Speaker President Northeast WI PASS Running, Hiking, Traveling 6/23/ TALAVANT. All Rights Reserved. 3

6 Data Monetization Data monetization, a form of monetization, may refer to the act of generating measurable economic benefits from available data sources (analytics). - 6/23/ TALAVANT. All Rights Reserved. 6

9 Data Monetization: Use Case Lockheed-Martin text mines project documentation and communications for leading indicators of project issues, leading to hundreds of millions of dollars in reduced cost overruns 6/23/ TALAVANT. All Rights Reserved. 9

Agenda 1. Data Monetization and Big Data 2. Data Lakes 3.

Data Lakes Lake of Egypt Southern Illinois

16 Data Lakes Azure Data Lake Store ERP RDBS Devices IoT RAW STAGE CURATED EXPLORATORY 2 Operational Value has been identified 1 Explorational Value is being discovered 6/23/ TALAVANT. All Rights Reserved. 16

17 Data Lakes Azure Data Lake Store Discover ERP RDBS RAW STAGE CURATED 2 Operational Value has been identified Tag / Explore Devices IoT EXPLORATORY 1 Explorational Value is being discovered 6/23/ TALAVANT. All Rights Reserved. 17

20 Roles RAW (1) STAGE (2) CURATED (3) EXPLORATION (0) Data Experts/Engineers Data Experts/Engineers ETL and BI Engineers / SME s / Analysts Data Scientist / Analysts TOOLS 6/23/ TALAVANT. All Rights Reserved. 20

21 Tagging RAW (1) STAGE (2) CURATED (3) EXPLORATION (0) Data Experts/Engineers Data Experts/Engineers ETL and BI Engineers / SME s / Analysts Data Scientist / Analysts AUTOMATED SME N/A TOOLS 6/23/ TALAVANT. All Rights Reserved. 21

s / Analysts Data Scientist / Analysts AUTOMATED SME N/A INGESTION

22 Purpose RAW (1) STAGE (2) CURATED (3) EXPLORATION (0) Data Experts/Engineers Data Experts/Engineers ETL and BI Engineers / SME s / Analysts Data Scientist / Analysts AUTOMATED SME N/A INGESTION CLEANSING DISTRIBUTION TOOLS 6/23/ TALAVANT. All Rights Reserved. 22

23 Azure Data Lake Store WebHDFS Data Store for housing data in it s Native Raw Format Built on Apache YARN Process and store Petabyte size files Designed and tuned specifically for analytics Enterprise Security through Azure Active Directory No storage limit 6/23/ TALAVANT. All Rights Reserved. 23

26 Azure Data Lake Store - Ingestion Azure Streaming Analytics Azure Data Factory SSIS Apache Sqoop Azure Event Hubs Azure Data Lake Analytics (Federated) Powershell ADLCopy DistCp Storage Explorer, Azure Portal, Visual Studio 6/23/ TALAVANT. All Rights Reserved. 26

29 Azure Data Lake Analytics Big Data Queries as a Service Analytics Query Federation Develop in U-SQL,.NET, R, and Python Cognitive Services Scale Instantly Pay Per Job 6/23/ TALAVANT. All Rights Reserved. 29

U-SQL KEY FEATURES Combines SQL and C# Patterned File Processing Thousands of Files Extensions: Python, R, Cognitive Query Data where it Lives (Federated Querying) Partition

30 U-SQL KEY FEATURES Combines SQL and C# Patterned File Processing Thousands of Files Extensions: Python, R, Cognitive Query Data where it Lives (Federated Querying) Partition and Distribution of Data for Massive Parallelism Manage Structure and Shared Programming through U-SQL Catalog U-SQL Procedures 6/23/ TALAVANT. All Rights Reserved. 30

U-SQL: Scripts Job Graph Expression Tree U-SQL

38 U-SQL: File-Sets Input Files DEFINE A FILE-PATH PATTERN /DataLake/A_RAW/Stocks/Stocks/{*}.txt /DataLake/A_RAW/Stocks/Stocks/{FileName}.txt /DataLake/A_RAW/Stocks/Stocks/{Date:yyyy}.txt 6/23/ TALAVANT. All Rights Reserved. 38

$U-SQL: File-Sets Output Files DEFINE A FILE-PATH PATTERN /DataLake/B_STAGE/Stocks/{*}.csv.$

40 U-SQL: Extract Query 1 = EXTRACT Field1 string, Field2 int, Field 3 int? FROM /datalake/01_raw/{*}.csv USING Extractors.Csv(); T-SQL CREATE TABLE mytable ( Field1 VARCHAR(100), Field2 INT, Field3 INT NOT NULL ); INSERT INTO mytable ( Field1, Field2, Field3) SELECT CAST(Field1 as varchar(100) as Field1, CAST(Field2 AS INT) as Field2, CONVERT(INT, Field3) as Field 3 FROM mytable 6/23/ TALAVANT. All Rights Reserved. 40

41 U-SQL: Extract Query 1 2 = EXTRACT Field1 string, Field2 int, Field 3 int? FROM /datalake/01_raw/{*}.csv USING = SELECT Field1, MAX(Field2) AS Field2 GROUP BY Field1; T-SQL CREATE TABLE mytable ( Field1 VARCHAR(100), Field2 INT, Field3 INT NOT NULL ); INSERT INTO mytable ( Field1, Field2, Field3) SELECT CAST(Field1 as varchar(100) as Field1, CAST(Field2 AS INT) as Field2, CONVERT(INT, Field3) as Field 3 FROM mytable 6/23/ TALAVANT. All Rights Reserved. 41

42 U-SQL: Extract Query 1 2 = EXTRACT Field1 string, Field2 int, Field 3 int? FROM /datalake/01_raw/{*}.csv USING = SELECT Field1, MAX(Field2) AS Field2 GROUP BY Field1; 3 TO /datalake/02_stage/myoutput.csv USING Outputters.Csv(); T-SQL CREATE TABLE mytable ( Field1 VARCHAR(100), Field2 INT, Field3 INT NOT NULL ); INSERT INTO mytable ( Field1, Field2, Field3) SELECT CAST(Field1 as varchar(100) as Field1, CAST(Field2 AS INT) as Field2, CONVERT(INT, Field3) as Field 3 FROM mytable 6/23/ TALAVANT. All Rights Reserved. 42

43 U-SQL: Extractors and Outputters (native) EXTRACTORS USING EXTRACTORS.Text() USING EXTRACTORS.Tsv() USING EXTRACTORS.Csv() USING EXTRACTORS.Parquet() USING EXTRACTORS.Orc() OUTPUTTERS USING OUTPUTTERS.Text() USING OUTPUTTERS.Tsv() USING OUTPUTTERS.Csv() USING OUTPUTTERS.Parquet() USING OUTPUTTERS.Orc() 6/23/ TALAVANT. All Rights Reserved. 43

45 U-SQL: Extractor and Outputter Parameters EXTRACT FROM /datalake/01_raw/{*}.csv USING Extractors.Csv(silent : true, delimiter :, ); CSV(); PARAMETERS Delimiter Encoding escapecharecter nullescape Quoting rowdelimiter Silent skipfirstnrows charformat 6/23/ TALAVANT. All Rights Reserved. 45

48 ADL & U-SQL: Best Practices U-SQL Constructs before custom code SELECT before PROCESS, Windowing Functions Use Tables and Partitioning for Big Performance Understand Skew for Performance Tuning Understand Data Profile Avoid many small files General good file size 500MB to 2GB Use Event Hubs or Streaming Analytics to pull larger file sets together Find your use case! Use Virtual Columns to filter file sets Avoid: Store Everything in Data Lake and analyze it later 6/23/ TALAVANT. All Rights Reserved. 48

Azure Data Lake: Use Case Azure Data Lake connects supply chain

50 Data Lake: Advice 1. Don t ingest everything in the beginning. 2. Depending on BI and Analytics maturity, determines data lake priority. 3. Then, the data lake augments the EDW. 6/23/ TALAVANT. All Rights Reserved. 50

Azure Data Lake Whitepaper https://www.talavant.

Azure Data Lake Store

Azure Data Lake Store Analytics 101 Kenneth M. Nielsen Data Solution Architect, MIcrosoft Our Sponsors About me Kenneth M. Nielsen Worked with SQL Server since 1999 Data Solution Architect at Microsoft