AZURE DATA FACTORY TRANSFERRING 40GB OF DATA EVERY DAY

Size: px

Start display at page:

Download "AZURE DATA FACTORY TRANSFERRING 40GB OF DATA EVERY DAY"

Bryan Warner
5 years ago
Views:

1 AZURE DATA FACTORY TRANSFERRING 40GB OF DATA EVERY DAY

2 Meet the Team Azure Architect, VISEO ALM Specialist, Architect Cloud Azure

Azure MVP Expert DevOps et ALM VISEO @LeontievIhor Lead Architect on all

Ukraine Azure Group" Organizer MSP Roadshow Ukraine DevOps trainer for

Azure (Ukraine and internationally) Video Courses on "Microsoft Channel9":

2 2 Meet the Team Azure Architect, VISEO ALM Specialist, Architect Cloud Azure (IaaS, PaaS) Consultant, Trainer Ihor Leontiev Architecte développeur logiciel Azure MVP Expert DevOps et ALM Lead Architect on all Blockchain and IoT projects of the group Associative Speaker at "Microsoft Ukraine Azure Group" Organizer MSP Roadshow Ukraine DevOps trainer for Franco-Ukrainian program "Businessinformatics" Author and co-author DevOps and Azure (Ukraine and internationally) Video Courses on "Microsoft Channel9": DevOpsUA Course in video "Introduction to ASP. Net Core " Youtube Video "Swift Dev-Ops with VSTS"

3 3 Where to start? Because choices are important

4 Overview of an Azure Data Factory Cloud based data integration Orchestration and transformation Automation Large volumes of data Part of Cortana Analytics Suite Information Management Fully managed service, scalable, reliable

5 Anatomy of an Azure Data Factory An Azure Data Factory is made up of:

6 Linked services Represents either a data store File system On-premises SQL Server Azure storage Azure DocumentDB Azure Data Lake Store etc. a compute resource HDInsight (own or on demand) Azure Machine Learning Endpoint Azure Batch Azure SQL Database Azure Data Lake Analytics

7 Data sets Named references to data Used for both input and output Identifies structure Files, tables, folders, documents Internal or external Use SliceStart and SliceEnd system variables to create distinct slices on output data sets, e.g., unique folder based on date

8 Activities Define actions to perform on data Zero or more input data sets One or more output data sets Unit of orchestration of a pipeline Activities for data movement data transformation data analysis Use WindowStart and WindowEnd system variables to select relevant data using a tumbling window

9 Pipelines Logical grouping of activities Provides a unit of work that performs a task Can set active period to run in the past to back fill data slices Back filling can be performed in parallel

10 Scheduling Data sets have an availability "availability": { "frequency": "Hour", "interval": 1 } Activities have a schedule (tumbling window) "scheduler": { "frequency": "Hour", "interval": 1 } Pipelines have an active period "start": " T08:00:00Z" "end": " T11:00:00Z OR end = start + 48 hours if not specified OR end : for indefinite

11 Data Lineage / Dependencies How does Azure Data Factory know how to link Pipelines? Uses Input and Output data sets On the Diagram in portal, can toggle data lineage on and off external required (and externaldata policy optional) for data sets created outside Azure Data Factory How does Azure Data Factory know how to link data sets that have different schedules? Uses starttime, endtime and dependency model

12 Functions Rich set of functions to Specify data selection queries Specify input data set dependencies [starttime, endtime] data set slice [f(starttime, endtime), g(starttime, endtime)] dependency period Use system variables as parameters Functions for text formatting and date/time selection Text.Format('{0:yyyy}',WindowStart) Date.AddDays(SliceStart, -7 - Date.DayOfWeek(SliceStart))

13 Data movement SOURCE Azure Blob Azure Table Azure SQL Database Azure SQL Data Warehouse Azure DocumentDB Azure Data Lake Store SQL Server on-premises / Azure IaaS File System on-premises / Azure IaaS Oracle Database on-premises / Azure IaaS MySQL Database on-premises / Azure IaaS DB2 Database on-premises / Azure IaaS Teradata Database on-premises / Azure IaaS Sybase Database on-premises / Azure IaaS PostgreSQL Database on-premises / Azure IaaS SINK Azure Blob Azure Table Azure SQL Database Azure SQL Data Warehouse Azure DocumentDB Azure Data Lake Store SQL Server on-premises / Azure IaaS File System on-premises / Azure IaaS

14 Data movement Uses the Copy activity and Data Movement Service or Data Management Gateway (for on-premises or Azure IaaS) Data Movement Service Source WAN WAN Sink Serialisation- Deserialisation Compression Column Mapping Source LAN/ WAN Serialisation- Deserialisation Data Management Gateway Compression Column Mapping LAN/ WAN Sink Globally available service for data movement (except Australia) executes at sink location, unless source is on-premises (or IaaS) then uses Data Management Gateway Exactly one input and exactly one output Support for securely moving between on-premises and the cloud Automatic type conversions from source to sink data types File based copy supports binary, text and Avro formats, and allows for conversion between formats Data Management Gateway supports multiple data sources but only a single Azure Data Factory

15 Focus Architecture

16 MICROSOFT REFERENCE ARCHITECTURE: ADAPTED VERSION ODS DWH

From Source to Target Data Sources Data Transport Data Storage Data transformation Business model Restitution to Cloud Source1 Source2 Source3 Sour ce 4 Azure Data Factory Purpose: Copy data to the

17 From Source to Target Data Sources Data Transport Data Storage Data transformation Business model Restitution to Cloud Source1 Source2 Source3 Sour ce 4 Azure Data Factory Purpose: Copy data to the Azure platform Transport of source data to Azure SQL-only formatting from all sources Compliance of data formats (dates, digital,...) ODS SQL Purpose: To store a unified image of the source data Limit the impact of Data processing on management applications Data format uniqueness (SQL) Base of all Data processing SSIS Purpose: Datawarehouse Power Supply Implementing business rules Historisation Contrôles de qualité de la Data Quality Controls and Release Management Business Data Modeling DataWarehouse Purpose: Provide the business with a reliable and modeled data environment for analysis Business and intelligible data representation Data repositories Bridges between P & C, ADP, Web, etc. Pre-Calculated Indicators Power BI Objective: Data tool Visualization of the group. Centralization and automation of reporting Data mining Knowledge sharing Source 5

18 ARCHITECTURE technique Cible avec environnements Sources Source1 DWH - Dev Dev Source2 Azure Data Factory ODS Dev Azure Dev/Test DWH - Rec Rec Source3 Source 4 Source 5 Azure Data Factory Azure Prod ODS Prod Azure Data Catalog DWH - Prod Prod

19 CLASSIC BI ARCHITECTURE - ODS + DWH: Forces

20 Tools focus

21 Developement tools Syntax verification Build Deploy Autocompletion (Intelisense)

Development tools Visual Studio Solution - Script de deployment pour VSTS - Project Azure DataFactory - Linked Services - Activities/Pipelines 25 Activities environ - Dataset Input/Output 150

22 Development tools Visual Studio Solution - Script de deployment pour VSTS - Project Azure DataFactory - Linked Services - Activities/Pipelines 25 Activities environ - Dataset Input/Output 150 Datasets environ - Cleaner Solution solution pour vide Azure Data Factory - Database project Destination Azure SQL Root folder ADF project ADF cleaner project Database project Deploy libraries/sctipts Powershell

23 Continuous Integration/ Continuous Deployment VSTS DEV PROD Provision Deploy 2 Build Configurations 1. Dev Debug pour la machine de dev 2. Release pour CI/CD pour build/release VSTS If json KO build rejected If T-SQL KO build rejected Adf1-DEV Adf2 - PROD Coordination +Monitoring <ProjectName>. dfproj <Import Project="$(AppData)\Microsoft\DataFactory Tools for Visual Studio\MsBuild\1.0\DataFactory.targets" Condition="'$(Configuration)' == 'Debug'" /> <Import Project="#{Targets}#" Condition="'$(Configuration)' == 'Release'" />

24 Continuous Integration/ Continuous Deployment Build.Targets De-ZIP* Deploy Folder in repository AzureDataFactoryVisualStu diotools-vs2015.vsix Build Steps Libraries 81 DLLs (only to check json!) (*)You may not reverse engineer, decompile or disassemble the software, except and only to the extent that applicable law expressly permits, despite this limitation; Adding dynamically local path for DLLs on the build machine Adding dynamically local path for changed build targets Build all solutions in repo (ADF, ADF cleaner, Database project) Copy results of build to Artifacts Directory Copy Powershell scripts for Deployment to Artifacts Directory Publish Build Artifacts

Continuous Integration/ Continuous Deployment Build.Targets De-ZIP* Deploy Folder in repository Libraries 81 DLLs (only to check json!) AzureDataFactoryVisualStu diotools-vs2015.

25 Continuous Integration/ Continuous Deployment Build.Targets De-ZIP* Deploy Folder in repository Libraries 81 DLLs (only to check json!) AzureDataFactoryVisualStu diotools-vs2015.vsix Build Steps Adding dynamically local path for DLLs on the build machine Adding dynamically local path for changed build targets Build all solutions in repo (ADF, ADF cleaner, Database project) Copy results of build to Artifacts Directory Copy Powershell scripts for Deployment to Artifacts Directory Publish Build Artifacts

Continuous Integration/ Continuous Deployment Build.Targets De-ZIP* Deploy Folder in repository AzureDataFactoryVisualStu diotools-vs2015.vsix Build Steps Libraries 81 DLLs (only to check json!

26 Continuous Integration/ Continuous Deployment Build.Targets De-ZIP* Deploy Folder in repository AzureDataFactoryVisualStu diotools-vs2015.vsix Build Steps Libraries 81 DLLs (only to check json!) (*)You may not reverse engineer, decompile or disassemble the software, except and only to the extent that applicable law expressly permits, despite this limitation; 3 days working with build.targets 3 days understanding DLL ordering 5 days build configuration

Continuous Integration/ Continuous Deployment Test deployment via Dev station For deployment Using Azure Data Factory Tools for Visual Studio Continuous

if they are exists in ADF, if not: -New-AzureRmDataFactoryLinkedService - Second deploy Input, Output Datasets the same logic with one difference checking if

27 Continuous Integration/ Continuous Deployment Test deployment via Dev station For deployment Using Azure Data Factory Tools for Visual Studio Continuous deployment via VSTS For Deployment via VSTS Custom Azure PowerShell Script with following logic: Release - First looking for all LinkedServices and verifying if they are exists in ADF, if not: -New-AzureRmDataFactoryLinkedService - Second deploy Input, Output Datasets the same logic with one difference checking if was the changing time of execution offset parameter is needed to delete activity which used the dataset, and all datasets for this activity - Third Deploy activity

Continuous Integration/ Continuous Deployment Continuous deployment via VSTS Environment specific configuration: Release - Build-in mechanism of Azure Data Factory Configuration Files has some

28 Continuous Integration/ Continuous Deployment Continuous deployment via VSTS Environment specific configuration: Release - Build-in mechanism of Azure Data Factory Configuration Files has some limitations: - A lot of code for one variable impossible to use extract variable like in classic.net approach from app.config or web config - An ADF JSON file with attribute values missing (because they come from config files) cannot be deployed using the Azure portal Author and Deploy blade. This will just fail validation as missing content. - An ADF JSON config file cannot be deployed using the Azure portal Author and Deploy blade. It is simply not understood by the browser based tool as a valid ADF schema. - As we working with json we can change parameters directly in structure, like a solution we are using Custom PowerShell Script

Continuous Integration/ Continuous Deployment Continuous deployment via VSTS Environment configuration Changing time of execution Custom Powershell -Verifying all json for correct time of execution

29 Continuous Integration/ Continuous Deployment Continuous deployment via VSTS Environment configuration Changing time of execution Custom Powershell -Verifying all json for correct time of execution if not changing with time indicated in environment variables Custom Azure Powershell ADF deployment Release 2 days First version of ADF Deployment Scipt 2 days Time Parameters Changing scrpt 5 days Second version of ADF Deployment Script 5 days testing of all deployment chain First deploy - Linked Services Second - Datasets Third - Activities Added logic with Time Parameters

30 Monitoring - PowerBI Dashboard - Accesing to data via API - Historical statics of slices run - Check if all the data was copied - OMS Currently not supporting

31 Development - Data Transfer Scenario Using the COPY Activity Problems of existing Copy data Wizard 1. Execution of Wizard for table with >200 columns >1 hour + 2. Types mismatch 3. 40% Non succsedeed execution of Wizzard 4. No possibility to create desintation table(f.ex. new table in Azure SQL) 5. 1 run for 1 table Solution Custom ADF Generator console application for generating SQL scripts and JSONs: 1. <Tablename>.json script Select sql 2. <Tablename>.sql script CREATE table in Azure SQL database with all columns of analyzed MySQL table notice after is need to execute manually 3. Input<Tablename>Dataset.json Generated input Dataset for analyzed MySQL table 4. Output<Tablename>Dataset.json Generated output Dataset for analyzed MySQL table 5. CopyPipeline-<Tablename>.json Generated copy activity for analyzed MySQL table for Azure SQL with script Select sql which stored in <Tablename>.json

32 Development - Data Transfer Scenario Custom ADF Generator console application Adf Copy Wizard Time of execution 1 table Console App ime of execution 100 tables >1 hour <10 minutes Failed Execution Failed Execution 40% < 1% More visual Less visual

Custom Select T-SQL in activity Types Mismatch MYSQL Types Recommend ed SQL Server Types Operation to be performed Bigint bigint blob varbinary(max) char(x) nchar(x) date date IF(xxxx >=

33 Custom Select T-SQL in activity Types Mismatch MYSQL Types Recommend ed SQL Server Types Operation to be performed Bigint bigint blob varbinary(max) char(x) nchar(x) date date IF(xxxx >= '1900/01/01', xxxx, NULL) as xxxx, datetime datetime IF(xxxx >= '1900/01/01', xxxx, NULL) as xxxx, decimal decimal double float enum nvarchar(max) CAST(xxxx AS VARCHAR) float float int int or bigint Depends on flag UNSIGNED MySQL : CAST (xxxx AS SIGNED BIGINT) or CAST(xxxx AS SIGNED INT) longblob varbinary(max) longtext mediumtext nvarchar(max) nvarchar(max) set nvarchar(max) CAST(xxxx AS NVARCHAR) smallint smallint or int Dépend du flag UNSIGNED MySQL : CAST (xxxx AS SIGNED INT) ou CAST(xxxx AS SIGNED SMALLINT) text nvarchar(max) time timestamp time smalldatetime tinyint tinyint Depends of flag UNSIGNED MySQL : CAST varchar(x) or smallint nvarchar(x) (xxxx AS SIGNED SMALLINT) or CAST(xxxx AS SIGNED TINYINT) Without This Operations ADF slices fails with no column type error Proviser Error 14 days understanding and looking for solution type per type 10 days Define Templates DataModels and Genereation base classes + dev custom solution 2 days Implementing custom select in solution

34 WHAT HAS BEEN TESTED

35 WHAT HAS BEEN TESTED Planification problems

Azure Data Factory VS. SSIS. Reza Rad, Consultant, RADACAD

Azure Data Factory VS. SSIS. Reza Rad, Consultant, RADACAD Azure Data Factory VS. SSIS Reza Rad, Consultant, RADACAD 2 Please silence cell phones Explore Everything PASS Has to Offer FREE ONLINE WEBINAR EVENTS FREE 1-DAY LOCAL TRAINING EVENTS VOLUNTEERING OPPORTUNITIES