@KATEGRASS. Let s Get Meta: ETL Frameworks Using Biml

Size: px

Start display at page:

Download "@KATEGRASS. Let s Get Meta: ETL Frameworks Using Biml"

Dustin Hubbard
5 years ago
Views:

1 Let s Get Meta: ETL Frameworks Using Biml

2 Please Support Our Sponsors SQL Saturday is made possible with the generous support of these sponsors. You can support them by opting-in and visiting them in the sponsor area.

3 Local User Groups Orange County User Group 2 rd Thursday of each month bigpass.pass.org Los Angeles User Group 3 rd Thursday of each odd month sql.la Malibu User Group 3 rd Wednesday of each month sqlmalibu.pass.org San Diego User Group 1 st & 3 rd Thursday of each month

) Over 15 years experience with SQL Server, currently consulting

4 Kate Grass: The Who, What and Where Digital nomad, dog dork, lover of data, hiker, biker, runner(?) Over 15 years experience with SQL Server, currently consulting full-time for firm specializing in Health & Human Service analytics Various forests and campgrounds around the US

5 Let s Get Meta: ETL Frameworks Using Biml ETL Framework Metadata Biml Biml Framework The Trifecta Metadata

6 Sam s Solar Systems

7 ETL Systems Investigation: The Good Solid ETL The Bad Minimal logging No auditing No error handling The Ugly Inconsisten t design Lack of standards

8 Lack of Standardization: The Problem Different developers => Different designs Different projects => Different designs Different day => Different designs

9 Lack of Standardization: The Effects Learning curve for new developers Development time required for changes Increased likelihood of mistakes Intelligent developers spending time doing copy/paste

10 Minimal Logging: The Problem Output file generated by SQL Agent job step Very minimal data Not persisted, overwritten every time!

11 Minimal Logging: The Effects Troubleshooting anything becomes a search for the needle in the haystack Difficult to pinpoint where or why performance has degraded

12 No Auditing: The Problem No record of ETL outcomes Row counts Table sizes Deletions

13 No Auditing: The Effects No proactive alerting when things get wonky Ex: Row count deltas from source <> target Difficult to answer questions about history Hard to explain changes in processing time that are found in the logs * Oh wait, there is no logging, good thing we don t have to worry about that last one

14 No Error Handling: The Problem Error file generated by SQL Agent job step, limited information and it s not persisted Programmatic handling of errors limited to restart attempts in SQL Agent job No restartability within an application

15 No Error Handling: The Effects Relying on SQL Agent error logs is cumbersome and not very helpful No error handling means SOMEONE has to handle the errors! A failure within one package resulted in the entire job step being re-run Can t learn from your mistakes if your mistakes aren t documented

16 The Solution Develop an ETL Framework Driven by Metadata Powered by Biml Biml Framework The Trifecta Metadata

17 The Solution: Build or Buy? The Goldilocks Dilemma: Options available but none were *just* right Commercial products Open source solutions Starting with SSIS 2012, the Integrations Services Catalog provides a great foundation for many frameworkey things Final decision was to make use of available features and build our own framework

18 Let s Get Meta: ETL Frameworks Using Biml ETL Framework Metadata Biml Framework The Trifecta Biml Metadata

19 Let s Get Meta: ETL Frameworks Using Biml What is a Framework? Characteristics Benefits Functional Components Design Process

20 What is a Framework? A group of design patterns, common and reusable tools and processes that facilitate the larger (ETL) process A set of foundational building blocks upon which (ETL) project are built

21 What are the Desired Characteristics of a Framework? as SIMPLE as possible Standardized Flexible Metadata Driven

22 What are the Benefits of a Framework? Broader Support Abstract Complexity Provide Transparency Easier Troubleshooting Standard Design Consistent Code Reduced Dev & Maintenance Time Fewer Mistakes

23 What are the Functional Components of a Framework? It Depends! Are you building an Oreo or an ipad? Business needs will dictate what is required and what is useful.

24 What are the Functional Components of a Framework? What is (or should be) repeated across projects? Possibilities include: Orchestration & Parallelism Reporting & Monitoring Deployment Templates Logging & Auditing Notifications Fault Tolerance / Recoverability

25 Framework Design Process: Component Selection Identify your pain points List all your wants Highlight your needs Review for feasibility Rank for Value vs Complexity Iterative process

26 Framework Design Process: System Mapping Components have been selected, now how do they relate? What s the order of operations? Get your hands off that keyboard! Pencil and paper first. And lots of eraser!

27 FrameworkInit.dt sx

28 OrchestrationRoot.dt sx

29 ThreadRoot.dtsx

30 Let s Get Meta: ETL Frameworks Using Biml ETL Framework Metadata Biml Framework The Trifecta Biml Metadata

31 Let s Get Meta: ETL Frameworks Using Biml What is Metadata? Abstracting your Metadata Utilizing your Metadata

32 What is Metadata? In it s simplest form Meta-X equals X about X Meta-cognition or the more fun, Meta-joke: A horse, a duck and a bear walk into a bar. The bartender says, what is this, some kind of joke? Metadata equals data about data! * Marko Ticak, Grammarly.com, 12/30/2016,

33 What is Metadata? all the information that defines and describes the structures, operations, and contents of the... system Technical metadata, business metadata and process metadata * The Kimball Group,

34 Abstracting your Metadata What are the definable properties shared across your ETL systems? Connection Strings Max Threads ETL pattern (Incremental vs Full Refresh) Database metadata Table names, column names, etc. Frequency of ETL process daily, weekly, etc. the list goes on

35 Abstracting your Metadata What are the definable properties within your systems? Directory or filename patterns Lag days or cut-off dates Various arguments for executing external tasks (Calling a zip program for example) the list goes on

36 Technical Metadata and Business Metadata

37 Utilizing your Metadata Inform ANY of the functional components of your framework: Orchestration & Parallelism Reporting & Monitoring Deployment Templates Logging & Auditing Notifications Fault Tolerance / Recoverability

38 Utilizing your Metadata At the highest level in this solution, metadata directs the orchestration. Maestro package determines its actions based on the metadata and sends info and commands on to the child packages.

39 OrchestrationRoot.dt sx

40 Child dtsx ThreadRoot.dtsx

41 Utilizing your Metadata Metadata can be used like a blueprint for your ETL projects, and you can build your packages manually. Or

42 You can it!!!

43 Let s Get Meta: ETL Frameworks Using Biml ETL Framework Metadata Biml Framework The Trifecta Biml Metadata

44 Let s Get Meta: ETL Frameworks Using Biml What is Biml? Options for using Biml BimlScript Biml on Steroids (or Metadata) Where to learn more

45 What is Biml? Created by Business Intelligence Markup Language: An XML dialect used to define BI assets, including relational models, SSIS and SSAS objects.

46 What is Biml? Human readable and human writable Compiles into the EXACT same.dtsx packages that are created via drag and drop in SSDT Biml is to SSIS Packages as HTML is to webpages

48 Options for using Biml free - FREE plugin for Visual Studio.com beta version currently (the software formerly known as Mist) All available through.com

49 Demo

52 This is a Biml file *** Insert an example of a simple Biml file here

54 What s better than Biml? BimlScript! Embed small nuggets of VB or C# code into your Biml Automatically generate more Biml!

55 Code Nuggets Control nuggets: Create variables, use conditional logic and control flow, call other code <# #> Text nuggets: Take a C# or VB expression, evaluate it, and replace with the string value! <#= #> Class nuggets: Write.net methods to reuse throughout a script file <#+ #>

56 Demo

59 Directives #> import: Reference a namespace (equivalent to C# using or VB Imports statement) <#@ import namespace= System.Data #> template: Specify language (C# is default) or tier <#@ template tier= 3 #> include: Dump an entire file s worth of code anywhere you want! <#@ include file= Inc_Variables.txt #>

60 <#=CallBimlScript #> The cooler, older sibling of the include directive References another file which it calls like a method, allowing you to specify parameters to be passed into the Callee script Evaluates the codes and replaces the directive with the string results Callee script uses the property directive to specify name and type of parameters <#@ property name= TableName type= String #>

61 Metadata Driven Biml What s better than BimlScript? Metadata Use the stored values of shared properties to create configurable packages

62 Metadata Driven Biml Values known at design-time can be used by Biml to create the packages. Values known at run-time can be fed into package variables.

63 Demo Adding Metadata to your BimlScript!

67 Learn More about Biml BimlScript.com SQL Server Central Stairway to Biml Biml blogs just ask Uncle Google SQL Saturdays

68 Thank You! Please complete evals! Questions or or

Let s Get Meta: ETL Frameworks Using Biml. Kate kategrass/

Let s Get Meta: ETL Frameworks Using Biml Kate Grass @kategrass https://www.linkedin.com/in/ kategrass/ www.kategrass.com Kate Grass Who: Digital nomad, dog dork, lover of data, hiker, biker, runner(ish)