Biml: I got the basics what s next?

Transcription

1 Biml: I got the basics what s next? WEITER BLICKEN. MEHR ERKENNEN. BESSER ENTSCHEIDEN.

2 Who are you? What have you done with Biml so far?

3 Who am I? Ben Weissman, Solisyon, Nuremberg, SQL Server since Version 6.5 Too stupid to learn C#

4 But first A tiny little wrap up

5 What is Biml? Your 3D Printer for (not just) SSIS

6 Frontends

7 Not just SSIS?! T-SQL/DBA Sample data generation Deployment SSAS Multidimensional Tabular Cube and dimension processing (think of AMO!)

8 BimlScript Coding <# Code Block #> <#+ Module Level Code Block #> <#= Inline Code Block #> Calls the.net ToString() method on whatever expression is in the block Can t be a statement, void, or null <#@ Directive #> All types that correspond to language elements are of the format Ast----Node Rootnode = Biml Metadata Store

9 The 3 use -cases when doing SSIS with Biml Use a stable meta-data model! Use LINQ! Use the Force!

10 What are we covering today? Stop re-inventing the (Biml)wheel Include Files Code Files CallBimlScript Adding load-timestamps Preserving indices Avoiding overhead by recreating the existing Working with flat files And if we have time (which we probably will I intentionally did not overload this session): let s discuss extension methods, SSAS, Metadata and other fun stuff with Biml

11 Warning What you will see is not necessarily a complete solution or the best way to achieve everything! Neither is it brand new!

12 Step 1 Stop re-inventing the wheel

13 Stop re-inventing the wheel: Include Files Think of copy + paste - Biml will simply replace your include directive with whatever is in your included file Handy for variable declarations across multiple files for example Will not result in an output (unless there is code producing output in the included file) Will not take any input parameters Syntax: <#@ include file="includedfile.biml" #>

14 Stop re-inventing the wheel: CallBimlScript Think of a function you call it, pass some parameters to it and get a result (meaning: Biml Code) Handy when you need the same task to be done with multiple variables and/or across multiple projects Will (pretty much) always result in an output Will take input parameters Syntax: <#=CallBimlScript("CallMeMaybe.biml", Parameter1, )#>

15 Stop re-inventing the wheel: CallBimlScript - Example Adding an extended property to a table Hint: use the designerbimlpath for Intellisense Calling Syntax: <#=CallBimlScript("Add_EP.biml", table,"bimlinitialized","true") #> Add_EP.biml:

16 Stop re-inventing the wheel: Code Files Develop and define your own classes, methods and functions in VB/C# and make them accessible for Biml without compiling them through Visual Studio Quite handy for A LOT but for example: Keep variables there (I know, we had that before!) Function for table naming conventions Will not result in an output or take any parameters by itself it s managed code so it will do what ever you tell it to do (uuuuh! Another 3D printer analogy )! Syntax: <#@ code file="thisisnotthecsharpyourelookingfor.vb" #>

17 Stop re-inventing the wheel: Code Files - Example Naming Convention 3 Parameters: Connection Schema Tablename Output: If Schema is DBO then: <Connection>_<Tablename> Else: <Connection>_<Schema>_<Tablename>

18 Uuuuuuh! Live code! Finally! Demo!

19 Step 2 Adding timestamps

20 Adding timestamps Ever wanted to add a timestamp to all of your staging tables? 2 Parts required: Add the column to the Biml meta data Add a derived column to your data flow task

21 Step 3 Preserving indices

22 Preserving Indices Let s assume, you have an index or 2 in your staging area Let s assume, these have been created by someone else Let s assume, you want to stay friends with that person That probably means: it would be a GREAT idea to keep those indices, even when you need to make changes to a table To get that done, we need a little checklist when building our meta data in Biml: Does the table already exist in the staging environment? If yes: does it have an index? If yes: do all columns for that index still exist?

23 Preserving Indices Example code Demo!

24 Step 4 Avoiding overhead

25 Avoiding overhead As long as the structure of a table does not change, there is no point to drop / re-create it. Agreed? That means: before generating a Drop and Create statement, we should check, if it is necessary To do so, we check: Does the table already exist? If it exists, is the column definition the same? If either of those 2 conditions is not met, we will create the table, otherwise we will just leave it as it is

26 Avoiding overhead What it usually looks like

27 Avoiding overhead Sample code Hint 1: Add a flag so you can also have a DTSX to only populate the new tables! Hint 2: Flag your SQL tables so you can automatically remove them if needed. Hint 3: This could go into 10 less codelines with LINQ

28 Avoiding overhead Sample code Little Demo on Hint 2!

29 Step 5 Flat files

30 Flat files Each flat file requires a flat file format and a flat file connection The connection is the link to a file The format describes the content of that file Multiple connections can share the same format vv. This becomes handy when looping over a folder with multiple files Hint: try to get your format from a metadata file to automate your process!

31 Flat files Sample Code

32 Flat files: For each loops Example Demo!

33 Flat files: Dynamic formats Example Another Demo!

34 What s next? Rather comprehensive list of Biml Blogs out there: All Demo code to be found at: Reach out to BimlHeroes here today or online Become a BimlHero yourself if I can do it, you can do it!

35 Do you have Questions?

36 Thank you! WEITER BLICKEN. MEHR ERKENNEN. BESSER ENTSCHEIDEN.