Level Up Your Biml: Best Practices and Coding Techniques. Cathrine Wilhelmsen

Transcription

1

2 Level Up Your Biml: Best Practices and Coding Techniques Cathrine Wilhelmsen

3 Session Description You already know how to use Biml to build a staging environment in an hour, so let's dive straight into some of the more advanced features of Biml. Attend this session for an overview of Biml best practices and coding techniques. Learn how to centralize and reuse code with include files and the CallBimlScript method. Make your code easier to read and write by utilizing LINQ (Language- Integrated Queries). Share code between files by using Annotations and ObjectTags. And finally, if standard Biml is not enough to solve your problems, you can create your own C# helper classes and extension methods to implement custom logic. Start improving your code today and level up your Biml in no time!

4 Cathrine Wilhelmsen Data Warehouse Architect Business Intelligence cathrinewilhelmsen.net

5 You Know basic Biml and BimlScript Completed BimlScript.com lessons Have created a staging environment?

6 Today Code Management Practical Biml Programming C# Classes and Methods :)

7 Quick Recap of Basic Biml

8 What is Biml? Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: Databases, Schemas, Tables, Views, Columns SSIS Packages SSAS Cubes Metadata

9 What do you need?

10 or you can use the new Biml tools

11 How does it work?

12 demo time! Let's generate some packages!

13 Ok, so we can go from Biml to SSIS

14 can we go from SSIS to Biml?

15 Yes! :)

16 demo time! Let's reverse-engineer some packages!

17 The magic is in the BimlScript! Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values BimlScript allows you to control and manipulate Biml code

18 BimlScript Code Nuggets <# #> Control Nuggets (Control logic) <#= #> Text Nuggets (Returns string) #> Directives (Compiler instructions) <#+ #> Class Nuggets (Create C# classes)

19 How does it work?

20 Yes, but how does it work?

21 Yes, but how does it actually work? <Biml xmlns=" <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml> <Biml xmlns=" <Packages> <Package Name="Load_Customer"/> <Package Name="Load_Product"/> <Package Name="Load_Sales"/> </Packages> </Biml>

22 Biml vs. BimlScript Flat XML "Just text" Automate, control and manipulate Biml with C#

23 demo time! Let's generate a lot of packages!

24 Code Management

25 Don't Repeat Yourself Move common code to separate files Centralize and reuse in many projects Update code once for all projects 1. Include files 2. CallBimlScript with Parameters 3. Tiered Biml files

26 BimlExpress vs. BimlOnline / BimlStudio "Black Box" Only SSIS packages visible Visual Editors All in-memory objects visible

27 Include Files Include common code in multiple files and projects Can include many file types:.biml.txt.sql.cs Use the include directive include file="commoncode.biml" #> The directive will be replaced by the included file Include pulls code from the included file into the main file Works like an automated Copy & Paste

28 Include Files

29 Include Files

30 Include Files

31 CallBimlScript with Parameters Works like a parameterized include File to be called (callee) specifies input parameters it accepts <#@ property name="parameter" type="string" #> File that calls (caller) passes input parameters <#=CallBimlScript("CommonCode.biml", Parameter)#> CallBimlScript pushes parameters from the caller to the callee, and the callee returns code

32 CallBimlScript with Parameters

33 CallBimlScript with Parameters

34 CallBimlScript with Parameters

35 CallBimlScript with Parameters

36 CallBimlScript with Parameters

37 Tiered Biml Files Split Biml code in multiple files and use the template directive: template tier="1" #> Create objects in-memory from lowest to highest tier to: Solve logical dependencies Simulate manual workflows In-memory objects are added to the RootNode Higher tiers can get objects added to RootNode in lower tiers

38 What is this RootNode? The RootNode contains all in-memory objects: Connections, Databases, Schemas, Tables Projects, Packages Annotations, Metadata Query the RootNode to loop over collections: <# foreach (var table in RootNode.Tables) { #> Query the RootNode to get specific objects: <#=RootNode.Tables["Product"].Schema#>

39 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

40 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

41 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

42 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

43 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

44 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

45 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

46 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

47 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

48 Inside the Black Box: Tiered Biml Files template tier="1" #> <Connections>...</Connections> template tier="2" #> <Packages>...</Packages> template tier="3" #> <Package>...</Package>

49 Inside the Black Box: Tiered Biml Files

50 How do you use Tiered Biml files? Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages 3

51 demo time! How does this actually work?

52 Debugging Biml

53 Debugging Biml BimlExpress is a "black box": You can only see the generated SSIS packages It is not possible to see the compiled Biml first Add a high-tier helper file to save compiled, flat Biml to file Check Biml For Errors to save flat Biml without generating packages

54 SaveFlatBimlToFile.biml Add the helper file to your project template tier="999" #> <# RootNode.GetBiml() ); #>

55 SaveFlatBimlToFile.biml with a high tier so it is executed as the last step <#@ template tier="999" #> <# RootNode.GetBiml() ); #>

56 SaveFlatBimlToFile.biml It creates a file <#@ template tier="999" #> <# RootNode.GetBiml() ); #>

57 SaveFlatBimlToFile.biml at the specified path template tier="999" #> <# RootNode.GetBiml() ); #>

58 SaveFlatBimlToFile.biml with all the Biml for all the object in RootNode template tier="999" #> <# RootNode.GetBiml() ); #>

59 How do you use this helper file? Create the helper file 2. Select all the Biml files and the helper file 3. Right-click and click Check Biml For Errors 3

60 demo time! How is this helper file used?

61 Annotations and ObjectTags

62 Annotations and ObjectTags Biml Annotations!= SSIS Annotations Annotations are string/string Key/Value pairs ObjectTags are string/object Key/Value pairs Use Annotations and ObjectTags to pass code between Biml files

63 Annotations Create annotations: <OleDbConnection Name="Destination" ConnectionString=" "> <Annotations> <Annotation Tag="Schema">AW2014</Annotation> </Annotations> </OleDbConnection> Use annotations: <# var destinationschema = RootNode.OleDbConnections["Destination"].GetTag("Schema"); #>

64 ObjectTags Create ObjectTags: <# #> RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"] = new List<string> {"Product","ProductSubcategory","ProductCategory"}; Use ObjectTags: <# #> var TableFilter = (List<string>) RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"];

65 LINQ

66 LINQ (Language-Integrated Query) One language to query: SQL Server Databases XML Documents Datasets Collections Two ways to write queries: SQL-like Syntax Extension Methods

67 LINQ Extension Methods Filter Sort Group Aggregate Where, OfType OrderBy, ThenBy GroupBy Count, Sum Check Collections All, Any, Contains Get Elements First, Last, ElementAt Project Collections Select, SelectMany..and many, many more!

68 LINQ Extension Methods var numconnections = RootNode.Connections.Count() foreach (var table in RootNode.Tables.Where( )) if (RootNode.Packages.Any( ))

69 LINQ and Lambda expressions Use lambda expressions to filter or specify values:.where(table => table.schema.name == "Production").OrderBy(table => table.name)

70 LINQ and Lambda expressions For each element in the collection.where(table => table.schema.name == "Production").OrderBy(table => table.name)

71 LINQ and Lambda expressions evaluate a criteria or get a value:.where(table => table.schema.name == "Production").OrderBy(table => table.name)

72 LINQ: Filter collections Where() Returns the filtered collection with all elements that meet the criteria RootNode.Tables.Where(t => t.schema.name == "Production") OfType() Returns the filtered collection with all elements of the specified type RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()

73 LINQ: Sort collections OrderBy() Returns the collection sorted by key RootNode.Tables.OrderBy(t => t.name) ThenBy() then sorted by secondary key RootNode.Tables.OrderBy(t => t.schema.name).thenby(t => t.name)

74 LINQ: Sort collections OrderByDescending() Returns the collection sorted by key RootNode.Tables.OrderByDescending(t => t.name) ThenByDescending() then sorted by secondary key RootNode.Tables.OrderBy(t => t.schema.name).thenbydescending(t => t.name)

75 LINQ: Sort collections Reverse() Returns the collection sorted in reverse order RootNode.Tables.Reverse()

76 LINQ: Group collections GroupBy() Returns a collection of key-value pairs where each value is a new collection RootNode.Tables.GroupBy(t => t.schema.name)

77 LINQ: Aggregate collections Count() Returns the number of elements in the collection RootNode.Tables.Count() RootNode.Tables.Count(t => t.schema.name == "Production")

78 LINQ: Aggregate collections Sum() Returns the sum of the (numeric) values in the collection RootNode.Tables.Sum(t => t.columns.count) Average() Returns the average value of the (numeric) values in the collection RootNode.Tables.Average(t => t.columns.count)

79 LINQ: Aggregate collections Min() Returns the minimum value of the (numeric) values in the collection RootNode.Tables.Min(t => t.columns.count) Max() Returns the maximum value of the (numeric) values in the collection RootNode.Tables.Max(t => t.columns.count)

80 LINQ: Check collections All() Returns true if all elements in the collection meet the criteria RootNode.Databases.All(d => d.name.startswith("a")) Any() Returns true if any element in the collection meets the criteria RootNode.Databases.Any(d => d.name.contains("dw"))

81 LINQ: Check collections Contains() Returns true if collection contains element RootNode.Databases.Contains(AdventureWorks2014)

82 LINQ: Get elements First() Returns the first element in the collection (that meets the criteria) RootNode.Tables.First() RootNode.Tables.First(t => t.schema.name == "Production") FirstOrDefault() Returns the first element in the collection or default value (that meets the criteria) RootNode.Tables.FirstOrDefault() RootNode.Tables.FirstOrDefault(t => t.schema.name == "Production")

83 LINQ: Get elements Last() Returns the last element in the collection (that meets the criteria) RootNode.Tables.Last() RootNode.Tables.Last(t => t.schema.name == "Production") LastOrDefault() Returns the last element in the collection or default value (that meets the criteria) RootNode.Tables.LastOrDefault() RootNode.Tables.LastOrDefault(t => t.schema.name == "Production")

84 LINQ: Get elements ElementAt() Returns the element in the collection at the specified index RootNode.Tables.ElementAt(42) ElementAtOrDefault() Returns the element in the collection or default value at the specified index RootNode.Tables.ElementAtOrDefault(42)

85 LINQ: Project collections Select() Creates a new collection from one collection A list of table names: RootNode.Tables.Select(t => t.name) A list of table and schema names: RootNode.Tables.Select(t => new {t.name, t.schema.name})

86 LINQ: Project collections SelectMany() Creates a new collection from many collections and merges the collections A list of all columns from all tables: RootNode.Tables.SelectMany(t => t.columns)

87 demo time! How is LINQ used in Biml projects?

88 C# Classes and Methods

89 C# Classes and Methods BimlScript and LINQ not enough? Need to reuse C# code? Create your own classes and methods!

90 C# Classes and Methods: From this public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { if (node.gettag(tag)!= "") { return true; } else { return false; } } }

91 C# Classes and Methods: to this public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { } return (node.gettag(tag)!= "")? true : false; } * For bools you can just use: return (node.gettag(tag)!= ""); But in this example we'll use the verbose, SSIS-like syntax because it can be reused with other data types, like

92 C# Classes and Methods: or this public static class HelperClass { public static string AnnotationTagExists(AstNode node, string tag) { return (node.gettag(tag)!= "")? "Yes" : "No"; } }

93 Where do you put your code? Inline code nuggets Included Biml files with code nuggets Reference code files

94 C# Classes and Methods: Inline <Biml xmlns=" <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>... <# } #> <# } #> </Biml> <#+ #> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { } } return (node.gettag(tag)!= "")? true : false;

95 C# Classes and Methods: Included Files include file="helperclass.biml" #> <Biml xmlns=" <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>... <# } #> <# } #> </Biml> <#+ public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.gettag(tag)!= "")? true : false; } } #>

96 C# Classes and Methods: Code Files code file="helperclass.cs" #> <Biml xmlns=" <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.gettag(tag)!= "")? true : false; } }

97 C# Extension Methods

98 Extension Methods "Make it look like the method belongs to an object instead of a helper class"

99 Extension Methods: From this code file="helperclass.cs" #> <Biml xmlns=" <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.gettag(tag)!= "")? true : false; } }

100 Extension Methods: to this code file="helperclass.cs" #> <Biml xmlns=" <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.gettag(tag)!= "")? true : false; } }

101 Extension Methods: to this code file="helperclass.cs" #> <Biml xmlns=" <# foreach (var table in RootNode.Tables) { #> <# if (table.annotationtagexists("sourceschema")) { #>... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.gettag(tag)!= "")? true : false; } }

102 Extension Methods: to this :) code file="helperclass.cs" #> <Biml xmlns=" <# foreach (var table in RootNode.Tables.Where(t =>... <# } #> <# } #> </Biml> t.annotationtagexists("sourceschema")) { #> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.gettag(tag)!= "")? true : false; } }

103 Questions?

104 Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often

105 Izpolnite anketo! Vam je bilo predavanje všeč? Ste se naučili kaj novega? Vaše mnenje nam veliko pomeni! Da bo NT konferenca prihodnje leto še boljša, vas prosimo, da izpolnite anketo o zadovoljstvu, ki jo najdete v svojem NTK spletnem profilu.

106 Biml resources and references: cathrinewilhelmsen.net slideshare.net/cathrinewilhelmsen linkedin.com/in/cathrinewilhelmsen

107