Power BI Custom Connectors Jeroen ter Heerdt

Transcription

1 Power BI Custom Connectors Jeroen ter Gebruikersdag 2018

2 Power Query

3 Power Query in Excel

4 Power Query in Power BI Desktop

5 Power Query in SQL Server Data Tools

6 Power Query Editor

7 Power Query Design Tenents

8 Design Tenets for M

9 Power Query Syntax and language flow

10 Syntax for a Simple Query SELECT Orders.OrderDate, Products.OrderID, Products.ProductSKU FROM Products INNER JOIN Orders ON Products.OrderID = Orders.OrderID ORDER BY Products.ProductSKU from p in Products join o in Orders on p.orderid equals o.orderid orderby p.productsku select new { o.orderdate, p.orderid, p.productsku } Products.Join(Orders, Products, o => o.orderid, p => p.orderid, (p, o) => new { o.orderdate, p.orderid, p.productsku } ).OrderBy( p => p.productsku ) let Joined = Table.Join( Products, "OrderID", Orders, "OrderID" ), Columns = Table.SelectColumns(Joined, {"OrderDate", "OrderID", "ProductSKU"}), Sorted = Table.Sort( Columns, "ProductSKU" ), in Sorted

11 Language Flow let in lazy evaluation Source = Web.Page(Web.Contents(" WebTable = Source{index}[Data], RenamedColumns = Table.RenameColumns(WebTable,{{"Column1", "Rank"}, {"Column2", "2013"}}) RenamedColumns let in Source = Sql.Database("localhost", "AdventureWorksDW2012"), DimCat = Source{[Schema="dbo",Item="DimProductCategory"]}[Data], DimSubCat = Source{[Schema="dbo",Item="DimProductSubcategory"]}[Data], DimCustomer = Source{[Schema="dbo",Item="DimCustomer"]}[Data], Joined = Table.NestedJoin(DimSubCat,{"CategoryKey"},DimCat,{"CategoryKey"},"Category",JoinKind.Inner) Joined

12 How it Actually Works let in Source = Sql.Database("localhost", "AdventureWorksDW2012"), DimCat = Source{[Schema="dbo",Item="DimProductCategory"]}[Data], DimSubCat = Source{[Schema="dbo",Item="DimProductSubcategory"]}[Data], DimCustomer = Source{[Schema="dbo",Item="DimCustomer"]}[Data], Joined = Table.NestedJoin(DimSubCat,{"CategoryKey"},DimCat,{"CategoryKey"},"Category",JoinKind.Inner) Joined DimSubCat Source Joined DimCat Source Values are lazy they are only evaluated when they are pulled

13 Power Query Query Folding

14 Query Folding (Delegation, Pushdown) let Source = Sql.Database("localhost", "AdventureWorksDW2012"), Products = Source{[Schema="dbo",Item="DimProduct"]}[Data], RemovedOtherColumns = Table.SelectColumns(Products,{"ProductKey", "Name"}), RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"Name", "Product"}}), FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10), Sorted = Table.Sort(FilteredRows,{{"ProductKey", Order.Ascending}}) in Sorted

15 Folding to Query Languages like SQL The Mashup Engine generates queries in the native language of the source, pushing down as much work as possible to the backend. Any remaining work is performed locally. Join's across sources can result in filters or join logic being passed from one side to the other. let Source = Sql.Database("localhost", "AdventureWorksDW2012"), Products = Source{[Schema="dbo",Item="DimProduct"]}[Data], RemovedOtherColumns = Table.SelectColumns(Products,{"ProductKey", "Name"}), RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"Name", "Product"}}), FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10), Sorted = Table.Sort(FilteredRows,{{"ProductKey", Order.Ascending}}) in Sorted Mashup Engine SELECT ProductKey, Name as [Product] FROM [dbo].[dimproduct] WHERE [ProductKey] < 10 ORDER BY [ProductKey]

16 Partial Folding and Compensation Query Mashup Engine Rename and sort done locally Filters pushed down as query string parameters gt 10 Query Rename, filter, and sort done locally File is read and processed locally

17 What Gets Folded?

18 Power Query Credentials and Privacy Levels

19 Credentials and Privacy Levels

20 Power Query Type system

21 Symbolic Operators Arithmetic (numbers, durations, and date-time values) +x -x x + y x - y x * y x / y Equality, inequality x = y x <> y Comparison x < y x <= y x > y x >= y Concatenation x & y (text & text, list & list, record & record, table & table, date & time) Field access (or null) r[f] r[f]? Projection (fill with null) r[[f1],[f2]] r[[f1],[f2]]? Item lookup (or null) l{i} l{i}? Item lookup by key (or null) l{[k1=v1, k2=v2]} l{[k1=v1, k2=v2]}? Function application f(a1, a2) Not-implemented error Self-recursive

22 Type System A small set of built-in types any, none null, logical, number, text, binary time, date, datetime, datetimezone, duration Complex Types list, record, table, function Sql.Database = (server as text, database as text, optional options as nullable record) as table MyCoolFunction = (index as number, category as text) as nullable table Date.StartOfDay = (datetime as any) as any

23 Ascribed Types A value's ascribed type is the type to which a value is declared to conform. When a value is ascribed a type, only a limited conformance check occurs. M does not perform conformance checking beyond a nullable primitive type. M program authors that choose to ascribe values with type definitions more complex than a nullable primitivetype must ensure that such values conform to these types. Commonly used ascribed types Byte.Type, Int8.Type, Int16.Type, Int32.Type, Int64.Type Single.Type, Double.Type, Decimal.Type, Currency.Type, Percentage.Type Character.Type

24 Simple Values SIMPLE VALUE Null LITERAL null Logical true, false Number Text 1, 1.2, 1.2e-3, #infinity, #nan "hello, world!" Date #date(2013, 3, 8) Time #time(15, 10, 0) DateTime #datetime(2013, 3, 8, 15, 10, 0) DateTimeZone #datetimezone(2013, 3, 8, 15, 10, 0, -8, 0) Duration #duration(1, 13, 59, 12.34)

25 Complex Values - Lists

26 Complex Values - Records

27 Complex Values Tables

28 Complex Values - Others COMPLEX VALUE LITERAL Function MyFunction = ( x, y, optional z ) => if z = null then x + y else (x + y) / z Type type table [ n = number, #"n^2" = number ] Binary #binary({0x68, 0x65, 0x6C, 0x6C, 0x6F})

29 Anatomy of a Table let in Source = Excel.Workbook(File.Contents("Sales.xlsx")), MyTable = Source{[Item="Data",Kind="Sheet"]}[Data] MyTable

30 Anatomy of a Table A column in the table is a list. MyTable[ProductKey] = { 217, 231, 485, 538, 480, 528, 480, 477 }

31 Anatomy of a Table A row in the table is a record. MyTable{2} = [ ProductKey = 485, OrderQuantity = 1, UnitPrice = 21.98, SalesAmount = 21.98, OrderDate = #date(2008, 1, 1) ]

32 Anatomy of a Table A table is a list of records MyTable = {[ProductKey = 217, ],[ProductKey = 231, ], }

33 Unary functions Many library functions take functions as arguments Table.SelectRows( table, (r) => r[manager] = r[buddy] ) Often, those parameter functions are unary A special syntactic form helps construct unary function values Table.SelectRows( table, each _[Manager] = _[Buddy] ) An each expression is just shorthand for a unary function The single parameter of an each function is named _ For conciseness and to get close to the DAX syntax, the _ can be omitted when accessing fields or columns Table.SelectRows( table, each [Manager] = [Buddy] )

34 Special Syntactic Forms Conditional expression Let expression Error expression (think: throw ) Terminates evaluation and raises error Try expression (think: catch ) Attempts evaluation, optional otherwise Encodes results and errors as record values if 1 < 2 then "hurray" else "sad face" let x = Number.Atan(3) in x * x error "Here I go wrong" error [ Reason = "Expression.Error", Message = "'t' should be positive", Detail = t ] try error "Bad" // [ HasError = true, // Error = [ // Reason = "Expression.Error", // Message = "Bad", // Detail = null // ] // ] try error "Bad" otherwise 42 // 42

35 Power Query Writing M and tips and tricks

36 Writing M Advanced View Access/edit the full query text in the Advanced View The formula for the current step is shown in the formula bar The fx button creates a new step, referencing the previous step by name Case Sensitive! Functions are typically CamelCase Library functions are usually named in the form of Type.Action Table.SelectColumns, Number.ToText, etc. Special syntactic forms are usually lower case if, then, etc.

37 Tips and Tricks: Leverage the UI Generate steps using the UI, then tweak the code Some things only work through the UI ( auto steps )

38 Tips and Tricks: Library Functions Use #shared to see all exported functions (and keywords) Typing in the function name will display its help and prompt for parameter

39 Tips and Tricks: Troubleshooting Use try/catch to isolate errors Select and remove rows with errors Table.Buffer will stop folding from occurring

40 Mashup Engine Extensibility teaching new tricks

41 M Extensibility Allows definition of new functions Dynamically loaded at runtime Can be used to add new Data Source functions Associated with a Data Source Kind and credential Extensions packaged as a zip file (.mez, or.pqx) Contains M definition, icons, and string resources

42 M Extensibility: how it works

43 Data Connector Extensibility Extend Power BI data source options through OData/ODBC/M Detects service capabilities Sandboxed, cloud deployable M Extension is a lightweight wrapper providing: Icons and branding Data Source Compensation and query generation Custom authentication flows

44 Simple OData Example section TripPin; // This record declares that TripPin.Feed should be recognized as data source function of kind "TripPin". The Publish record // indicates that it should be shown in the Power Query "Get Data dialog. [DataSource.Kind="TripPin", Publish="TripPin.Publish"] shared TripPin.Feed = (url as text) => let // Wrapping another data source function gives us all the capabilities of that function // (Paging, OData $metadata, query folding). source = OData.Feed(url) in source; // Data Source Kind description TripPin = [ // Declares the supported type(s) of authentication. In this case, Implicit = Anonymous web access. Authentication = [ Implicit = [] ], // Assigns a label to the data source credential. This will be displayed in the "Manage Data Sources" dialog. Label = "TripPin Part 1 - OData" ]; // Data Source UI publishing description TripPin.Publish = [ Beta = true, Category = "Other", ButtonText = { "TripPin OData", "TripPin OData" } ];

45 Mashup Engine Extensibility demo

46 Resources

47 Bedankt! Vergeet niet de evaluatie in te vullen! Scan de QR code. Gebruikersdag 2018