Excel. Self Service BI: Power Query ABSTRACT: By Eric Russo

Transcription

1 Self Service BI: Excel Power Query ABSTRACT: By Eric Russo Microsoft Power BI is a self service solution for your data needs using Excel. It incorporates different tools for data discovery, analysis and visualization. Based in the cloud it provides you insight into almost any type of data in Excel, which is one of a DBAs best tools. Excel has evolved so much over the years it s difficult to stay abreast of additions and fancy tools. With the introduction of a number of Power capabilities including Power Pivot, Power View and Power Maps, it s Excel s Power Query that s worth looking into further. Power Query helps end users find and prep data for analysis. Power Query makes it possible for queries to be published and reused, but the script is easier to maintain. In this white paper we will look at Power Query s shaping capabilities when working with data inside Excel 2013.

2 Self Service BI: Excel Power Query

3 content Query Editor... 1 Loading Data After Query... Options & Sharing Data with Power Query... Pivot or Unpivot Data Scientific Operations... 3 Working with Text Transforms... Expanding Columns Options... Power Query Shaping Functions... Connect to Azure Options... How & Why - Power Query with Power BI... Overview for Sharing a Query... How to Share a Query... Conclusion About Datavail... 8

4 Data shaping is a common task in Excel that involves reducing and rearranging one or more tables into a subject table that matches your data analysis requirements. Microsoft Power Query for Excel provides an intuitive user interface to help you define data shaping and transformation steps with simple user actions over a sample of data. Power Query enables you to import data into Microsoft Excel from a wide variety of data sources including other Microsoft Excel Workbooks, SQL Server, Oracle, IBM DB2, SharePoint, Azure, HDInsights and other structured data. Once you connect to a data source, you use the Power Query query editor to shape your data into a useful format. As you shape your data source, Power Query automatically creates a sequence of query steps associated with a specific editor activity. For example, a sequence of query steps is created while you work in the query editor to connect to a data source, promote a row to column headers, sort rows, and filter a table. These steps are automatically run every time a query is refreshed and can be modified later to match changes in the data or in your analysis needs. You can shape data from multiple data sources by adding, removing or editing query steps. Query steps replace the need to manually acquire and reshape data in Excel. You will often hear queries built with Power Query referred to as 'Mashups'. Query Editor The Microsoft Power Query for Excel query editor enables you to navigate, define, and perform data transform operations over a data source. To display the Query Editor dialog box, create a query by connecting to a data source or click Filter & Shape on the Query Settings pane. The query editor has the following user interface elements: 1. Query title: This is the area for your query name. To change the query name, select the existing title by doubleclicking it and then type in the change. The Query title area also shows the time of the last data download. 2. Navigator pane: This pane enables you to browse structured data sources to find the data table that you want to query. Not all data sources support navigation. 3. Preview pane: This pane displays a preview of data from the results of each query step. You interact within the Preview pane to shape data to reduce and rearrange one or more tables into a subject table to match your data analysis requirements. 4. Context menu: This menu is specific to an element such as a table column, within the editor preview pane, allowing options such as Filter, Sort, Unpivot Columns, Hide Columns, etc. These options are what enable the data transformation capabilities of the Query Editor. 5. Step pane: This pane displays a sequence of query steps. A step is each of the data acquisition or data transformation tasks applied on a query. 6. Formula bar: This bar contains the Power Query function for the selected Step. Its visibility is toggled from the Settings option located at the bottom of the Query Editor window. Now that we are clear about using Query Editor options, let us look at some of the interesting techniques we can work while working with Power Query editor. Page 1 Self Service BI: Excel Power Query 2015 Datavail, Inc. All rights reserved.

5 Loading Data After Query There are few important options one needs to be aware when working with Power Query loading into Excel: 1. Enable download: This option controls whether or not the query downloads data. When this option is turned off for a query, the data result of the query is not downloaded but the query can still be combined with other queries in order to build the desired result. 2. Load to worksheet: In addition to loading query results into an Excel worksheet, Power Query enables you to load a query result into an Excel data model. After you load data into an Excel data model, you can use PowerPivot and Power View for further data analysis. If the data from the query is going to only be used from the Excel data model, you can turn the Load to worksheet option off to prevent the unnecessary increase in the size of the workbook. This option is necessary for query results that exceed the volume limitations for loading to the Excel sheet. 3. Load to data model: This option is available for loading the query result into an Excel data model. When the query result has been loaded to the Excel data model, the Query Settings pane will show Loaded to data model instead of Load to data model for the query. If you do not want the data to be in both the worksheet and the data model, you must turn off Load to worksheet prior to choosing Load to data model. If you have already loaded to the data model with the Load to worksheet option on, the Load to worksheet option will be grayed out. To change it, you must remove the data from the data model by turning the Enable download option for the query off. You can then turn Enable download back on, turn Load to worksheet off and then choose Load to data model. Options & Shaping Data With Power Query Excel is a tool that allows us to exploit and play with data even in its native mode. Introduction of Power Query just made it formidable and powerful. In this section let me take a moment to talk about few options that were introduced with Power Query editor. These are not exhaustive but representative to what we can do when working with Power Query. Pivot or Unpivot Data Some of these functions have been in the SQL Server and Access world for a while. But some of these transformations were hidden living inside Excel too. Region Salesman Order Date Units Sold East Sales 01 1/31/ West Sales 01 2/28/ East Sales 02 1/31/ Self Service BI: Excel Power Query Page 2

6 Gets transformed into Region 1/31/2015 2/28/2015 3/31/2015 East West South With Power Query, when we import data from an external source we can use the Pivot Column or Unpivot Column based on our requirements to transform the data. This is a classic scenario to transform from rows to columns and vice versa in an instant. Once selected, we have a nice dialog that will help us complete the steps as shown in figure above. Scientific Operations When working with numeric values, Power Query allows you to use a number of scientific and statistical functions that can be applied to the data under edit mode. At a statistical function point of view it is functions like (SUM, Max, Min, Standard Deviation, Average, Count and others). From Standard functions point of view, we have Add, multiply, subtract, divide and so on. We also have scientific functions like Power, Square root, Logarithmic, Exponential and others. Page 3 Self Service BI: Excel Power Query 2015 Datavail, Inc. All rights reserved.

7 Working with Text Transforms The next part worth a mention are the transformations we can do with Text column data. Sometimes we try to split columns based on standard or custom delimiter, change the case of data to pull the report, parse the text and more. In my opinion this is a powerful part of working with Power Query and textual data that is pulled from the internet or csv based formats. Expanding Columns Options We don t want to bring in the column headers that were in the source system. This Options part of objects definition inside column expansion is a great addition. In the above figure, we have used the Facebook graph API to select an object, and we have options to expand the object as-is or we can customize the columns extracted. Self Service BI: Excel Power Query Page 4

8 Power Query Shaping Functions Each shaping option chosen in the query editor generates a formula for the step in the query using functions. The following sections show examples of the formulas generated for the corresponding menu selection in the query editor. JOIN FUNCTION When using the query editor to Merge queries, the formula generated uses the AddJoinColumn function. A Merge produces the following expression in the formula bar: Menu Selection Options Chosen Formula Generated Auto Filter Selected only 'Pinal' from the list Table.SelectRows(Orders, each ([CustomerID] = "Pinal")) Number Filter Date/Time Filter Text Filter Keep Top 100 Rows Show rows where: Freight Greater Than 0 Between Show rows where: OrderDate Greater Than Or Equal To 8/1/ :00:00 AM And Less Than Or Equal To 8/30/ :00:00 AM Show rows where: ShipCity Equals Redmond Or Equals LA Table.SelectRows(Orders, each [Freight] > 0) Table.SelectRows(Orders, each [OrderDate] >= #datetime(2014, 8, 1, 0, 0, 0) and [OrderDate] <= #datetime(2014, 8, 30, 0, 0, 0)) Table.SelectRows(Orders, each [ShipCity] = "Redmond" or [ShipCity] = "LA") Table.FirstN(Orders,100) Keep Range of Rows First Row = 10, Number of Rows 25 Table.Range(Orders,9,25) Remove Top Rows Number of Rows = 100 Table.Skip(Orders,100) Remove Alternate Rows First Row to Remove = 1, Number of Rows to Remove = 1, Number of Rows to Keep = 1 Table.AlternateRows(Orders,0,1,1) Remove Remove Other Columns Split Column Split Column Right-clicked column heading on ShipVia column and chose Remove Multi-selected OrderID, Freight and ShipName columns. Right-clicked the selection and chose Remove Other Columns By Delimiter Select or Enter Delimiter: Comma Split: At the left-most delimiter By Number of Characters Number of Characters: 5 Split: Once, as far left as possible. Table.RemoveColumns(Orders,{"ShipVia"}) Table.SelectColumns(Orders,{"OrderID", "Freight", "ShipName"}) Table.SplitColumn(Source,"Name",Splitter. SplitTextByEachDelimiter({","}, null, false),{"name.1", "Name.2"}) Table.SplitColumn(Source,"Name",Splitter.SplitTextByPositions({0, 5}, false),{"name.1", "Name.2"}) Sort Ascending Sort Descending Table.Sort(Orders,{{"CustomerID", Order.Ascending}}) Table.Sort(Orders,{{"CustomerID", Order.Descending}}) Page 5 Self Service BI: Excel Power Query 2015 Datavail, Inc. All rights reserved.

9 </> Table.AddJoinColumn(Products,{"ProductID"},()=>#"Total Sales",{"Order_Details.ProductID"},"NewColumn") The AddJoinColumn function generates a left outer join of the two sources involved. Other join types are possible in Power Query using the Table.Join function, which is not available without a custom query step. Table.Join defaults to an inner join but other join types can be specified. Connect to Azure Options Since working on Azure is a special case scenario, in this section we will see what options available when selecting the Azure source. For folks who are not aware what a Marketplace source is, this is a repository of datasets published by organizations which can be consumed publically either in paid mode or free dataset. If you are looking at creating some nice dashboards and were looking at some datasets to start with, well marketplace is a great place to start. How & Why - Power Query with Power BI Signing in to Power BI using your organizational account enables you to share queries within your organization and also to discover and use queries shared by other users within your organization. You will not be able to access these features until you sign in. You need to have a license assigned in Power BI in order to sign in. In Excel, on the POWER QUERY tab, click Sign In. In the Sign In dialog box, click Sign In, and then enter your organizational account information associated with your Power BI tenant. Self Service BI: Excel Power Query Page 6

10 Overview for Sharing a Query After you have created a query, you can share the metadata of the resultant dataset as a query in Power Query with all users or a specific set of users within the organization. Shared queries appear in search results when users perform an online search in Power Query. This enables the users to easily discover and reuse the shared queries and their underlying data for their data analysis and reporting. When you share a query, the metadata for the query is stored in the Power BI Data Catalog in the cloud. This information in the Power BI Data Catalog helps users in finding and using the shared queries. When you use a shared query in Power Query, the query definition is executed in Excel to import the data referenced by the query into a new worksheet in Excel. How to Share a Query 1. Sign in to Power BI. In Excel, on the POWER QUERY tab, click Sign In. In the Sign In dialog box, click Sign In, and then enter your organizational account information that was used for signing up for Power BI. 2. After signing in, select the query you want to share. Then, on the QUERY tab in Excel, click the down arrow on the Share icon, and then click Share. 3. In the Share Query dialog box: a. Specify a name and description for your query. b. The Share Query dialog box displays information about the underlying data sources for the query you are sharing. c. To certify the query, select the Certify this query for others check box. d. Under the Share with area, specify the users with whom you want to share the query within your organization. Clicking Just me does not share the query with anyone except you. Clicking My organization shares the query with all the users within your organization who have access to Power BI. Clicking People and security groups enables you to specify the specific users or security groups within your organization with which you want to share the query. e. To include support documentation for your query, specify the URL of the documentation in the Documentation URLfield. This is an optional field, and you can skip it. f. To include a preview of the top rows when others search for the query, ensure that the Previews check box is selected. g. Click Share a Copy. Once this step is done, the query we entered is queryable inside the search of Excel Power Query window. If you don t have a Power BI tenant to play, you can skip this section. The standalone Power Query editor is powerful and has a number of capabilities that are worth a look. We have discussed a partial set of features and capabilities available with Power Query in this whitepaper. Page 7 Self Service BI: Excel Power Query 2015 Datavail, Inc. All rights reserved.

11 Conclusion As I take a moment to wrap up this whitepaper, it is important to look at the additional capabilities one can get when working with Power BI and especially Excel Power Query. If you are a power user of Excel and work with data, then Power Query enhances your ability to work with repetitive work to the next level. All the Power Query transformations are stored as part of Excel and when new data is refreshed as part of Power Query, these transformations are applied automatically as part of the pull. This makes Excel consume data without making many changes to the reports built on top of them. Biography Eric Russo Vice President and Practice Leader of SQL Server and MySQL Client Services for Datavail Experienced in the management of large DBA teams involving offshore staff, Eric specializes in ITIL certification, management of local and remote staff, project coordination, database administration, change management, server OS administration, programming and scripting. Eric s years of experience as a DBA programmer, ITIL implementer, and change control manager allow him to run high-performance DBA teams providing exceptional customer support. In his off time Eric enjoys spending time with his family, snowboarding, karate, running and computer games. About Datavail Datavail Corporation is the largest pure-play provider of remote database administration (DBA) services in North America, offering database design and architecture, administration and 24x7 support. The company specializes in Oracle, Oracle E-Business Suite, Microsoft SQL Server, MySQL, MongoDB, DB2 and SharePoint, and provides flexible on-site/off-site, onshore/offshore service delivery options to meet each customer s unique business needs. Contact Us General Inquiries: Fax Number: info@datavail.com Corporate Headquarters: Datavail Corporation Ridge Parkway Suite 125 Broomfield, CO Database Operations Control Center: Datavail Infotech Pvt. Ltd 3rd Floor, Unit No. B-3 Ashar IT Park, Road No. 16Z Wagale Estate Thane (West), Thane Direct Telephone Number: Bangalore Office Datavail Infotech Pvt. Ltd Concept Business Park #319/9, 1st floor, Block A Hosur Main Road Bommanahalli, Bangalore Self Service BI: Excel Power Query Page 8

12