Learn By Doing Populating the Fact Tables in the Manufacturing Data Mart

Transcription

1 Learn By Doing Populating the Fact Tables in the Manufacturing Data Mart Dr. Joshi: We did this in class, but in case you missed it: Change the BatchName field in your DimBatch Table to nvarchar(50). Feature Highlighted Populating a fact table with Integration Services Business Need Obviously, these data marts don t do Maximum Miniatures any good unless they contain facts! Therefore, we need to populate the fact tables. The information for the Manufacturing fact table is in the BatchInfo.csv file. In an actual production environment, this file would probably have the production results for one day or perhaps one week at the most. The Integration Services package that processes this file would be run daily or weekly. However, because we want to have a good deal of sample data to work from, our copy of BatchInfo.csv has three years worth of data. The design of the Integration Services package used to import this information remains unchanged. We also have the little issue of one dimension table that has not been populated. This is the DimBatch table. The only source we have for the batch number information is the BatchInfo.csv file. Therefore, we include a data flow in our Integration Services package to populate the dimension table right before the fact table is populated. We now have a second fact table in this data mart as well: the Inventory fact table. The information for this fact table comes from the order processing database. Our Integration Services package also includes a data flow to handle this. To complete this Learn By Doing activity, you need the BatchInfo.csv file and Note the Maximum Miniatures Order Processing database. If you have not done so already, go to and search for the book s page using the ISBN, which is Use the Code Download link to download the Zip file containing the book s material. Follow the instructions within the zipped folders in the Zip file to install the Maximum Miniatures Order Processing database. Steps to Create a Project Connection Manager 1. Open SQL Server Data Tools. 2. Reopen the MaxMinManufacturingDMLoad project. 3. Open the DimensionLoad.dtsx package, if it is not already open. 4. In the Connections tray, right-click the MaxMinManufacturingDM connection, and select Convert to Project Connection in the context menu. The MaxMinManufacturingDM connection is moved to the Connection Managers

2 folder in the Solution Explorer window. A reference to the project-level MaxMinManufacturingDM connection is placed in the Connection tray. This reference name is preceded by (project) to indicate it is a link to a project connection, not a package connection. A project connection can be shared by several packages within the project. A Note package connection can be promoted to a project connection, as was done in this example. You can also create a project connection directly by rightclicking the Connection Managers folder in the Solution Explorer window and selecting New Connection Manager from the context menu. Steps to Create the Load DimBatch Data Flow 1. In the Solution Explorer window, right-click the SSIS Packages folder, and select New SSIS Package from the context menu. 2. Right-click the Package1.dtsx entry in the Solution Explorer window, and select Rename from the context menu. Enter FactLoad.dtsx for the package name and press ENTER. 3. Drag a Data Flow Task item from the SSIS Toolbox and drop it on the Control Flow tab. 4. Right-click the Data Flow Task item you just created and select Rename from the context menu. Enter Load DimBatch and press ENTER. Double-click the Load DimBatch item. This takes you to the Data Flow tab. 5. Right-click in the Connections tray and select New Flat File Connection from the context menu. The Flat File Connection Manager Editor dialog box appears. 6. Enter BatchInfo.CSV File for Connection Manager Name. Click Browse. The Open dialog box appears. 7. Select CSV Files (*.csv) in the file type drop-down list in the lower-right corner of the dialog box. 8. Browse to the BatchInfo.csv file that you downloaded from the book s webpage. Select this file and click Open to exit the Open dialog box. 9. Check Column names in the first data row, if it is not checked by default. 10. Select Columns in the page selector on the left side of the dialog box. Note the content of the columns being read from the CSV text file. 11. Select Advanced in the page selector. 12. Change the data type for the BatchNumber, MachineNumber, ProductCode, NumberProduced, and NumberRejected columns to four-byte signed integer [DT_I4]. Change the data type for the TimeStarted and TimeStopped columns to database timestamp [DT_DBTIMESTAMP]. 13. Click OK to exit the Flat File Connection Manager Editor dialog box. 14. Drag a Flat File Source item from the Other Sources section of the SSIS Toolbox, and drop it on the Data Flow tab. 15. Double-click the Flat File Source item you just created. The Flat File Source Editor dialog box appears. 16. The BatchInfo.csv File data connection should be selected in the Flat File Connection Manager drop-down list. 17. Select Columns in the page selector on the left side of the dialog box. 18. Uncheck all of the columns in the Available External Columns list except the BatchNumber column.

3 19. Click OK to exit the Flat File Source Editor dialog box. 20. Drag a Sort item from the Common section of the SSIS Toolbox, and drop it on the Data Flow tab. 21. Click the Flat File Source item. Click the blue data flow arrow, drag it on top of the Sort item, and drop it on this item. This connects the source to the transformation. 22. Double-click the Sort item you just created. The Sort Transformation Editor dialog box appears. 23. Check the box to the left of BatchNumber in the Available Input Columns list. 24. Check Remove rows with duplicate sort values in the lower-left corner of the dialog box. This gives us a distinct list of batch numbers that we can use to populate our dimension table. 25. Click OK to exit the Sort Transformation Editor dialog box. 26. Drag a Derived Column item from the SSIS Toolbox and drop it on the Data Flow tab. 27. Click the Sort item. Click the blue data flow arrow, drag it on top of the Derived Column item, and drop it on this item. This connects the two transformations. 28. Double-click the Derived Column item you just created. The Derived Column Transformation Editor dialog box appears. 29. Enter BatchName in the first row under Derived Column Name. 30. Enter the following in the first row under Expression: (DT_WSTR, 50)[BatchNumber] 31. Click OK to exit the Derived Column Transformation Editor dialog box. 32. Drag a SQL Server (Dr. Joshi: OLE DB) Destination item from the Other Destinations section of the SSIS Toolbox, and drop it on the Data Flow tab. 33. Click the Derived Column item. Click the blue data flow arrow, drag it on top of the SQL Server Destination item, and drop it on this item. This connects the transformation to the destination. 34. Double-click the SQL Server (Dr. Joshi: OLE DB) Destination item. The SQL (Dr. Joshi: OLE DB) Destination Editor dialog box appears. 35. The MaxMinManufacturingDM connection should already be selected in the Connection manager drop-down list. Select [dbo].[dimbatch] from the Use a table or view drop-down list. 36. Click Mappings to view the Mappings page. The columns from the data source (which are Available Input Columns to this data destination item) should be automatically mapped to the columns in the destination (the Available Destination Columns). The following mappings should be in place: Available Input Columns Available Destination Columns BatchNumber to BatchNumber BatchName to BatchName

4 Available Input Columns Available Destination Columns 37. Click OK to exit the SQL Destination Editor dialog box. The Data Flow tab should appear as shown in Figure Figure 8-17: The Data Flow tab for the Load DimBatch Data Flow task Steps to Begin Creating the Load ManufacturingFact Data Flow 1. Click the Control Flow tab. Drag another Data Flow Task item onto the Control Flow tab. Rename this new Data Flow Task item Load ManufacturingFact. 2. Click the Load DimBatch item. Drag the precedence arrow onto the Load ManufacturingFact item and drop it there. (The DimBatch table must be loaded successfully before the ManufacturingFact table can be loaded.) 3. Double-click the Load ManufacturingFact item. This takes you to the Data Flow tab with Load ManufacturingFact selected in the Data Flow Task drop-down list. 4. Drag a Flat File Source item from the SSIS Toolbox and drop it on the Data Flow tab.

5 5. Double-click the Flat File Source item you just created. The Flat File Source Editor dialog box appears. 6. The BatchInfo.csv File data connection should be selected in the Flat File Connection Manager drop-down list. Click OK to exit the Flat File Source Editor dialog box. Steps to Create a Derived Column Data Flow Item 1. Drag a Derived Column item from the SSIS Toolbox and drop it on the Data Flow tab. 2. Click the Flat File Source item. Click the blue data flow arrow, drag it on top of the Derived Column item, and drop it on this item. This connects the source to the transformation. 3. Double-click the Derived Column item you just created. The Derived Column Transformation Editor dialog box appears. 4. Our flat file does not explicitly provide us with the number of products accepted and the elapsed time of production. Instead, we need to calculate this information from the data that is provided in the flat file. The flat file also includes a time portion on TimeStarted. This causes problems when we use this field to build our time dimension. We can solve these issues by adding three derived columns to the data flow. Enter AcceptedProducts in the first row under Derived Column Name. Enter the following in the first row under Expression: [NumberProduced] - [NumberRejected] (You can expand the Columns folder in the upper-left corner of the dialog box, and then drag-and-drop the fields onto the expression, if you like.) 5. Enter ElapsedTimeForManufacture in the second row under Derived Column Name. Enter the following in the second row under Expression: (DT_NUMERIC,6,2)DATEDIFF("mi", [TimeStarted], [TimeStopped]) (You can expand the Date/Time Functions folder in the upper-right corner of the dialog box, and then drag-and-drop the DATEDIFF function onto the expression. You can also drag-and-drop the fields onto the expression.) 6. Enter DateOfManufacture in the third row under Derived Column Name. Enter the following in the third row under Expression: (DT_DBTIMESTAMP)SUBSTRING((DT_WSTR,25)[TimeStarted],1,10) This expression converts TimeStarted into a string and selects the first ten characters of that string (the date portion, but not the time portion). This string is

6 then converted back into a datetime, without the time portion, so time defaults to midnight. (You can expand the Type Casts and String Functions folders in the upper-right corner of the dialog box and drag and drop the (DT_WSTR) type cast, SUBSTRING function, and (DT_DBTIMESTAMP) type cast onto the expression. You can also drag-and-drop the fields onto the expression.) 7. Click OK to exit the Derived Column Transformation Editor dialog box. Steps to Create Lookup Data Flow Items 1. Drag a Lookup item from the SSIS Toolbox and drop it on the Data Flow tab. 2. Change the name of this item to Lookup Batch Number. 3. Click the Derived Column item. Click the blue data flow arrow, drag it on top of the Lookup Batch Number item, and drop it on this item. This connects the two transformations. 4. Double-click the Lookup Batch Number item. The Lookup Transformation Editor dialog box appears. 5. In the Specify how to handle rows with no matching entries drop-down list, select Redirect rows to no match output. 6. Click Connection to view the Connection page. 7. Select [dbo].[dimbatch] from the Use a table or a view drop-down list. 8. Click Columns to view the Columns page. 9. Click BatchNumber in the Available Input Columns list, and drag-and-drop it on BatchNumber in the Available LookupColumns list. 10. Click OK to exit the Lookup Transformation Editor dialog box. 11. Drag another Lookup item from the SSIS Toolbox and drop it on the Data Flow tab. 12. Change the name of this item to Lookup Machine Number. 13. Click the Lookup Batch Number item. Click the blue data flow arrow, drag it on top of the Lookup Machine Number item, and drop it on this item. The Input Output Selection dialog box appears. 14. Select Lookup Match Output from the Output drop-down list and click OK. 15. Double-click the Lookup Machine Number item. The Lookup Transformation Editor dialog box appears. 16. In the Specify how to handle rows with no matching entries drop-down list, select Redirect rows to no match output. 17. Click Connection to view the Connection page. 18. Select [dbo].[dimmachine] from the Use a table or a view drop-down list. 19. Click Columns to view the Columns page. 20. Click MachineNumber in the Available Input Columns list, and drag-and-drop it on MachineNumber in the Available LookupColumns list. 21. Click OK to exit the Lookup Transformation Editor dialog box. 22. Repeat Steps 11 to 21 in this Steps to Create Lookup Data Flow Items section to add the following Lookup item to the data flow:

7 Item Name Table Available Input Column Available Lookup Columns Lookup Product [dbo].[dimproduct] ProductCode ProductCode Steps to Create a SQL Server Data Destination Item 1. Drag a SQL Server (Dr. Joshi: OLE DB) Destination item from the SSIS Toolbox and drop it on the Data Flow tab. 2. Click the Lookup Product item. Click the blue data flow arrow, drag it on top of the SQL Server (Dr. Joshi: OLE DB) Destination item, and drop it on this item. The Input Output Selection dialog box appears. 3. Select Lookup Match Output from the Output drop-down list, and click OK. 4. Double-click the SQL Server (Dr. Joshi: OLE DB) Destination item. The SQL (Dr. Joshi: OLE DB) Destination Editor dialog box appears. 5. Select [dbo].[manufacturingfact] from the Use a table or view drop-down list. 6. Click Mappings to view the Mappings page. The columns from the data source (which are Available Input Columns to this data destination item) should be automatically mapped to the columns in the destination (the Available Destination Columns). The following mappings should be in place: Available Input Columns Available Destination Columns BatchNumber to BatchNumber MachineNumber to MachineNumber ProductCode to ProductCode AcceptedProducts to AcceptedProducts ElapsedTimeForManufacture to ElapsedTimeForManufacture DateOfManufacture to DateOfManufacture 7. Use drag-and-drop to add the following mapping: Available Input Columns Available Destination Columns NumberRejected to RejectedProducts 8. Click OK to exit the SQL Destination Editor dialog box. Steps to Create a Data Destination for the Rows in Error 1. Drag a Union All item from the SSIS Toolbox and drop it on the Data Flow tab. 2. Click the Lookup Batch Number item. Click the unattached blue data flow arrow, drag it on top of the Union All item, and drop it on this item. Note that the Lookup

8 No Match Output is connected to the Union All item. This means the unmatched records are going to follow this data flow. 3. Click the Lookup Machine Number item. Click the unattached blue data flow arrow, drag it on top of the Union All item, and drop it on this item. 4. Repeat this process with the Lookup Product item. 5. Drag a Flat File Destination item from the SSIS Toolbox and drop it on the Data Flow tab. 6. Change the name of this item to Write Error Records to a Text File. 7. Click the Union All item. Click the blue data flow arrow, drag it on top of the Write Error Records item, and drop it on this item. 8. Double-click the Write Error Records item. The Flat File Destination Editor dialog box appears. 9. Click New to create a new Flat File Connection Manager. The Flat File Format dialog box appears. 10. Leave the format set to delimited, and click OK. The Flat File Connection Manager Editor appears. 11. Enter Error Output for Connection manager name. 12. Click Browse. The Open dialog box appears. 13. Browse to an appropriate location for this error text file. Enter MfgInfoErrors.txt for File name. 14. Click Open to exit the Open dialog box. 15. Check the Column names in the first data row check box. 16. Click OK to exit the Flat File Connection Manager Editor. 17. Click Mappings to view the Mappings page of the Flat File Destination Editor dialog box. Mappings have been created for all columns. 18. Click OK to exit the Flat File Destination Editor dialog box. The Data Flow tab should appear as shown in Figure 8-18.

9 Figure 8-18: The data flow to load the Manufacturing fact table 19. Hold down CTRL small and click the following items in the data flow: Lookup Batch Number Lookup Machine Number Lookup Product Union All Write Error Records to a Text File 20. Right-click the Lookup Batch Number item, and select Group from the context menu. The selected items are all placed inside a grouping in the data flow. 21. Click the ^ in the upper-right corner of the Group box to contract the group. 22. Drag the group so it is between the Derived Column item and the SQL Server (Dr. Joshi: OLE BD ) Destination item. Position and size the items appropriately. 23. Rename the Group item to Validate Batch Number, Machine Number, and Product. The Data Flow should appear similar to Figure 8-19.

10 Figure 8-19: The data flow to load the Manufacturing fact table with the Validation Grouping Note A Group can be used in a control flow or in a data flow to help organize items and improve organization and readability. They do not have any effect on the operation of the SSIS package. You can undo a grouping by right-clicking the group item and selecting Ungroup from the context menu. Steps to Begin Creating the InventoryFact Data Flow 1. Click the Control Flow tab. Drag a third Data Flow Task item onto the Control Flow tab. Rename this new Data Flow Task item Load InventoryFact. (You do not need to set precedence between the Load InventoryFact item and the other Data Flow Task items. It does not matter when the InventoryFact table is filled relative to the operations being done on the other tables.) 2. Double-click the Load InventoryFact item. This takes you to the Data Flow tab. 3. Right-click in the Connections tray, and select New OLE DB Connection from the context menu. Click New in the Configure OLE DB Connection Manager dialog box to create a new data connection. 4. Enter the name of the server where the Maximum Miniatures Order Processing database is installed in Server name. Select the appropriate method for accessing this server in the Log on to the server section. Enter credentials if necessary. Select OrderProcessingSystem from the Select or enter a database name drop-down list. (If the drop-down list is empty, either your server name or your login credentials are incorrect.) Click OK to return to the Configure OLE DB Connection Manager dialog box. Click OK again to exit this dialog box. 5. Drag an OLE DB Source item from the SSIS Toolbox and drop it on the Data Flow tab.

11 6. Double-click the OLE DB Source item you just created. The OLE DB Source Editor dialog box appears. 7. Select the OrderProcessingSystem data connection that you just created in the OLE DB connection manager drop-down list. Leave the Data access mode dropdown list set to Table or view. Select [dbo].[inventory] from the Name of the table or the view drop-down list. Click OK. Steps to Create a Lookup Data Flow Item 1. Drag a Lookup item from the SSIS Toolbox and drop it on the Data Flow tab. 2. Change the name of this item to Lookup Product. 3. Click the OLE DB Source item. Click the blue data flow arrow, drag it on top of the Lookup Product item, and drop it on this item. This connects the source to the transformation. 4. Double-click the Lookup Product item. The Lookup Transformation Editor dialog box appears. 5. Select Redirect rows to no match output in the Specify how to handle rows with no matching entries drop-down list. 6. Click Connection to view the Connection page. 7. Select the MaxMinManufacturingDM data connection in the OLE DB connection manager drop-down list. 8. Select [dbo].[dimproduct] from the Use a table or a view drop-down list. 9. Click Columns to view the Columns page. 10. Click Code in the Available Input Columns list and drag-and-drop it on ProductCode in the Available LookupColumns list. 11. Click OK to exit the Lookup Transformation Editor dialog box. Steps to Create a Data Conversion Data Flow Item Dr. Joshi: (you do not need to convert the data type) 1. Drag a Data Conversion item from the SSIS Toolbox and drop it on the Data Flow tab. 2. Click the Lookup Product item. Click the blue data flow arrow, drag it on top of the Data Conversion item, and drop it on this item. The Input Output Selection dialog box appears. 3. Select Lookup Match Output from the Output drop-down list. Click OK. 4. Double-click the Data Conversion item. The Data Conversion Transformation Editor dialog box appears. 5. Check the check box next to Material in the Available Input Columns area. 6. Enter Material-Unicode in the Output Alias column. 7. Select Unicode string [DT_WSTR] from the drop-down list in the Data Type column. 8. Click OK to exit the Data Conversion Transformation Editor dialog box. Steps to Create a Data Destination Data Flow Item 1. Drag a SQL Server (Dr. Joshi: OLE DB) Destination item from the SSIS Toolbox and drop it on the Data Flow tab.

12 2. Click the Data Conversion (Dr. Joshi: Lookup Product) item. Click the blue data flow arrow, drag it on top of the SQL Server (Dr. Joshi: OLE DB) Destination item, and drop it on this item. This connects the transformation to the destination. 3. Double-click the SQL Server (Dr. Joshi: OLE DB) Destination item. The SQL (Dr. Joshi: OLE DB) Destination Editor dialog box appears. 4. Select the MaxMinManufacturingDM data connection in the OLE DB Connection Manager drop-down list. Select [dbo].[inventoryfact] from the Use a table or view drop-down list. 5. Click Mappings to view the Mappings page. Delete the line that joins the Material column in both lists. 6. Use drag-and-drop to add the following mappings: Available Input Columns Available Destination Columns Code to ProductCode InventoryDate to DateOfInventory AmountOnHand to InventoryLevel AmountBackordered to NumberOnBackorder Material-Unicode Material to Material 7. Click OK to exit the SQL Destination Editor dialog box. Steps to Create a Data Destination for the Rows in Error 1. Drag a Flat File Destination item from the SSIS Toolbox and drop it on the Data Flow tab. 2. Change the name of this item to Write Inventory Error Records to a Text File. 3. Click the Lookup Product item. Click the blue data flow arrow, drag it on top of the Write Inventory Error Records item, and drop it on this item. 4. Double-click the Write Inventory Error Records item. The Flat File Destination Editor dialog box appears. 5. Click New to create a new Flat File Connection Manager. The Flat File Format dialog box appears. 6. Leave the format set to delimited, and click OK. The Flat File Connection Manager Editor appears. 7. Enter Inventory Error Output for Connection manager name. 8. Click Browse. The Open dialog box appears. 9. Browse to the same folder you selected for the manufacturing error text file. Enter InvInfoErrors.txt for File name. 10. Click Open to exit the Open dialog box. 11. Check the Column names in the first data row check box. 12. Click OK to exit the Flat File Connection Manager Editor. 13. Click Mappings to view the Mappings page of the Flat File Destination Editor dialog box.

13 14. Click OK to exit the Flat File Destination Editor dialog box. The Data Flow tab should appear as shown in Figure Figure 8-20: The data flow to load the Inventory fact table Note: You will not have the Data Conversion item in your Data Flow 15. Click the Save All button on the toolbar to save the completed package. 16. Click the Start Debugging button on the toolbar to execute the completed package. When the execution is complete, click Stop Debugging or the Package Execution Completed link to return to Design mode. 17. Close the project.