Dynamically build connection objects for Microsoft Access databases in SQL Server Integration Services SSIS

Transcription

1 Dynamically build connection objects for Microsoft Access databases in SQL Server Integration Services SSIS Problem As a portion of our daily data upload process, we receive data in the form of Microsoft Access files (*.mdb) which needs to get uploaded to a SQL Server 2005 database. We want to build a SQL Server 2005 Integration Services package that loads all our Microsoft Access files and uploads data based on parameters in a table in our database. We want to make this an automated process, but the number of Access files may change over time. Do you have any solutions on how to build the SSIS package such that we do not have to modify our SSIS package for each new file? Solution In this example, let us assume that our sales force sends their sales order information as an Access file and each each sales team has their file stored in a different folder. All available Access sales files need to be loaded in the same process to keep the sales information current. The company is continually adding new sales teams, but not all sales teams will submit a file, so we want to be able to indicate the files (by location) that we want processed dynamically when the SQL Server 2005 Integration Services (SSIS) Package is executed. The first step is to build a listing of the Microsoft Access files that need to be processed. For this particular solution, the active Access file locations will be stored in a table in the SQL Server database. Here is the simple create table script to hold the file information. Sample Access Table Definition CREATE TABLE [dbo].[accessdatafiles]( [ID] [int] IDENTITY(1,1) NOT NULL, [FileLocation] [varchar](500) NOT NULL, [Updated] [datetime] NOT NULL CONSTRAINT [DF_AccessDataFiles_Updated] DEFAULT (G ETDATE()) ) ON [PRIMARY] When populated, the table will look something like below. SQL Server Integration Services Package For the Integration Services Package, our solution include the following steps: 1. Execute a SQL query to select records containing the file locations. 2. Iteration over the records returned, creating a connection string for each Access file. 3. Set the variables that are assigned to the OLE DB connection object for each the Access file. 4. Execution of the data flow operation to extract data from an Access table into a SQL Server table. To implement this solution, create an SSIS project in the SQL Server 2005 Business Intelligence Development Studio. Create a connection to one of the Access files (call it

2 AccessOrderMDB). Next, create an OLE DB connection to the SQL Server database (call it AccessUploadDB). With our connections in place, we are ready to build the package. Step 1. Retrieving the file locations During this step, we will use SSIS to create a record set from a SQL statement that will be used during the looping step. Add a variable called maccessfiles at the package level with the data type of Object. Add an Execute SQL Task to the Control Flow surface. Set the following properties in the General tab (double click on the object to launch the Execute SQL Task Editor): Parameter ResultSet Connection SQLSourceType SQLStatement Value Full result set AccessUploadDB Direct input Select FileLocation From dbo.accessdatafiles Select the "Result Set" tab on the Execute SQL Task Editor and set the following properties:

3 Result Name Variable Name 0 User::mAccessFiles Step 2. Configure the ForEach loop Add a Foreach task to the Control Flow design surface. Within the Foreach task add a Data Flow task. Your solution should look like the following: In the Foreach Loop task, select the Foreach Loop Editor (double click on the task or select Edit from the context menu).

4 Select the Collection tab, and in the Enumerator property, set the drop down value to Foreach ADO Enumerator. In the Enumeration configuration settings, under ADO object source variable, select the variable we previously set in the proceeding step, User::mAccessFiles. Also, select Rows in the first table. This makes it possible to iterate over the returned rows stored in our variable as an ADO object.

5 In the variable mappings map the variable as follows:

6 Step 3. Dynamically configure the Access connection object through Expressions To successfully connect to an Access file both the connection string and ServerName values must be set correctly. When you use the connection wizard, these values are properly populated. Unfortunately, when you want to dynamically configure your connection to the Access files, you must set both. Note the Properties values below.

7 To set these values dynamically the properties should be changed as follows: Property ConnectionString ServerName Value "Data Source=" + ";Provider=Microsoft.Jet.OLEDB.4.0;User Step 4. Set Dataflow Properties Now we simply configure our data flow using our OLE DB Source object from our Access table to pump data into our SQL Server database OLE DB Destination table. The OLE DB connection manager should point to our Access connection object. Since these steps are typical data flow operations, the details will be left to the reader. Next Steps Configuring your SSIS package to support dynamic connections to multiple Access databases can be achieved, but it is important to remember that you need to set two properties to successfully connect to the Access files, the ConnectionString and the ServerName.

8 Passing dynamic parameter values to SQL Server Integration Services SSIS Problem When using SQL Server Integration Services (SSIS) the ideal situation is to make the code as re-useable as possible, so the same code-set can be used to handle multiple situations instead of having a hard-coded solution. In a previous tip, "Dynamic Flat File Connections in SQL Server Integration Services" we looked at how to create a dynamic file source based on some variable settings within the SSIS package. This solution was great, but how do I take this further and pass in a dynamic value into an SSIS package? Solution As with Data Transformation Services (DTS) in SQL Server 2000, SSIS also gives you the ability to pass in parameter values directly to the SSIS package at run time. With SSIS the syntax is different than with DTS, but the options that you have are much greater than what was available in DTS. To show you some simple examples we are going to create a flat file source that will import data into a SQL Server table. The first solution will be a hardcoded option and then we will call the SSIS package from the command line and pass in new parameter values for multiple parts of the package. We first create a new SSIS package and use a "Data Flow Task". Then we add a "Flat File Source" and an "OLE DB Destination". The flat file source is a CSV file that has these columns ID, Name, Address, City, State and Zip. The SQL Server table has these exact same columns. With the values hardcoded the package executes without a problem. To take advantage of the dynamic aspects of an SSIS package we create a new variable called "filename' which will take the path and the name of the file we are passing into the

9 SSIS package. For more information on setting up a dynamic flat file source take a look at this tip "Dynamic Flat File Connections in SQL Server Integration Services". First we create a new variable called "filename". After the variable has been created we create an Expression for the flat file connection, which is shown below. At this point our filename that we are importing is called "c:\temp\test2.csv" (which is a non-existent file) this is based on the variable that we just set. If we execute the package now the package fails with the following message. The reason the variable was set to this non-existent file was just to illustrate that the package will fail, but will run when we pass in a good file name. The following is syntax to execute the SSIS package along with passing in the variable value for the filename variable. This syntax creates the command to run the SSIS package, but uses the 'C:\temp\test.csv" filename instead of "C:\temp\test2.csv. This code can be pasted into a query window and executed. (note: you must have xp_cmdshell enabled to run the following code. Take a look at this tip on how to enable xp_cmshell in SQL Server 2005.) varchar(1000) varchar(1000) varchar(1000)

10 = 'C:\temp\Package.dtsx' = 'C:\temp\test.csv' = 'dtexec /F "' + '"' + ' /SET \Package.Variables[User::fileName].Properties[Value];"' + '"' exec When this code is run the end of the query output will look something like this, to show the package ran successfully. To take this a step further, let's say you want to pass two variables to the package. You need to do pretty much the same thing. Here we now have a variable for filename and another one for filepath. If we look at the expression, it still works the same way, but we have both variables making up the expression now. The code to pass the two variables to the package is shown below. For the most part the syntax is identical. We now have two /SET commands to pass in each variable.

11 (Note: notice that we had to use a \\ at the end of the filepath. If you only use one such as "C:\temp\" the package will give you an execution error. At this point you can execute the package and the two variables we be passed into the SSIS package. varchar(1000) varchar(1000) varchar(1000) varchar(1000) = 'C:\temp\Package2.dtsx' = 'C:\temp\\' = 'test.csv' = 'dtexec /F "' + '"' + ' /SET \Package.Variables[User::filePath].Properties[Value];"' + '"' + ' /SET \Package.Variables[User::fileName].Properties[Value];"' + '"' exec We only hit upon passing user defined variables to a package, but you can pass all sorts of different values dynamically to control how your SSIS package behaves. Next Steps Start making your SSIS packages as reusable and portable as possible by passing in dynamic variable values Take a look at these other SSIS tips