LAB: Replicating SharePoint Online List to SQL Server

Transcription

1 Get things done. LAB: Replicating SharePoint Online List to SQL Server Copy data from a SharePoint Online list to a SQL Server database in the cloud, and keep the database in sync automatically when data changes in the SharePoint list using Enzo Data Pipelines. Created by: Herve Roggero, Azure MVP Released On: 8/15/2018 info@enzounified.com

2 The purpose of this lab is to demonstrate the steps needed to replicate a SharePoint Online list to a SQL Server database in the cloud, with automatic synchronization. Synchronizing data from a SharePoint list to a SQL Server database gives administrators the ability build custom reports on top of database tables and a simpler data integration mechanism. This Lab uses the following technologies: Enzo Online SharePoint Online Azure SQL Database SQL Server Management Studio (or equivalent database management tool) o You will be running a few SQL statements in this lab, so a database management tool will be helpful (but not required) NOTE: This lab does not require any coding. Pre-Requisites Before starting this lab you should have the following: Services o An Enzo Online account ( You can create a free account o A SharePoint Online site ( You can create a free account o A cloud hosted SQL Server database ( You can use Microsoft Azure or AWS RDS for example You can create a free account To follow instructions on how to create a database in Azure, see this documentation: Overview This lab uses Enzo Data Pipelines to replicate SharePoint Online lists to a SQL Server database in the cloud. Enzo Data Pipelines is a feature of Enzo Online that provides copy and synchronization between SharePoint and destination systems, such as a database, a messaging system, or even a cloud message bus. Once Enzo Data Pipelines are configured, data synchronization happens automatically at a scheduled interval.

3 Preliminary Setup In order to go through this lab successfully you will need to have a specific SharePoint list and a corresponding database table created. The database table should be created with specific columns so that the data can be saved successfully. This lab will show you how to obtain the exact statement to execute when creating the database; the statement will be provided by Enzo Online so you can run it as-is on your database. To create the SharePoint list we will use for this lab, login to your SharePoint Online portal and follow these steps: Go to your Team Sites content ( Select New -> List Enter a name for the list: My Contacts and click Create Add the following columns: o Company (single line of text) o (single line of text) o Family (Yes/No, default No) o Age (Number)

4 That s it you now have a simple SharePoint Online list that you can use to enter some data. Add a handful of records to your list before processing. Here is some sample data: Herve hroggero@bluesyntax.net Blue Syntax No 40 Jim Nastic jimnastic@ .com Yes 39 Rose Mary rmary@ .com Rose Inc. No 30 Sultan Peper speper@ .com Yes 22 Rose Beef rbeef@ .com Rose Inc. No 21 Enzo Online Configuration Configure SharePoint Now that you have a SharePoint site configured with a list, let s register the site in Enzo Online so that Enzo Data Pipelines can read from SharePoint. Login to Enzo Online ( and follow these steps: Click on SharePoint on the left bar Click Create and enter the following information: Name spconfig This is a friendly configuration name for this SharePoint site Enzo Instance <pick one> This is the server on which this configuration will be saved SharePoint Site URL This is the URL of the Site where the List was created User Name Your Login Name for SharePoint This is usually an address that you use when you login Password Your SharePoint password This is the password for the User Name provided Use Lookup IDs YES Check this box to receive reference links instead of names (better for data replication) Click Save Configure a Connection String Next we will configure a database connection string so that Enzo Data Pipeline can send data.

5 Click on Connection Strings on the left bar Click on Create Enter the following information: Name sqlconfig This is a friendly configuration name for this connection string Enzo Instance <pick one> This is the server on which this Connection String Click Save Server=DBSERVER;Initial Catalog=DBNAME;User ID=USERID;Password=PWD; Encrypt=True; configuration will be saved This is the full connection string to your database, including the user name and password Create A Data Pipeline You are now ready to create a data pipeline that will communicate to SharePoint and extract data. A data pipeline works independently from the destination systems; the pipeline keeps two data stores: an INIT and a CDC store. The INIT store allows destination systems to receive a copy of the data source as it was when the pipeline was created; the CDC store allows destination systems to receive changes that took place over time since the pipeline was created. To create a data pipeline, click on Enzo Pipeline on the left bar in the Enzo Portal; then follow these steps: Enter the following information: Name SP Source My Contacts This is a friendly configuration name for this pipeline Enzo Instance <pick one> This is the server on which this configuration will be saved Source System SharePoint The source system to sync from Configuration spconfig The configuration name to use for SharePoint Source Table My Contacts This list is pulled from SharePoint directly. If no names are available, there may be an issue with your SharePoint settings Schedule 15 minutes This is how often data will be replicated Pipeline Mode Sync + CDC Indicates that both the INIT and CDC stores should be created for the pipeline Click Create

6 Your screen should look like this: Once created, your screen should look like this (NOTE: you may only have a single pipeline in your list; the free edition of Enzo Online only allows you to create one pipeline and one listener). Create a Listener Now that the pipeline is created, we will create the listener that will forward the information to the SQL Server database. A listener can forward data captured by the pipeline. The availability of the INIT and CDC stores depend on the Pipeline Mode selected at the time the pipeline was created. Since we selected Sync + CDC, both are available for this listener.

7 Click on the Add button (circled in the preview screenshot), and follow these steps: Enter the following information, but do not click Create immediately: Name SQL Dest 1 This is a friendly configuration name for this listener Destination System Database The Database destination systems allows you to select the previously created connection string Destination Configuration sqlconfig The database connection string previously created Destination Object/Table sharepoint.mycontacts Enter the name of a database table in the following format: SCHEMA.TABLE this table will be created later Processing Mode Call destination for each record This indicates that the CDC data will be played back one record at a time Initialize Yes The listener will restore the INIT data first Delete CDC Yes The listener will playback all Delete CDC records Upsert CDC Yes The listener will playback all inserts and updates Column List Age, Company, Created, , Family, ID, Modified, Title Click the Edit button to select these fields. It is not recommended to select all fields for performance reasons Max Retries 3 Number of retries before CDC records error out Click on the View/Execute DDL Statement button; this represents the DDL SQL command to execute on your database to create the schema and table where the data will be saved. The DDL Statement provided uses the Column List selected so that only the necessary fields are created in the destination table. Before clicking Create, you must create the destination table or the Listener will fail. To create the destination database table, you can either click on the EXECUTE button on the Database Schema popup screen, or use SQL Server Management Studio and run the DDL statement provided.

8 NOTE: a database must already be created; this script does not create a database automatically. Run the DDL Script by clicking on the EXECUTE button, then close the window. For Azure SQL Databases, do not specify the Database Name; it is already provided as part of the database connection string. Click Create Running and Monitoring the Pipeline We will now force-start the synchronization and monitor the pipeline and the listener. Back on the Enzo Pipeline main screen, click Run Now on your pipeline. The pipeline and the listener will execute within a couple of minutes.

9 Then click on the pipeline name link; this opens up another page with three tabs; click on the Statistic tabs. Depending on the amount of data is being captured, the Pipeline status will be RUNNING. In the screenshot below, the pipeline is executing, and the listener has not yet run once. To refresh the screen simply reload the page. Once the listener has picked up the INIT records and completed pushing the data to the destination database, the listener will be in a READY state. You can also verify that the records have been pushed to the database using SQL Server Management Studio using this SQL command: select * from sharepoint.mycontacts

10 Additional Capabilities Now that your pipeline is running, and a listener is forwarding data to a database, you can try a few additional things on your own. Add/Update/Edit SharePoint Items You can modify your SharePoint list and wait for the pipeline to pick up the changes at the next scheduled interval. Replay INIT and/or CDC Changes Replaying changes can be performed for both the INIT and/or the CDC store. If you add a new record in SharePoint, let the record synchronize with the database, and then delete the record in the database, replaying the CDC log will reinstate the missing record. If you want to completely rebuild the table, you would replay the entire log, which replays the INIT and the CDC log. Create SMS Listeners Another feature is the ability to forward CDC changes to a phone using SMS for example; it is possible (and simple) to configure a new Listener that forwards records to Twilio for example.