Contents Getting Started... 3 About Scribe Online and Connectors... 3 Scribe Online Services... 3 CDK Components... 3 Audience... 4 Prerequisites...

Transcription

1 Contents Getting Started... 3 About Scribe Online and Connectors... 3 Scribe Online Services... 3 CDK Components... 3 Audience... 4 Prerequisites... 4 Requirements... 4 CDK Workflow... 5 Scribe Online Architecture... 6 Application Architecture... 6 Connector Roles... 6 Key Concepts... 8 Scribe Online Replication Services (RS)... 8 Scribe Online Synchronization Services (SYS)... 8 RS Source... 8 RS Target... 8 SYS Source / SYS Target... 8 Administrator... 9 User... 9 Design Considerations SYS Connectors RS Source Connectors RS Target Connectors Sample Projects Using the RS Source Sample Connector Using the RS Target Sample Connector Using the SYS Sample Connector Setting Up the Sample Database About the Sample Project Unit Tests Testing Unit Testing Overview of the Unit Testing Process Example Code Interpreting Test Results Attached Debugger Testing Setting up Verbose Logging User Acceptance Testing Developing Connectors Enabling the Scribe Online API in your Development Environment Enabling the Scribe Online API in Microsoft Visual Studio Connector Basics Developing Replication Service Sources RS Source Connector Overview RS Source Methods UI Generated Connection Role Testing and Debugging Setting up Verbose Logging RS Source Pseudo Code Developing Replication Services Targets

2 RS Target Connector Overview RS Target Methods UI Generated Connection Role Testing and Debugging Setting up Verbose Logging RS Target Pseudo Code Developing a Synchronization Service SYS Connector Overview SYS Methods Connection UI Generation Connection Role Testing and Debugging Setting up Verbose Logging SYS Pseudo Code

3 Getting Started Welcome to the Scribe Online Connector Development Kit (CDK). This CDK includes fully functional sample connector projects, unit test programs, and reference documentation for the Scribe Online API. About Scribe Online and Connectors Scribe Online is a cloud-based integration platform that enables customers and partners to integrate applications in cloud-to-cloud environments, or in hybrid environments integrating a cloud-based system with an on-site application. Scribe Online is architected to provide a core integration engine and an extensible set of connectors. Connectors enable Scribe Online to communicate with specific applications using the appropriate technology and API for each application. Connectors insulate the Scribe Online integration engine from the specific technologies needed to communicate with each application. Connectors also enable the integration engine to provide a common interface experience to end-users, allowing them to focus on the business processes required to complete an integration rather than the specific technologies involved. Scribe Online Services The Scribe Online integration platform provides multiple services. Connectors need to use different methods in the Scribe Online API depending upon the services you choose. Scribe Online Replication Service (RS) provides basic data replication and gives you the ability to easily maintain a copy of your source data (in a proprietary format such as Microsoft Dynamics CRM or Salesforce.com) in a Microsoft SQL Server database. Scribe Online Synchronization Service (SYS) provides two-way integration between source and target data stores. Scribe Online SYS enables you to map, filter, and transform data from Microsoft Dynamics CRM, Salesforce.com or Microsoft SQL Server as part of your data integration. CDK Components This CDK includes the following major components: Getting Started, Scribe Online Architecture, and Design Considerations Developing Connectors and Connector Basics Complete the sample connector projects in ScribeOnlineCDKSamples.zip Unit Testing Connector Reference 3

4 Audience This document has two intended audiences: 1. Business Analysts or Architects who need to understand the capabilities of the Scribe Online CDK. 2. Developers who need to implement a Scribe Online Connector for their application. Prerequisites Scribe recommends that the connector developers be competent in.net development and have a thorough understanding of the application API or technology being exposed to Scribe Online. Developers will also benefit from ancillary skills with web services, relational databases, and testing. Additional skills should include general API / SDK development using C++, C#, or Visual Basic; and supporting tools. All sample code and examples are written in C#. Developers do not need to have any previous experience with Scribe Online technologies, architecture, or internal data structures. Requirements The development computer must meet the following requirements: Processor (Pentium or equivalent): 800 MHz (Minimum); 1GHz (Recommended) Memory: 1 GB (Minimum); 4 GB (Recommended) Hard Disk: Up to 100 GB of available space Display: 800 x 600, 256 colors (Minimum); 1024 x 768 high color, 32-bit (Recommended) Operating Systems: Windows 7, Windows Server 2003, Windows Server 2008, Windows Server 2008 R2. Web Browsers: Microsoft Internet Explorer 7.x, 8.x,9.x; Mozilla Firefox 4.x, 5.x, 6.x; Google Chrome 15. To develop a Scribe Online Connector and make use of the CDK, you must have the following development environment available: Microsoft Visual Studio 2010 Microsoft.Net 4.0 The sample project scripts require SQL server. If you do not have access to an SQL Server instance, download a free copy of Microsoft SQL Server Express Edition. 4

5 CDK Workflow To develop connectors using this CDK, complete the following major steps in order: 1. Identify how your connector will be used to determine what Scribe Online role to assign to it. For more information see Connector Roles. All connectors need to be assigned a role. A role determines which Scribe Online services the connector can work with and whether the connector can be used as a source, as a target, or both. 2. Implement the methods required to support the connector roles. There are specific methods that need to be used in a connector depending on the connector s role. 3. Exercise unit tests to make sure the new connectors are functionally accurate and complete. This CDK includes unit testing programs to help you test your connector before you deploy it to use with Scribe Online and before you do user acceptance testing. 4. Deploy your connector locally so you can test it with Scribe Online. Deploying requires you to copy the connector to a specific folder on the computer where your Scribe Online agent is installed. Then after restarting your Scribe Online agent, Scribe Online will recognize your connector and let you use it in Scribe Online. 5. Complete User Acceptance Testing to make sure the connectors meet your expectations and the user s needs in the target environment. Scribe recommends that you test the connector by using it in several Scribe Online solutions that exercise all of the connector capabilities. This is the best way to know that your connector is working properly with Scribe Online and that it meets all of your requirements. 6. Distribute the connector. When your connector is ready for general use, you can make it available to your customers. 5

6 Scribe Online Architecture Scribe Online is a cloud-based integration platform that enables customers and partners to build integrations between any combination of cloud-based and on-premise applications. The API takes advantage of the latest Microsoft technologies, including.net 4.0, Silverlight, Enterprise Service Bus, Entity Framework, RIA Services and Windows Identify Framework. Scribe Online is hosted using Microsoft Azure Services. Scribe Online is architected to provide a core integration engine and an extensible set of connectors. Connectors enable Scribe Online to communicate with specific applications using the most appropriate technology and API for each application. Connectors insulate the Scribe Online integration engines from the details of any specific technology involved. Connectors also enable the integration engines to expose a common user experience to end-users, helping them focus on building the business processes required for a particular integration rather than the details of specific technologies involved. Application Architecture The following graphic shows the interaction between the components of a typical Scribe Online deployment. The Scribe Online API provides a means for developers to create custom connectors joining the components of a Scribe Online deployment. Connector Roles Scribe Online uses a concept called Connector Roles to identify different ways of using connectors. A connector s role is defined by the developer of the connector and is based on how they want the connector to be used. When Scribe Online loads a connector, the role indicates where the connector is selectable for use in the Scribe Online user interface. The role also defines which methods need to be implemented from the Scribe Online API. Connectors are not limited to a single role; in-fact a Synchronization Service connector is required to have 6

7 two roles (Scribe.IS.Target and Scribe.IS.Source). If a connector is available for both Replication Service (Scribe.RS.Target and Scribe.RS.Source) and Synchronization Service roles it is up to the connector developer to decide if the data store and their architecture of the connector can handle multiple role capabilities. 7

8 Key Concepts You should understand the following Scribe Online concepts before you begin developing your own connector. Scribe Online Scribe Online provides data integration capability to the user in the form of solutions. Each solution is designed enable a user to solve a set data integration problem, such as data replication or data synchronization. Users can configure any number instances from any of the available solutions within the context of a solution instance. Scribe Online Replication Services (RS) The Scribe Online Replication Service solution (RS) enables you to replicate data from either Microsoft Dynamics CRM 2011 Online, other cloud-based applications such as Salesforce, or on-premise deployments; to a Microsoft SQL Server database. Scribe Online Synchronization Services (SYS) The Scribe Online Synchronization Service solution (SYS) allows you to synchronize data between various online and on-premise data stores. SYS offers entity mapping, field transformation, source filtering and net change, and support for the standard create, read, update, and delete operations. Connector Roles You must develop connectors to support one or more of the following roles: Agents RS Source The Scribe Online RS Source is the origin of the data to be replicated and can be any data store that you want to replicate. Connectors with this role are available in Source connection selection for creation or editing of Scribe Online RS solutions. RS Target The Scribe Online RS Target is the destination of the data to be replicated and can be any data store that you want to store the replicated data. Connectors with this role are available in Target connection selection for Scribe Online RS solutions. SYS Source / SYS Target The Scribe Online IS Source and IS Target may be used as both a destination and origin of data to be synchronized using synchronization services (SYS). Connectors with these two roles are available for selection as a source and as a target in Scribe Online SYS solutions Scribe Online Agents facilitate communication between the data stores used by a solution. The Agent provides secure communications to Scribe Online without opening security holes in your corporate firewall. The Agent communicates with your source and target systems as well as with Scribe Online in the cloud. 8

9 Connections A connection is the login and other parameters you need to connect to any data store used by Scribe Online. Every connection is named, to allow you to organize and track your Scribe Online connections. Each connection provides access to a single data store; such as a Microsoft Dynamics CRM Organization or a SQL Server database. Organizations An Organization represents the company (or other entity) that is using this Scribe Online Solution. Users Scribe Online has two different user roles: Administrator A Scribe Online Administrator has full access to Scribe Online capability, including tasks such as creating and managing Organizations and Solutions Instances, installing Agents, and inviting other users to join Organizations. User Non-administrative users become members of an Organization by being invited to join by an Administrator. Users can perform many of the tasks that Administrators can, including creating and managing Organizations and Solution Instances, and installing Agents. However, a Scribe Online User cannot make changes to the Organization or its members. 9

10 Design Considerations SYS Connectors Scribe Online SYS Connectors complete the four fundamental create-read-update-delete (CRUD) operations. An SYS Connector parses metadata to determine if an object allows any or all of the CRUD operations. SYS Connectors require the following conditions: The metadata must provide an object s child-to-parent relationship information so that complex queries can be generated and executed through the connector. The metadata must provide property-level available actions, such as whether a field is read-only, has a default value, and must be included in a create operation. Update and delete operations must allow multiple objects to be accessed and modified with one call to the connector. Query executions to the connector must allow queried data to be organized in either ascending or descending order, for any specified field. For example, a DateTime field must be able to provide records newest to oldest in ascending order and oldest to newest in descending order. Query executions from the connector must be able to query against a root object for any and all data from the root object, and/or one or all related parent object data. RS Source Connectors Determine whether or not the data source you are connecting to already has a process in place to track changes in data. Specifically, does the data source use a timestamp to indicate a change in data? If the data source does not have a process for tracking changes, can a process be implemented via connector interaction with the data source? Connectors require a single timestamp per record that is changed the last time the data was synchronized to retrieve all records created, updated, or deleted since the last replication. If a process is not already in place to track data changes using a timestamp, you can implement change tracking, on either a global level and/or an object-by-object basis. For example: if the source is a database and does not track any changes, the connector must make supporting schema changes. Before replicating, execute InitReplication to create a Change History table. The Change History table tracks table names, timestamps, and primary key values. This table is updated by triggers created using InitReplicationObject on each table in the database. InitReplicationObject can also create table columns to track the date that the record was last modified. If no changes are required, returning success in either the InitReplication or InitReplicationObject method is sufficient. When a connector retrieves metadata, it must specify which field in the source contains the time stamp. The GetReplicationData method retrieves all replication data since the last time the 10

11 data was synchronized, and passes it using the method input properties from Scribe Online. If this is an initial replication, it sends the minimum date of 1/1/0000. The connector must determine whether or not to change this date, if it is not supported. In the example provided, the SQL database supports a minimum date of 1/1/1753 so the minimum date is reset to use this value. When a connector retrieves data from a source, it must be converted into a common data type to enable transfer between systems. Connectors must use.net standard data types as a universal means to transfer data. RS Source connectors must convert data to.net standard data types. RS Target connectors receive data as.net data types, and must convert data to the expected format of the target system receiving the replicated data. RS Target Connectors Before each replication, the agent compares metadata between the source and target connections. The target connector is responsible for detecting and handling changes in metadata. For example: if the target is a database, the target connector executes the CreateOrUpdateObjectForReplication method to detect changes such as a new column in the table. The table must be deleted and recreated with the new schema information. During replication, the table is refilled with the new data. If a new table was added, this method would also be responsible for creating a table that does not exist. 11

12 Sample Projects To help you develop connectors, Scribe Software has created a sample database and several sample projects available in a zip file: ScribeOnlineCDKSamples.zip. Extract the contents of the ScribeOnlineCDKSamples.zip file to your hard drive and browse for the following folders: \Binaries\ Contains the Scribe.Core.ConnectorApi file as well as all dependent libraries. Please reference the Scribe.Core.ConnectorApi to access the API features and include the supporting files with your project. \SampleRSSourceDatabase\ Contains SQL scripts that you can run from the Query Browser in Microsoft SQL Server Management Studio to create and populate a sample source database. You can use this database to test the ability of your RS Source connectors to communicate with a commonly used source. \Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.RS_Source\ Contains a complete sample connector for an RS source. You can use this connector as a template to develop RS source connectors. \Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.Source.RS_Test\ Contains C # unit tests for each Replication Service source connector class in the API: metadata providers, methods, operations, and connections. The test scripts validate both positive and negative use cases. You can use these scripts to confirm that your connectors are working properly. \Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.RS_Target\ Contains a complete sample connector for an RS target. You can use this connector as a template to develop RS target connectors. \Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.RS_Target.Test\ Contains C # unit tests for each Replication Service target connector class in the API: methods, operations, and connections. The test scripts validate both positive and negative use cases. Use these scripts to confirm that your connectors are working properly. \Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.SYS\ Contains a complete sample connector for Synchronization Service. You can use this connector as a template to develop SYS connectors. \Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.SYS.Test\ Contains C # unit tests for each Synchronization Service connector class in the API: metadata providers, operations, queries and connections. The test scripts validate both positive and negative use cases. Use these scripts to confirm that your connectors are working properly. 12

13 Using the RS Source Sample Connector The RS Source sample connector can be used with Scribe Online for exploration through user acceptance testing scenarios or through unit testing scenarios. You must first compile the RS Sample source project. If you are using the sample connector for exploration by using it with Scribe Online, you must deploy the connector by copying the.dll file to a directory under where the Scribe Online Agent is installed. If you are using the sample connector for exploration through unit testing scenarios, you do not have to copy the.dll file because it is referenced by the unit test project. Regardless of how you are using the RS Source sample connector, you must copy the following files into the directory where the RS Source sample connector resides: ScribeChangeHistory_Create.sql ScribeDelete_Trigger.sql These files are located in the \Samples\SampleRSSourceDatabase\ directory. You must also set up the sample database as described in Setting up the Sample Database. Using the RS Target Sample Connector The RS Source target connector can be used with Scribe Online for exploration through user acceptance testing scenarios or through unit testing scenarios. You must first compile the RS Target project. If you are using the sample connector for exploration by using it with Scribe Online, you must deploy the connector by copying the.dll file to a directory under where the Scribe Online Agent is installed. If you are using the sample connector for exploration through unit testing scenarios, you do not have to copy the.dll file because it is referenced by the unit test project. The unit testing project references the sample database. If the sample database is not installed, you must install it to unit test the RS Sample Target connector. Refer to Setting up the sample database. If you are going to use the RS Target Sample Connector in Scribe Online in a user acceptance testing scenario or in an attached debugger scenario, you must create an empty SQL Server database and you must have login credentials for a user that has full rights to the database. 13

14 Using the SYS Sample Connector The CDK includes a sample connector for SYS that functions as both a source and a target. You can use the sample SYS connector with Scribe Online for exploration through user acceptance testing scenarios or through unit testing scenarios. To use the SYS connector you must first compile the SYS Connector project. If you are using the sample connector for exploration by using it with Scribe Online, you must deploy the connector by copying the.dll file to a directory under where the Scribe Online Agent is installed. In this scenario you can use the connector to connect to any SQL Server database. If you are using the sample connector for exploration through unit testing scenarios, you do not have to copy the.dll file because it is referenced by the unit test project. The unit testing project references the sample database. If the sample database is not installed, you must install it to unit test the RS Sample Target connector. See Setting up the Sample Database for details. Setting Up the Sample Database Before setting up the database, make sure the supporting database applications are installed as described in the Requirements. To create the sample database needed to test Scribe Online connectors: 1. Start the Query Browser in Microsoft SQL Server Management Studio and open the \Samples\SampleRSSourceDatabase\ folder. 2. Run the first script SampleRSSource.sql. This script creates the sample database, ScribeSampleRSSource. 3. Run the second script SampleRSSourceData.sql. This script populates the database with sample data. You can repeat this step any time during testing, such as after testing the delete connector, to remove all previous changes and repopulate the database. Note: You must run these scripts in order. The order of operations mirrors how the Scribe Online production databases are created and populated. The database change history is required because new records are automatically discovered, but finding deleted records without a change history would require an inefficient and time-consuming line-by-line comparison between the source and target databases. 14

15 About the Sample Project Unit Tests Sample test code (when executed) only makes changes to the ScribeSampleRSSource database. The test code does not alter any other files. Some tests can only be run once (per database refresh) in order to return useful results. For example, a delete test deletes a database record, which cannot be deleted again. To refresh the sample database between tests, repeat Step 3 in Setting Up the Database. Test connectors make changes to both source and target databases. The source tests create a trigger for a specified table. This trigger updates the change history table with information about each deleted row in the specified database. The test code also deletes rows from the change history table once the deleted information has been sent to the target. The target requires an empty SQL database in a location that can be accessed and discovered by the sample test code. This database must be referenced in the connection information parameter through the connect method. The target has the ability to create and update table schema, as well as perform all create-read-update-delete (CRUD) operations. 15

16 Testing Scribe recommends that connector developers follow a testing process to verify that the connector is giving the expected results. The best practice for this is to do three levels of testing: unit testing, attached debugger testing, and user acceptance testing. This CDK provides a number of resources as examples to help you set up this testing process. Unit Testing This CDK provides sample projects that you can use as a model to set up unit tests. You can use the unit test projects included with the CDK that work with the sample connector projects and adapt the unit test projects to your connector. The sample unit test projects include tests for metadata, methods, operations, and connection. See Sample Projects for more information. Overview of the Unit Testing Process Complete the following process to develop and run tests: Download and open the Sample Projects. Create unit tests for methods Setting Up Tests. Test on the code Running Tests. View test results and fix errors in the code Interpreting Test Results. Note: You can build and run the sample connectors and unit tests included in the CDK. Refer to the Scribe Sample Projects and the Microsoft Developer Network Testing site for guidance. Example Code Scribe provides extensive unit tests for each of the CDK sample projects. These tests include examples of all expected method inputs and return values for the Scribe.Core.ConnectorApi, IConnector, and IMetadataProvider interfaces. This example code shows the basics elements of a connector s Connect() method for testing purposes. View any of the sample project code for more details. /// <summary> ///This is to test the third-party connection /// </summary> [TestMethod] public void SourceConnectorConnectionValidTest() //create a new instance of the sample connector var sourceconnector = new SourceConnector(); //setup the initial parameters for data connection Dictionary<string, string> properties = new Dictionary<string, string> "Provider", "SQLNCLI10", "Server", "localhost", "Database", "ScribeSampleRSSource", 16

17 ; "UserName", "sa" //encrypt the connection password using the shared key string encryptedpassword = Encryptor.Encrypt_AesManaged("sa", CryptoKey); properties.add("password", encryptedpassword); //call the connect method from the connector and pass //in the connection properties dictionary sourceconnector.connect(properties); //do a check that the IsConnected flag is true //and the connection has been opened Assert.IsTrue(sourceConnector.IsConnected); Interpreting Test Results When the test is complete, see the Test Results window for information about the success or failure of the test. For more information, see Analyzing Test Results in the MSDN Library. Attached Debugger Testing In addition to Unit Tests, connectors may be tested by attaching the connector project to the running Scribe Online agent process. This allows for precise testing of a connector that is being used by a running Scribe Online solution instance. To perform attached debugger testing, download and install the Scribe Online Agent from Scribe Online. To deploy your connector: 1. Stop the Scribe Online Agent Windows service. 2. Navigate to one of the following folders, depending upon your operating system: For 32-bit (X86) \Program Files\Scribe Software\Scribe Online Agent\Connectors\ -OR- For 64-bit (X64) \Program Files (x86)\scribe Software\Scribe Online Agent\Connectors\ 3. In this folder, create a new subfolder for the connector to test, such as Scribe.Connector.Cdk.Sample.Source or any unique name. 4. Copy all files built in debug mode from the Visual Studio project into the newly created folder. For example, Scribe.Connector.Cdk.Sample.RS_Source includes the following files in its folder: Scribe.Connector.Cdk.Sample.RS_Source.dll Scribe.Connector.Cdk.Sample.RS_Source.pdb ScribeChangeHistory_Create.sql ScribeDelete_Trigger.sql 17

18 5. Start the Scribe Online Agent Windows service. 6. Create a connection using the newly installed connector. It should appear in the list of available connections in the Scribe Online user interface. 7. Set up a Scribe Online solution that uses the new connection. 8. With the connector solution open in Visual Studio: a. Select Debug and Attach to Process. b. From the list of running processes, select Scribe.Core.ProcessorService.exe. c. Click Attach. See the Scribe Online Help Center for more information about working with Scribe Online Agents, which describes installing an agent, creating connections, and creating a solution instance. Setting up Verbose Logging All Scribe Online connectors include extensive logging messages that provide an accurate and detailed description of what occurs during replication. By default, installing a Scribe Online Agent activates standard logging. To turn one verbose logging: 1. Stop the Scribe Online Agent Windows service. 2. Navigate to one of the following folders, depending upon your operating system: For 32-bit (X86) \Program Files\Scribe Software\Scribe Online Agent\ -OR- For 64-bit (X64) \Program Files (x86)\scribe Software\Scribe Online Agent\ 3. Open the file Scribe.Core.ProcessorService.exe.config in NotePad with Administrator privileges, or make sure you are logged in as an account with Administrator privileges before opening the file. 4. Set Debug=true in the Scribe.Logger.LogEntrySeverities application setting. The edited line should match the following example: <add key="scribelogger.logentryseverities" value="none=true,debug=true,info=true,warning=true,fatal=true,error=true" /> 5. Save and close the file. 6. Start the Scribe Online Agent Windows service. 18

19 User Acceptance Testing User acceptance testing should be completed after unit testing, and attached debugger testing. Perform user acceptance testing on a different computer from the development computer. Install the connector as described in the Testing and Debugging section for each connector. Make sure the connector shows up in Scribe Online and only in the places appropriate to its declared role. For example, if the connector is for RS Source only, then it should only show up in an RS solution on the source connection list. Use the connector in a Scribe Online solution and make sure it performs as expected: o Connection dialog works and you can connect. o Connection data persists and can be edited. o Provides metadata. o Metadata is refreshed when schema changes are made against that connection. o If the connector is a source connector for SYS, does it provide source data? Does the net change feature work? Does the filter feature work? o If the connector is a source connector for SYS and DB has relationships defined, make sure that the parent relationships can be used in mapping. o If the connector is a target connector for SYS, does it perform all the CRUD operations it should support? o If the connector is a target for SYS, does it shows correct actions/operations the entity can support (part of metadata)? o If the connector is a source for RS, does it replicates read appropriate data into a target connection on both INITIAL replication and subsequent replications, with inserts, updates, and deletes in the source prior to the next run? o If the connector is a source for RS, can the user can select custom entities to be replicated? o If the connector is a target for RS, make sure that both initial and subsequent replications are replicated in Target RS? 19

20 Developing Connectors To develop your own Scribe Online Connectors, use Microsoft.NET tools and technologies. Each Connector consists of two basic components, the Connector itself, which is installed and runs on the Scribe Online Agent, and a User Interface (UI) component which can collect connection information from the user. A minimal connector requires that you implement two interfaces, requiring a total of 16 combined properties and methods. For this release of Scribe Online, Connector developers will have access to a dynamically generated user interface, which creates a UI similar to the following. Each connector uses the PreConnect method to generate this interface. Enabling the Scribe Online API in your Development Environment The Scribe Online API comes in the form of.dll files that you must reference from your development environment. The API includes the following files: Scribe.Core.Common.dll Provides basic Scribe Online functionality. Place a copy of this file in the build directory for your connectors. Scribe.Core.ConnectorAPI.dll Provides connector-specific functionality. Note: These files must exist in the same directory. 20

21 Enabling the Scribe Online API in Microsoft Visual Studio 1. Create a new development project from Visual Studio. 2. In the Solution Explorer tree, right-click on References and select Add Reference. The Add Reference dialog box opens. 3. Select the Browse tab and navigate to the folder containing the API.dll files. 4. Select Scribe.Core.ConnectorApi.dll and click OK. The Add Reference dialog box closes and the.dll file now appears listed in the references for the project. Visual Studio now has access to the components of the Scribe Online API. 21

22 Connector Basics When developing a Connector for Scribe Online RS, you must provide functionality to support using the Connector as either a replication source or a target. This section describes the Connector Interfaces and both the RS Source Methods, RS Target Methods, and SYS Methods. For a complete listing of the classes, interfaces, and enumerated lists that make up the Scribe Online Connector API see Scribe Online Connector Reference. The API, template connectors, and test scripts are available at: Connecter Interfaces The Scribe Online CDK uses two interfaces: IMetadataProvider Provides the metadata details to the Scribe Online RS Source Connector. IConnector Provides the runtime link between Scribe Online and the targeted application API or technology. While both interfaces must be fully implemented, many of the methods can be stubbed out depending on if you are developing an RS Source or Target Connector. The sections below detail what methods must be implemented for each type of Connector. Developing Replication Service Sources The Scribe Online RS Source is the origin of the data to be replicated and can be any data store that you want to replicate. Before developing an RS source connector, determine whether or not the data source you are connecting to already has a process in place to track data changes. Specifically, does the data source use a timestamp to indicate a change in data? If the data source does not have a process for tracking changes, can a process be implemented via connector interaction with the data source? Connectors require a single timestamp per record that is changed the last time the data was synchronized to retrieve all records created, updated, or deleted since the last replication. If a process is not already in place to track data changes using a timestamp, you can implement change tracking, on either a global level and/or an object-by-object basis. For example: if the source is a database and does not track any changes, the connector must make supporting schema changes. Before replicating, execute InitReplication to create a Change History table. The Change History table tracks table names, timestamps, and primary key values. This table is updated by triggers created using InitReplicationObject on each table in the database. InitReplicationObject can also create table columns to track the date that the record was last modified. If no changes are required, returning success in either the InitReplication or InitReplicationObject method is sufficient. 22

23 When a connector retrieves metadata, it must specify which field in the source contains the time stamp. The GetReplicationData method retrieves all replication data since the last time the data was synchronized, and passes it using the method input properties from Scribe Online. If this is an initial replication, it sends the minimum date of 1/1/0000. The connector must determine whether or not to change this date, if it is not supported. For example, if the SQL database supports a minimum date of 1/1/1753, the minimum date is reset to use this value. When a connector retrieves data from a source, it must be converted into a common data type to enable transfer between systems. Connectors must use.net standard data types as a universal means to transfer data. RS Source connectors must convert data to.net standard data types. RS Target connectors receive data as.net data types, and must convert data to the expected format of the target system receiving the replicated data. RS Source Connector Overview The following graphic details the interaction between the various methods, the agent, and a source connector: 23

24 RS Source Methods To develop a Scribe Online RS Source Connector, implement the RS Source methods for each Interface. An RS Source Connecter requires the following interfaces and properties: IConnector.PreConnect IConnector.Connect IConnector.Disconnect IConnector.ExecuteOperation IConnector.ExecuteMethod IMetadataProvider.RetrieveObjectDefinitions IConnector.Preconnect Preconnect is called before a connection has been established so that information can be exchanged between the Connector and its UI. Currently, this method is used solely to allow the adapter to configure the dynamic Connector UI that will be used for all Connectors. IConnector.Connect Connect instructs the Connector to connect to its data store. The Connector can choose to maintain or drop this connection as desired while the Connector is in use. At a minimum, implement this method to test the Connection when called. IConnector.Disconnect Disconnect is called to instruct the Connector to disconnect from its data store and free any resources that may be associated with the connection. IConnector.ExecuteOperation The ExecuteOperation method performs one or more operations, such as Create, Update or Delete, on a given data object type (such as an Account). The OperationInput object that is passed in provides the operation name, object name, data values and other information needed to perform the operation. The returned OperationResult object returns the results of the operation, including data or error information as needed. To support the Scribe Online RS Source role, only the Delete operation is required. Delete Operation ScribeOnline executes this method after synchronization has occurred. If data stored in the change history object is no longer needed it will be removed using the Delete operation. The LastSyncDate determines the range of records that need to be removed and is provided by Scribe Online through the LookupCondition (part of OperationInput object.) If this feature is not available in this Connector, create a placeholder method that returns success. 24

25 IConnector.ExecuteMethod ExecuteMethod is called by Scribe Online to perform any Connector-specific logic. Unlike operations, which always operate on a single object type such as an Account, the ExecuteMethod methods are capable of operating on and returning any number of different object types. For information about implementing this method, see the Scribe Online Connector Development Guide. For a Connector to support the Scribe Online RS Source role, this method must support the following ExecuteMethod source methods: GetObjectDefinition Get the specified 'Object' name, from the data source. Include the primary key or identifying item in the each of the objects. GetObjectDefinitionList Get the list of 'Object' names, (table names, in this case) from the data source. Include the primary key or identifying item in the each of the objects. GetReplicationData Scribe Online executes this source side method to retrieve data that will be replicated. Three pieces of information are provided to this method from Scribe Online that use properties found in the MethodInput object: ObjectName For SQL, the table that is requested for replication. ModificationDateFullName The field name containing the time that the record was last updated. LastSyncDate The date the system was last synchronized. GetChangeHistoryData This method is executed by ScribeOnline to retrieve change history information from a specific object. The MethodInput provides the name of the object that the information is being retrieved from (ObjectName) as well as the last synchronization date (LastSyncDate). InitReplication This method is executed by ScribeOnline to create an Object that keeps track of changes in the data source. For SQL, this object will be filled using created triggers. If triggers are not available for this connector, have the method return a success in the method result every time. InitReplicationObject This method is executed by Scribe Online to create a trigger function inside the data source that will be used to keep track of changes. This method should also be able to detect if the trigger already exists as well as call the InitReplication method to ensure that the object for replication exists. If this is not possible for this connector, the method should return a success in the method result every time. IMetadataProvider.RetrieveObjectDefinitions RetrieveObjectDefinitions is a required method called by Scribe Online to ask the Connector for a complete list of metadata object definitions. For example, a SQL Connector must provide table information that includes all columns and properties for each column. UI Generated Scribe Online Connector API allows for generation of connector specific UI. The call for the generic UI is done through the IConnector.PreConnect method. The developer must use the FormDefinition provided in the ConnectionUI namespace. Please see the sample connectors for an example of usage. 25

26 Connection When Scribe Online requests a connection the information required in the connection UI is sent through the method IConnector.Connect. It is up to the connector to handle the information that it requires for connecting to the origin of the data including decryption of sensitive data and establish a connection. Once the connection has been established, the IConnector.IsConnected flag must be set. The connector must also handle any retry logic that may be required if a connection fails or times out. Role The role of the Connector indicates what methods are being implemented from Scribe.Core.ConnectorApi, whether the connector will be accessing the data s source, the data s origin and which scribe solution type it will support. Roles are specified through the supportedsolutionroles parameter of ScribeConnectorAttribute. For replication service source connectors add Scribe.RS.Source to the array of supported solution roles. When the Connector is recognized by Scribe Online, the supported solution role will be recognized and ConnectorTypeName specified for the connector will only show up in the appropriate connection user interfaces. For example, if the role of a connector is Scribe.RS.Source, then the ConnectorTypeName will only appear for replication services and may only be used as a source connector. Scribe Online does allow for multiple roles in a connector. SYS connectors, for example, must support both Scribe.IS.Source and Scribe.IS.Target roles, and may support one or both of the RS roles. Note: Scribe Online does allow for multiple roles in a connector, in fact SYS connectors must support both Scribe.IS.Source and Scribe.IS.Target roles, and may support one or both of the RS roles. Testing and Debugging Once you have compiled your connector, you might want to install it and then test it. What follows here is an overview of how to do execute those processes. Refer to the Testing section to learn more about testing using the sample connector projects. To install a connector: 1. Stop the Scribe Online Agent Windows service. 2. Navigate to one of the following folders, depending upon your operating system: For 32-bit (X86) \Program Files\Scribe Software\Scribe Online Agent\Connectors\ -OR- For 64-bit (X64) \Program Files (x86)\scribe Software\Scribe Online Agent\Connectors\ 3. In this folder, create a new subfolder for the connector to test. 4. Copy all files built in debug mode from the Visual Studio project into the newly created folder. 5. Start the Scribe Online Agent Windows service. 6. The connector appears in the list of available connections in the Scribe Online user interface. 26

27 7. Set up a connection with the newly installed connector. 8. Set up a Scribe Online solution that uses the new connection. See the Scribe Online Help Center for more information about working with Scribe Online Agents, which describes installing an agent, creating connections, and creating a solution instance. Setting up Verbose Logging All Scribe Online connectors include extensive logging messages, including basic and verbose logging, which provide an accurate and detailed description of what occurs during replication. You can see how this type logging is implemented by exploring the sample connector projects. When connectors are running, they inherit the logging settings of the agent. By default, installing a Scribe Online Agent activates standard logging. To turn on verbose logging for the agent and therefore your connector: 1. Stop the Scribe Online Agent Windows service. 2. Navigate to one of the following folders, depending upon your operating system: For 32-bit (X86) \Program Files\Scribe Software\Scribe Online Agent\ -OR- For 64-bit (X64) \Program Files (x86)\scribe Software\Scribe Online Agent\ 3. Open the file Scribe.Core.ProcessorService.exe.config in Notepad with Administrator privileges, or make sure you are logged in as an account with Administrator privileges before opening the file. 4. Set Debug=true in the Scribe.Logger.LogEntrySeverities application setting. The edited line should match the following example: <add key="scribelogger.logentryseverities" value="none=true,debug=true,info=true,warning=true,fatal=true,error=true" /> 5. Save and close the file. 6. Start the Scribe Online Agent Windows service. 27

28 RS Source Pseudo Code To help you understand the flow of the Scribe Online RS solution, refer to the following pseudo code. This example illustrates the flow of the RS process and the interaction between the RS Source and Target Connectors: void MainProcess() // connect SourceConnector.Connect(); TargetConnector.Connect(); // inform source that we are starting replication SourceConnector.InitReplication(null); // get a list of the object types to be replicated var objectdeflist = SourceConnector.GetObjectDefinitionList(null); foreach(var objectdef in objectdeflist) // make sure target replication object has been created and is current var sourcefullobjectdef = SourceConnector.GetObjectDefinition(objectDef); var targetfullobjectdef = TargetConnector.GetObjectDefinition(objectDef); if (targetfullobjectdef == null) // target replication object not found // inform source that we are replicating this object SourceConnector.InitReplication(objectDef); // create target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); else if (targetfullobjectdef!= sourcefullobjectdef) // update target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); // call next process to replicate the data for the current object ReplicateData(objectDef); // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); void ReplicateData(object objectdef) // get the timestamp of the last entity synced from the target var lastreplicationdate = TargetConnector.GetLastReplicationSyncDate(objectDef); // query the source for all data changed since the lastreplicationdate 28

29 var dataentities = SourceConnector.GetReplicationData(lastReplicationDate); // walk the query results replicating the data foreach(var dataentity in dataentities) // attempt to insert the data into the target var success = TargetConnector.Create(dataEntity); if(!success) // insert failed, attempt to replace the row // remove TargetConnector.Delete(dataEntity); // replace success = TargetConnector.Create(dataEntity); // log an error on failure if (!success) LogRowError(objectDef); // call next process to soft delete any records deleted since last execution ProcessDeletes(objectDef, lastreplicationdate); void ProcessDeletes(object objectdef,datetime lastreplicationdate) if (lastreplicationdate!= DateTime.MinValue) // we have valid replication data // query the source for all deleted data since the lastreplicationdate var deletedentities = SourceConnector.GetChangeHistoryData(objectDef, lastreplicationdate); // walk the query results marking the deleted records as deleted in the target foreach (var deletedentity in deletedentities) TargetConnector.Update(deletedEntity); else // either an initial sync or an object with no last modified timestamp field // update all records not updated by the current execution as // deleted in the target TargetConnector.Update(objectDef, lastreplicationdate); 29

30 Developing Replication Services Targets RS target connectors are responsible for providing entity level, and property level metadata information to Scribe Online after initial synchronization. For each entity the connector must also specify a property that gets updated when a change occurs to a record. If this property does not exist in the destination then one must be created. This date is important to identify the point at which the last replication was run for the given entity. Before each replication, the agent compares metadata between the source and target connections. The target connector is responsible for detecting and handling changes in metadata. On initial replication it is up to the target connector to create a copy of the source metadata provided by Scribe Online. For example: if the target is a database, the target connector executes the CreateOrUpdateObjectForReplication method to detect changes such as a new column in the table. The table must be deleted and recreated with the new schema information. During replication, the table is refilled with the new data. If a new table was added, this method would also be responsible for creating a table that does not exist. 30

31 RS Target Connector Overview The following graphic details the interaction between the various methods, the agent, and a target connector: RS Target Methods To develop a Scribe Online RS Target Connector, implement each of the following IConnector methods: IConnector.PreConnect IConnector.Connect IConnector.Disconnect IConnector.ExecuteOperation IConnector.ExecuteMethod IConnector.Preconnect Preconnect is called before a connection has been established so that information can be exchanged between the Connector and its UI. Currently, this method is used solely to allow the adapter to configure the dynamic Connector UI that will be used for all Connectors. 31

32 IConnector.Connect Connect instructs the Connector to connect to its data store. The Connector can choose to maintain or drop this connection as desired while the Connector is in use. At a minimum, implement this method to test the Connection when called. IConnector.Disconnect Disconnect is called to instruct the Connector to disconnect from its data store and free any resources that may be associated with the connection. IConnector.ExecuteOperation ExecuteOperation is called by Scribe Online to perform one or more operations, such as create, update, or delete, on a given data object type (an Account, for example). For the Scribe Online RS Target role, you must implement all three operations (create, update, and delete). These operations are passed to the OperationInput object which stores the filtering information for that operation. These LookupConditions may contain information from a simple where clause to multiple comparison operations. The IConnector.ExecuteOperation target operations are: Create The Create operation allows a new row of data to be inserted into the target data source. Create is called when new data has been detected in the source during replication. Delete The Delete operation is executed by Scribe Online when an entity (row) of data has been deleted from the source data. Using properties stored in the Operation input this operation must allow for multiple deletions from a single Object (Table). Update The Update operation is executed when a data entity (row) has been changed in the RS source data. This operation uses filtering defined by the OperationInput.LookupCondition properties to update the selected fields. IConnector.ExecuteMethod Scribe Online calls ExecuteMethod to perform Connector-specific logic. Unlike operations which always operate on a single object type such as an Account, methods are capable of consuming and returning any number of different object types. The MethodInput object that is passed in provides the method name and other information needed to execute the method. The returned MethodResult object returns the results of the method execution, including data or error information as needed. The IConnector.ExecuteMethod target methods are: GetObjectDefinition ScribeOnline executes this method to retrieve a specific Object's definition, which includes any attributes and supporting object properties. CreateOrUpdateObjectForReplication ScribeOnline executes this method to create or update an object that is being replicated. The method input will provide the definition of the object and the property definitions in the MethodInput parameter. 32

33 Three things are considered when developing this method. First check if the object already exists and create a new one if it does not. Second compare the property definitions against the existing object and update the object when a change has been detected. Lastly if no change is detected no changes to the data source is made and note in the MethodInput that this is the case by defining a property SchemaChanged and setting it to false. GetLastReplicationSyncDate Scribe Online executes this target side method to retrieve the last date data was replicated for a specific object. Two pieces of information are provided to this method from ScribeOnline use properties found in the MethodInput object. First the ObjectName in the case of SQL this would be the table that is requested for replication. Second is the ModificationDateFullName this is the column or field name that an update time for a row of data has been updated. UI Generated Scribe Online Connector API allows for generation of connector specific UI. The call for the generic UI is done through the IConnector.PreConnect method. The developer must use the FormDefinition provided in the ConnectionUI namespace. See the sample connectors for an example of usage. Connection When Scribe Online requests a connection the information required in the connection UI is sent through the method IConnector.Connect. It is up to the connector to handle the information that it requires for connecting to the origin of the data including decryption of sensitive data and establish a connection. Once the connection has been establish the IConnector.IsConnected flag must be set. The connector must also handle any retry logic that may be required if a connection fails or times out. Role The role of the Connector indicates what methods are being implemented from Scribe.Core.ConnectorApi, whether the connector will be accessing the data s source, the data s origin and which scribe solution type it will support. Roles are specified through the supportedsolutionroles parameter of ScribeConnectorAttribute. For replication service target connectors add Scribe.RS.Target to the array of supported solution roles. When the Connector is recognized by Scribe Online the supported solution role will be recognized and ConnectorTypeName specified for the connector will only show up in the appropriate connection user interfaces. For example, if the role of a connector is Scribe.RS.Source then the ConnectorTypeName will only appear for replication services and may only be used as a source connector. Scribe Online does allow for multiple roles in a connector, in fact SYS connectors must support both Scribe.IS.Source and Scribe.IS.Target roles, and may support one or both of the RS roles. 33

34 Testing and Debugging Once you have compiled your connector, Scribe recommends installing and testing the new connector. This section provides an overview of how to complete this process. Refer to the Testing section to learn more about testing using the sample connector projects. To install a connector: 1. Stop the Scribe Online Agent Windows service. 2. Navigate to one of the following folders, depending upon your operating system: For 32-bit (X86) \Program Files\Scribe Software\Scribe Online Agent\Connectors\ -OR- For 64-bit (X64) \Program Files (x86)\scribe Software\Scribe Online Agent\Connectors\ 3. In this folder, create a new subfolder for the connector to test. 4. Copy all files built in debug mode from the Visual Studio project into the newly created folder. 5. Start the Scribe Online Agent Windows service. 6. The connector appears in the list of available connections in the Scribe Online user interface. 7. Set up a connection with the newly installed connector. 8. Set up a Scribe Online solution that uses the new connection. See the Scribe Online Help Center for more information about working with Scribe Online Agents, which describes installing an agent, creating connections, and creating a solution instance. Setting up Verbose Logging All Scribe Online connectors include extensive logging messages, including basic and verbose logging, which provides an accurate and detailed description of what occurs during replication. You can see how this type logging is implemented by exploring the sample connector projects. When connectors are running, they inherit the logging settings of the agent. By default, installing a Scribe Online Agent activates standard logging. To turn on verbose logging for the agent and therefore your connector: 1. Stop the Scribe Online Agent Windows service. 2. Navigate to one of the following folders, depending upon your operating system: For 32-bit (X86) \Program Files\Scribe Software\Scribe Online Agent\ -OR- For 64-bit (X64) \Program Files (x86)\scribe Software\Scribe Online Agent\ 3. Open the file Scribe.Core.ProcessorService.exe.config in Notepad with Administrator privileges, or make sure you are logged in as an account with Administrator privileges before opening the file. 4. Set Debug=true in the Scribe.Logger.LogEntrySeverities application setting. The edited line should match the following example: 34

35 <add key="scribelogger.logentryseverities" value="none=true,debug=true,info=true,warning=true,fatal=true,error=true" /> 5. Save and close the file. 6. Start the Scribe Online Agent Windows service. RS Target Pseudo Code To help you understand the flow of the Scribe Online RS solution, refer to the following pseudo code. This example illustrates the flow of the RS process and the interaction between the RS Source and Target Connectors: void MainProcess() // connect SourceConnector.Connect(); TargetConnector.Connect(); // inform source that we are starting replication SourceConnector.InitReplication(null); // get a list of the object types to be replicated var objectdeflist = SourceConnector.GetObjectDefinitionList(null); foreach(var objectdef in objectdeflist) // make sure target replication object has been created and is current var sourcefullobjectdef = SourceConnector.GetObjectDefinition(objectDef); var targetfullobjectdef = TargetConnector.GetObjectDefinition(objectDef); if (targetfullobjectdef == null) // target replication object not found // inform source that we are replicating this object SourceConnector.InitReplication(objectDef); // create target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); else if (targetfullobjectdef!= sourcefullobjectdef) // update target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); // call next process to replicate the data for the current object ReplicateData(objectDef); // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); 35

36 void ReplicateData(object objectdef) // get the timestamp of the last entity synced from the target var lastreplicationdate = TargetConnector.GetLastReplicationSyncDate(objectDef); // query the source for all data changed since the lastreplicationdate var dataentities = SourceConnector.GetReplicationData(lastReplicationDate); // walk the query results replicating the data foreach(var dataentity in dataentities) // attempt to insert the data into the target var success = TargetConnector.Create(dataEntity); if(!success) // insert failed, attempt to replace the row // remove TargetConnector.Delete(dataEntity); // replace success = TargetConnector.Create(dataEntity); // log an error on failure if (!success) LogRowError(objectDef); // call next process to soft delete any records deleted since last execution ProcessDeletes(objectDef, lastreplicationdate); void ProcessDeletes(object objectdef,datetime lastreplicationdate) if (lastreplicationdate!= DateTime.MinValue) // we have valid replication data // query the source for all deleted data since the lastreplicationdate var deletedentities = SourceConnector.GetChangeHistoryData(objectDef, lastreplicationdate); // walk the query results marking the deleted records as deleted in the target foreach (var deletedentity in deletedentities) TargetConnector.Update(deletedEntity); 36

37 else // either an initial sync or an object with no last modified timestamp field // update all records not updated by the current execution as // deleted in the target TargetConnector.Update(objectDef, lastreplicationdate); Developing a Synchronization Service Scribe Online SYS Connectors complete the four fundamental create-read-update-delete (CRUD) operations. An SYS Connector parses metadata to determine if an object allows any or all of the CRUD operations. SYS Connectors require the following conditions: The metadata must provide an object s child-to-parent relationship information so that complex queries can be generated and executed through the connector. The metadata must provide property-level available actions, such as whether a field is read-only, has a default value, and must be included in a create operation. Update and delete operations must allow multiple objects to be accessed and modified with one call to the connector. Query executions to the connector must allow queried data to be organized in either ascending or descending order, for any specified field. For example, a DateTime field must be able to provide records newest to oldest in ascending order and oldest to newest in descending order. Query executions from the connector must be able to query against a root object for any and all data from the root object, and/or one or all related parent object data. 37

38 SYS Connector Overview The following graphic details the interaction between the various methods, the agent, and a synchronization connector: SYS Methods To develop a Scribe Online SYS Synchronization Connector, implement each of the following IConnector methods: IConnector.PreConnect IConnector.Connect IConnector.Disconnect IConnector.ExecuteOperation IConnector.ExecuteQuery IConnector.GetMetadataProvider IMetadataProvider.RetrieveActionDefinitions IMetadataProvider.RetrieveObjectDefinitions IMetadataProvider.RetrieveObjectDefinition IMetadataProvider.ResetMetadata 38