SAP Sybase Replication Server Change DATA Capture Configuration Example Configuration
TABLE OF CONTENTS 1 INTRODUCTION... 3 1.1 Scope... 3 1.2 Purpose... 3 2 CONFIGURATION... 4 2.1 Requeriments... 4 2.2 Notes... 4 2.2 Configuration steps... 4 2
1 INTRODUCTION New SAP solution called SAP Sybase Replication Server Change Data Capture combines SAP Sybase Replication Server, one of the most used and powerful tools for real-time data replication, with SAP Data Services, one of the most used and powerful ETL tools. Implementing this new solution gives you the ability to perform real-time replication with complex data transformations for your business processes. 1.1 Scope This document is useful for people interested in installing this solution and checking how to configure SAP Sybase Replication Server to capture data changes and send them to a CDC staging database to be processed by SAP Data Services. 1.2 Purpose The basic purpose of the document is be used as cookbook to show how to model a SRS process with PowerDesigner 16.5 to create a CDC staging database is ASE 15.7. PowerDesigner will be able to generate data model for the CDC staging DB and also data movement model and will create all scripts needed (both at DB level and at SRS level) to active data replication to the CDC staging DB. 3
2 CONFIGURATION 2.1 Requeriments 2.2 Notes SAP Sybase Repliation Server 15.7.1 SAP ASE 15.7 SAP Data Services 4.2 PowerDesigner 16.5 For this example/demo primary DB (called testdb ) will be located in an SAP ASE 15.7 (called ASE157) but it can be any source DBMS supported by SRS. CDC staging database will be located in the same ASE and will be called testdb_stage. Replication server is supposed to be installed, running and with the connections to the primary DB and staging DB created. This doc describes configuration related to SAP Sybase related products and not Data Services. 2.2 Configuration steps 1. testdb and testdb_stage will be located in same ASE (ASE157). In testdb there is a table t1 that will be replicated (noticed that the table must have a primary key). testb_stage database has just been created and no user tables in it (only SRS tables). 4
2. Replication server is configured, up and running with connections to both testdb and testdb_stage created: 5
3. Power Designer automates the process of configuring Sybase Replication Server, which includes marking the source table as replicated, creating data model for the CDC staging database and creating replication definitions and subscriptions. The extended PD model for Data Services CDC distributed with Data Services 4.2 has to be copied to the extended models definitions folder in PowerDesigner (the.xem file is called SybaseRepServerDataServicesCDC100.xem): 6
4. PDM model created for the primary db: 5. Create a new Data Movement Model: 7
6. Attach extensions SAP Data Services CDC and Replication Server 15.7.1 SP100 (see select extensions button on the right-bottom side of the window): 7. Run the Replication Wizard. It creates the replication server process, source physical data model and replicated physical data model. In this case physical model for the source has been created previously. From PowerDesigner Tools Replication Wizard Source Database is ASE157.testdb 8
8. Choose Select existing Physical Data Model and check ASE157-testdb that this the model for the source database 9. 10. Use the repserver name as name for the replication process, in this case r157: 9
11. In Replication Wizard: Publication Selection, selects Create or select a single publication to gather replicated tables and give a name for the publication: 12. Select objects to replicate, in this case only table t1: 10
13. Connection to the CDC staging database: 14. Create a new physical data model for CDC staging DB for ASE 157: 11
15. Finish and get the Data Movement Model: 12
16. After completion of Replication Wizard, Power Designer will create default replication definitions (called Articles) and PDM for staging database with all selected tables from source PDM and corresponding table columns and properties. Double click on Replication Server process Publications & Replication Definitions to see the publication: 13
17. Check properties for the publication to see the articles included in it: Go further to check article properties (columns, where clauses etc), it is possible to check the generated code for the publication and articles in the preview tab: 14
15
18. Configure a connection to CDC staging DB: On the object Browser, right click ASE157.testdb_stage (CDC staging DB) and select properties 19. Inside Database Properties window select Database Connection tab, click Select a Data Source and select the DSN for the connection to staging DB or create it if it does not exists: 16
Click OK 20. Save DMM and PDM. 17
21. Select Replication Process icon in DMM diagram and choose Add Data Services CDC Definitions from context menu to run Data Services extension to create: a. CDC replication definitions, b. CDC staging tables, c. CDC system tables (DS_CDC_TABLES_MAP, DS_CHANGE_RETENTION and DS_PURGE_TIMESTAMP), d. PURGE_OLD_CHANGE_DATA stored procedure and e. User SAPDS. 18
22. Check CDC system tables, stating tables, PURGE_OLD_CHANGE_DATA stored proc, user SAPDS and publications/replication definitions have been created, it is possible to check function strings too: 19
23. Running Add Data Services CDC Definitions command automatically changes owner of replicated tables, Purge stored procedure, CDC staging table and CDC system tables to SAPDS. It defines CDC staging table name same as Remote Table name of article with suffix _SAPCDC. It creates or overrides function strings - rs_delete, rs_insert, rs_update and rs_get_textptr for blob columns. The CDC staging table defines additional columns in addition to RemoteTable columns to store Replication Server system variables: a. DS_SRCDB - stores system variable rs_origin_db b. DS_SRCDB_SRV stores system variable rs_origin_ds c. DS_SRCDB_ID stores system variable rs_origin d. DS_SRCDB_COMMIT_TIME - stores system variable rs_origin_commit_time e. DS_SRCDB_ORIGIN_QID stores rs_origin_qid f. DS_SRCDB_OP_TYPE stores change operation types I insert, B - Before image of update, U - After image of update, D -Delete g. DS_CDCDB_PUBLISH_TIME stores publish datetime of CDC database when record get inserted by Replication Server 20
24. SAP Data Services CDC extension added two extended attributes to Article object. They are Retention Period and CDC Staging Table. Default value Retention Period extended attributes is 5 days, You can change this value. Add Data Services CDC Definitions command sets CDC Staging Table attributes with CDC staging table name for each article. Physical data models and Data Movement model are now ready. To finish Data Services Sybase Replication Server CDC environment configuration these 4 steps are needed: 1. Create objects in the CDC staging DB. 2. Run Data Services Staging database runtime script. 3. Deploy replication definitions and subscriptions into SRS. 4. Configuring primary DB, mark source tables for replication. 21
1. CREATE OBJECTS IN THE CDC STAGING DB 1.1 Create SAPDS login in ASE157 and add SAPDS user in CDC staging DB: 1.2 Generate staging DB DDL script Select the Data Services staging database PDM in Power Designer Data Movement Model and right click the context to select "Generate Database" to generate the CDC database DDL script. This automatically generates DDL of all the database objects that are required for Data Services CDC Staging database. The CDC database script is generated as a file: 22
1.3 Execute the script with isql and check that objects are created properly and give permissions to maintenance user in staging tables: 23
2. RUN DATA SERVICES STAGING DATABASE RUNTIME SCRIPT 2.1 Generate script Select the Data Services staging database PDM in Power Designer Data Movement Model and right click the context to select "Generate Script". In the pop-up window select SAP Data Services CDC extension and proceed. It generates script file suffixed with _DataServices_CDC_DB_Runtime_Script.sql which creates Data Services CDC staging database runtime environment. This script provides following functionalities: a. Validates the staging database to find out if this database is added to replication server by Replication Server rs_init utility. b. Validates if Data Services CDC staging database create script is run. c. Creates Replication Server maintenance user that defined in Data Movement Model if it is not created before. DBA can change the password after creation to protect un-authorized access. d. Adds maintenance user to database. e. Add TargetTable name and CDC staging table name to DS_CDC_TABLES_MAP table for each replication definition. f. Add CDC staging table name and retention period to DS_CHANGE_RETENTION table for each replication definition. g. Grants select, insert, update and delete privileges to CDC staging tables. h. Grant execute privilege to PURGE_OLD_CHANGE_DATA stored procedure. i. Grants Replication Server maintenance user privileges to execute Purge Old Change Data scheduled job in Sybase ASE Job Scheduler database. j. Creates Purge Old Change Data scheduled job in Sybase ASE Job Scheduler database. Purge Old Change Data job name is SAPDS_<CDC Staging Database Name>_PURGE_CHANGE_OLD_DATA. 2.1 Execute the script with isql: 24
3. DEPLOY REPLICATION DEFINITIONS AND SUBSCRIPTIONS INTO SRS 3.1 Define a primary DB for the subscriptions: Double click Replication Server process in Power Designer Data Movement Model and select subscriptions, double click in subscription to get its properties and set Primary Connection to ASE157.testdb: 25
3.2 In Publications & Replication definitions (general tab) check that type is publication : 26
3.3 Generate script: To generate Replication Server RCL script file select the Replication Process in Power Designer Data Movement Model and right click the context to select "Generate Script", set the folder and click OK to generate Replication Definitions, CDC Function Strings definitions and subscriptions. Connection has been created previously so Create Connection should be set to false: 27
3.4 Execute script with isql: 4. CONFIGURING PRIMARY DB, MARK SOURCE TABLES FOR REPLICATION 4.1 Generate script: Select the primary databases in Power Designer Data Movement Model and right click the context to select "Generate Script" and select Replication Server 15.7SP100 : Only the script with sp_setreptable is needed, no need to generate the other ones: 28
Click Edit to check and be sure that owner on in set in the sp_setreptable call: 4.2 Execute script in primary DB testdb (removing // lines if they appear) to avoid syntax error: 29
Everything is now configured and changes in t1 at primary DB should be captured into the CDC staging table: 30
www.sap.com 2013 SAP AG. All rights reserved. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, ianywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company. Crossgate, m@gic EDDY, B2B 360, and B2B 360 Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.