Disaster Recovery Procedure for a RulePoint Setup Spanning Across Geographical Locations

Transcription

1 Disaster Recovery Procedure for a RulePoint Setup Spanning Across Geographical Locations 2014 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. All other company and product names may be trade names or trademarks of their respective owners and/or copyrighted materials of such owners.

2 Abstract In a RulePoint setup that is highly available and spans different geographies, you can perform disaster recovery steps to restore continuity of service after an outage. This article describes the procedure to restore RulePoint installations after a failure. Supported Versions RulePoint 6.2 Table of Contents Overview Configuration 1: Primary Customer Source Database Is Replicated... 3 Setting Up Customer Source Database in the Data Centers Restoring the RulePoint Setup After the Primary Data Center Fails Recovery Actions to Reassign the Primary Role to a Host Machine that Recovers After a Failure Configuration 2: Primary Customer Source Database and RulePoint Metadata Database Are Replicated Setting Up the Customer Source Database and the RulePoint Metadata Database in the Data Centers... 6 Restoring the RulePoint Setup After the Primary Data Center Fails Recovery Actions to Reassign the Primary Role to a Host Machine that Recovers After a Failure Overview A RulePoint installation in a data center can be backed up by another data center set up in a different geographical location to minimize downtime before you restore the services after an outage. When an outage occurs, you move the operations from the primary data center to the standby data center to resume the services. After the primary data center is back in service, you can move the operations from the standby data center back to the primary data center. RulePoint Setup The RulePoint setup in the primary and standby data centers consists of the following components: Customer source database. The customer source database stores customer records. RulePoint installation. The RulePoint installation consists of a primary host machine and multiple secondary host machines. The host machine on which the RulePoint installer installs the installation files is referred to as the primary host machine. When you configure a highly available setup, you copy the RulePoint installation files from the primary host machine to the secondary host machines in the data center. RulePoint metadata database. The RulePoint metadata database stores records of RulePoint events. Database Configuration You need to set up the database configuration with the objective of performing disaster recovery actions if an outage occurs. The disaster recovery actions that you need to perform to restore the services depend on the 2

3 database configurations that you have included in the RulePoint setup. You can perform disaster recovery actions for the following database configurations: The customer source database in the standby data center replicates the customer source database in the primary data center. If there is an outage in the primary data center, you can access the standby data center to resume the RulePoint services. The customer source database and the RulePoint metadata database in the standby data center replicate the customer source database and the RulePoint metadata database in the primary data center. Recovery Process Perform the disaster recovery actions depending on the following situations: The host system in the primary data center fails and does not recover. The host system in the primary data center recovers after a failure and needs to be reassigned the primary role. Configuration 1: Primary Customer Source Database Is Replicated In this configuration, the customer source database in the standby data center replicates the customer source database in the primary data center. The following image illustrates the replication setup deployed between the data centers: 3

4 Setting Up Customer Source Database in the Data Centers The following procedure describes how to set up the customer source database across the data centers to prepare for disaster recovery. Note: The steps in this procedure refer to the commands that are applicable for the Microsoft SQL Server. However, you can apply these steps to configure any supported database. 1. Perform the following steps to set up the source database replication: a. Ensure that the database that contains the tables that are read by the RulePoint SQL source in the primary data center is replicated by the database in the standby data center. b. To track the processed state of incoming events, add the column <ProcessedColumnName> with the initial value as 0 in the source database tables in the primary data center. For example, ProcessedInd = 0 2. In the standby data center, ensure that the RulePoint host agent, topology, and the design-time processes are not running. 3. Configure the following settings for the RulePoint source in the primary and standby data centers: a. Create an SQL connection to access the database and read the tables. b. Create a topic. c. Create an SQL source for the connection and topic that you created. Use the following parameters to configure the source for the SQL database: SQL query: select * from <tablename> where <ProcessedColumnName> = 0 Update SQL database: update <tablename> set <ProcessedColumnName> = 1 Note: If you need to update multiple tables, use the update statements in a transaction and enter commit after the update. d. Create a responder and a response to send out the required alerts. e. Create a rule to process the events and send alerts for the response that you created. For example, when 1 sql_topic s with s.cust_id > 1001 the RTAM_Response Note: In the standby data center, use Windows services to start the RulePoint design-time service to create the RulePoint objects. After the objects are created, stop the design-time service. You can also export the RulePoint objects from the primary data center to the standby data center. 4. Verify the updates for the database table in the primary data center: a. Log in to RulePoint and deploy all the objects. b. Verify whether the table that you updated in step 1 has the <ProcessedColumnName> value set to 1. The updates to the tables continue as long as the events are generated by the source. Restoring the RulePoint Setup After the Primary Data Center Fails In this scenario, the primary data center fails and the host systems do not recover after the failure. The database tables are left with unprocessed rows after the failure. For example, the SQL source processes only 2000 rows out of rows, and only the processed rows have the <ProcessedColumnName> value set to 1. Perform the following steps in the standby data center: 1. Start the host agent, topology, and design-time services. 2. Log in to the RulePoint user interface and deploy all the objects. Ensure that the unprocessed rows in the database source tables have the <ProcessedColumnName> value set to 1 and the required responses are generated. 4

5 Recovery Actions to Reassign the Primary Role to a Host Machine that Recovers After a Failure You can assign the primary role to the standby data center if there is an outage in the primary data center. Even after the primary data center recovers, the RulePoint setup in the standby data center continues to perform the primary role. To resume operations from the primary data center, you need to stop the RulePoint instances at the standby data center and start the services in the primary data center. The primary data center is reassigned the primary role to function as it was before the failure. Perform the following steps to reassign the primary role to the primary data center: 1. Reset RulePoint in the primary data center: a. On a host machine, log in to the RulePoint user interface. b. Undeploy all objects. c. Stop the RulePoint topology, host agent, and design-time services. d. Delete the topology stored in the <Topology Name>_UM-Store directory to clear stale messages, if any. 2. Stop the RulePoint instance in the standby data center: a. On a host machine, log in to the RulePoint user interface. b. Undeploy all objects. c. Stop the RulePoint topology, host agent, and design-time services. 3. Perform the following actions in the primary data center: a. Start the RulePoint topology, host agent, and design-time services. b. Deploy all the objects that you undeployed in step 1. c. Ensure that the processing of data from the tables resumes from the point where the standby data center host stopped. Configuration 2: Primary Customer Source Database and RulePoint Metadata Database Are Replicated In this configuration, the RulePoint installation consists of a primary data center which is backed up by a standby data center at a different region. The customer source database in the primary data center is replicated by the database in 5

6 the standby data center. In addition, the RulePoint metadata database in the primary data center is also backed up by a database in the standby data center. The following image illustrates the RulePoint setup between the data centers: Setting Up the Customer Source Database and the RulePoint Metadata Database in the Data Centers The following procedure describes how to set up the customer source database and the RulePoint metadata database across the data centers to prepare for disaster recovery. Note: The steps in this procedure refer to the commands that are applicable for the Microsoft SQL Server. However, you can apply these steps to configure any supported database. 1. Perform the following steps to set up the source database replication: a. Ensure that the customer source database that contains the tables that are read by the RulePoint SQL source in the primary data center is replicated by the database in the standby data center. b. To track the processed state of incoming events, add the column <ProcessedColumnName> with the initial value as 0 in the source database tables in the primary data center. For example, ProcessedInd = 0 6

7 c. Ensure that the RulePoint metadata database in the primary data center is replicated by the RulePoint metadata database in the standby data center. 2. In the standby data center, ensure that the RulePoint host agent, topology, and the design-time processes are not running. 3. Configure the following settings for the RulePoint source in the primary and standby data centers: a. Create an SQL connection to access the database and read the tables. b. Create a topic. c. Create an SQL source for the connection and topic that you created. Use the following parameters to configure the source: Use the following parameters to configure the source for the SQL database: SQL query: select * from <tablename> where <ProcessedColumnName> = 0 Update SQL database: update <tablename> set <ProcessedColumnName> = 1 Note: If you need to update multiple tables, use the update statements in a transaction and enter commit after the update. d. Create a responder and a response to send out the required alerts. e. Create a rule to process the events and send alerts for the response that you created. For example, when 1 sql_topic s with s.cust_id > 1001 the RTAM_Response 4. Verify the updates for the database table in the primary data center: a. Log in to RulePoint and deploy all the objects. b. Verify whether the table that you updated in step 1 has the <ProcessedColumnName> value set to 1. The updates to the tables continue as long as the events are generated by the source. Restoring the RulePoint Setup After the Primary Data Center Fails In this scenario, the primary data center fails and the host systems do not recover after the failure. After the failure, the database tables are left with unprocessed rows. For example, the SQL source processes only 2000 rows out of rows and only the processed rows have the <ProcessedColumnName> value set as 1. Verify the following conditions before you perform the recovery actions: Only the standby customer database is running. None of the components of RulePoint on the host machines in the standby data center is running. To restore service continuity, perform the following steps in the standby data center: 1. Log in to the database to edit the RULEPOINT_TOPOLOGYSTATE schema and run the following scripts to clear the run-time topology tables: DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."HA_SOURCE_STATE"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_DEPLOY_SEQUENCE"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_DEPLOYMENT_PACKAGE"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_FAILOVER"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_LEADER"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_CONTROLLERS"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_UM_LBMRD"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_UM_STORES"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_CEPEU"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_SERVERS"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_NODE"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_TOPOLOGY"; UPDATE RULEPOINT_TOPOLOGYSTATE.T_RPSERVER_GROUP_DETAILS SET PRIMARY_INSTANCE_ID =null 7

8 Note: The run-time topology tables are populated when you start the topology on the server. 2. Log in to the RULEPOINT_TOPOLOGY schema. 3. In the primary data center, run the following script to replace the host IP address with the IP address of the host machine in the standby data center: UPDATE RULEPOINT_TOPOLOGY.HOST SET HOSTNAME='<<PRIMARY HOST IP>>' Replace <<PRIMARY HOST IP>> with the IP address of the primary host machine in the standby data center. 4. Start the RulePoint design instance. 5. Log in to the RulePoint user interface. 6. Select Administration > Topology and then select the appropriate topology. 7. On the Topology view, select Edit from the list in the row for the topology. The Edit Topology dialog box appears. 8. In the Properties section, update the database details. Verify whether the IP address of the primary host machine in the standby data center appears as the primary host IP address. 9. Go to <RULEPOINT_HOME> and delete the node_name, stored_name, and lbmrd_name folders. Where, <RULEPOINT_HOME> refers to the installation directory which the RulePoint installer creates during the installation phase to install the RulePoint components. The default location of the RulePoint installation directory in Windows is C:\RulePoint_ Start the host agent with the new IP address. 11. Start the topology. Recovery Actions to Reassign the Primary Role to a Host Machine that Recovers After a Failure You can assign the primary role to the standby data center if there is an outage in the primary data center. Even after the primary data center recovers from a failure, the standby data center continues to perform the primary role. To resume operations from the primary data center, you need to stop the RulePoint instances in the standby data center and shift the services to the primary data center. The primary data center is reassigned the primary role to function as it was before the failure. Perform the following steps to reassign the primary role to the host machine in the primary data center: 1. Reset RulePoint in the primary data center: a. Log in to a host machine. b. Shut down all the RulePoint components. c. Delete the topology stored in the <RULEPOINT HOME>/<Topology Name>_UM-Store directory to clear stale messages, if any, that were created when the primary data center was down. 2. Stop the RulePoint instance in the standby data center: a. Log in to the RulePoint user interface. b. Undeploy all objects. c. Stop the RulePoint topology, host agent, and design-time services. 3. Log in to the database in RULEPOINT_TOPOLOGYSTATE schema and run the following scripts to clear the runtime topology tables: DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."HA_SOURCE_STATE"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_DEPLOY_SEQUENCE"; 8

9 DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_DEPLOYMENT_PACKAGE"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_FAILOVER"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_LEADER"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_CONTROLLERS"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_UM_LBMRD"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_UM_STORES"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_CEPEU"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_SERVERS"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_NODE"; DELETE FROM "RULEPOINT_TOPOLOGYSTATE"."T_RPSERVER_TOPOLOGY"; UPDATE RULEPOINT_TOPOLOGYSTATE.T_RPSERVER_GROUP_DETAILS SET PRIMARY_INSTANCE_ID =null Note: The run-time topology tables are populated when you start the topology on the server. 4. Log in to the RULEPOINT_TOPOLOGY schema. 5. In the primary data center, run the following script to update the host IP addresses: UPDATE RULEPOINT_TOPOLOGY.HOST SET HOSTNAME='<<YOUR HOST PRIMARY IP>>' Replace <<YOUR HOST PRIMARY IP>> with the IP address of the primary host machine in the primary data center. 6. Perform the following steps on a host machine in the primary data center: a. Start the RulePoint topology, host agent, and design-time services. b. Deploy all the objects that you undeployed. 7. Ensure that the processing of data from the tables resumes from the point where the standby data center host stopped. Author Shilpa Bhat Lead QA Engineer Acknowledgements The author would like to acknowledge the Informatica CEP team and the Documentation team for their contributions to this article. 9