SOA Cloud Service Disaster Recovery

Transcription

1 SOA Cloud Service Disaster Recovery Production and DR in the Cloud O R A C L E W H I T E PAP E R A U G U S T

2 SOA CLOUD SERVICES DISASTER RECOVERY

3 Table of Contents Introduction 1 Disaster Protection Considerations for Oracle SOA Cloud Service 3 Service Level Requirements 5 Security Requirements 5 Configuration Requirements 6 SOACS DR Deployment 7 OPTION1 Pre-condition: primary SOACS System already exists 8 1. Create virtual hostname for the frontend OTD and update frontend address in primary site 9 2. Update t3/rmi urls (if used) Set up and verify Data Guard Provision SOACS in secondary location pointing to snapshot standby database Create virtual hostname for frontend OTD and update frontend address in secondary site Update t3/rmi urls in the secondary site (if used) Copy and update fmwconfig bootstrap artifacts to secondary location Replace schema prefix in the standby datasources Convert the snapshot standby database back to the physical standby database Switchover Database Verify SOACS in secondary site 17 SOA CLOUD SERVICE DISASTER RECOVERY

4 12. Switchback to original site. 17 SOACS DR Lifecycle Procedures 18 Switchover 18 Failover 18 Applying domain configuration changes to the system 19 a) Repeating domain configuration changes in both sites 19 b) Using DBFS for configuration propagation 20 Scaling the SOACSDR system 34 Conclusion 36 Appendix A - OPTION2 37 A. Provision DBaaS in both locations 37 B. Set up and verify Data Guard 37 C. Provision SOACS in primary location 37 D. Create virtual hostname for front end OTD and update front end address in primary site 38 E. Update t3/rmi urls (if used) 38 F. Switchover Database 38 G. Provision SOACS in secondary location 38 H. Create virtual hostname for front end OTD and update front end address in secondary site 38 I. Update t3/rmi urls in the secondary site (if used) 38 J. Copy and update fmwconfig bootstrap artifacts to secondary location 38 K. Replace schema prefix in datasources 38 L. Verify SOACS in secondary site 38 SOA CLOUD SERVICES DISASTER RECOVERY

5 M. Switchback to original site. 38 Appendix B - Setting up Data Guard 39 Appendix C - Ceating and dispersing wallets in 12c 55. SOA CLOUD SERVICES DISASTER RECOVERY

6 Introduction Oracle s Maximum Availability Architecture (Oracle MAA) is the best practices blueprint for data protection and availability of Oracle products (Database, Fusion Middleware, Applications) deployed on on-premises, private, public or hybrid clouds. Implementing Oracle Maximum Availability Architecture best practices is one of the key requirements for any Oracle deployment. Oracle Fusion Middleware and Oracle Databases include an extensive set of high availability features, such as process death detection and restart, clustering, server migration, clusterware integration, GridLink, load balancing, failover, backup and recovery, rolling upgrades, and rolling configuration changes, which protects application deployments from unplanned downtime and minimize planned downtime. Oracle Service-Oriented Architecture (SOA) Cloud Service is a Platform as a Service (PaaS) computing platform solution for running Oracle SOA Suite in the cloud. This computing platform uses Oracle Compute Cloud Service, Oracle Database Cloud Service and Oracle Java Cloud Service as its basic infrastructure Oracle SOA Suite requires an Oracle Database to store Oracle Platform Security Services information, instance tracking, composite and document metadata and other Oracle FMW Infrastructure schemas. In a typical Oracle SOA deployment the application data (such as applicationspecific schemas, jms stores etc.) and the SOA-specific schemas are stored in the same database for transactional consistency and simplified administration reasons. In a SOA Cloud Service instance an Oracle Database Cloud Service instance is used to store these schemas. All Oracle SOA deployments need protection from unforeseen disasters and natural calamities. This protection is also required for systems deployed in the Cloud and it needs to address both the middle tier (Oracle SOA Suite Cloud Service) and the data tier (Database Cloud Service). The solution involves setting up a standby system at a geographically different Oracle cloud data center than the production site. The standby system may have equal or fewer services and resources compared to the production site. The standby system is normally in a passive mode and is activated when the primary site becomes unavailable. This deployment model is sometimes referred to as an active-passive model. This model is usually adopted when the two sites are connected over a WAN and network latency does not allow clustering across the two sites. Oracle SOA Cloud Service (SOACS) provides the best Recovery Time Objective (RTO) and Recovery Point Objective (RPO) by utilizing high availability and disaster protection capabilities provided by Oracle Fusion Middleware and Oracle Database. While there are some unique considerations to a cloud disaster recovery (DR) configuration, it follows the same Oracle MAA best practices as any Oracle Fusion Middleware (FMW) and Oracle Database deployment. This Oracle MAA blueprint details Oracle MAA Best Practices and provides a procedural overview for deploying DR for SOA 1 ENTER TITLE OF DOCUMENT HERE

7 Cloud Service. Oracle SOA Cloud Service Disaster Recovery solution is achieved by replicating a limited set of configuration files that are required to bootstrap SOA components. The application may require additional configuration files to be replicated. Options are provided in this paper to suit different application paradigms. Disaster protection for Oracle Database Cloud Service used by Oracle SOA Cloud Service is provided through Oracle Data Guard. This paper is intended for a technical audience having knowledge of Oracle Weblogic Server, Oracle FMW SOA, Oracle Database, Data Guard and Oracle Database backup and recovery. This paper also assumes a basic understanding of services offered on the Oracle Cloud SOA CLOUD SERVICES DISASTER RECOVERY

8 Disaster Protection Considerations for Oracle SOA Cloud Service Oracle SOA Cloud Service uses DBCS (Oracle Database Cloud Service) to host the metadata and SOA instance information. There are two options for protecting Oracle Database Cloud Services against disasters: Data Guard utilizing Enterprise Edition Service or High Performance Service. Active Data Guard utilizing the Extreme Performance Service or Exadata Cloud Service. For more information on Data Guard and Active Data Guard please refer to the Data Guard home page on the Oracle Technology Network and the Active Data Guard white paper 2. Oracle FMW SOA Suite can take advantage of the automatic block repair, fast incremental backups, fast rolling upgrades, Far Sync, and most features of Active Data Guard. However, SOA servers cannot, however, use Oracle Active Data Guard s read-only query capabilities. This is because Oracle SOA Servers cannot run in read-only mode. As soon as a SOA Server starts up, it connects to the database and tries to process business flows (i.e. they write to the database right away). The only way to start the SOA servers in the standby site is by either 1) performing a switchover/failover of the database or 2) by converting the standby database to a snapshot standby which makes the standby read-writable temporarily for testing purposes. Redo received from the primary are stored but not applied. The changes that are done are discarded at the end of testing where it can resume applying the redo from the primary database. It needs to be noted that additional storage is required for the fast recovery area when a standby is in snapshot mode (to hold archived redo received from the primary production database for later use and current redo and flashback logs generated by the snapshot standby). In a SOACS DR set up, the snapshot standby method is used to replicate configuration changes that were applied in the production site when WLS domain configuration is not changed too frequently. The procedure (which will be described later on in detail) consists in converting the standby database to a snapshot standby and applying again the changes using the WLS Admin Console in the secondary location. This updates the file system artifacts (ear files, deployment plans etc) to be the same as in the primary site. It is also a best practice to periodically place the standby in read/write mode (using Data Guard Snapshot Standby) to validate its readiness to support read-write production workloads. The snapshot standby, however, needs to be used carefully with SOA servers since undesired processing of pending instances may occur when the SOA servers have access to a dehydration store. A snapshot standby may also be used for a final level of pre-production functional and performance testing of patches and upgrades since the DR system is frequently sized similar to the production system Please refer to Oracle documentation for additional details on Data Guard Snapshot Standby 3 Beyond this, the Oracle Cloud provides all backend infrastructure and capabilities required for disaster recovery should the primary database become unavailable for any reason. This includes: 1. Ability to monitor the standby database and alert on major issues. 2. Ability to activate the standby to validate DR readiness and then convert back to a synchronized standby. 3. Utilization of essentially the same Oracle MAA best practices as on-premises deployments. This paper describes the few considerations that are specific to cloud deployments SOA CLOUD SERVICES DISASTER RECOVERY

9 4. Ability to switchover (planned event) or failover (unplanned event) production to the standby database in the cloud during a planned maintenance or an unplanned outage. Once the failed primary database is repaired, it can be resynchronized with the new production database in the cloud and then roles can be switched back to the original status. 5. Ability to failover both the SOA/middle tier and database to enable production applications to run in the Oracle Cloud when there is a complete site outage. In the middle tier, the architecture of a SOACS DR system consists of two SOACS instances deployed in the same sites as the DBaaS instances. Each SOACS instance uses a SOA Cluster, an Administration Server and OTD as front-end load balancer. A single Database (DBaaS) instance is used to host both the SOACS schemas (MDS, composite deployment, SOAINFRA schemas etc.) the JMS persistent stores, the Transaction Logs persistent stores and any application specific data. Figure 1 describes this architecture: Figure 1: SOACS Dr architecture 4 SOA CLOUD SERVICES DISASTER RECOVERY

10 Service Level Requirements Oracle SOA Cloud Service is a user-managed environment. The user must determine service level expectations for availability, data protection, and performance that are practical for a given configuration and application. Service Levels must be established for each of three dimensions relevant to disaster recovery that are applicable to any Data Guard configuration:» Availability: Recovery Time Objective (RTO) describes the maximum acceptable downtime should an outage occur. This includes time required to detect the outage and to failover the database, the Web tier and SOA servers so that service is resumed.» Data Protection: Recovery Point Objective (RPO) describes the maximum amount of data loss that can be tolerated. In SOA s case this is especially related to transaction logs, JMS messages and SOA instance information which all resides in the same database. The actual achievable RPO depends upon:» Available network bandwidth.» Network reliability.» Data Guard transport method used: either asynchronous for near-zero data loss protection, or synchronous for zero data loss protection.» Performance: Database and Middle Tier response time may be different after failover if less capacity compute, memory, I/O, etc., are provisioned at the standby system than in the primary system. This occurs when users purposefully under-configure standby resources to reduce cost (accepting reduced service level while in DR mode). MAA best practices recommend configuring symmetrical capacity at both primary and standby in the web, application and database tiers so there is no change in response time after failover. Rapid provisioning available with the cloud can enable a middle ground where less capacity is initially deployed, but where the new primary is rapidly scaled-up should a failover be required. Note: Independent of the service levels related to DR, all database instances created in the Oracle cloud conform to the service descriptions defined by the applicable Database Cloud Service 4. Security Requirements Oracle MAA best practice recommends using Oracle Transparent Data Encryption (TDE) to encrypt primary and standby databases at rest. Conversion to TDE enables automatic encryption at rest for all DATA/INDEX tablespaces and encryption-in-flight of user data redo changes during replication to the cloud. Oracle Net encryption is also required for encryption-in-flight for other redo changes that are not encrypted by TDE (e.g. data from unencrypted tablespaces such as SYSTEM and SYSAUX). Note: Data Guard and Active Data Guard use redo-based replication a process that transmits redo generated by a primary database and applies those changes to a standby database using continuous media recovery. This means that primary and standby databases are block for block identical copies of each other. Using TDE to encrypt a standby database also requires that the primary database be encrypted with TDE SOA CLOUD SERVICES DISASTER RECOVERY

11 Using TDE to protect data is an important part of improving the security of the system. Users should however be aware of certain considerations when using any encryption solution, including:» Additional CPU overhead: Encryption requires additional CPU cycles to calculate encrypted and decrypted values. TDE minimizes the overhead by taking advantage of database caching capabilities and leveraging hardware acceleration for AES on Intel and SPARC CPUs. Most TDE users see little performance impact on their production systems after enabling TDE. Please refer to the Oracle Database Advanced Security Guide 5 for more details.» Lower data compression: Encrypted data compresses poorly because it must reveal no information about the original plain text data. Thus, any compression applied to TDE encrypted data will have low compression ratios. Hence, redo transport compression should not be enabled when TDE encryption is used. However, when TDE is used in conjunction with Oracle database compression technologies such as Advanced Compression or Hybrid Columnar Compression, compression is performed before the encryption occurs, and the benefits of compression and encryption are both achieved.» Key management: Encryption is only as strong as the key used to encrypt. Furthermore, the loss of the encryption key is tantamount to losing all data protected by that key. If encryption is enabled on a few databases, keeping track of the key and its lifecycle is relatively easy. As the number of encrypted databases grows, managing keys becomes an increasingly difficult problem. For users with a large number of encrypted databases, it is recommended that Oracle Key Vault 6 on-premise be used to store and manage TDE master keys. When DBFS is used to maintain WLS domain configuration, using the appropriate encryption at the DB level guarantees also the security of configuration propagation across sites. Configuration Requirements Beyond these runtime-related aspects, the following considerations are key to the SOACS DR set up: A default database instance is automatically created when you create an Oracle Database Cloud Service. On the standby site, the database initially created cannot be used as a Data Guard standby database. It can be deleted or reused for a different purpoxse The SOACS instance name in both primary & standby domains/locations should be the same. The instance name is used to construct the hostnames, WLS server names and domain name where SOACS will run. Since the domain configuration is not copied entirely from one site to the other, using the same instance name guarantees consistency and allows the recovery of JMS messages and Tlogs. It also simplifies customizations and operations in both sites. MDS, Composite deployments and policies are automatically synchronized across sites by Data Guard since they are stored in the database. The SOACS WLS domain configuration is not globally synced across sites with the following considerations: o Each SOACS system will maintain the original JDBC URLs used to connect to the DB even after the DR set up has completed. Only the schema prefix will be altered so that both locations point to the same schemas SOA CLOUD SERVICES DISASTER RECOVERY

12 o o All the fmwconfig-opss in the WLS domain configuration is copied from one site to the other and the JDBC string is modified after the copy so that OPSS points to the local DB Application deployments (workflow task ears, custom ear files, deployment plan updates, redeployments etc) are not synced automatically across sites. The SOACS DR design offers two alternatives for addressing this: Systems with highly frequent domain configuration changes or application deployments can use DBFS (stored in the database) to replicate the domain artifacts across sites. Systems with infrequent domain configuration changes or application deployments can convert the standby database to snapshot and apply the configuration change also in the secondary site. Once Data Guard has been configured and for systems in production, conversion of standby to snapshot should be done only without starting the SOA servers in the standby site. This is because SOA servers will try to process SOA instances and complete pending work (callbacks, async invocation etc.) that may have been processed already on the primary site. Only the administration server should be brought up pointing to a snapshot standby once the system is in production. SOACS DR Deployment In this document we assume as a starting point that the primary site, consisting of a single instance DBaaS, SOACS Cluster and Oracle Traffic Director (OTD) is live and a dual site geographically distributed DR configuration will be created. Refer to the SOACS Documentation for details about how to provision the initial set up. In summary and leaving aside the Compute Shapes selected for SOA, OTD and the DB) similar options to the following should be used: The SOACS DR configuration makes failover/switchover transparent to end user or systems accessing the services by making SOA endpoints agnostic to the site that is being used. For this to be possible, the front end address of the existing SOA Cluster is altered to use a virtual hostname that can be assigned to either DR location s OTD once the configuration is completed. Appropriate DNS services (global DNS, local DNS server or manual file hostname resolution) need to be used for the front end url to be mapped to either site. If any composites are deployed/used 7 SOA CLOUD SERVICES DISASTER RECOVERY

13 before this front end address change is performed, it may be required to alter the endpoint address to match this new virtual hostname (NOTE: By default end point addresses are constructed based on the SOA/WLS cluster s HTTP front end parameter). Since the production SOACS may already be running production workloads, the DR configuration process should cause as minimum downtime as possible. There are two deployment options: 1. Option #1 is where a primary SOACS already exists. 2. Option #2 is a tiered approach where the Database tier is set up first and verified for DR and then the middle tier is added on top. This approach is described in Appendix A. This section describes the summary steps for set up process for Option #1: OPTION1 Pre-condition: Primary SOACS System already exists 1. Create virtual hostname for front end OTD and update front end address in primary site 2. Update t3/rmi urls (if used) 3. Provision DBaaS in secondary location 4. Set up and verify DataGuard (service downtime) 5. Provision SOACS in secondary location pointing to snapshot DB 6. Create virtual hostname for front end OTD and update front end address in secondary site 7. Update t3/rmi urls (if used) 8. Copy and update fmwconfig bootstrap artifacts to secondary location 9. Replace schema prefix in datasources 10. Convert snapshot to standby 11. Switchover Database (service downtime) 12. Verify SOACS in secondary site 13. Switchback to original site (service downtime) If the primary SOACS site is running it may be the case that callbacks and instances remain pending to be executed when the secondary site is configured. Follow the indications in the next sections to avoid duplications when using this approach. As an alternative and for those cases where the standby DR site is being created from scratch (i.e. the primary SOACS system is not live yet and DR is being implemented at the same time as the main production system), it is recommended to configure DBaaS and Data Guard first and then provision SOACS in both sites. This allows performing an initial Data Guard verification without altering the dependent SOACS services and also does not use the databases on both sites concurrently during the set up. The first option (referred to as OPTION 1 in this document) incurs in less downtime during the set up. The following sections describe in detail the set up process. The second option ( OPTION 2 ) provides a tiered approach where the DB tier is set up and verified for DR first and then the middle tier is added on top. This approach is described in Appendix A. OPTION1 Pre-condition: primary SOACS System already exists It is expected that the primary site has already been provisioned following the standard SOACS procedures: 8 SOA CLOUD SERVICES DISASTER RECOVERY

14 Create an SSH key pair Create an Oracle Storage Cloud Service Container Create an Oracle Database Cloud Service instance Run the SOACS provisioning wizard It is also expected that the required VNC access has been configured to the VMs in the primary site. This is needed because an X11 session is used for installing Grid Infrastructure in the DBaaS compute. Once the above steps have been followed and the SOACS instance is functioning, follow these steps: 1. Create virtual hostname for the frontend OTD and update frontend address in primary site By default SOACS s provisioning sets the value of the SOA cluster s front end address to the IP of OTD s listen address. This IP needs to be replaced with a virtual hostname and this hostname needs to be resolvable by both the external clients and by the WLS servers that OTD is routing requests to. To resolve the virtual hostname externally a local file or DNS resolution may be used. To resolve the virtual hostname in the scope of the SOACS instance, a local file resolution must be used. A. To make the virtual hostname available in the SOACS nodes, follow these steps: Login to the WebLogic Server Administration Console for your SOACS instance. In the left pane, choose Environment in the Domain Structure window and then choose Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster. Select HTTP Note down the IP used in the Frontend Host filed. NOTE: Alternatively you can get this IP also from the SOACS instance screen information. This IP would be listed as the PUBLIC IP for the Load Balancer: 9 SOA CLOUD SERVICES DISASTER RECOVERY

15 In a command shell prompt for your SOACS nodes, sudo to the root user and edit /etc/hosts to map this IP to a virtual hostname (if a DNS is going to be used for the external resolution of this IP, this hostname will have to be used here. If you are going to use file-based (/etc/hosts) hostname resolutions, this would be your own choice). For example: [root@soacsdr-jcs-wls-1~]# cat /etc/hosts localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain soalbrdr soalbrdr.example.com Repeat these steps in the other SOACS server that are used by the SOACS Cluster B. To make the virtual hostname available in the external clients either update your DNS server or modify the hosts file for those clients as above. For example in your on-premises windows client, edit your C:\Windows\System32\drivers\etc\hosts file as above NOTE: This configuration for external clients will work if direct connections from the internet are used. Connections from custom intranets will need to account for this hostname to be added to the required proxy server used by the browsers or http clients C. Once the appropriate hostname resolution is in place per the steps above, modify the front end address in the SOACS Cluster: Login to the WebLogic Server Administration Console for your SOACS instance. In the left pane, choose Environment in the Domain Structure window and then choose Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster. Select HTTP Set the value for the Frontend Host=virtual_hostname_created, in the example above this would be soalbrdr Click Save. To activate the changes, click Activate Changes in the Change Center section of the Administration Console. Restart the SOACS cluster for the front end change to be effective 2. Update t3/rmi urls (if used) The urls used for RMI invocations in the SOACS cluster need to be agnostic to the IPs or hostnames used by each site in the SOACS DR configuration. Instead of using the typical host:port,host:port JNDI urls, change them to use the cluster syntax 7. The cluster syntax is as follows: cluster:t3://cluster_name. For example, to modify the JMS adapter factory properties to use this syntax, follow these steps: A. Log into your Oracle WebLogic Server Administration Console for your SOACS instance B. Click Deployments in the left pane for Domain Structure. 7 Obviously this is feasible only for inter-domain invocations. T3/rmi clients that are external to the SOACS domain will not be able to use this approach and will have to either use a tcp load balancer for JNDi resolution or use the appropriate DNS mapping of the host:port list in the secondary site 10 SOA CLOUD SERVICES DISASTER RECOVERY

16 C. Click JmsAdapter under Summary of Deployments on the right pane. D. Click the Configuration tab. E. Click the Outbound Connection Pools tab and expand oracle.tip.adapter.jms.ijmsconnectionfactory to see the configured connection factories. F. Click the specific instance you are using (for example, eis/wls/queue). The Outbound Connection Properties for the connection factory opens. G. Click Lock & Edit. H. In the FactoryProperties field (click on the corresponding cell under Property value), alter the java.naming.provider.url filed to use the cluster syntax (leave the rest of the fields as they were): java.naming.provider.url= cluster:t3://cluster_name; I. Click Save after you update the properties. The Save Deployment Plan page appears. J. Enter a location for the deployment plan. K. Copy the deployment plan from your SOACS node1 to your SOACS node2 in the exact same directory/location NOTE: since the ssh key for the system pertains the opc user, yuo need to copy the file as opc user to a tmp Location in node2 where both opc and oracle users have access rights and then copy form that directory to eh deployment plan home using the oracle user. L. Click Save and Activate. M. Click Deployments. N. Click Lock & Edit O. Select the JMS Adapter. P. Click Update. Q. Select Update this application in place with new deployment plan changes (A deployment plan must be specified for this option.) and select the deployment plan saved in a shared storage location; all servers in the cluster must be able to access the plan. R. Click Finish. S. Activate the changes. Similarly, any other custom JNDI urls used in the SOACS system should also be updated so that when a switchover/failover occurs in the SOACSDR system, the urls are valid also in the secondary site. 11 SOA CLOUD SERVICES DISASTER RECOVERY

17 3. Set up and verify Data Guard The steps to set up Data Guard across two locations are generic for any DbaaS Database. Refer to Appendix B for details Provision SOACS in secondary location pointing to snapshot standby database Once Data Guard has been configured the secondary SOACS system can be provisioned. The following are key considerations for this set up: 1. In order to use the standby database as the container for SOACS schemas, the physical standby needs to be converted to a snapshot standby. 2. Oracle SOACS provisions SOA schemas using a prefix that is specific to each cloud services/instance. This means that in the initial provisioning the secondary location SOACS servers will point to the same Database but will use different schemas. This is critical for running systems because this will prevent the execution of composites/flows by the initial SOACS domain in the secondary location. It is needed that only one site has active SOA servers pointing to an available database at any point in time. Otherwise message and callback duplications could occur leading the SOA system to inconsistencies. Follow these steps to provision the secondary SOACS system: A. Convert the physical standby database in the secondary site into a snapshot standby IMPORTANT: Once the secondary location JDBC strings are updated to point to the same schemas as production, the SOA servers in the secondary location will see the same data that the production ones were seeing when the snapshot conversion occurred. If any SOA flows, callbacks etc. are pending, the servers in the secondary location will try to complete those. Thus, it is important that instances are drained and completed on the primary site before converting the standby database to snapshot or duplications could occur. Alternatively the SOA servers in the primary location may be stopped and the database can be switched over to the secondary location (incurs in higher downtime). From the primary database host and as user oracle execute the following commands: [oracle@soacsdb ~]$. ~/script.env 8 Apply fix for bug to your specific database version or, as a temporary workaround, adjust the PDB Service to the standby domain after the first switchover. To do this, connect to sqplus as sysdba on the standby and execute the following: update cdb_service$ set con_id#=3 where name='pdb1.opcdomain ; where opc_domain is the value returned by show parameter db_domain in the standby database. For example: SQL> show parameter db_domain NAME TYPE VALUE db_domain string esoracle25875.oraclecloud.inte rnal SQL> update cdb_service$ set con_id#=3 where name='pdb1.esoracle25875.oraclecloud.internal'; 1 row updated. SQL> commit; Commit complete. 12 SOA CLOUD SERVICES DISASTER RECOVERY

18 ~]$dgmgrl DGMGRL> CONVERT DATABASE orclb to SNAPSHOT STANDBY; Converting database "orclb" to a Snapshot Standby database, please wait... Database "orclb" converted successfully B. Provision the SOACS instance using the SOACS provisioning wizard: Follow the steps in the SOACS documentation to create the secondary site SOACS system. Use the EXACT same name for the SOACS service that you are using in your primary location. The following table summarizes the wizard options for the set up SOACS Configuration Primary Site Secondary Site Oracle Cloud Domain domaina domainb Oracle Cloud User joe@acme.com joe@acme.com Oracle Cloud Password Acme1234# Acme1234# SOACS Service Type SOA Cluster SOA Cluster SOACS Software Version SOACS Instance Name soacsdr soacsdr SOACS DB Configuration soacsdb soacsdbb SOACS Cluster Size 2 2 SOACS Compute Shape OC2M OC2M WebLogic User weblogic weblogic WebLogic Password Acme1234# Acme1234# Load Balancer Provisioning Yes Yes Load Balancer Policy Least Connection Count Least Connection Count Load Balancer Compute Shape OC3 OC3 SOACS Backup Container soacscontainer soacsbcontainer Oracle recommends that the exact same capacity and compute configuration is used on both primary and standby locations for the ideal failover/switchover behavior (otherwise, the required request-throttling in OTD and sizing of SOACS nodes needs to be done on the secondary location). Once the provisioning process completes the SOA servers can be sanity verified. 5. Create virtual hostname for frontend OTD and update frontend address in secondary site Follow the same steps as in section Create virtual hostname for frontend OTD and update frontend address in primary site above using the public IP of the OTD instance in the secondary site to map the virtual host. NOTE: Despite the hostname alias being the same in both sites, the required trust stores/certificates will have to be recreated in the standby locations and the standby s OTD/LBRS certificate will have to be imported in the apporpirate trust store if any SSL invocations are executed from the Cloud servers to the front end LBR. Refer to the Enterprise Deployment Guide for for Oracle SOA Suite for the required steps 6. Update t3/rmi urls in the secondary site (if used) Follow the same steps as in section Update t3/rmi urls in the primary site above using the same cluster name as in primary. 13 SOA CLOUD SERVICES DISASTER RECOVERY

19 7. Copy and update fmwconfig bootstrap artifacts to secondary location The FMW configuration in the secondary site needs to be updated so that both locations reuse the same policy and JPS configuration. For this, it is required to copy the domain_name/config/fmwconfig artifacts from the primary SOACS Administration servers to the secondary one and update the JPS JDBC url of the primary site with the address that the WLS servers in the secondary site will use to access the DB. The entire domain does NOT need to be copied (and, due to firewall and access restrictions across Cloud datacenters, cannot be copied). Follow these steps: A. From the primary location s Admin Server host as user oracle execute the following: [oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config [oracle@soacsdr-jcs-wls-1~]$ tar -czvf /u01/data/fmwconfig-site1.gz./fmwconfig/ [oracle@soacsdr-jcs-wls-1~]$ scp -i key_for_ssh.ppk /u01/data/fmwconfig-site1.gz secondary_site_public_ip:/tmp/ It is expect that the key_for_ssh.ppk file generated as prescribed in the SOACS Documentation is available in both locations. The key, can be copied itself from a third party node to the compute nodes themselves to enable ssh between them. B. In the secondary s site, execute the following steps as the oracle user a. Stop the Administration and Managed servers using the Weblogic Administration Console b. Move the fmwconfig directory to a backup name [oracle@soacsdr-jcs-wls-1~]$ mv /u01/data/domains/domain_name/config/fmwconfig/ /u01/data/domains/domain_name/config/fmwconfigbackup c. Replace the fmwconfig directory with the copy from the primary site. [oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config [oracle@soacsdr-jcs-wls-1~]$ tar xzvf /tmp/fmwconfig-site1.gz d. Replace the identity domain and database name in the JDBC connect string used by OPSS: [oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config/fmwconfig [oracle@soacsdr-jcs-wls-1~]$find./*.xml xargs sed i 's/primary_database_node:1521\/pdb1.primary_domain_name.oraclecloud.internal/secondary_d atabase_node:1521\/pdb1.secondary_domain_name.oraclecloud.internal/g' Where primary_domain_name and secondary_domain_name are the cloud identity domains of each site. For example: [oracle@soacsdr-jcs-wls-1~]$find./*.xml xargs sed i 's/soacsdb:1521\/pdb1.us6z12opcsoa01.oraclecloud.internal/soacdbb:1521\/pdb1.esoracle oraclecloud.internal/g' 8. Replace schema prefix in the standby datasources As described in previous sections, each site will use a local database address in the JDBC connect string, however it is required to update the schema names so that both sites use the same schema. Execute these steps to replace the schema prefix in the secondary site. 14 SOA CLOUD SERVICES DISASTER RECOVERY

20 A. Make a copy of your DataSource configuration. As the oracle user in the secondary Administration Server node [oracle@soacsdr-jcs-wls-1~]$ cp -R /u01/data/domains/domain_name/config/jdbc/ /u01/data/domains/domain_name/config/jdbcbackup B. Identify the schema prefix in the primary site. As the oracle user in the primary Administration Server node [oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config/jdbc/ [oracle@soacsdr-jcs-wls-1~]$ cat SOADataSource-jdbc.xml grep _SOAINFRA <value>sp _soainfra</value> The prefix is the value preceding the _SOAINFRA string (in this example SP ) C. Identify the schema prefix in the secondary site. As the oracle user in the secondary Administration Server node [oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config/jdbc/ [oracle@soacsdr-jcs-wls-1~]$ cat SOADataSource-jdbc.xml grep _SOAINFRA <value>sp _soainfra</value> The prefix is the value preceding the _SOAINFRA string (in this example SP on the standby) D. Replace the initial schema prefix in the secondary site with the prefix of the primary site. As the oracle user in the secondary Administration Server node [oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config/jdbc/ find /u01/data/domains/domain_name/config/jdbc -name '*.xml' xargs sed -i 's/secondary_location_prefix/primary_location_prefix/g' For example (using the values returned above): find /u01/data/domains/domain_name/config/ *.xml xargs sed -i 's/ SP /SP /g' IMPORTANT: Up to this point, the SOA servers in the secondary location have been pointing to empty SOAINFRA schemas with no composites deployed, no policies and no flows pending of execution. Once the secondary location JDBC strings have been updated to point to the same schemas as production per the above steps, the SOA servers in the secondary location will see the same data that the production ones were seeing. If any flows, callbacks etc were pending to be executed; the secondary location servers will try to complete those at this point if started. Thus, it is important that instances are drained and completed on the primary site before converting to snapshot the standby database as already indicated above. E. If DBFS is being used for sharing files across the SOACS node in acluster, it is needed to recreate the wallets and tns connect strings used for the local HA dbfs mounts11. a. As the oracle user in the Admin Server node in the secondary location update the contents of domain_location/dbfs/tnsnames.ora to match those of Dataguard set up. Make sure the tnsnames.ora file points to the right name in standby 11 SOACS provides default DBFS mount points to share files across the members of the SOA cluster in the scope of a single Datacenter. These DBFS mounts are used by components like MFT, B2B and the File Adapter. 15 SOA CLOUD SERVICES DISASTER RECOVERY

21 cat /u01/data/domains/maa2_domainbackup/dbfs/tnsnames.ora ORCLB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = drdb2a)(port = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PDB1.secondarydomain.oraclecloud.internal) b. Re-create the wallet [oracle@soacsdr-jcs-wls-1~]$mv /u01/data/domains/maa2_domain/dbfs/wallet/ /u01/data/domains/maa2_domain/dbfs/wallet_orig [oracle@soacsdr-jcs-wls-1~]$unset ORACLE_HOME [oracle@soacsdr-jcs-wls-1~]/u01/app/oracle/middleware/oracle_common/bin/mkstore -wrl /u01/data/domains/maa2_domain/dbfs/wallet create [oracle@soacsdr-jcs-wls-1~]/u01/app/oracle/middleware/oracle_common/bin/mkstore -wrl /u01/data/domains/maa2_domain/dbfs/wallet createcredential ORCLB primary_location_prefix_dbfs Cloud_Schema_Password12 c. Remount the DBFS directories witht the wallet [oracle@soacsdr-jcs-wls-1~]fusermount u /u01/soacs/dbfs_directio [oracle@soacsdr-jcs-wls-1~]fusermount u /u01/soacs/dbfs [oracle@soacsdr-jcs-wls-1~]export ORACLE_HOME=/u01/app/oracle/middleware/dbclient/ [oracle@soacsdr-jcs-wls-1~]$oracle_home/bin/dbfs_client -o wallet /@ORCLB -o direct_io /u01/soacs/dbfs_directio [oracle@soacsdr-jcs-wls-1~]$oracle_home/bin/dbfs_client -o wallet /@ORCLB /u01/soacs/dbfs 9. Convert the snapshot standby database back to the physical standby database Execute these steps as oracle user in the primary Database host: [oracle@soacsdb ~]$. ~/script.env [oracle@soacsdb ~]$. dgmgrl sys/${passwd}@${a_dbnm} DGMGRL> CONVERT DATABASE orclb to PHYSICAL STANDBY; Converting database "orclb" to a Physical Standby database, please wait... Oracle Clusterware is restarting database "orclb"... Continuing to convert database "orclb"... Database "orclb" converted successfully NOTE: Give some time for the redo-apply to catch up on standby before continuing with the next steps and make sure the configuration is correct in dgmgrl 10. Switchover Database Execute these steps as oracle user in the primary Database host: [oracle@soacsdb ~]$. ~/script.env [oracle@soacsdb ~]$. dgmgrl sys/${passwd}@${a_dbnm} DGMGRL> switchover to orclb Performing switchover NOW, please wait... Operation requires a connection to instance "ORCL" on database "orclb" Connecting to instance "ORCL"... Connected as SYSDBA. 12 This is the password provided for the SOACS service creation 16 SOA CLOUD SERVICES DISASTER RECOVERY

22 New primary database "orclb" is opening... Oracle Clusterware is restarting database "orcl"... Switchover succeeded, new primary is "orclb" 11. Verify SOACS in secondary site At this point the SOA servers in both sites are pointing to the same schemas and the database is open in the secondary location. Start the servers and verify composite deployments and composite instances (if any) in the new active site a. Start the Administration and Managed servers in the secondary site a. Logon to the Enterprise Manager FMW Control Console for the secondary SOACS instance and verify the soa-infra systems in both servers in the cluster. It is a good practice to perform an endpoint test to confirm the correct behavior of the system 12. Switchback to original site. As an optional step and to revert the system back to its original state, switchback the database and restart SOA servers in the primary location. 17 SOA CLOUD SERVICES DISASTER RECOVERY

23 SOACS DR Lifecycle Procedures Switchover In a switchover operation an administrator reverts the roles of the two sites in a planned operation. The roles change from the primary to the standby as well as from standby to primary. To perform a switchover in a SOACS DR configuration follow these steps: a. Stop the managed servers in the SOACS primary site. Use the SOACS documentation to stop the Administration and Managed servers in the primary site b. If configuration changes have been applied recently to the primary SOA/Weblogic domain, make sure to propagate those to the standby location (Refer to section Applying domain configuration changes to the system bellow) c. Perform the required DNS changes to point consumers to the new primary OTD Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host resolution to point the front end address of the system to the public IP used by OTD in site2 d. Perform a database switchover using Data Guard Broker From the primary site s DBaaS node and as oracle user: [oracle@soacsdb ~]$. ~/script.env [oracle@soacsdb ~]$ dgmgrl sys/${passwd}@${a_dbnm} DGMGRL> switchover to orclb Performing switchover NOW, please wait... Operation requires a connection to instance "ORCL" on database "orclb" Connecting to instance "ORCL"... Connected as SYSDBA. New primary database "orclb" is opening... Oracle Clusterware is restarting database "orcl"... Switchover succeeded, new primary is "orclb" e. Start Managed servers in the new SOACS primary site Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site. (the Administration server and Node Manager can be kept up on standby) Failover In a failover operation, the primary site becomes unavailable and an administrator fails over the Database and starts managed servers in the secondary site. You can role-transition a standby database to a primary database when the original primary database fails and there is no possibility of recovering the primary database in a timely manner. This is known as a manual failover. There may or may not be data loss depending upon whether your primary and target standby databases were transactionally consistent at the time of the primary database failure. To perform a switchover in a SOACS DR configuration follow these steps: a. Perform the required DNS changes to point consumers to the new primary OTD Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host resolution to point the front end address of the system to the public IP used by OTD in Site2 18 SOA CLOUD SERVICES DISASTER RECOVERY

24 b. Perform a database failover using Data Guard Broker. Start Data Guard broker on the secondary DBaaS node. As oracle user execute these steps: [oracle@soacsdbb ~]$. ~/script.env [oracle@soacsdbb ~]$ dgmgrl sys/${passwd}@${a_dbnm} DGMGRL> failover to orclb Performing failover NOW, please wait... Failover succeeded, new primary is "orclb" c. Start Managed servers in the new SOACS primary site Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site Applying domain configuration changes to the system The fact that file system synchronization across Cloud datacenters is not allowed, precludes WLS domain config changes from being automatically propagated to the secondary site (as it would occur in standard on-premise Active Passive DR deployments). Two main approaches can be used to maintain the same configuration (ear deployments, WLS domain configuration, deployment plans etc) in both locations. The applicability of each depends on how frequently this file-system-resident configuration is modified. a) For those SOACS cases where the domain configuration is infrequently altered (notice that composite deployments, domain and WSM policies and MDS updates do not fall into this category as they are stored in the Database) it is recommended to simply apply the configuration changes manually twice, once in production and once in standby. b) For those SOACS cases where the file system configuration is modified regularly, Oracle Database File System (DBFS) can be used to synchronize the configuration using Dataguard (DBFS provides a file system view of data that is stored in the Database). Using DBFS for configuration replication has implications from the set up, disk space allocation and lifecycle perspective and oracle recommends using it only when it is strictly necessary to replicate configuration changes very frequently. There are other alternatives to DBFS such as direct use of rsync across sites, but those present other risks including lack of transactional control in the copy and possible corruption of the domain structure in the event of a failure during the copy operations. Both approaches are described below. a) Repeating domain configuration changes in both sites To maintain the file system configuration synchronized by repeating the config change in both sites, follow these steps: A. Apply the configuration change normally in the primary site Use the WLS Administration Console in the primary location to apply the configuration change. Activate the change, restart the required SOACS servers if needed and verify that the change is working as expected. B. Convert the standby database to a snapshot standby Changes to a reduced number of configuration artifacts in SOA and OSB may require the servers to be up in order to be applied, in these cases a switchover and a start of the managed servers will be needed. Refer to the specific product documentation to identify these artifacts. 19 SOA CLOUD SERVICES DISASTER RECOVERY

25 Execute these steps as oracle user in the primary Database host: ~]$. ~/script.env ~]$ dgmgrl DGMGRL> CONVERT DATABASE orclb to PHYSICAL STANDBY; Converting database "orclb" to a Physical Standby database, please wait... Oracle Clusterware is restarting database "orclb"... Continuing to convert database "orclb"... Database "orclb" converted successfully C. Start (if it wasn t started) the Administration Server on the secondary site Follow the steps in the Oracle Cloud documentation to start the administration server. It is important that ONLY the administration server and not the managed servers is started on the secondary location. The reason being that started SOA managed could resume/recover any pending flows, callbacks or and lead to duplicates if the primary site remains up also. D. Repeat the configuration change in the secondary site Use the WLS Administration Console in the primary location to apply the configuration change. Activate the change, restart the required SOACS servers if needed and verify that the change is working as expected. This change should not alter any of the configurations in the database, only WLS domain configuration or application deployments (not ear deployments). Any modifications in the database will be overwritten by the primary when the Db is reverted to physical standby in the next step. E. Revert the database to physical standby Execute these steps as oracle user in the primary Database host: DGMGRL> CONVERT DATABASE orclb to PHYSICAL STANDBY; Converting database "orclb" to a Physical Standby database, please wait... Oracle Clusterware is restarting database "orclb"... Continuing to convert database "orclb"... Database "orclb" converted successfully b) Using DBFS for configuration propagation Database File System (DBFS) uses database features to store files and manage relational data and expose them as a standard file system that can be accessed by any operating system. The Oracle Database File System (DBFS) creates a standard file system interface on top of files and directories that are stored in database tables. DBFS is similar to NFS in that it provides a shared network file system that looks like a local file system. Like NFS, both a server component and a client component are required to run DBFS. Given the restrictions in file replication across Oracle Cloud data centers, DBFS can be used with Oracle Data Guard to copy files from a primary site to a secondary site. When the system s lifecycle involves frequents updates to the domain file system, DBFS can be used to replicate the WLS domain configuration across sites. However, the SOACS domain Configuration cannot reside directly on the DBFS mount because that would make the middle tier dependent on the DBFS infrastructure in order to come up (the dependency is not only on the database but also on FUSE libraries, mount points etc). Note also that the SOACS Weblogic domain configuration cannot be copied as is in this design since each site in the SOACS DR solution contains references to the Cloud identity domain and the local DB service in the JDBC connect strings. The configuration has to be modified after it is copied to each site. In this approach, a directory with a copy of the primary site s domain configuration is kept up to date with Data Guard in the standby location. This directory is not available to the standby site unless the DB is open in read-only mode (the 20 SOA CLOUD SERVICES DISASTER RECOVERY