Oracle Optimized Solution for Disaster Recovery

Transcription

1 Oracle Optimized Solution for Disaster Recovery Oracle E-Business Suite Using Oracle Solaris Cluster Geo Edition Implementation Guide A U G U S T E

2 Table of Contents 1. Overview Architecture... 2 Sites Configuration with Oracle E-Business Suites Hostnames and IP Address Information The /etc/hosts File The FND_NODES Table... 4 Sites Configuration with Oracle Solaris Cluster Geo Edition Configure the Oracle E-Business Suite DB Context Files... 7 Edit the /etc/hosts Files... 8 Set the s_virtual_hostname Variable... 8 Run AutoConfig on the Primary Cluster... 9 Run AutoConfig on the Secondary Cluster Verify the Contents of the FND_NODES Table Configure Flashback on the Database Configure Oracle Solaris Cluster Geo Edition Install Geographic Edition Software Prepare Each Zone Cluster for Partner Membership Set Up the Network Address for Each Zone Cluster Set Up Security on the Primary Cluster Set Up Security on the Secondary Cluster Enable Geographic Edition Software Configure Partnerships Configure Trust Between Partner Clusters Create Partnerships Configure the Secondary Cluster to Join the Partnership Configure Protection Groups Create the First Protection Group (opmn-ebs-pg... 56

3 Create the Second Protection Group (cm-ebs-pg Create the Third Protection Group (cm-odg-pg Display the Oracle Solaris Cluster Listings Configure Sites and Multigroups Create and Configure the santaclara Site Create and Configure the hillsboro Site Create and Configure a DNS Server Configure the DNS Service Configure the DNS Update Scripts Create the dns_update.sh Script Verify the DNS Update Scripts on the DNS Server Configure the DNS Client Configure the Protection Group Role Change Script Load Testing with Oracle Application Testing Suite (OATS The Test Configuration OLTP Load Generator: the OATS Machine OATS Testing Information Concurrent Manager Reports Switchover ebs-mg Multigroup to the Secondary Cluster Switchover Operation Switchover Timing Information Takeover ebs-mg Multigroup from the Secondary Cluster Takeover Operation Takeover Timing Information Restore the Primary Site Apply SuperCluster Quarterly Maintenance Updates Overview ii

4 Overview ii

5 1. Overview This document provides a step-by-step guide to set up disaster recovery for an Oracle E-Business Suite installation on Oracle SuperCluster. The configuration uses Oracle Solaris Cluster Geo Edition to switch Oracle E- Business Suite to a standby Oracle SuperCluster in the event of a failure. The primary and standby SuperClusters can be in different geographical locations. The document assumes that you have installed Oracle E-Business Suite and configured Oracle Data Guard and Oracle s ZFS Storage Appliance by following the instructions provided in the following guides (available on My Oracle Support: Oracle Optimized Solution for Oracle E-Business Suite on Oracle SuperCluster Example Implementation Guide (Doc ID Oracle Optimized Solution for Disaster Recovery Oracle E-Business Suite Example Implementation Guide (Doc ID The guides contain procedures for duplicating database content and applications between primary and standby Oracle SuperClusters. This document builds on the configurations from the implementation guides by deploying Oracle Cluster Geo Edition to automatically orchestrate the stopping and starting of Oracle E-Business Suite services after a failover to the standby SuperCluster. Furthermore, in the configuration of the primary and standby SuperClusters, logical hostnames are used for the APP Tiers. Likewise, an alias hostname for each Oracle RAC VIP is used as a virtual hostname for each Oracle RAC database host. Thus, in the event of a switchover, you do not need to repeatedly run AutoConfig beyond initial configuration. The topics for a disaster recovery solution are presented as follows: 1. Description of the reference architecture. 2. Configuration of Oracle E Business Suite in both the primary and secondary SuperClusters. 3. Installation and configuration of Oracle Solaris Cluster Geo Edition. 4. Description of the switchover process between the primary and standby SuperClusters which ends with the standby SuperCluster taking subsequent control. Overview 1

6 2. Architecture This section describes the reference architecture for the implementation of the disaster recovery program. Sites Configuration with Oracle E-Business Suites The configuration of Oracle E-Business Suites in the primary and standby sites is illustrated in the following figure: Figure 1- Starting Configuration of Primary and Standby Sites Note: In this document, the terms standby and secondary are used interchangeably. As shown in Figure 1, each site consists of four Oracle VM SPARC Server domains. Two clustered database domain containing Oracle RAC Two clustered application domains containing Oracle Solaris Cluster, which is configured with the following zone clusters: o o Zone cluster containing a Web Entry Point (Web and 2 Oracle Process Manager and Server (OPMN1 and OPMN2 services Zone cluster containing Concurrent Manager (CM1 services Architecture 2

7 Additionally, redo log shipping and ZFSSA continuous replication are configured for both primary and standby sites. Db_unique_names are assigned for the two sites: VIS at the primary site VISION at the standby site In this configuration, Oracle Solaris Cluster Geo Edition is not yet installed on either site. Only the primary site is active. Note: In the configuration on which this document is based, a symlink is set up to an ENV file. This is in place for each database node. Below is an example. root@etc18cn1dbiodom1:~# su oracle Oracle Corporation SunOS December 2015 oracle@etc18cn1dbiodom1:~$ oracle@etc18cn1dbiodom1:~$ ls l VIS.env lrwxrwxrwx 1 oracle oinstall 67 Oct VIS.env -> /u01/app/oracle/product/ /dbhome_1/vis1_etc18cn1dbiodom1.env The file enables you to access the SQL and the DGMGRL prompts. Accessing either prompt is part of the subsequent procedures in this document. When accessing the DGMGRL prompt, make sure that you use the correct DB sys user password. For example, after sourcing the ENV file, type: oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password> Hostnames and IP Address Information The Oracle SuperCluster provides an external 10 gigabit Ethernet (10GbE network and internal Internet Protocol over InfiniBand (IPoIB network. The latter is used as a private non-routable network ( /16 prefix that is only accessible from within the Oracle SuperCluster. One of the tasks in this document to implement disaster recovery is to map an alias hostname for each Oracle RAC VIP as a virtual hostname for each Oracle RAC database host. After you complete the task, the mapping of hostnames, IP addresses, and the aliases appears in both the /etc/hosts files of the DB and APP tier nodes as well as FND_NODES table for both sites The /etc/hosts File For example, on the primary site, the /etc/hosts file on each DB and APP Tier contains the following entries: etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip idb1-vip.us.oracle.com idb1-vip etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip idb2-vip.us.oracle.com idb2-vip Architecture 3

8 On the standby site, the /etc/hosts file on each DB and APP Tier contains the following entries: etc18db01-ibvip.us.oracle.com etc18db01-ibvip idb1-vip.us.oracle.com idb1-vip etc18db02-ibvip.us.oracle.com etc18db02-ibvip idb2-vip.us.oracle.com idb2-vip In these sample /etc/hosts files, the alias hostnames idb1-vip and idb2-vip map to the same IP addresses at the primary and standby sites, respectively, and The FND_NODES Table The mapping also appears in the FND_NODES table within the APPS schema as shown in this example. su oracle Oracle Corporation SunOS December VIS.env sqlplus /as sysdba SQL *Plus: Release Production Copyright (c 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> SQL> select node_name, server_address, virtual_ip from fnd_nodes; NODE_NAME SERVER_ADDRESS VIRTUAL_IP EWEB-LH m.m.m.68 IOPMN2-LH ETC11-CN01 ETC18CN1DBIODOM1 ICM1-LH ETC11-CN02 ETC18CN2DBIODOM1 AUTHENTICATION * idb1-vip idb1-vip idb2-vip idb2-vip IOPMN1-LH rows selected. SQL> Architecture 4

9 Note: In all the examples in this documentation, some IP addresses have been masked on the first 3 octets to indicate that these should reflect valid addresses that are actually used at your company s configuration. For the primary cluster, the masked address is n.n.n.#, such as n.n.n.27. For the secondary cluster, the masked address is m.m.m.#, such as m.m.m.68.the distinction also indicates the typical configuration where primary and secondary clusters are on different subnets. The FND_NODES table provides the following information: Node names for the nodes in the Cluster configuration. Oracle Solaris Cluster manages the location of the logical hostname for the node names. Consequently, the logical hostname is hosted only on one node at any one time. The hostnames are prefixed with either E or I. The E prefix designates that the logical hostname is an external address on the 10GbE network. The I prefix indicates that the hostname is in an internal address on the IPoIB network. The same logical hostnames for the services are used at the primary and standby sites, specifically EWEB- LH, IOPMN1-LH, IOPMN2-LH and ICM1-LH. Two Oracle SuperCluster DB hostname pairs exist: o o ETC11-CN01 and ETC11-CN02 on the primary site. ETC18CN1DBIODOM1 and ETC18CN2DBIODOM1 on the standby site. Each pair of Oracle SuperCluster DB hostnames is mapped to corresponding virtual IP names in the internal IPoIB network as follows: o o ETC11-CN01 and ETC18CN1DBIODOM1 are mapped to idb1-vip. ETC11-CN02 and ETC18CN2DBIODOM1 are mapped to idb2-vip. Architecture 5

10 Sites Configuration with Oracle Solaris Cluster Geo Edition The following figure shows the primary and standby sites with Oracle Solaris Cluster Geo Edition installed and configured. Figure 2- Intended Final Configuration of Primary and Standby Sites Figure 2 still reflects the basic architecture as illustrated in Figure 1. However, this figure also shows how partnerships are configured through Oracle Solaris Cluster Geo Edition: Partnership 1 consists of the OPMN zone clusters of both sites Partnership 2 consists of the CM zone clusters of both sites. Likewise, protection groups are also configured for the partnerships: Protection Group 1 consists of the WEB and OPMN services. Protection Group 2 consists of the CM services and ZFS Storage Appliance Protection Group 3 consists of the Oracle Data Guard Note that Protection Groups 2 and 3 are set for Partnership 2. The steps to configure partnerships and protection groups are provided later in the section Configure Oracle Solaris Cluster Geo Edition. With this configuration, Oracle Solaris Cluster Geo Edition provides the capability that orchestrates the stopping and starting of Oracle E-Business Suite services after a switchover or failover from the primary site to the Architecture 6

11 standby site. In summary, the architecture resembles the configurations described in the Oracle Optimized Solution for Oracle E-Business Suite on Oracle SuperCluster Example Implementation Guide (Doc ID and Oracle Optimized Solution for Disaster Recovery Oracle E-Business Suite Example Implementation Guide (Doc ID Configure the Oracle E-Business Suite DB Context Files This section provides the procedure to configure the Oracle E-Business Suite DB context files to prepare for disaster recovery setup. Based on the reference architecture described previously, the following DB and APP Tier nodes exist in the current configuration. Refer to this table while performing the configurations as instructed: On the Primary Cluster Tier Node RAC VIP DB (RAC etc11-cn01 etc11-01-ib-vip DB (RAC etc11-cn02 etc11-02-ib-vip APP (opmn-zc eopmn1 APP (opmn-zc eopmn2 APP (cm-zc ecm1 APP (cm-zc ecm2 On the Secondary Cluster Tier Node RAC VIP DB (RAC DB (RAC APP (dr-opmn-zc APP (dr-opmn-zc APP (dr-cm-zc APP (dr-cm-zc etc18cn1dbiodom1 etc18db01-ibvip etc18cn2dbiodom1 etc18db02-ibvip deopmn1 deopmn2 decm1 decm2 In the procedure that follows, you would perform the steps on all 12 nodes as indicated. Configure the Oracle E-Business Suite DB Context Files 7

12 Edit the /etc/hosts Files This step maps the hostnames and their addresses to the virtual IP names idb1-vip and idb2-vip. The revisions are in bold. On the primary cluster, edit the /etc/hosts files to reference the virtual IP names etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip idb1- vip.us.oracle.com idb1-vip etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip idb2- vip.us.oracle.com idb2-vip On the secondary cluster, edit the /etc/hosts files to reference the virtual IP names etc18db01-ibvip.us.oracle.com etc18db01-ibvip idb1- vip.us.oracle.com idb1-vip etc18db02-ibvip.us.oracle.com etc18db02-ibvip idb2- vip.us.oracle.com idb2-vip Set the s_virtual_hostname Variable This step populates the FND_NODES table and subsequent TNS entries at each DB and APP Tier. On the primary cluster, perform the following steps. a. On etc11-cn01, set s_virtual_hostname to idb1-vip. /u01/app/oracle/product/ /dbhome_1/vis1_etc11- cn01.env echo $CONTEXT_FILE /u01/app/oracle/product/ /dbhome_1/appsutil/vis1_etc11-cn01.xml grep s_virtual_hostname $CONTEXT_FILE <host oa_var="s_virtual_hostname">idb1-vip</host> b. On etc11-cn02, set s_virtual_hostname to idb2-vip. /u01/app/oracle/product/ /dbhome_1/vis2_etc11- cn02.env echo $CONTEXT_FILE /u01/app/oracle/product/ /dbhome_1/appsutil/vis2_etc11-cn02.xml Configure the Oracle E-Business Suite DB Context Files 8

13 grep s_virtual_hostname $CONTEXT_FILE <host oa_var="s_virtual_hostname">idb2-vip</host> On the secondary cluster, perform the following steps: a. On etc18cn1dbiodom1, set s_virtual_hostname to idb1-vip. /u01/app/oracle/product/ /dbhome_1/vis1_etc18cn1dbiodom1.env echo $CONTEXT_FILE /u01/app/oracle/product/ /dbhome_1/appsutil/vis1_etc18cn1dbiodom1.xml grep s_virtual_hostname $CONTEXT_FILE <host oa_var="s_virtual_hostname">idb1-vip</host> b. On etc18cn2dbiodom1, set s_virtual_hostname to idb2-vip. /u01/app/oracle/product/ /dbhome_1/vis2_etc18cn2dbiodom1.env echo $CONTEXT_FILE /u01/app/oracle/product/ /dbhome_1/appsutil/vis2_etc18cn2dbiodom1.xml grep s_virtual_hostname $CONTEXT_FILE <host oa_var="s_virtual_hostname">idb2-vip</host> Run AutoConfig on the Primary Cluster On etc11-cn01, run the adautocfg.sh script. /u01/app/oracle/product/ /dbhome_1/vis1_etc11- cn01.env echo $ORACLE_HOME /u01/app/oracle/product/ /dbhome_1 cd $ORACLE_HOME/appsutil/scripts/VIS1_etc11-cn01 Enter the APPS user password: The log file for this session is located at: /u01/app/oracle/product/ /dbhome_1/appsutil/log/ VIS1_etc11-cn01/ /adconfig.log AutoConfig is configuring the Database environment... Configure the Oracle E-Business Suite DB Context Files 9

14 AutoConfig will consider the custom templates if present. Using ORACLE_HOME location : /u01/app/oracle/product/ /dbhome_1 Classpath : :/u01/app/oracle/product/ /dbhome_1/ jdbc/lib/ojdbc5.jar:/u01/app/oracle/product/ /dbhome_1/ appsutil/java/xmlparserv2.jar:/u01/app/oracle/product/ / dbhome_1/appsutil/java:/u01/app/oracle/product/ /dbhome_1/ jlib/netcfg.jar:/u01/app/oracle/product/ /dbhome_1/jlib/ ldapjclnt11.jar Using Context file : /u01/app/oracle/product/ /dbhome_1/appsutil/vis1_etc11-cn01.xml Context Value Management will now update the Context file Updating Context file...completed Attempting upload of Context file and templates to database...completed Updating rdbms version in Context file to db112 Updating rdbms type in Context file to 64 bits Configuring templates from ORACLE_HOME... AutoConfig completed successfully. oracle@etc11-cn01:~$ On etc11-cn02, run the adautocfg.sh script. oracle@etc11-cn02:~$. /u01/app/oracle/product/ /dbhome_1/vis2_etc11- cn02.env oracle@etc11-cn02:~$ echo $ORACLE_HOME /u01/app/oracle/product/ /dbhome_1 oracle@etc11-cn02:~$ oracle@etc11-cn02:~$ cd $ORACLE_HOME/appsutil/scripts/VIS2_etc11-cn02 oracle@etc11-cn02:~$./adautocfg.sh Enter the APPS user password: The log file for this session is located at: /u01/app/oracle/product/ / dbhome_1/appsutil/log/vis2_etc11-cn02/ /adconfig.log AutoConfig is configuring the Database environment... AutoConfig will consider the custom templates if present. Using ORACLE_HOME location : /u01/app/oracle/product/ /dbhome_1 Classpath : :/u01/app/oracle/product/ /dbhome_1/jdbc/lib/ojdbc5.jar:/u01/app/ oracle/product/ /dbhome_1/appsutil/java/xmlparserv2.jar:/ u01/app/oracle/product/ /dbhome_1/appsutil/java:/u01/app/oracle/ Configure the Oracle E-Business Suite DB Context Files 10

15 product/ /dbhome_1/jlib/netcfg.jar:/u01/app/oracle/product/ / dbhome_1/jlib/ldapjclnt11.jar Using Context file : /u01/app/oracle/product/ /dbhome_1/appsutil/vis2_etc11-cn02.xml Context Value Management will now update the Context file Updating Context file...completed Attempting upload of Context file and templates to database...completed Updating rdbms version in Context file to db112 Updating rdbms type in Context file to 64 bits Configuring templates from ORACLE_HOME... AutoConfig completed successfully. oracle@etc11-cn02:~$ On etc11-cn01, run the adautocfg.sh script. This second run of AutoConfig on the node updates the $ORACLE_HOME/VIS1_etc11- cn01/tnsnames.ora file to make its entries for the RAC instance on etc11-cn02 reflect the idb2-vip alias. The file can only be updated after AutoConfig is run on etc11-cn02 in the preceding step. In the following example, the output is intentionally truncated. oracle@etc11-cn01:~$. /u01/app/oracle/product/ /dbhome_1/vis1_etc11- cn01.env oracle@etc11-cn01:~$ echo $ORACLE_HOME /u01/app/oracle/product/ /dbhome_1 oracle@etc11-cn01:~$ oracle@etc11-cn01:~$ cd $ORACLE_HOME/appsutil/scripts/VIS1_etc11-cn01 oracle@etc11-cn01:~$./adautocfg.sh Enter the APPS user password: The log file for this session is located at: /u01/app/oracle/product/ /dbhome_1/appsutil/log/vis1_etc11- cn01/ /adconfig.log AutoConfig is configuring the Database environment... On eopmn1 (eweb-lh, run the adautocfg.sh script. bash-4.1$. /ebsfs/apps/vis/apps/apps_st/appl/appsvis_eweb-lh.env -bash-4.1$ -bash-4.1$ echo $ADMIN_SCRIPTS_HOME Configure the Oracle E-Business Suite DB Context Files 11

16 /ebsfs/apps/vis/inst/apps/vis_eweb-lh/admin/scripts -bash-4.1$ -bash-4.1$ echo $CONTEXT_FILE /ebsfs/apps/vis/inst/apps/vis_eweb-lh/appl/admin/vis_eweb-lh.xml -bash-4.1$ -bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh Enter the APPS user password: The log file for this session is located at: /ebsfs/apps/vis/inst/apps/ VIS_eweb-lh/admin/log/ /adconfig.log AutoConfig is configuring the Applications environment... AutoConfig will consider the custom templates if present. Using CONFIG_HOME location : /ebsfs/apps/vis/inst/apps/vis_eweb-lh Classpath : /ebsfs/apps/vis/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/vis/ apps/apps_st/comn/java/classes Using Context file : /ebsfs/apps/vis/inst/apps/vis_eweb-lh.xml Context Value Management will now update the Context file Updating Context file...completed Attempting upload of Context file and templates to database...completed Configuring templates from all of the product tops... Configuring AD_TOP...COMPLETED Configuring FND_TOP...COMPLETED Configuring ICX_TOP...COMPLETED Configuring MSC_TOP...COMPLETED Configuring IEO_TOP...COMPLETED Configuring BIS_TOP...COMPLETED Configuring AMS_TOP...COMPLETED Configuring CCT_TOP...COMPLETED Configuring WSH_TOP...COMPLETED Configuring CLN_TOP...COMPLETED Configuring OKE_TOP...COMPLETED Configuring OKL_TOP...COMPLETED Configuring OKS_TOP...COMPLETED Configuring CSF_TOP...COMPLETED Configuring IGS_TOP...COMPLETED Configuring IBY_TOP...COMPLETED Configuring JTF_TOP...COMPLETED Configuring MWA_TOP...COMPLETED Configuring CN_TOP...COMPLETED Configuring CSI_TOP...COMPLETED Configuring WIP_TOP...COMPLETED Configure the Oracle E-Business Suite DB Context Files 12

17 Configuring CSE_TOP...COMPLETED Configuring EAM_TOP...COMPLETED Configuring FTE_TOP...COMPLETED Configuring ONT_TOP...COMPLETED Configuring AR_TOP...COMPLETED Configuring AHL_TOP...COMPLETED Configuring OZF_TOP...COMPLETED Configuring IES_TOP...COMPLETED Configuring CSD_TOP...COMPLETED Configuring IGC_TOP...COMPLETED AutoConfig completed successfully. -bash-4.1$ On eopmn1 (iopmn1-lh, run the adautocfg.sh script. -bash-4.1$. /ebsfs/apps/vis/apps/apps_st/appl/appsvis_iopmn1-lh.env -bash-4.1$ -bash-4.1$ echo $ADMIN_SCRIPTS_HOME /ebsfs/apps/vis/inst/apps/vis_iopmn1-lh/admin/scripts -bash-4.1$ -bash-4.1$ echo $CONTEXT_FILE /ebsfs/apps/vis/inst/apps/vis_eweb-lh/appl/admin/vis_iopmn1-lh.xml -bash-4.1$ -bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh Enter the APPS user password: The log file for this session is located at: /ebsfs/apps/vis/inst/apps/vis_iopmn1-lh/admin/log/ /adconfig.log AutoConfig is configuring the Applications environment... AutoConfig will consider the custom templates if present. Using CONFIG_HOME location : /ebsfs/apps/vis/inst/apps/vis_iopmn1-lh Classpath : /ebsfs/apps/vis/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/vis/ apps/apps_st/comn/java/classes Using Context file appl/admin/vis_iopmn1-lh.xml : /ebsfs/apps/vis/inst/apps/vis_iopmn1-lh/ Context Value Management will now update the Context file Updating Context file...completed Attempting upload of Context file and templates to database...completed Configure the Oracle E-Business Suite DB Context Files 13

18 Configuring templates from all of the product tops... Configuring AD_TOP...COMPLETED Configuring FND_TOP...COMPLETED Configuring ICX_TOP...COMPLETED Configuring MSC_TOP...COMPLETED Configuring IEO_TOP...COMPLETED Configuring BIS_TOP...COMPLETED Configuring AMS_TOP...COMPLETED Configuring CCT_TOP...COMPLETED Configuring WSH_TOP...COMPLETED Configuring CLN_TOP...COMPLETED Configuring OKE_TOP...COMPLETED Configuring OKL_TOP...COMPLETED Configuring OKS_TOP...COMPLETED Configuring CSF_TOP...COMPLETED Configuring IGS_TOP...COMPLETED Configuring IBY_TOP...COMPLETED Configuring JTF_TOP...COMPLETED Configuring MWA_TOP...COMPLETED Configuring CN_TOP...COMPLETED Configuring CSI_TOP...COMPLETED Configuring WIP_TOP...COMPLETED Configuring CSE_TOP...COMPLETED Configuring EAM_TOP...COMPLETED Configuring FTE_TOP...COMPLETED Configuring ONT_TOP...COMPLETED Configuring AR_TOP...COMPLETED Configuring AHL_TOP...COMPLETED Configuring OZF_TOP...COMPLETED Configuring IES_TOP...COMPLETED Configuring CSD_TOP...COMPLETED Configuring IGC_TOP...COMPLETED AutoConfig completed successfully. -bash-4.1$ On eopmn2 (iopmn2-lh, run the adautocfg.sh script. -bash-4.1$. /ebsfs/apps/vis/apps/apps_st/appl/appsvis_iopmn2-lh.env -bash-4.1$ -bash-4.1$ echo $ADMIN_SCRIPTS_HOME /ebsfs/apps/vis/inst/apps/vis_iopmn2-lh/admin/scripts -bash-4.1$ -bash-4.1$ echo $CONTEXT_FILE /ebsfs/apps/vis/inst/apps/vis_eweb-lh/appl/admin/vis_iopmn2-lh.xml -bash-4.1$ Configure the Oracle E-Business Suite DB Context Files 14

19 -bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh Enter the APPS user password: The log file for this session is located at: /ebsfs/apps/vis/inst/apps/vis_iopmn2- lh/admin/log/ /adconfig.log AutoConfig is configuring the Applications environment... AutoConfig will consider the custom templates if present. Using CONFIG_HOME location : /ebsfs/apps/vis/inst/apps/vis_iopmn2-lh Classpath : /ebsfs/apps/vis/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/vis/apps/ apps_st/comn/java/classes Using Context file lh/appl/admin/vis_iopmn2-lh.xml : /ebsfs/apps/vis/inst/apps/vis_iopmn2- Context Value Management will now update the Context file Updating Context file...completed Attempting upload of Context file and templates to database...completed Configuring templates from all of the product tops... Configuring AD_TOP...COMPLETED Configuring FND_TOP...COMPLETED Configuring ICX_TOP...COMPLETED Configuring MSC_TOP...COMPLETED Configuring IEO_TOP...COMPLETED Configuring BIS_TOP...COMPLETED Configuring AMS_TOP...COMPLETED Configuring CCT_TOP...COMPLETED Configuring WSH_TOP...COMPLETED Configuring CLN_TOP...COMPLETED Configuring OKE_TOP...COMPLETED Configuring OKL_TOP...COMPLETED Configuring OKS_TOP...COMPLETED Configuring CSF_TOP...COMPLETED Configuring IGS_TOP...COMPLETED Configuring IBY_TOP...COMPLETED Configuring JTF_TOP...COMPLETED Configuring MWA_TOP...COMPLETED Configuring CN_TOP...COMPLETED Configuring CSI_TOP...COMPLETED Configuring WIP_TOP...COMPLETED Configuring CSE_TOP...COMPLETED Configuring EAM_TOP...COMPLETED Configuring FTE_TOP...COMPLETED Configure the Oracle E-Business Suite DB Context Files 15

20 Configuring ONT_TOP...COMPLETED Configuring AR_TOP...COMPLETED Configuring AHL_TOP...COMPLETED Configuring OZF_TOP...COMPLETED Configuring IES_TOP...COMPLETED Configuring CSD_TOP...COMPLETED Configuring IGC_TOP...COMPLETED AutoConfig completed successfully. -bash-4.1$ On ecm1 (icm1-lh, run the adautocfg.sh script. -bash-4.1$. /ebsfs/apps/vis/apps/apps_st/appl/appsvis_icm1-lh.env -bash-4.1$ -bash-4.1$ echo $ADMIN_SCRIPTS_HOME /ebsfs/apps/vis/inst/apps/vis_icm1-lh/admin/scripts -bash-4.1$ -bash-4.1$ echo $CONTEXT_FILE /ebsfs/apps/vis/inst/apps/vis_eweb-lh/appl/admin/vis_icm1-lh.xml -bash-4.1$ -bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh Enter the APPS user password: The log file for this session is located at: /ebsfs/apps/vis/inst/apps/vis_icm1- lh/admin/log/ /adconfig.log AutoConfig is configuring the Applications environment... AutoConfig will consider the custom templates if present. Using CONFIG_HOME location : /ebsfs/apps/vis/inst/apps/vis_icm1-lh Classpath : /ebsfs/apps/vis/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/vis/apps/ apps_st/comn/java/classes Using Context file admin/vis_icm1-lh.xml : /ebsfs/apps/vis/inst/apps/vis_icm1/appl/ Context Value Management will now update the Context file Updating Context file...completed Attempting upload of Context file and templates to database...completed Configuring templates from all of the product tops... Configuring AD_TOP...COMPLETED Configuring FND_TOP...COMPLETED Configure the Oracle E-Business Suite DB Context Files 16

21 Configuring ICX_TOP...COMPLETED Configuring MSC_TOP...COMPLETED Configuring IEO_TOP...COMPLETED Configuring BIS_TOP...COMPLETED Configuring AMS_TOP...COMPLETED Configuring CCT_TOP...COMPLETED Configuring WSH_TOP...COMPLETED Configuring CLN_TOP...COMPLETED Configuring OKE_TOP...COMPLETED Configuring OKL_TOP...COMPLETED Configuring OKS_TOP...COMPLETED Configuring CSF_TOP...COMPLETED Configuring IGS_TOP...COMPLETED Configuring IBY_TOP...COMPLETED Configuring JTF_TOP...COMPLETED Configuring MWA_TOP...COMPLETED Configuring CN_TOP...COMPLETED Configuring CSI_TOP...COMPLETED Configuring WIP_TOP...COMPLETED Configuring CSE_TOP...COMPLETED Configuring EAM_TOP...COMPLETED Configuring FTE_TOP...COMPLETED Configuring ONT_TOP...COMPLETED Configuring AR_TOP...COMPLETED Configuring AHL_TOP...COMPLETED Configuring OZF_TOP...COMPLETED Configuring IES_TOP...COMPLETED Configuring CSD_TOP...COMPLETED Configuring IGC_TOP...COMPLETED AutoConfig completed successfully. -bash-4.1$ Verify that the contents of the tnsnames.ora file are correct. Ensure that the file s entries correctly reference the idb1-vip and idb2-vip aliases. Confirm also that FNDFS_APPLTOP_AT_eweblh references all the other APP Tiers. If an address entry is missing on an APP Tier node entry, rerun AutoConfig on that node. -bash-4.1$ cat $TNS_ADMIN/tnsnames.ora ############################################################### # # This file is automatically generated by AutoConfig. It will be read and # overwritten. If you were instructed to edit this file, or if you are not # able to use the settings created by AutoConfig, refer to Metalink Note # for assistance. # Configure the Oracle E-Business Suite DB Context Files 17

22 #$Header: NetServiceHandler.java /03/09 08:11:36 jmajumde ship $ # ############################################################### VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS1 VIS1= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS1 VIS1_FO= (DESCRIPTION_LIST= (LOAD_BALANCE=NO (FAILOVER=YES (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS1 (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS1 VIS2= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb2-vip.us.oracle.com(PORT=1521 (ADDRESS=(PROTOCOL=tcp(HOST=idb2-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS2 VIS2_FO= (DESCRIPTION_LIST= Configure the Oracle E-Business Suite DB Context Files 18

23 (LOAD_BALANCE=NO (FAILOVER=YES (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb2-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS2 (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb2-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS2 VIS_FO= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS (INSTANCE_NAME=VIS1 VIS_BALANCE= (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=YES (FAILOVER=YES (ADDRESS=(PROTOCOL=tcp(HOST=idb2-vip.us.oracle.com(PORT=1521 (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (ADDRESS=(PROTOCOL=tcp(HOST=idb2-vip.us.oracle.com(PORT=1521 (ADDRESS=(PROTOCOL=tcp(HOST=idb1-vip.us.oracle.com(PORT=1521 (CONNECT_DATA= (SERVICE_NAME=VIS FNDFS_IOPMN1-LH= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN1-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS FNDFS_IOPMN1-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN1-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS Configure the Oracle E-Business Suite DB Context Files 19

24 FNDFS_VIS_IOPMN1-LH= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN1-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS FNDFS_VIS_IOPMN1-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN1-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS FNDFS_IOPMN2-LH= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN2-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS FNDFS_IOPMN2-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN2-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS FNDFS_VIS_IOPMN2-LH= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN2-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS FNDFS_VIS_IOPMN2-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN2-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS FNDFS_EWEB-LH= (DESCRIPTION= Configure the Oracle E-Business Suite DB Context Files 20

25 (ADDRESS=(PROTOCOL=tcp(HOST=EWEB-LH.us.oracle.com(PORT=1627 (CONNECT_DATA= (SID=FNDFS FNDFS_EWEB-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=EWEB-LH.us.oracle.com(PORT=1627 (CONNECT_DATA= (SID=FNDFS FNDFS_VIS_EWEB-LH= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=EWEB-LH.us.oracle.com(PORT=1627 (CONNECT_DATA= (SID=FNDFS FNDFS_VIS_EWEB-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=EWEB-LH.us.oracle.com(PORT=1627 (CONNECT_DATA= (SID=FNDFS FNDFS_ICM1-LH= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=ICM1-LH(PORT=1627 (CONNECT_DATA= (SID=FNDFS FNDFS_ICM1-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=ICM1-LH(PORT=1627 (CONNECT_DATA= (SID=FNDFS FNDFS_VIS_ICM1-LH= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=ICM1-LH(PORT=1627 (CONNECT_DATA= (SID=FNDFS Configure the Oracle E-Business Suite DB Context Files 21

26 FNDFS_VIS_ICM1-LH.us.oracle.com= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=ICM1-LH(PORT=1627 (CONNECT_DATA= (SID=FNDFS FNDSM_IOPMN1-LH_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN1-LH(PORT=1626 (CONNECT_DATA= (SID=FNDSM FNDSM_IOPMN1-LH.us.oracle.com_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN1-LH(PORT=1626 (CONNECT_DATA= (SID=FNDSM FNDSM_IOPMN2-LH_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN2-LH(PORT=1626 (CONNECT_DATA= (SID=FNDSM FNDSM_IOPMN2-LH.us.oracle.com_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN2-LH(PORT=1626 (CONNECT_DATA= (SID=FNDSM FNDSM_EWEB-LH_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=EWEB-LH.us.oracle.com(PORT=1627 (CONNECT_DATA= (SID=FNDSM FNDSM_EWEB-LH.us.oracle.com_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=EWEB-LH.us.oracle.com(PORT=1627 (CONNECT_DATA= Configure the Oracle E-Business Suite DB Context Files 22

27 (SID=FNDSM FNDSM_ICM1-LH_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=ICM1-LH(PORT=1627 (CONNECT_DATA= (SID=FNDSM FNDSM_ICM1-LH.us.oracle.com_VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp(HOST=ICM1-LH(PORT=1627 (CONNECT_DATA= (SID=FNDSM FNDFS_APPLTOP_AT_eweblh= (DESCRIPTION= (ADDRESS_LIST= (ADDRESS=(PROTOCOL=tcp(HOST=EWEB-LH.us.oracle.com(PORT=1627 (ADDRESS=(PROTOCOL=tcp(HOST=ICM1-LH(PORT=1627 (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN1-LH(PORT=1626 (ADDRESS=(PROTOCOL=tcp(HOST=IOPMN2-LH(PORT=1626 (CONNECT_DATA= (SID=FNDFS IFILE=/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/ora/10.1.2/network/admin/VIS_eweb-lh_ifile.ora -bash-4.1$ Run AutoConfig on the Secondary Cluster Repeat the step to run AutoConfig on the secondary cluster to populate the FND_NODES table as well as update the relevant tnsnames.ora entries for the DB and APP nodes. Access Oracle Data Guard broker. su oracle Oracle Corporation SunOS December VIS.env dgmgrl sys/<password> Configure the Oracle E-Business Suite DB Context Files 23

28 DGMGRL for Solaris Version bit Production Copyright (c 2000, 2009, Oracle. All Rights reserved. Welcome to DGMGRL, type help for information. Connected. DGMGRL> Switchover the database. This step causes VISION to become the primary database. DGMGRL> switchover to 'VISION' Performing switchover NOW, please wait... New primary database "VISION" is opening... Operation requires shutdown of instance "VIS2" on database "VIS" Shutting down instance "VIS2"... ORACLE instance shut down. Operation requires startup of instance "VIS2" on database "VIS" Starting instance "VIS2"... ORA-32004: obsolete or deprecated parameter(s specified for RDBMS instance ORACLE instance started. Database mounted. Switchover succeeded, new primary is "VISION" Display database configuration. DGMGRL> show configuration Configuration - DR Orchestration for EBS Protection Mode: MaxAvailability Databases: VISION - Primary database VIS - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS DGMGRL> Configure the Oracle E-Business Suite DB Context Files 24

29 Run the adautocfg.sh script on the nodes. a. etc18cn1dbiodom1 b. etc18cn2dbiodom1 c. etc18cn1dbiodom1 d. deopmn1 (eweb-lh e. deopmn1 (iopmn1-lh f. deopmn2 (iopmn2-lh g. deopmn2 (icm1-lh Verify that the contents of the tnsnames.ora file are correct. Verify the Contents of the FND_NODES Table The table within the APPS schema should contain relevant hostnames for the primary and secondary clusters as follows: su oracle Oracle Corporation SunOS December VIS.env sqlplus /as sysdba SQL *Plus: Release Production Copyright (c 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> SQL> select node_name, server_address, virtual_ip from fnd_nodes; NODE_NAME SERVER_ADDRESS VIRTUAL_IP EWEB-LH m.m.m.68 IOPMN2-LH ETC11-CN01 idb1-vip Configure the Oracle E-Business Suite DB Context Files 25

30 ETC18CN1DBIODOM1 ICM1-LH ETC11-CN02 ETC18CN2DBIODOM1 idb1-vip idb2-vip idb2-vip AUTHENTICATION * IOPMN1-LH rows selected. SQL> 4. Configure Flashback on the Database You can configure the database with flashback retention. The flashback logs can be used to recover the failed primary database to a point in time before the switchover to the standby database occurred and convert that database to a physical standby database. By default, flashback retention is set to 1440 minutes or 1 day. You can set it to a different value depending on how much flashback data is kept in the fast recovery area. You define the flashback retention time through the db_flashback_retention_target parameter. Do the following steps: Check the status of the databases. The output shows that as a consequence of Step 2 in Section 3.4, VISION is the primary database while VIS is the standby database. oracle@etc18cn1dbiodom1:~$ srvctl status db d vision v Instance VIS1 is running on node etc18cn1dbiodom1. Instance status: Open. Instance VIS1 is running on node etc18cn2dbiodom1. Instance status: Open. oracle@etc11-cn01:~$ srvctl status db d vision v Instance VIS1 is running on node etc11-cn01. Instance status: Mounted (Closed. Instance VIS1 is running on node etc11-cn02. Instance status: Mounted (Closed. Access the primary database. root@etc18cn1dbiodom1:~# su oracle Oracle Corporation SunOS December 2015 oracle@etc18cn1dbiodom1:~$ oracle@etc18cn1dbiodom1:~$. VIS.env oracle@etc18cn1dbiodom1:~$ sqlplus /as sysdba Configure Flashback on the Database 26

31 SQL *Plus: Release Production Copyright (c 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> Configure the flashback retention as shown. SQL> alter database flashback on; SQL> select flashback_on from v$database; FLASHBACK_ON YES SQL> SQL> alter system set db_flashback_retention_target=240 scope=both; System altered. SQL> show parameters db_flashback_retention_target NAME TYPE VALUE db_flashback_retention_target integer 240 SQL> Access Oracle Data Guard broker. root@etc18cn1dbiodom1:~# su oracle Oracle Corporation SunOS December 2015 oracle@etc18cn1dbiodom1:~$ oracle@etc18cn1dbiodom1:~$. VIS.env oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password> DGMGRL for Solaris Version bit Production Configure Flashback on the Database 27

32 Copyright (c 2000, 2009, Oracle. All Rights reserved. Welcome to DGMGRL, type help for information. Connected. DGMGRL> Display standby database information. DGMGRL> show database VIS Database VIS Role: PHYSICAL STAND BY Intended State: APPLY-ON Transport Lag: 0 seconds Apply Lag: 0 seconds Real Time Query: OFF Instance(s: VIS1 (apply instance VIS2 Database Status: SUCCESS Switchover the database. This step causes VIS to become the primary database. DGMGRL> switchover to 'VIS' Performing switchover NOW, please wait... New primary database "VIS" is opening... Operation requires shutdown of instance "VIS2" on database "VISION" Shutting down instance "VIS2"... ORACLE instance shut down. Operation requires startup of instance "VIS2" on database "VISION" Starting instance "VIS2"... ORA-32004: obsolete or deprecated parameter(s specified for RDBMS instance ORACLE instance started. Database mounted. Switchover succeeded, new primary is "VIS" Configure Flashback on the Database 28

33 Display database configuration. DGMGRL> show configuration Configuration - DR Orchestration for EBS Protection Mode: MaxAvailability Databases: VIS - Primary database VISION - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS DGMGRL> Display both VIS and VISION database information. DGMGRL> show database VIS Database VIS Role: Primary Intended State: TRANSPORT-ON Instance(s: VIS1 VIS2 Database Status: SUCCESS DGMGRL> show database VISION Database VISION Role: PHYSICAL STANDBY Intended State: APPLY-ON Transport Lag: 0 seconds Apply Lag: 0 seconds Real Time Query: OFF Instance(s VIS1 (apply instance VIS2 Database Status SUCCESS Configure Flashback on the Database 29

34 Check the status of the databases. srvctl status db d vis v Instance VIS1 is running on node etc11-cn01. Instance status: Open Instance VIS2 is running on node etc11-cn02. Instance status: Open oracle@etc18cn1dbiodom1:~$ srvctl status db d vision v Instance VIS1 is running on node etc18cn1dbiodom1. Instance status: Mounted (Closed. Instance VIS2 is running on node etc18cn2dbiodom1. Instance status: Mounted (Closed. On the new primary database, repeat Steps 2 and 3 to turn on flashback retention. 5. Configure Oracle Solaris Cluster Geo Edition Oracle Solaris Cluster Geo Edition manages the disaster recovery orchestration by using a cluster partnership, protection groups, multigroups and sites. Install Geographic Edition Software Reference documentation: How to Install Geographic Edition Software Install the Oracle Solaris Cluster Geo Edition software on each Oracle Solaris Cluster node. root@eopmn1:~# pkg install ha-cluster/group-package/ha-cluster-geo-full Verify that the package installed successfully on each Oracle Solaris Cluster node. root@eopmn1:~# pkg info ha-cluster/group-package/ha-cluster-geo-full Name: ha-cluster/group-package/ha-cluster-geo-full Summary: Oracle Solaris Cluster Geographic Edition full group package Description: Oracle Solaris Cluster Geographic Edition full group package Category: Meta Packages/Group Packages State: Installed Publisher: ha-cluster Version: 4.3 (Oracle Solaris Cluster Geographic Edition Build Release: 5.11 Branch: Configure Oracle Solaris Cluster Geo Edition 30

35 Packaging Date: August 26, :33:37 PM Size: 5.53 kb FMRI: Prepare Each Zone Cluster for Partner Membership Reference documentation: How to Prepare a Zone Cluster for Partner Membership Set Up the Network Address for Each Zone Cluster Check first that the correct entries are in the /etc/hosts file before you set the address for the zone clusters. On the primary cluster, set the network address for opmn-zc and cm-zc. a. Verify zone cluster entries on the /etc/hosts file. root@eopmn1:~# grep opmn-zc /etc/hosts n.n.n.115 opmn-zc.us.oracle.com opmn-zc root@eopmn1:~# root@eopmn2:~# grep opmn-zc /etc/hosts n.n.n.115 opmn-zc.us.oracle.com opmn-zc root@eopmn2:~# root@ecm1:~# grep cm-zc /etc/hosts n.n.n.116 cm-zc.us.oracle.com cm-zc root@ecm1:~# root@ecm2:~# grep cm-zc /etc/hosts n.n.n.116 cm-zc.us.oracle.com cm-zc root@ecm2:~# b. Set the zone clusters network address. root@etc11cn01-appiodom1:~# clzc configure opmn-zc clzc:opmn-zc> add net clzc:opmn-zc:net> set address=opmn-zc clzc:opmn-zc:net> end clzc:opmn-zc> verify clzc:opmn-zc> commit Configure Oracle Solaris Cluster Geo Edition 31

36 clzc:opmn-zc> exit clzc configure cm-zc clzc:cm-zc> add net clzc:cm-zc:net> set address=cm-zc clzc:cm-zc:net> end clzc:cm-zc> verify clzc:cm-zc> commit clzc:cm-zc> exit On the secondary cluster, set the network address for dr-opmn-zc and dr-cm-zc. a. Verify zone cluster entries on the /etc/hosts file. grep dr-opmn-zc /etc/hosts m.m.m.81 dr-opmn-zc.us.oracle.com dr-opmn-zc grep dr-opmn-zc /etc/hosts m.m.m.81 dr-opmn-zc.us.oracle.com dr-opmn-zc grep dr-cm-zc /etc/hosts m.m.m.82 dr-cm-zc.us.oracle.com dr-cm-zc grep dr-cm-zc /etc/hosts m.m.m.82 dr-cm-zc.us.oracle.com dr-cm-zc b. Set the zone clusters network address. clzc configure dr-opmn-zc clzc:dr-opmn-zc> add net clzc:dr-opmn-zc:net> set address=dr-opmn-zc clzc:dr-opmn-zc:net> end clzc:dr-opmn-zc> verify clzc:dr-opmn-zc> commit clzc:dr-opmn-zc> exit clzc configure dr-cm-zc clzc:dr-cm-zc> add net clzc:dr-cm-zc:net> set address=dr-cm-zc clzc:dr-cm-zc:net> end clzc:dr-cm-zc> verify clzc:dr-cm-zc> commit Configure Oracle Solaris Cluster Geo Edition 32

37 clzc:dr-cm-zc> exit Set Up Security on the Primary Cluster In this step, you copy the security files for the common agent container to all zone cluster nodes on the primary cluster. For the opmn-zc cluster nodes, perform the following steps: a. On eopmn1, issue the commands as shown. cacaoadm status default instance is DISABLED at system startup. default instance is not running. cacaoadm stop cacaoadm create-keys --force cd /etc/cacao/instances/default tar cf /tmp/security.tar./security scp /tmp/security.tar iopmn2:/tmp Password: SECURITY.tar 100% ***************************** 984 KB 00:00 cd cacaoadm set-param network-bind-address= cacaoadm enable cacaoadm start b. On eopmn2, issue the commands as shown. cacaoadm status default instance is DISABLED at system startup. default instance is not running. cacaoadm stop cd /etc/cacao/instances/default tar xf /tmp/security.tar cd cacaoadm set-param network-bind-address= cacaoadm enable cacaoadm start Configure Oracle Solaris Cluster Geo Edition 33

38 For the cm-zc nodes, perform the following steps: a. On ecm1, issue the commands as shown. cacaoadm status default instance is DISABLED at system startup. default instance is not running. cacaoadm stop cacaoadm create-keys --force cd /etc/cacao/instances/default tar cf /tmp/security.tar./security scp /tmp/security.tar icm2:/tmp Password: SECURITY.tar 100% ***************************** 984 KB 00:00 cd cacaoadm set-param network-bind-address= cacaoadm enable cacaoadm start b. On ecm2, issue the commands as shown. cacaoadm status default instance is DISABLED at system startup. default instance is not running. cacaoadm stop cd /etc/cacao/instances/default tar xf /tmp/security.tar cd cacaoadm set-param network-bind-address= cacaoadm enable cacaoadm start Set Up Security on the Secondary Cluster In this step, you copy the security files for the common agent container to all zone cluster nodes on the secondary cluster. For the dr-opmn-zc cluster nodes, perform the following steps: a. On deopmn1, issue the commands as shown. cacaoadm status default instance is ENABLED at system startup. Smf monitoring process: Configure Oracle Solaris Cluster Geo Edition 34

39 15284 Uptime: 0 day(s, 22:56 root@deopmn1:~# cacaoadm stop root@deopmn1:~# cacaoadm create-keys --force root@deopmn1:~# cd /etc/cacao/instances/default root@deopmn1:/etc/cacao/instances/default# tar cf /tmp/security.tar./security root@deopmn1:/etc/cacao/instances/default# cd root@deopmn1:~# scp /tmp/security.tar deopmn2:/tmp Password: SECURITY.tar 100% ***************************** 985 KB 00:00 root@deopmn1:/etc/cacao/instances/default# cd root@deopmn1:~# cacaoadm set-param network-bind-address= root@deopmn1:~# cacaoadm enable root@deopmn1:~# cacaoadm start b. On deopmn2, issue the commands as shown. root@deopmn2:~# cacaoadm status default instance is ENABLED at system startup. Smf monitoring process: Uptime: 0 day(s, 22:56 root@deopmn2:~# cacaoadm stop root@deopmn2:~# cd /etc/cacao/instances/default root@deopmn2:/etc/cacao/instances/default# tar xf /tmp/security.tar root@deopmn2:/etc/cacao/instances/default# cd root@deopmn2:~# cacaoadm set-param network-bind-address= root@deopmn2:~# cacaoadm enable root@deopmn2:~# cacaoadm start For the dr-cm-zc cluster nodes, perform the following steps: a. On decm1, issue the commands as shown. root@decm1:~# cacaoadm status default instance is ENABLED at system startup. Smf monitoring process: Smf monitoring process: Uptime: 0 day(s, 22:55 root@decm1:~# cacaoadm stop root@decm1:~# cacaoadm create-keys --force Configure Oracle Solaris Cluster Geo Edition 35

40 cd /etc/cacao/instances/default tar cf /tmp/security.tar./security scp /tmp/security.tar decm2:/tmp Password: SECURITY.tar 100% ***************************** 984 KB 00:00 cd cacaoadm set-param network-bind-address= cacaoadm enable cacaoadm start b. On decm2, issue the commands as shown. cacaoadm status default instance is ENABLED at system startup. Smf monitoring process: Uptime: 0 day(s, 22:55 root@decm2:~# cacaoadm stop root@decm2:~# cd /etc/cacao/instances/default root@decm2:/etc/cacao/instances/default# tar xf /tmp/security.tar root@decm2:/etc/cacao/instances/default# cd root@decm2:~# cacaoadm set-param network-bind-address= root@decm2:~# cacaoadm enable root@decm2:~# cacaoadm start Enable Geographic Edition Software Reference documentation: How to Enable the Geographic Edition Framework This step enables the Geographic Edition infrastructure. You must enable it on all the clusters of the geographically separated clusters. Enable the Geographic Edition framework on the primary cluster. a. For the zone cluster opmn-zc, issue the commands on eopmn1 as shown. root@eopmn1:~# geoadm start... checking for management agent management agent check done starting product infrastructure... please wait... Registering resource type <SUNW.HBmonitor>...done. Registering resource type <SUNW.SCGeoInitSvc>...done. Configure Oracle Solaris Cluster Geo Edition 36

41 Resource type <SUNW.scmasa> has been registered already Registering resource type <SUNW.SCGeoZC>...done. Creating scalable resource group <geo-clusterstate>...done. Creating service tag management resource <geo-servicetag>... Service tag management resource created successfully... Creating failover resource group <geo-infrastructure>...done. Creating logical host resource <geo-clustername>... Logical host resource created successfully... Creating resource <geo-hbmonitor>...done. Creating resource <geo-failovercontrol>...done. Bringing RG <geo-clusterstate> to managed state...done. Bringing resource group <geo-infrastructure> to managed state...done. Enabling resource <geo-clustername>...done. Enabling resource <geo-hbmonitor>...done. Enabling resource <geo-failovercontrol>...done. Node eopmn1: Bringing resource group <geo-infrastructure> online...done. Oracle Solaris Cluster Geographic Edition infrastructure started successfully. geoadm show --- CLUSTER LEVEL INFORMATION --- Oracle Solaris Cluster Geographic Edition software is active on : node eopmn1, cluster opmn-zc Command ran successfully. root@eopmn1:~# root@eopmn1:~# clrg status geo-clusterstate geo-infrastructure === Cluster Resource Groups === Group Name Node Name Suspended Status geo-clusterstate eopmn1 No Online eopmn2 No Online geo-infrastructure eopmn1 No Online eopmn2 No Offline root@eopmn1:~# clrs status -g geo-clusterstate, geo-infrastructure === Cluster Resources === Resource Name Node Name State Status Message geo-zc-sysevent eopmn1 Online_not_monitored Online_not_monitored eopmn2 Online_not_monitored Online_not_monitored geo_servicetag eopmn1 Online_not_monitored Online_not_monitored eopmn2 Online_not_monitored Online_not_monitored geo-failovercontrol eopmn1 Online Online - Service is online. eopmn2 Offline Offline Configure Oracle Solaris Cluster Geo Edition 37

42 geo-hbmonitor eopmn1 Online Online - Daemon OK eopmn2 Offline Offline geo-clustername eopmn1 Online Online - LogicalHostname online. root@eopmn1:~# eopmn2 Offline Offline b. For the zone cluster cm-zc, issue the commands on ecm1 as shown. root@ecm1:~# geoadm start... checking for management agent management agent check done starting product infrastructure... please wait... Registering resource type <SUNW.HBmonitor>...done. Registering resource type <SUNW.SCGeoInitSvc>...done. Resource type <SUNW.scmasa> has been registered already Registering resource type <SUNW.SCGeoZC>...done. Creating scalable resource group <geo-clusterstate>...done. Creating service tag management resource <geo-servicetag>... Service tag management resource created successfully... Creating failover resource group <geo-infrastructure>...done. Creating logical host resource <geo-clustername>... Logical host resource created successfully... Creating resource <geo-hbmonitor>...done. Creating resource <geo-failovercontrol>...done. Bringing RG <geo-clusterstate> to managed state...done. Bringing resource group <geo-infrastructure> to managed state...done. Enabling resource <geo-clustername>...done. Enabling resource <geo-hbmonitor>...done. Enabling resource <geo-failovercontrol>...done. Node ecm1: Bringing resource group <geo-infrastructure> online...done. Oracle Solaris Cluster Geographic Edition infrastructure started successfully. Smf monitoring process: Uptime: 0 day(s, 23:11 root@ecm1:~# root@ecm1:~# geoadm show --- CLUSTER LEVEL INFORMATION --- Oracle Solaris Cluster Geographic Edition software is active on : node ecm1, cluster dr-cmzc Command ran successfully. root@ecm1:~# root@ecm1:~# clrg status geo-clusterstate geo-infrastructure Configure Oracle Solaris Cluster Geo Edition 38

43 === Cluster Resource Groups === Group Name Node Name Suspended Status geo-clusterstate ecm1 No Online ecm2 No Online geo-infrastructure ecm1 No Online ecm2 No Offline clrs status -g geo-clusterstate, geo-infrastructure === Cluster Resources === Resource Name Node Name State Status Message geo-zc-systevent ecm1 Online_not_monitored Online_not_monitored ecm2 Online_not_monitored Online_not_monitored geo-servicetag ecm1 Online_not_monitored Online_not_monitored ecm2 Online_not_monitored Online_not_monitored geo-failovercontrol ecm1 Online Online - Service is online. Ecm2 Offline Offline geo-hbmonitor ecm1 Online Online - Daemon OK ecm2 Offline Offline geo-clustername ecm1 Online Online - LogicalHostname online. Ecm2 Offline Offline root@ecm1:~# Enable the Geographic Edition framework on the secondary cluster. a. For the zone cluster dr-opmn-zc, issue the commands on deopm1 as shown. root@deopmn1:~# geoadm start... checking for management agent management agent check done starting product infrastructure... please wait... Registering resource type <SUNW.HBmonitor>...done. Registering resource type <SUNW.SCGeoInitSvc>...done. Resource type <SUNW.scmasa> has been registered already Registering resource type <SUNW.SCGeoZC>...done. Creating scalable resource group <geo-clusterstate>...done. Creating service tag management resource <geo-servicetag>... Service tag management resource created successfully... Creating failover resource group <geo-infrastructure>...done. Creating logical host resource <geo-clustername>... Logical host resource created successfully... Configure Oracle Solaris Cluster Geo Edition 39

44 Creating resource <geo-hbmonitor>...done. Creating resource <geo-failovercontrol>...done. Bringing RG <geo-clusterstate> to managed state...done. Bringing resource group <geo-infrastructure> to managed state...done. Enabling resource <geo-clustername>...done. Enabling resource <geo-hbmonitor>...done. Enabling resource <geo-failovercontrol>...done. Node deopmn1: Bringing resource group <geo-infrastructure> online...done. Oracle Solaris Cluster Geographic Edition infrastructure started successfully. geoadm show --- CLUSTER LEVEL INFORMATION --- Oracle Solaris Cluster Geographic Edition software is active on : node deopmn1, cluster dropmn-zc Command ran successfully. root@deopmn1:~# root@deopmn1:~# clrg status geo-clusterstate geo-infrastructure === Cluster Resource Groups === Group Name Node Name Suspended Status geo-clusterstate deopmn1 No Online deopmn2 No Online geo-infrastructure deopmn1 No Online deopmn2 No Offline root@deopmn1:~# clrs status -g geo-clusterstate, geo-infrastructure === Cluster Resources === Resource Name Node Name State Status Message geo-zc-sysevent deopmn1 Online_not_monitored Online_not_monitored deopmn2 Online_not_monitored Online_not_monitored geo-servicetag deopmn1 Online_not_monitored Online_not_monitored deopmn2 Online_not_monitored Online_not_monitored geo-failovercontrol deopmn1 Online Online - Service is online. deopmn2 Offline Offline geo-hbmonitor deopmn1 Online Online - Daemon OK deopmn2 Offline Offline geo-clustername deopmn1 Online Online - LogicalHostname online. deopmn2 Offline Offline Configure Oracle Solaris Cluster Geo Edition 40

45 b. For the zone cluster dr-cm-zc, issue the commands on decm1 as shown. geoadm start... checking for management agent management agent check done starting product infrastructure... please wait... Registering resource type <SUNW.HBmonitor>...done. Registering resource type <SUNW.SCGeoInitSvc>...done. Resource type <SUNW.scmasa> has been registered already Registering resource type <SUNW.SCGeoZC>...done. Creating scalable resource group <geo-clusterstate>...done. Creating service tag management resource <geo-servicetag>... Service tag management resource created successfully... Creating failover resource group <geo-infrastructure>...done. Creating logical host resource <geo-clustername>... Logical host resource created successfully... Creating resource <geo-hbmonitor>...done. Creating resource <geo-failovercontrol>...done. Bringing RG <geo-clusterstate> to managed state...done. Bringing resource group <geo-infrastructure> to managed state...done. Enabling resource <geo-clustername>...done. Enabling resource <geo-hbmonitor>...done. Enabling resource <geo-failovercontrol>...done. Node decm1: Bringing resource group <geo-infrastructure> online...done. Oracle Solaris Cluster Geographic Edition infrastructure started successfully. geoadm show --- CLUSTER LEVEL INFORMATION --- Oracle Solaris Cluster Geographic Edition software is active on : node decm1, cluster drcm-zc --- LOCAL NODE INFORMATION --- Node decm1 does not host active product module. Command ran successfully. root@decm1:~# root@decm1:~# clrg status geo-clusterstate geo-infrastructure === Cluster Resource Groups === Group Name Node Name Suspended Status geo-clusterstate decm1 No Online decm2 No Online Configure Oracle Solaris Cluster Geo Edition 41

46 geo-infrastructure decm1 No Online decm2 No Offline clrs status -g geo-clusterstate, geo-infrastructure === Cluster Resources === Resource Name Node Name State Status Message geo-zc-sysevent decm1 Online_not_monitored Online_not_monitored decm2 Online_not_monitored Online_not_monitored geo-servicetag decm1 Online_not_monitored Online_not_monitored decm2 Online_not_monitored Online_not_monitored geo-failovercontrol decm1 Online Online - Service is online. decm2 Offline Offline geo-hbmonitor decm1 Online Online - Daemon OK decm2 Offline Offline geo-clustername decm1 Online Online - LogicalHostname online. decm2 Offline Offline root@decm1:~# Configure Partnerships The procedures in this section cover the following topics: Configuration of partnership trusts Creation of partnerships Joining of partner clusters into the partnership Configure Trust Between Partner Clusters Reference documentation: How to Configure Trust Between Two Clusters Before creating a partnership between two clusters, you must first configure Geographic Edition software for secure communication between the two clusters. The configuration must be reciprocal. On the primary cluster, configure partnership trusts for opmn-zc. a. On eopmn1, create trust between opmn-zc and the remote dr-opmn-zc. root@eopmn1:~# geops add-trust -c dr-opmn-zc Configure Oracle Solaris Cluster Geo Edition 42

47 Local cluster : opmn-zc Local node : eopmn1 Cleaning up certificate files in /etc/cacao/instances/default/security/jsse on eopmn1 Retrieving certificates from dr-opmn-zc... Done New Certificate: Owner: CN=deopmn1_agent Issuer: CN=deopmn1_ca Serial number: 701b5d1c Valid from: Wed May 14 23:36:48 PDT 1969 until: Sun Oct 14 23:36:48 PDT 2035 Certificate fingerprints: MD5: 0E:D6:7A:73:47:8D:7F:7F:7A:5A:38:AA:1B:5C:2D:62 SHA1: 57:62:C6:F2:AF:84:8A:AE:B0:18:2E:AE:B5:D8:AB:31:E3:87:7C:59 SHA256: 7C:16:92:01:94:0F:80:39:D3:56:30:86:5C:56:EF:69:97:70:EA:60:85:2F:FB: 6E:6C:EF:C0:72:28:0B:27:26 Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: Criticality=false BasicConstraints:[ ] CA:false PathLen: undefined Do you trust this certificate? [y/n] y Adding certificate to truststore on eopmn1... Done Adding certificate to truststore on eopmn2... Done New Certificate: Owner: CN=deopmn1_ca Issuer: CN=deopmn1_ca Serial number: 3b Valid from: Wed May 14 23:36:47 PDT 1969 until: Sun Oct 14 23:36:47 PDT 2035 Certificate fingerprints: MD5: 8C:3C:9E:A9:4B:7E:E6:40:96:B5:E0:4F:29:64:29:0F SHA1: 8B:D1:8D:0C:01:9D:11:75:E9:65:CE:C9:BA:B1:68:55:F4:96:78:0B SHA256: ED:58:C4:42:C9:55:12:37:5C:64:C1:34:4F:D3:E9:FB:37:68:7E:6D:87:7F:C2:8F:96: Configure Oracle Solaris Cluster Geo Edition 43

48 06:FF:D3:6B:15:41:1A Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: Criticality=true BasicConstraints:[ CA:true ] PathLen:1 Do you trust this certificate? [y/n] y Adding certificate to truststore on eopmn1... Done Adding certificate to truststore on eopmn2... Done Operation completed successfully. All certificates are added to truststore on nodes of cluster opmn-zc root@eopmn1:~# b. On both eopmn1 and eopmn2, verify that the trust was created. root@eopmn1:~# geops verify-trust -c dr-opmn-zc Local cluster : opmn-zc Local node : eopmn1 Retrieving information from cluster dr-opmn-zc... Done Verifying connections to nodes of cluster dr-opmn-zc: eopmn1 -> deopmn2 / m.m.m OK eopmn1 -> deopmn1 / m.m.m OK Operation completed successfully, able to establish secure connections from eopmn1 to all nodes of cluster dr-opmn-zc root@eopmn1:~# root@eopmn2:~# geops verify-trust -c dr-opmn-zc Local cluster : opmn-zc Local node : eopmn2 Retrieving information from cluster dr-opmn-zc... Done Verifying connections to nodes of cluster dr-opmn-zc: Configure Oracle Solaris Cluster Geo Edition 44

49 eopmn2 -> deopmn2 / m.m.m OK eopmn2 -> deopmn1 / m.m.m OK Operation completed successfully, able to establish secure connections from eopmn2 to all nodes of cluster dr-opmn-zc. root@eopmn2:~# On the primary cluster, configure partnership trusts for cm-zc. a. On ecm1, create trust between cm-zc and the remote dr-cm-zc. root@ecm1:~# geops add-trust -c dr-cm-zc Local cluster : cm-zc Local node : ecm1 Cleaning up certificate files in /etc/cacao/instances/default/security/jsse on ecm1 Retrieving certificates from dr-cm-zc... Done New Certificate: Owner: CN=decm1_agent Issuer: CN=decm1_ca Serial number: 63ee75ca Valid from: Wed May 14 23:40:29 PDT 1969 until: Sun Oct 14 23:40:29 PDT 2035 Certificate fingerprints: MD5: BD:4C:9A:55:BB:68:FB:61:95:A9:3F:8E:80:6A:B9:DE SHA1: 71:62:6E:DF:E0:CA:A2:52:D8:D2:CC:63:5E:D3:2F:1B:6C:A1:BF:AB SHA256: 19:87:53:BE:E8:93:F7:2D:36:D5:C7:04:17:45:70:0F:1F:A4:32: 47:39:F6:80:1E:52:0A:95:35:B0:96:78:1E Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: Criticality=false BasicConstraints:[ ] CA:false PathLen: undefined Do you trust this certificate? [y/n] y Adding certificate to truststore on ecm1... Done Configure Oracle Solaris Cluster Geo Edition 45

50 Adding certificate to truststore on ecm2... Done New Certificate: Owner: CN=decm1_ca Issuer: CN=decm1_ca Serial number: Valid from: Wed May 14 23:40:28 PDT 1969 until: Sun Oct 14 23:40:28 PDT 2035 Certificate fingerprints: MD5: 32:8D:3B:3F:83:8E:20:B1:55:95:5F:FC:87:76:57:55 SHA1: CE:18:C4:0D:AD:39:E4:04:A0:9D:1D:7B:03:E4:2B:C4:8A:9E:9D:89 SHA256: 52:0D:B8:C4:0E:BD:3E:D5:E6:8B:49:DB:66:9D:38:22:11: 0F:35:4D:B9:8B:BA:78:B1:26:C8:4F:21:0B:AD:67 Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: Criticality=true BasicConstraints:[ CA:true ] PathLen:1 Do you trust this certificate? [y/n] y Adding certificate to truststore on ecm1... Done Adding certificate to truststore on ecm2... Done Operation completed successfully. All certificates are added root@ecm1:~# to truststore on nodes of cluster cm-zc b. On both ecm1 and ecm2, verify that the trust was created. root@ecm1:~# geops verify-trust -c dr-cm-zc Local cluster : cm-zc Local node : ecm1 Retrieving information from cluster dr-cm-zc... Done Verifying connections to nodes of cluster dr-cm-zc: ecm1 -> decm1 / m.m.m OK ecm1 -> decm2 / m.m.m OK Configure Oracle Solaris Cluster Geo Edition 46

51 Operation completed successfully, able to establish secure connections from ecm1 to all nodes of cluster dr-cm-zc geops verify-trust -c dr-cm-zc Local cluster : cm-zc Local node : ecm2 Retrieving information from cluster dr-cm-zc... Done Verifying connections to nodes of cluster dr-cm-zc: ecm2 -> decm1 / m.m.m OK ecm2 -> decm2 / m.m.m OK Operation completed successfully, able to establish secure connections from ecm2 to all nodes of cluster dr-cm-zc root@ecm2:~# On the secondary cluster, configure partnership trusts for dr-opmn-zc. a. On deopmn1, create trust between dr-opmn-zc and the remote opmn-zc. root@deopmn1:~# geops add-trust -c opmn-zc Local cluster : dr-opmn-zc Local node : deopmn1 Cleaning up certificate files in /etc/cacao/instances/default/security/jsse on deopmn1 Retrieving certificates from opmn-zc... Done New Certificate: Owner: CN=eopmn1_agent Issuer: CN=eopmn1_ca Serial number: 4855faab Valid from: Wed May 14 23:32:46 MeDT 1969 until: Sun Oct 14 22:32:46 MeST 2035 Certificate fingerprints: MD5: 90:02:22:E5:06:FB:9C:6A:6C:BA:1C:EE:1F:55:14:6A SHA1: 4A:8D:FD:7B:7D:29:54:06:70:A1:52:27:63:5A:13:C7:56:EA:AA:62 SHA256: 24:6D:88:5F:E6:AD:B5:06:00:84:96:FE:B9:C0:2E:B2:C3:B0:26:23:66: D6:7A:33:13:F5:06:0C:21:EE:FE:2C Signature algorithm name: SHA1withRSA Version: 3 Configure Oracle Solaris Cluster Geo Edition 47

52 Extensions: #1: ObjectId: Criticality=false BasicConstraints:[ CA:false ] PathLen: undefined Do you trust this certificate? [y/n] y Adding certificate to truststore on deopmn1... Done Adding certificate to truststore on deopmn2... Done New Certificate: Owner: CN=eopmn1_ca Issuer: CN=eopmn1_ca Serial number: 5adf1784 Valid from: Wed May 14 23:32:45 MeDT 1969 until: Sun Oct 14 22:32:45 MeST 2035 Certificate fingerprints: MD5: AB:9F:B8:79:A0:4B:66:48:26:64:62:43:1D:AD:00:47 SHA1: 18:5E:E8:C8:A2:FC:3C:B1:D2:AD:92:CF:1B:2F:AB:D4:12:44:04:5E SHA256: 52:E6:97:A9:22:16:98:71:43:21:10:43:97:39:45:05:05:36:77: 99:AB:0D:65:0E:C6:55:7D:19:AB:26:9A:4A Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: Criticality=true BasicConstraints:[ ] CA:true PathLen:1 Do you trust this certificate? [y/n] y Adding certificate to truststore on deopmn1... Done Adding certificate to truststore on deopmn2... Done Operation completed successfully. All certificates are added to truststore on nodes of cluster dr-opmn-zc root@deopmn1:~# Configure Oracle Solaris Cluster Geo Edition 48

53 b. On both deopmn1 and deopmn2, verify that the trust was created. geops verify-trust -c opmn-zc Local cluster : dr-opmn-zc Local node : deopmn1 Retrieving information from cluster opmn-zc... Done Verifying connections to nodes of cluster opmn-zc: deopmn1 -> eopmn1 / n.n.n OK deopmn1 -> eopmn2 / n.n.n OK Operation completed successfully, able to establish secure connections from deopmn1 to all nodes of cluster opmn-zc root@deopmn1:~# root@deopmn2:~# geops verify-trust -c opmn-zc Local cluster : dr-opmn-zc Local node : deopmn2 Retrieving information from cluster opmn-zc... Done Verifying connections to nodes of cluster opmn-zc: deopmn2 -> eopmn1 / n.n.n OK deopmn2 -> eopmn2 / n.n.n OK Operation completed successfully, able to establish secure connections from deopmn2 to all nodes of cluster opmn-zc root@deopmn2:~# On the secondary cluster, configure partnership trusts for dr-cm-zc. a. On decm1, create trust between dr-cm-zc and the remote cm-zc. root@decm1:~# geops add-trust -c cm-zc Local cluster : dr-cm-zc Local node : decm1 Cleaning up certificate files in /etc/cacao/instances/default/security /jsse on decm1 Retrieving certificates from cm-zc... Done New Certificate: Configure Oracle Solaris Cluster Geo Edition 49

54 Owner: CN=ecm1_agent Issuer: CN=ecm1_ca Serial number: 3e192ee1 Valid from: Wed May 14 23:13:50 MeDT 1969 until: Sun Oct 14 22:13:50 MeST 2035 Certificate fingerprints: MD5: AB:25:0D:95:56:6B:14:DB:49:98:D2:D0:E1:30:71:E2 SHA1: DF:44:CC:6A:C5:3C:E7:D6:3B:DC:16:BB:BE:47:1C:F5:61:79:AB:00 SHA256: B8:87:52:7F:B4:32:2E:5E:9D:35:D1:22:75:44:FC:08:35:19:05: E6:6A:D5:97:69:22:86:56:21:DA:1E:F7:A9 Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: Criticality=false BasicConstraints:[ CA:false ] PathLen: undefined Do you trust this certificate? [y/n] y Adding certificate to truststore on decm1... Done Adding certificate to truststore on decm2... Done New Certificate: Owner: CN=ecm1_ca Issuer: CN=ecm1_ca Serial number: 19e8a71e Valid from: Wed May 14 23:13:48 MeDT 1969 until: Sun Oct 14 22:13:48 MeST 2035 Certificate fingerprints: MD5: B7:31:32:F3:7B:D3:6F:73:41:46:FE:88:B1:37:69:AB SHA1: 1B:14:98:BC:C0:62:EE:21:31:2D:34:02:68:FC:A7:6F:68:D0:AC:06 SHA256: 3B:52:5E:47:10:FE:DB:E0:F1:B7:BA:42:A4:FB:AB:48:0F:43: 4A:A4:D7:E5:9C:6B:11:06:CF:0F:71:18:CE:E4 Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: Criticality=true BasicConstraints:[ CA:true ] PathLen:1 Configure Oracle Solaris Cluster Geo Edition 50

55 Do you trust this certificate? [y/n] y Adding certificate to truststore on decm1... Done Adding certificate to truststore on decm2... Done Operation completed successfully. All certificates are added to root@decm1:~# truststore on nodes of cluster dr-cm-zc b. On both decm1 and decm2, verify that the trust was created. root@decm1:~# geops verify-trust -c cm-zc Local cluster : dr-cm-zc Local node : decm1 Retrieving information from cluster cm-zc... Done Verifying connections to nodes of cluster cm-zc: decm1 -> ecm1 / n.n.n OK decm1 -> ecm2 / n.n.n OK Operation completed successfully, able to establish secure connections from decm1 to all nodes of cluster cm-zc root@decm1:~# root@decm2:~# geops verify-trust -c cm-zc Local cluster : dr-cm-zc Local node : decm2 Retrieving information from cluster cm-zc... Done Verifying connections to nodes of cluster cm-zc: decm2 -> ecm1 / n.n.n OK decm2 -> ecm2 / n.n.n OK Operation completed successfully, able to establish secure connections from decm2 to all nodes of cluster cm-zc :~# Configure Oracle Solaris Cluster Geo Edition 51

56 Create Partnerships Reference documentation: How to Create a Partnership A partnership establishes heartbeat monitoring between two clusters that are running the Geographic Edition framework. Partnered clusters exchange heartbeats to monitor each other's presence and health. A partnership can involve only two clusters, and only one partnership can be defined between the clusters. The steps that follow create partnerships between the two clusters as reflected in Figure 2: Partnership 1 involves the primary zone cluster opmn-zc and the standby zone cluster dr-opmn-zc. This partnership is created as opmn-ps. Partnership 2 involves the primary zone cluster cm-zc and the standby zone cluster dr-cm-zc. This partnership is created as cm-ps. The procedure includes verification that the partnerships are reflected in both the primary and secondary clusters. On the primary cluster, configure the partnerships. a. On eopmn1, create a partnership between zone clusters opmn-zc and dr-opmn-zc. root@eopmn1:~# geops create -c dr-opmn-zc opmn-ps Partnership between local cluster "opmn-zc" and remote cluster "dr-opmn-zc" successfully created. root@eopmn1:~# geops status Partnership "opmn-ps" Partner clusters Synchronization ICRM Connection : dr-opmn-zc Heartbeat "hb_opmn-zc~dr-opmn-zc" monitoring "dr-opmn-zc" Plug-in "ping_plugin" : Inactive Plug-in "tcp_udp_plugin" root@eopmn1:~# b. On eopmn2, verify the created partnership. root@eopmn2:~# geops status Partnership "opmn-ps" Partner clusters Synchronization ICRM Connection : dr-opmn-zc Heartbeat "hb_opmn-zc~dr-opmn-zc" monitoring "dr-opmn-zc" Configure Oracle Solaris Cluster Geo Edition 52

57 Plug-in "ping_plugin" Plug-in "tcp_udp_plugin" : Inactive root@eopmn2:~# c. On ecm1, create a partnership between zone clusters cm-zc and dr-cm-zc. root@ecm1:~# geops create -c dr-cm-zc cm-ps Partnership between local cluster "cm-zc" and remote cluster "dr-cm-zc" successfully created. root@ecm1:~# geops status Partnership "cm-ps" Partner clusters Synchronization ICRM Connection : dr-cm-zc Heartbeat "hb_cm-zc~dr-cm-zc" monitoring "dr-cm-zc" Plug-in "ping_plugin" : Inactive Plug-in "tcp_udp_plugin" root@ecm1:~# d. On ecm2, verify the created partnership. root@ecm2:~# geops status Partnership "cm-ps" Partner clusters Synchronization ICRM Connection : dr-cm-zc Heartbeat "hb_cm-zc~dr-cm-zc" monitoring "dr-cm-zc" Plug-in "ping_plugin" : Inactive Plug-in "tcp_udp_plugin" root@ecm2:~# On the secondary cluster, verify the partnerships. a. On deopmn1 and deopmn2, verify the partnership for the zone cluster dr-opmn-zc. root@deopmn1:~# geops status Configure Oracle Solaris Cluster Geo Edition 53

58 Partnership "opmn-ps" Partner clusters Synchronization ICRM Connection : opmn-zc Heartbeat "hb_dr-opmn-zc~opmn-zc" monitoring "opmn-zc" Plug-in "ping_plugin" : Inactive Plug-in "tcp_udp_plugin" root@deopmn1:~# root@deopmn2:~# geops status Partnership "opmn-ps" Partner clusters Synchronization ICRM Connection : opmn-zc Heartbeat "hb_dr-opmn-zc~opmn-zc" monitoring "opmn-zc" Plug-in "ping_plugin" : Inactive Plug-in "tcp_udp_plugin" root@deopmn2:~# b. On decm1 and decm2, verify the partnership for dr-cm-zc. root@decm1:~# geops status Partnership "cm-ps" Partner clusters Synchronization ICRM Connection : cm-zc Heartbeat "hb_dr-cm-zc~cm-zc" monitoring "cm-zc" Plug-in "ping_plugin" : Inactive Plug-in "tcp_udp_plugin" root@decm1:~# root@decm2:~# geops status Partnership "cm-ps" Partner clusters Synchronization ICRM Connection : cm-zc Configure Oracle Solaris Cluster Geo Edition 54

59 Heartbeat "hb_dr-cm-zc~cm-zc" monitoring "cm-zc" Plug-in "ping_plugin" : Inactive Plug-in "tcp_udp_plugin" root@decm2:~# Configure the Secondary Cluster to Join the Partnership Reference documentation: How to Join a Partnership In this step, you complete the partnership configuration by defining the secondary cluster to become the second member of the partnership. You perform this step on the secondary cluster. On deopmn1, configure the local cluster to join the cluster opmn-zc in the partnership opmn-ps. root@deopmn1:~# geops join-partnership opmn-zc opmn-ps Local cluster "dr-opmn-zc" is now partner of cluster "opmn-zc". root@deopmn1:~# Verify the partnership s status. root@deopmn1:~# geops show opmn-ps Partnership: opmn-ps Description : Local heartbeat Remote heartbeat Notification_ActionCmd : Notification_ Addrs : root@deopmn1:~# : hb_dr-opmn-zc~opmn-zc : hb_opmn-zc~dr-opmn-zc On decm1, configure the local cluster to join the cluster cm-zc in the partnership cm-ps. root@decm1:~# geops join-partnership cm-zc cm-ps Local cluster "dr-cm-zc" is now partner of cluster "cm-zc". root@decm1:~# Verify the partnership s status. root@decm1:~# geops show cm-ps Configure Oracle Solaris Cluster Geo Edition 55

60 Partnership: cm-ps Description : root@decm1:~# Local heartbeat : hb_dr-cm-zc~cm-zc Remote heartbeat : hb_cm-zc~dr-cm-zc Notification_ActionCmd : Notification_ Addrs : Configure Protection Groups Protection groups enable a set of clusters to tolerate and recover from disaster by managing the resource groups for services. Protection groups can exist only in a partnership. Thus, you must first create partnerships before you can define protection groups. In a protection group for a specific partnership, one partner cluster becomes the primary cluster while the other partner becomes the secondary cluster. A protection group contains application resource groups and properties for managing data replication for those application resource groups. The configuration for a protection group is identical on partner clusters. Therefore, partner clusters must have the application resource groups of the protection group defined in their configuration. The Geographic Edition framework propagates protection group configurations between partners Create the First Protection Group (opmn-ebs-pg Reference documentation: Creating a Protection Group That Does Not Require Data Replication Based on Figure 2, Protection Group 1 manages Web Entry Point (WEB and Oracle Process Manager and Server (OPMN1 & OPMN2 services. Thus, the group is called opmn-ebs-pg. This group is set up for Partnership 1 (opmn-ps. Ensure that the required resource groups are online. root@eopmn1:~# clrg online -em opmn1-lh-rg opmn2-lh-rg root@eopmn1:~# clrg online -em web-rg opmn1-rg opmn2-rg root@eopmn1:~# clrg status === Cluster Resource Groups === Group Name Node Name Suspended Status web-rg eopmn1 No Online eopmn2 No Offline Configure Oracle Solaris Cluster Geo Edition 56

61 opmn1-lh-rg eopmn1 No Online eopmn2 No Offline opmn2-lh-rg eopmn2 No Online eopmn1 No Offline opmn1-rg eopmn1 No Online opmn2-rg eopmn2 No Online scalmnt-rg eopmn1 No Online eopmn2 No Online geo-clusterstate eopmn1 No Online eopmn2 No Online geo-infrastructure eopmn1 No Online eopmn2 No Offline On the primary cluster, perform the following steps: a. Create the group. geopg list *** No protection group configured *** geopg create -s opmn-ps -o primary opmn-ebs-pg Protection group "opmn-ebs-pg" successfully created. geopg validate opmn-ebs-pg Protection group "opmn-ebs-pg" successfully validated. b. Configure the group. clrg set -p Auto_start_on_new_cluster=false web-rg opmn1-rg opmn2-rg geopg set-prop -p External_Dependency_Allowed=true opmn-ebs-pg Protection group "opmn-ebs-pg" successfully modified. geopg status opmn-ebs-pg Configure Oracle Solaris Cluster Geo Edition 57

62 Protection group "opmn-ebs-pg" Partnership : opmn-ps Synchronization Cluster opmn-zc Role Activation state Configuration Data replication Resource groups : Primary : Activated : None : None Cluster dr-opmn-zc Role Activation state Configuration Data replication Resource groups : Secondary : Activated : None : None c. Add resource groups to the protection group. root@eopmn1:~# geopg add-resource-group web-rg,opmn1-rg,opmn2-rg opmn-ebs-pg Following resource groups successfully added: "web-rg,opmn1-rg,opmn2-rg". root@eopmn1:~# geopg validate opmn-ebs-pg Protection group "opmn-ebs-pg" successfully validated. root@eopmn1:~# root@eopmn1:~# geopg show opmn-ebs-pg Protection Group: opmn-ebs-pg Partnership name : opmn-ps Local Role Deployed on clusters Data Replication type Configuration status Synchronization status Creation signature Last update : Primary : opmn-zc, dr-opmn-zc : NONE : opmn-zc Oct 16, :42:31 PM PDT : Oct 16, :54:19 PM MeST Local properties : None Global properties : Configure Oracle Solaris Cluster Geo Edition 58

63 Description : Timeout : 3600 seconds RoleChange_ActionCmd : External_Dependency_Allowed : true RoleChange_ActionArgs : Protected resource groups: opmn2-rg web-rg opmn1-rg *** No device groups defined in protection group "opmn-ebs-pg" *** On the secondary cluster, create the same protection group by retrieving the configuration from the partner cluster. root@deopmn1:~# geopg get -s opmn-ps opmn-ebs-pg Protection group "opmn-ebs-pg" successfully created. root@deopmn1:~# geopg validate opmn-ebs-pg Protection group "opmn-ebs-pg" successfully validated. On the primary cluster, start the protection group. root@eopmn1:~# geopg start -e global opmn-ebs-pg Processing operation... The timeout period for this operation on each cluster is 3600 seconds... Protection group "opmn-ebs-pg" successfully started. root@eopmn1:~# root@eopmn1:~# geopg validate opmn-ebs-pg Protection group "opmn-ebs-pg" successfully validated. root@eopmn1:~# root@eopmn1:~# geopg status opmn-ebs-pg Protection group "opmn-ebs-pg" Partnership : opmn-ps Synchronization Configure Oracle Solaris Cluster Geo Edition 59

64 Cluster opmn-zc Role Activation state Configuration Data replication Resource groups : Primary : Activated : None Cluster dr-opmn-zc Role Activation state Configuration Data replication Resource groups : Secondary : Activated : None Create the Second Protection Group (cm-ebs-pg Reference documentation: How to Create and Configure an Oracle ZFS Storage Appliance Protection Group Based on Figure 2, Protection Group 2 manages Concurrent Manager (CM1 services as well as providing data replication for the Oracle ZFS Storage Appliance. Thus, the group is called cm-ebs-pg. This group is set up for Partnership 2 (cm-ps. You create this protection group by creating and configuring the Oracle ZFS Storage Appliance replication. On the primary cluster, perform the following steps: a. Create a resource of SUNW.ScalMountPoint for the Concurrent Manager Logs. root@ecm1:~# grep cmlogs /etc/vfstab etc11-storib:/export/ebslogs/ebsfs_cmlogs - /ebsfs_cmlogs nfs - no rw,bg,hard,nointr,nosuid,rsize=131072,wsize=131072,proto=tcp,vers=3 root@ecm2:~# grep cmlogs /etc/vfstab etc11-storib:/export/ebslogs/ebsfs_cmlogs - /ebsfs_cmlogs nfs - no rw,bg,hard,nointr,nosuid,rsize=131072,wsize=131072,proto=tcp,vers=3 root@ecm1:~# mount /ebsfs_cmlogs root@ecm1:~# df -h /ebsfs_cmlogs Filesystem Size Used Available Capacity Mounted on etc11-storib:/export/ebslogs/ebsfs_cmlogs 13T 31K 13T 1% /ebsfs_cmlogs Configure Oracle Solaris Cluster Geo Edition 60

65 mount /ebsfs_cmlogs df -h /ebsfs_cmlogs Filesystem Size Used Available Capacity Mounted on etc11-storib:/export/ebslogs/ebsfs_cmlogs 13T 31K 13T 1% /ebsfs_cmlogs clrg create -S scalmnt2-rg clrs create -d -g scalmnt2-rg -t ScalMountPoint -x MountPointDir=/ebsfs_cmlogs -x FileSystemType=nas -x TargetFileSystem=etc11- storib:/export/ebslogs/ebsfs_cmlogs cm-logs mkdir -p /var/geo/config/ echo "ebs-apps-cmlogs any ecm2,ecm1" > /var/geo/config/geozfssaconfig grep vnic1 /etc/hosts n.n.n.90 etc11-vnic1.us.oracle.com etc11-vnic1 m.m.m.35 etc18-vnic1.us.oracle.com etc18-vnic1 mkdir -p /var/tmp/geo cp /opt/orclscgrepzfssa/etc/zfssa_geo_config /var/tmp/geo/ grep -v ^# /var/tmp/geo/zfssa_geo_config grep -v ^$ PS=cm-ps PG=cm-ebs-pg REPCOMP=ebs-apps-cmlogs REPRS=zfssa-rep-rs REPRG=zfssa-rep-rg DESC="Geo ZFSSA" APPRG=scalmnt2-rg,cm-rg CONFIGFILE=/var/geo/config/geozfssaconfig CLUSTER_DGS= clrg offline cm-rg clrg offline scalmnt2-rg /opt/orclscgrepzfssa/util/zfssa_geo_register -f /var/tmp/geo/zfssa_geo_config sourcing /var/tmp/geo/zfssa_geo_config Creating protection group "cm-ebs-pg" This operation might take a while... Protection group "cm-ebs-pg" successfully created. Configure Oracle Solaris Cluster Geo Edition 61

66 Adding the replication component "ebs-apps-cmlogs" to the protection group "cm-ebs-pg" Password for property local_service_password : Password for property remote_service_password : Script-based plug-in configuration "ebs-apps-cmlogs" successfully added to the protection group "cm-ebs-pg". Run "geopg get --partnership cm-ps cm-ebs-pg" on the partner cluster if this has not been done. root@ecm1:~# b. Add resource groups to the protection group. root@ecm1:~# geopg add-resource-group cm-rg cm-ebs-pg Following resource groups successfully added: "cm-rg". root@ecm1:~# root@ecm1:~# geopg show cm-ebs-pg Protection Group: cm-ebs-pg Partnership name : cm-ps Local Role Deployed on clusters Data Replication type Configuration status Synchronization status Creation signature Last update : Primary : cm-zc, dr-cm-zc : SBP : cm-zc Oct 16, :14:17 AM PDT : Oct 16, :31:10 AM PDT Local properties : None Global properties : create_config_script : /opt/orclscgrepzfssa/bin/zfssa_geo_control Timeout Description add_app_rg_args : remove_app_rg_args : RoleChange_ActionCmd : : 3600 seconds : Geo ZFSSA remove_config_script : /opt/orclscgrepzfssa/bin/zfssa_geo_control RoleChange_ActionArgs : switchover_script : /opt/orclscgrepzfssa/bin/zfssa_geo_control Configure Oracle Solaris Cluster Geo Edition 62

67 takeover_script : /opt/orclscgrepzfssa/bin/zfssa_geo_control start_replication_script : /opt/orclscgrepzfssa/bin/zfssa_geo_control remove_app_rg_script : /bin/true stop_replication_script : /opt/orclscgrepzfssa/bin/zfssa_geo_control add_app_rg_script : /opt/orclscgrepzfssa/bin/zfssa_geo_control External_Dependency_Allowed : true configuration_file : /var/geo/config/geozfssaconfig Protected resource groups: cm-rg scalmnt2-rg SBP Script-based plug-in configurations: ebs-apps-cmlogs ebs-apps-cmlogs_create_config_args : -O ebs-apps-cmlogs -r zfssa-reprs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1 ebs-apps-cmlogs_start_replication_args : -O ebs-apps-cmlogs -r zfssarep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18- vnic1 ebs-apps-cmlogs_switchover_args : -O ebs-apps-cmlogs -r zfssa-rep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1 ebs-apps-cmlogs_local_service_password : ******** ebs-apps-cmlogs_stop_replication_args : -O ebs-apps-cmlogs -r zfssarep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18- vnic1 ebs-apps-cmlogs_remote_service_password : ******** ebs-apps-cmlogs_remove_config_args : -O ebs-apps-cmlogs -r zfssa-reprs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1 ebs-apps-cmlogs_takeover_args : -O ebs-apps-cmlogs -r zfssa-rep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1 root@ecm1:~# root@ecm1:~# geopg validate cm-ebs-pg Protection group "cm-ebs-pg" successfully validated. On the secondary cluster, create the same protection group by retrieving the configuration from the partner cluster. root@decm1:~# geopg get -s cm-ps cm-ebs-pg Protection group "cm-ebs-pg" successfully created. Configure Oracle Solaris Cluster Geo Edition 63

68 geopg validate cm-ebs-pg Protection group "cm-ebs-pg" successfully validated. On the primary cluster, start the protection group. geopg start -e global cm-ebs-pg Processing operation... The timeout period for this operation on each cluster is 3600 seconds... Protection group "cm-ebs-pg" successfully started. geopg validate cm-ebs-pg Protection group "cm-ebs-pg" successfully validated. geopg status cm-ebs-pg Protection group "cm-ebs-pg" Partnership : cm-ps Synchronization Cluster cm-zc Role Activation state Configuration Data replication Resource groups : Primary : Activated Cluster dr-cm-zc Role Activation state Configuration Data replication Resource groups : Secondary : Activated Create the Third Protection Group (cm-odg-pg Reference documentation: How to Create and Configure an Oracle Data Guard Protection Group Configure Oracle Solaris Cluster Geo Edition 64

69 Based on Figure 2, Protection Group 3 provides data replication for Oracle Data Guard. Thus, the group is called cm-odg-pg. This group is also set up for Partnership 2 (cm-ps. On the primary cluster, perform the following steps: a. Create the group. root@ecm1:~# geopg create -s cm-ps -o primary -d odg cm-odg-pg Protection group "cm-odg-pg" successfully created. root@ecm1:~# geopg validate cm-odg-pg Protection group "cm-odg-pg" successfully validated. root@ecm1:~# b. Configure the protection group. This step creates the following for Oracle Solaris Cluster: EBS-oracle-external-proxy-shadow-rg resource group EBS-oracle-external-proxy-shadow-rs resource root@ecm1:~# cat /var/cluster/geo/odg/ebs_config ORACLE_HOME=/u01/app/oracle/product/ /dbhome_1 root@ecm1:~# root@ecm1:~# geopg add-replication-component \ -p local_database_name=vis \ -p local_db_service_name=vis_dgmgrl \ -p remote_database_name=vision \ -p remote_db_service_name=vision_dgmgrl \ -p standby_type=physical \ -p local_oracle_svr_rg_name=oep-rg \ -p remote_oracle_svr_rg_name=oep-rg \ -p sysdba_username=sys \ -p sysdba_password= \ -p replication_mode=maxavailability EBS cm-odg-pg Password for property sysdba_password : Oracle Data Guard configuration "EBS" successfully added to the protection group "cm-odg-pg". c. Add resource groups to the protection group. root@ecm1:# geopg add-resource-group EBS-oracle-external-proxy-shadow-rg cm-odg-pg Following resource groups successfully added: "EBS-oracle-external-proxy-shadow-rg". root@ecm1:# root@ecm1:~# geopg validate cm-odg-pg Protection group "cm-odg-pg" successfully validated. Configure Oracle Solaris Cluster Geo Edition 65

70 geopg show cm-odg-pg Protection Group: cm-odg-pg Partnership name : cm-ps Local Role Deployed on clusters Data Replication type Configuration status Synchronization status Creation signature Last update : Primary : cm-zc, dr-cm-zc : ODG : cm-zc Oct 16, :31:09 AM PDT : Oct 20, :42:00 AM MeST Local properties : None Global properties : Description : Timeout : 3600 seconds RoleChange_ActionCmd : External_Dependency_Allowed : true RoleChange_ActionArgs : Protected resource groups: EBS-oracle-external-proxy-shadow-rg ODG Oracle Data Guard configurations: EBS EBS_remote_db_service_name : VIS_DGMGRL EBS_remote_rac_proxy_svr_rg_name : oep-rg EBS_replication_mode : maxavailability EBS_sysdba_username : sys EBS_local_db_service_name : VISION_DGMGRL EBS_local_rac_proxy_svr_rg_name : oep-rg EBS_sysdba_password : ******** EBS_local_database_name EBS_standby_type : VISION : physical EBS_remote_database_name : VIS Configure Oracle Solaris Cluster Geo Edition 66

71 On the secondary cluster, create the same protection group by retrieving the configuration from the partner cluster. geopg get -s cm-ps cm-odg-pg Protection group "cm-odg-pg" successfully created. geopg validate cm-odg-pg Protection group "cm-odg-pg" successfully validated. On the primary cluster, start the protection group. geopg start -e global cm-odg-pg Processing operation... The timeout period for this operation on each cluster is 3600 seconds... Protection group "cm-odg-pg" successfully started. geopg validate cm-odg-pg Protection group "cm-odg-pg" successfully validated. geopg status cm-odg-pg Protection group "cm-odg-pg" Partnership : cm-ps Synchronization Cluster cm-zc Role Activation state Configuration Data replication Resource groups : Primary : Activated Cluster dr-cm-zc Role Activation state Configuration Data replication Resource groups : Secondary : Activated Display the Oracle Solaris Cluster Listings After you have completed the configuration of Oracle Solaris Cluster Geo Edition, display the listings. Configure Oracle Solaris Cluster Geo Edition 67

72 Display the listings on the primary cluster. a. Check the status of the opmn-zc zone cluster. clrg status === Cluster Resource Groups === Group Name Node Name Suspended Status opmn1-lh-rg eopmn1 No Online eopmn2 No Offline opmn2-lh-rg eopmn2 No Online eopmn1 No Offline scalmnt-rg eopmn1 No Online eopmn2 No Online geo-clusterstate eopmn1 No Online eopmn2 No Online geo-infrastructure eopmn1 No Online eopmn2 No Offline web-rg eopmn1 No Online eopmn2 No Offline opmn1-rg eopmn1 No Online opmn2-rg eopmn2 No Online clrs status === Cluster Resources === Resource Name Node State Status Message ebs-web-lsr eopmn1 Online Online - Service is online. eopmn2 Offline Offline ebs-web eopmn1 Online Online - Service is online. eopmn2 Offline Offline ebs-web-lh eopmn1 Online Online - LogicalHostname online. eopmn2 Offline Offline - LogicalHostname offline. ebs-opmn1-lh eopmn1 Online Online - LogicalHostname online. eopmn2 Offline Offline - LogicalHostname offline. ebs-opmn2-lh eopmn2 Online Online - LogicalHostname online. eopmn1 Offline Offline ebs-opmn1 eopmn1 Online Online Configure Oracle Solaris Cluster Geo Edition 68

73 ebs-opmn1-lsr eopmn1 Online Online - Service is online. ebs-opmn2 eopmn2 Online Online ebs-opmn2-lsr eopmn2 Online Online - Service is online. ebs-storage eopmn1 Online Online eopmn2 Online Online geo-zc-sysevent eopmn1 Online_not_monitored Online_not_monitored eopmn2 Online_not_monitored Online_not_monitored geo-servicetag eopmn1 Online_not_monitored Online_not_monitored eopmn2 Online_not_monitored Online_not_monitored geo-failovercontrol eopmn1 Online Online - Service is online. eopmn2 Offline Offline geo-hbmonitor eopmn1 Online Online - Daemon OK eopmn2 Offline Offline geo-clustername eopmn1 Online Online - LogicalHostname online. eopmn2 Offline Offline b. Check the status of the cm-zc zone cluster. root@ecm1:~# clrg status === Cluster Resource Groups === Group Name Node Name Suspended Status oep-rg ecm1 No Online ecm2 No Online scalmnt-rg ecm1 No Online ecm2 No Online cm-rg ecm1 No Online ecm2 No Offline scalmnt2-rg ecm1 No Online ecm2 No Online geo-clusterstate ecm1 No Online ecm2 No Online geo-infrastructure ecm1 No Online ecm2 No Offline Configure Oracle Solaris Cluster Geo Edition 69

74 cm-odg-pg-odg-rep-rg ecm No Online ecm2 No Offline EBS-oracle-external-proxy-shadow-rg ecm1no Online ecm2 No Online zfssa-rep-rg ecm1 No Online ecm2 No Offline clrs status === Cluster Resources === Resource Name Node Name State Status Message oep-rs ecm Online Online - Service VIS is UP [Instance is OPEN] OPEN] ecm2 Online Online - Service VIS is UP [Instance is ebs-storage ecm1 Online Online ecm2 Online Online ebs-cm ecm1 Online Online - Service is online. ecm2 Offline Offline ebs-cm-lsr ecm1 Online Online - Service is online. ecm2 Offline Offline ebs-cm-lh ecm1 Online Online - LogicalHostname online. ecm2 Offline Offline - LogicalHostname offline. cm-logs ecm1 Online Online ecm2 Online Online geo-zc-sysevent ecm1 Online_not_monitored Online_not_monitored ecm2 Online_not_monitored Online_not_monitored geo-servicetag ecm1 Online_not_monitored Online_not_monitored ecm2 Online_not_monitored Online_not_monitored geo-failovercontrol ecm1 Online Online - Service is online. ecm2 Offline Offline geo-hbmonitor ecm1 Online Online - Daemon OK ecm2 Offline Offline geo-clustername ecm1 Online Online - LogicalHostname online. ecm2 Offline Offline Configure Oracle Solaris Cluster Geo Edition 70

75 EBS-odg-rep-rs ecm1 Online Online - Data protection mode is MaxAvailability ecm2 Offline Offline EBS-oracle-external-proxy-shadow-rs ecm1 Online Online - Service VIS is UP [Instance is OPEN] (Data Guard Primary ecm2 Online Online - Service VIS is UP [Instance is OPEN] (Data Guard Primary zfssa-rep-rs ecm1 Online Online - Sending update ecm2 Offline Offline Display the listings on the secondary cluster. a. Check the status of the dr-opmn-zc zone cluster. root@deopmn1:~# clrg status === Cluster Resource Groups === Group Name Node Name Suspended Status web-rg deopmn1 No Unmanaged deopmn2 No Unmanaged opmn1-lh-rg deopmn1 No Online deopmn2 No Offline opmn2-lh-rg deopmn2 No Online deopmn1 No Offline scalmnt-rg deopmn1 No Online deopmn2 No Online opmn1-rg deopmn1 No Unmanaged opmn2-rg deopmn2 No Unmanaged geo-clusterstate deopmn1 No Online deopmn2 No Online geo-infrastructure deopmn1 No Online deopmn2 No Offline root@deopmn1:~# root@deopmn1:~# clrs status === Cluster Resources === Resource Name Node Name State Status Message Configure Oracle Solaris Cluster Geo Edition 71

76 ebs-web deopmn1 Offline Offline deopmn2 Offline Offline ebs-web-lsr deopmn1 Offline Offline deopmn2 Offline Offline ebs-web-lh deopmn1 Offline Offline deopmn2 Offline Offline ebs-opmn1-lh deopmn1 Online Online - LogicalHostname online. deopmn2 Offline Offline ebs-opmn2-lh deopmn2 Online Online - LogicalHostname online. deopmn1 Offline Offline ebs-storage deopmn1 Offline Online deopmn2 Offline Online ebs-opmn1 deopmn1 Offline Offline ebs-opmn1-lsr deopmn1 Offline Offline ebs-opmn2 deopmn2 Offline Offline ebs-opmn2-lsr deopmn2 Offline Offline geo-zc-sysevent deopmn1 Online_not_monitored Online_not_monitored deopmn2 Online_not_monitored Online_not_monitored geo-servicetag deopmn1 Online_not_monitored Online_not_monitored deopmn2 Online_not_monitored Online_not_monitored geo-failovercontrol deopmn1 Online Online - Service is online. deopmn2 Offline Offline geo-hbmonitor deopmn1 Online Online - Daemon OK deopmn2 Offline Offline geo-clustername deopmn1 Online Online - LogicalHostname online. deopmn2 Offline Offline b. Check the status of the dr-cm-zc zone cluster. root@decm1:~# clrg status === Cluster Resource Groups === Group Name Node Name Suspended Status Configure Oracle Solaris Cluster Geo Edition 72

77 scalmnt-rg decm1 No Online decm2 No Online oep-rg decm1 No Online decm2 No Online cm-rg decm1 No Unmanaged decm2 No Unmanaged geo-clusterstate decm1 No Online decm2 No Online geo-infrastructure decm1 No Online decm2 No Offline cm-odg-pg-odg-rep-rg decm1 No Online decm2 No Offline EBS-oracle-external-proxy-shadow-rg decm1 No Unmanaged decm2 No Unmanaged zfssa-rep-rg decm1 No Online decm2 No Offline scalmnt2-rg decm1 No Unmanaged decm2 No Unmanaged clrs status === Cluster Resources === Resource Name Node Name State Status Message ebs-storage decm1 Online Online decm2 Online Online oep-rs decm1 Online Online - Service VIS is UP [Instance is MOUNTED] decm2 Online Online - Service VIS is UP [Instance is MOUNTED] geo-zc-sysevent decm1 Online_not_monitored decm2 Online_not_monitored Online_not_monitored Online_not_monitored geo-servicetag decm1 Online_not_monitored Configure Oracle Solaris Cluster Geo Edition 73

78 decm2 Online_not_monitored Online_not_monitored Online_not_monitored geo-failovercontrol decm1 Online Online - Service is online. decm2 Offline Offline geo-hbmonitor decm1 Online Online - Daemon OK decm2 Offline Offline geo-clustername decm1 Online Online - LogicalHostname online. decm2 Offline Offline EBS-odg-rep-rs decm1 Online Online - Data protection mode is MaxAvailability decm2 Offline Offline zfssa-rep-rs decm1 Online Online - Receiving update decm2 Offline Offline ebs-cm decm1 Offline Offline decm2 Offline Offline ebs-cm-lsr decm1 Offline Offline decm2 Offline Offline ebs-cm-lh decm1 Offline Offline - LogicalHostname offline. decm2 Offline Offline - LogicalHostname offline. EBS-oracle-external-proxy-shadowrs decm1 Offline Offline decm2 Offline Offline cm-logs decm1 Offline Offline decm2 Offline Offline Configure Sites and Multigroups A multigroup is a set of protection groups that is preconfigured for group migration between predetermined sets of clusters called sites. A switchover or takeover of all protection groups in a multigroup is performed in a single operation. Configure Oracle Solaris Cluster Geo Edition 74

79 A protection group in a multigroup can be configured in a dependency chain, which defines which protection groups have a dependency on another protection group. Likewise, a multigroup can contain multiple dependency chains. The dependency chain configuration determines the sequence protection groups must be taken offline and brought online. This sequence is based on the groups' roles in the dependency chain. For example, an application that depends on a database needs to be stopped first before the database is shut down. In contrast, during restart operations after a switchover, the database takes precedence over the application. Based on the Figure 1 and Figure 2, the two sites to be created are santaclara and hillsboro. Reference documentation: How to Create a Site How to Create a Multigroup Create and Configure the santaclara Site Perform these steps on the primary cluster. Create the site. On this site, the zone cluster opmn-zc is the controller while cm-zc is a member. root@eopmn1:~# geosite create -m cm-zc santaclara Processing operation...might take several minutes to complete. Site "santaclara" successfully created. root@eopmn1:~# root@ecm1:~# geosite join opmn-zc santaclara Processing operation...might take several minutes to complete. Local cluster "cm-zc" successfully joined site "santaclara" on cluster "opmnzc". root@ecm1:~# Create the multigroup at the site. root@eopmn1:~# geomg create -s santaclara ebs-mg Processing operation...might take several minutes to complete. Multigroup "ebs-mg" successfully created. root@eopmn1:~# Add the protection groups to the multigroup. This syntax means that later on, during multigroup switchover or takeover, opmn-ebs-pg and cm-ebs-pg can execute in parallel, but after the execution of cm-odg-pg. This arrangement makes sure that the database Configure Oracle Solaris Cluster Geo Edition 75

80 services are recovered before the application services. geomg add-protection-group "(opmn-zc:opmn-ebs-pg,cm-zc:cm-ebspg/cm-zc:cm-odg-pg" ebs-mg The following protection group list was successfully added to multigroup "ebsmg": "(opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg/cm-zc:cm-odg-pg". Validate the multigroup. geomg validate ebs-mg Multigroup "ebs-mg" successfully validated. Display the multigroup s status. root@eopmn1:~# geomg status ebs-mg Multigroup: ebs-mg Description : Site name : santaclara Creation signature Last update Configuration status : opmn-zc Oct 16, :14:52 PM PDT : Oct 16, :15:50 PM PDT Synchronization status with : cm-zc Timeout : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg/cm- Protection group list zc:cm-odg-pg : seconds Status of protection group "opmn-zc:opmn-ebs-pg" : Global status Role Activation state : Primary : Activated Status of protection group "cm-zc:cm-ebs-pg" : Global status Role : Primary Activation state : Activated Status of protection group "cm-zc:cm-odg-pg" : Global status Role : Primary Configure Oracle Solaris Cluster Geo Edition 76

81 Activation state : Activated root@eopmn1:~# Create and Configure the hillsboro Site Perform these steps on the secondary cluster. Create the site. On this site, the zone cluster dr-opm-zc is the controller while the dr-cm-zc is a member. root@deopmn1:~# geosite create -m dr-cm-zc hillsboro Processing operation...might take several minutes to complete. Site "hillsboro" successfully created. root@deopmn1:~# root@decm1:~# geosite join dr-opmn-zc hillsboro Processing operation...might take several minutes to complete. Local cluster "dr-cm-zc" successfully joined site "hillsboro" on cluster "dropmn-zc". root@decm1:~# Create the multigroup at the site. root@deopmn1:~# geomg create -s hillsboro ebs-mg Processing operation...might take several minutes to complete. Multigroup "ebs-mg" successfully created. root@deopmn1:~# Add the protection groups to the multigroup. root@deopmn1:~# geomg add-protection-group "(dr-opmn-zc:opmn-ebs-pg,dr-cmzc:cm-ebs-pg/dr-cm-zc:cm-odg-pg" ebs-mg The following protection group list was successfully added to multigroup "ebs-mg": "(dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-pg/dr-cm-zc:cm-odg-pg". root@deopmn1:~# Configure Oracle Solaris Cluster Geo Edition 77

82 Validate the multigroup. geomg validate ebs-mg Multigroup "ebs-mg" successfully validated. Display the multigroup s status. root@deopmn1:~# geomg status ebs-mg Multigroup: ebs-mg Description : Site name : hillsboro Creation signature : dr-opmn-zc Oct 16, :19:33 PM MeST Last update : Oct 16, :29:16 PM MeST Configuration status Synchronization status with : dr-cm-zc Timeout : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs- Protection group list pg/dr-cm-zc:cm-odg-pg : seconds Status of protection group "dr-cm-zc:cm-odg-pg" : Global status Role : Secondary Activation state : Activated Status of protection group "dr-cm-zc:cm-ebs-pg" : Global status Role Activation state : Secondary : Activated Status of protection group "dr-opmn-zc:opmn-ebs-pg" : Global status Role : Secondary Activation state : Activated root@deopmn1:~# 6. Create and Configure a DNS Server This section provides steps in preparation for a switchover and takeover from the primary cluster to the secondary Create and Configure a DNS Server 78

83 cluster. Note: This section is a simplified illustration of how you can remap network host names by using a dedicated DNS server that can be remotely updated. However, you must choose a method appropriate to the network administration of your own data center. Before a switchover and takeover, the client address eweb-lh needs to be changed from the IP address that is used by the primary cluster to the IP address that is used by the secondary cluster. A script is provided that modifies the DNS server so that the client address eweb-lh refers to the correct address. Whenever a role change occurs, modifying the DNS server is required to ensure that the primary cluster always refers to the correct IP address for eweb-lh. For example, if the primary cluster is running at the Santa Clara site, the DNS server resolves the client address eweb-lh to the IP address n.n.n.27. If the primary cluster is running at the Hillsboro site, the DNS server resolves eweb-lh to the IP address m.m.m.68. In the procedures that follow, the examples use ebsdns as the name of the DNS server. Configure the DNS Service Install the DNS package. root@ebsdns:~# pkg install service/network/dns/bind Creating Plan (Solver setup: Creating Plan (Evaluating mediators: \ Verify the installed packages. root@ebsdns:~# pkg list grep -i bind network/dns/bind service/network/dns/bind i i-- Create the /etc/named.conf file. root@ebsdns:~# cat /etc/named.conf options { directory "/var/named"; dump-file "/var/named/dump/named_dump.db"; statistics-file "/var/named/named.stats"; }; zone "." in { }; type hint ; file "named.root" ; zone "us.oracle.com" in { Create and Configure a DNS Server 79

84 }; type master ; file "domain.us.oracle.com"; zone "in-addr.arpa" in { type master ; }; file "rdomain.us.oracle.com"; zone " in-addr.arpa" in { }; type master ; file "loopback"; logging { category "default" { "debug"; }; category "general" { "debug"; }; category "database" { "debug"; }; category "security" { "debug"; }; category "config" { "debug"; }; category "resolver" { "debug"; }; category "xfer-in" { "debug"; }; category "xfer-out" { "debug"; }; category "notify" { "debug"; }; category "client" { "debug"; }; category "unmatched" { "debug"; }; category "network" { "debug"; }; category "update" { "debug"; }; category "queries" { "debug"; }; category "dispatch" { "debug"; }; category "dnssec" { "debug"; }; category "lame-servers" { "debug"; }; channel "debug" { file "/tmp/nameddbg" versions 2 size 50m; print-time yes; print-category yes; }; }; Create a forward lookup file for the DNS server. In this file, the client address eweb-lh resolves to n.n.n.27. root@ebsdns:~# cat /var/named/domain.us.oracle.com ; Forward map for us.oracle.com $TTL 3h Create and Configure a DNS Server 80

85 @ in soa ebsdns.us.oracle.com. root.ebsdns.us.oracle.com. ( in ns ebsdns.oracle.com. in mx 10 mail.us.oracle.com. ebsdns in A n.n.n.20 localhost in A mail in CNAME ebsdns www in CNAME ebsdns ftp in CNAME ebsdns eweb-lh in A n.n.n.27 root@ebsdns:~# Create a reverse lookup file for the DNS server. root@ebsdns:~# cat /var/named/loopback ; Reverse map for in-addr.arpa. $TTL in soa ebsdns.oracle.com. root.ebsdns.oracle.com. ( in ns ebsdns.oracle.com. 1 in ptr localhost.oracle.com. root@ebsdns:~# Enable the DNS server. root@ebsdns:~# svcadm enable dns/server root@ebsdns:~# svcs dns/server STATE STIME FMRI online 1:09:58 svc:/network/dns/server:default root@ebsdns:~# Create and Configure a DNS Server 81

86 Configure the DNS Update Scripts Perform this configuration on each node in the zone clusters opmn-zc and dr-opmn-zc Create the dns_update.sh Script On the primary cluster, the script remotely runs the dns_change_to_primaryip.sh script in the DNS server. On the secondary cluster, the same script runs the dns_change_to_drip.sh script. Passwordless SSH has already been configured for nodes of the opmn-zc and dr-opmn-zc clusters to access the DNS server. On the primary cluster, create the scripts with execute permissions on the eopmn1 and eopmn2 nodes. cat /var/geo/scripts/dns_update.sh #!/bin/ksh ssh ebsdns /var/geo/scripts/dns_change_to_primaryip.sh cat /var/geo/scripts/dns_update.sh #!/bin/ksh ssh ebsdns /var/geo/scripts/dns_change_to_primaryip.sh On the secondary cluster, create the scripts with execute permissions on the deopmn1 and deopmn2 nodes. cat /var/geo/scripts/dns_update.sh #!/bin/ksh ssh ebsdns /var/geo/scripts/dns_change_to_drip.sh cat /var/geo/scripts/dns_update.sh #!/bin/ksh ssh ebsdns /var/geo/scripts/dns_change_to_drip.sh On the DNS server ebsdns, create two scripts with execute permissions as indicated. a. Create the dns_change_to_drip.sh script. cat /var/geo/scripts/dns_change_to_drip.sh #!/bin/ksh # # This script sets eweb-lh hostname to m.m.m.68 # Create and Configure a DNS Server 82

87 DB_hosts=/var/named/domain.us.oracle.com DB_ip2host=/var/named/rdomain.us.oracle.com /usr/bin/ed ${DB_ip2host} << EOF /^27.n.n.n/s/27.n.n.n/68.m.m.m/ w q EOF /usr/bin/ed ${DB_hosts} << EOF /^eweb-lh/s/n.n.n.27/m.m.m.68/ w q EOF svcadm restart dns/server root@ebsdns:~# b. Create the dns_change_to_primaryip.sh script. root@ebsdns:~# cat /var/geo/scripts/dns_change_to_primaryip.sh #!/bin/ksh # # This script sets eweb-lh hostname to n.n.n.27 # DB_hosts=/var/named/domain.us.oracle.com DB_ip2host=/var/named/rdomain.us.oracle.com /usr/bin/ed ${DB_ip2host} << EOF /^68.m.m.m/s/68.m.m.m/27.n.n.n/ w q EOF /usr/bin/ed ${DB_hosts} << EOF /^eweb-lh/s/m.m.m.68/n.n.n.27/ w q EOF svcadm restart dns/server root@ebsdns:~# Create and Configure a DNS Server 83

88 Verify the DNS Update Scripts on the DNS Server Run these commands. cd /var/geo/scripts ls -l total 4 -rwxr-xr-x 1 root root 497 Oct dns_change_to_drip.sh -rwxr-xr-x 1 root root 496 Oct dns_change_to_primaryip.sh root@ebsdns:/var/geo/scripts# root@ebsdns:/var/geo/scripts#./dns_change_to_drip.sh root@ebsdns:/var/geo/scripts# root@ebsdns:/var/geo/scripts# nslookup eweb-lh Server: Address: n.n.n.20 n.n.n.20#53 Name: eweb-lh.us.oracle.com Address: m.m.m.68 root@ebsdns:/var/geo/scripts#./dns_change_to_primaryip.sh root@ebsdns:/var/geo/scripts# root@ebsdns:/var/geo/scripts# nslookup eweb-lh Server: Address: n.n.n.20 n.n.n.20#53 Name: eweb-lh.us.oracle.com Address: n.n.n.27 root@ebsdns:/var/geo/scripts# Configure the DNS Client In the two zone clusters opmn-zc and dr-opmn-zc, their respective nodes have a constant mapping of eweb-lh to the IP address in their /etc/hosts file. This mapping ensures that the eweb-lh within the cluster resolves to the correct IP address for that cluster. Create and Configure a DNS Server 84

89 On each zone cluster node, add the DNS server to /etc/hosts. The nodes are eopmn1, eopmn2, deopmn1, and deopmn2. grep ebsdns /etc/hosts n.n.n.20 ebsdns grep ebsdns /etc/hosts n.n.n.20 ebsdns grep ebsdns /etc/hosts n.n.n.20 ebsdns grep ebsdns /etc/hosts n.n.n.20 ebsdns On each DNS client machine, configure the name server. Note: In your own company configuration, multiple name servers might exist for redundancy. root# svccfg -s network/dns/client setprop config/nameserver = net_address: n.n.n.20 root# svccfg -s network/dns/client setprop config/domain = astring: us.oracle.com root# svccfg -s network/dns/client setprop config/search = astring: us.oracle.com root# svccfg -s name-service/switch setprop config/ipnodes = astring: '"files dns"' root# svccfg -s name-service/switch setprop config/host = astring: '"files dns"' root# svccfg -s network/dns/client listprop config config application config/value_authorization astring solaris.smf.value.name-service.dns.client config/nameserver net_address n.n.n.20 config/domain astring us.oracle.com config/search astring us.oracle.com root# Enable the DNS client on each client machine. root# svcadm enable dns/client root# svcs dns/client STATE STIME FMRI online Nov_08 svc:/network/dns/client:default root# Create and Configure a DNS Server 85

90 On each DNS client machine, verify a name server lookup for eweb-lh. root# nslookup eweb-lh Server: n.n.n.20 Address: n.n.n.20#53 Name: eweb-lh.us.oracle.com Address: n.n.n.27 root# nslookup n.n.n.27 Server: n.n.n.20 Address: n.n.n.20#53 27.n.n.n.in-addr.arpa name = eweb-lh.us.oracle.com. root# Configure the Protection Group Role Change Script Add the dns_update.sh script to the RoleChange_ActionCmd property for the opmn-ebs-pg protection group. The system sets this property on the local cluster, and then propagates the value to the partner cluster. root@eopmn1:~# geopg set-prop -p RoleChange_ActionCmd=/var/geo/scripts/dns_update.sh opmn-ebs-pg Protection group "opmn-ebs-pg" successfully modified. root@eopmn1:~# 7. Load Testing with Oracle Application Testing Suite (OATS This section describes a test performed using the disaster recovery configuration implemented in this documentation. In this testing scenario, 600 virtual users perform online transactions using Oracle E-Business Suite The test and subsequent graphs show the time disruption users experience during an outage and through the disaster recovery implementation, the recovery speed for all 600 virtual users to resume normal operations. There is a separate SQL script to submit batch jobs to the concurrent managers. The Test Configuration The test configuration is shown in the following figure: Load Testing with Oracle Application Testing Suite (OATS 86

91 Figure 3 - Primary and Standby Sites Configuration with Oracle Application Testing Suites The test configuration consists of a Windows server that is using the Oracle Application Testing Suite (OATS as a load generator. In its DNS configuration, the Windows server is set up to use the DNS Server ebsdns. In a disaster recovery operation, the Oracle Solaris Cluster Geo Edition software modifies the eweb-lh entry on the DNS Server ebsdns. After that modification, the Oracle E-Business Suite virtual users are directed to the correct SuperCluster where Oracle E-Business Suite services are running. Load Testing with Oracle Application Testing Suite (OATS 87

92 OLTP Load Generator: the OATS Machine Figure 4 shows that OATS is configured with 4 OATS scripts. Each script simulates 150 users across 4 different types of transactions for a total of 600 virtual users. Figure 4 - Load Testing Configuration The following list shows the parameters of the scripts: Create_Query_Cash_Expenses Create_Query_Credit_Expenses TC_Entry View_Payslip 150 Virtual Users and OLT Server 150 Virtual Users and OLT Server 150 Virtual Users and OLT Server 150 Virtual Users and OLT Server Load Testing with Oracle Application Testing Suite (OATS 88

93 OATS Testing Information The test was run for approximately 15 minutes to measure the performance of Oracle E-Business Suite in handling the workload of sustained usage by 600 virtual users. Figure 5 shows the efficiency with which Oracle E-Business Suite performed in supporting the transactions of 600 virtual users before and after disaster recovery. Figure 5- OATS Test Performance Graph Load Testing with Oracle Application Testing Suite (OATS 89

94 Concurrent Manager Reports In the test, an additional task was added to run concurrent to the constant OLTP transactions of the 600. The task simulated a DBA user logged to the SuperCluster DB node and repeatedly running a script that submits an Orderto-Cash job. Figure 6 - Concurrent Manager Reports Figure 6 shows the Oracle E-Business Suite System Administration screen for completed jobs. The screen displays the report name, status and links to the report details and output. The Output column links to access the output files. In the test, part of the recovery verification was actual accessing of the output links. Overall, the entire test demonstrates the continuous replication feature of the ZFS SA for the Concurrent Manager output files based on the configuration prescribed in the architecture. 8. Switchover ebs-mg Multigroup to the Secondary Cluster This section describes procedures for performing a switchover of the ebs-mg multigroup to the standby. The section first describes the steps for the switchover. Then timing information regarding the switchover operation is provided. Switchover ebs-mg Multigroup to the Secondary Cluster 90

95 Switchover Operation Check the status first before performing the switchover. On the primary cluster, perform the following steps: a. Check the status of the ebs-mg multigroup. The command output shows that the santaclara site is activated and has the primary role for the three protection groups. Consequently, the hillsboro site has the secondary role for the protection groups. geomg status ebs-mg Multigroup: ebs-mg Description : Site name : santaclara Creation signature Last update Configuration status : opmn-zc Oct 16, :14:52 PM PDT : Oct 16, :15:50 PM PDT Synchronization status with : cm-zc : seconds : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg/cm- Timeout Protection group list zc:cm-odg-pg Status of protection group "opmn-zc:opmn-ebs-pg" : Global status Role : Primary Activation state : Activated Status of protection group "cm-zc:cm-ebs-pg" : Global status Role : Primary Activation state : Activated Status of protection group "cm-zc:cm-odg-pg" : Global status Role : Primary Activation state : Activated root@eopmn1:~# Switchover ebs-mg Multigroup to the Secondary Cluster 91

96 b. Check the status of the standby site. geosite show -v hillsboro Site: hillsboro Description : Creation signature : dr-opmn-zc Oct 16, :18:02 PM MeST Last update Timeout Query_interval : Oct 16, :19:33 PM MeST : 30 seconds : 120 seconds Controller "dr-opmn-zc" Configuration Member "dr-cm-zc" Configuration Local heartbeat : hb_dr-opmn-zc~hillsboro~dr-cm-zc Multigroup: ebs-mg Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-pg/dr-cmzc:cm-odg-pg The command output shows that for this site, the dr-opmn-zc is the controller. On the secondary cluster, perform the following steps: a. Check the status of the ebs-mg multigroup. This step shows that prior to the switchover, the ebs-mg multigroup on the standby site has the secondary role. root@deopmn1:~# geomg status ebs-mg Multigroup: ebs-mg Description : Site name : hillsboro Creation signature : dr-opmn-zc Oct 16, :19:33 PM MeST Last update : Oct 16, :29:16 PM MeST Configuration status Synchronization status with : dr-cm-zc Timeout : seconds Switchover ebs-mg Multigroup to the Secondary Cluster 92

97 : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs- Protection group list pg/dr-cm-zc:cm-odg-pg Status of protection group "dr-cm-zc:cm-odg-pg" : Global status Role : Secondary Activation state : Activated Status of protection group "dr-cm-zc:cm-ebs-pg" : Global status Role : Secondary Activation state : Activated Status of protection group "dr-opmn-zc:opmn-ebs-pg" : Global status Role Activation state : Secondary : Activated root@deopmn1:~# b. Switch the ebs-mg multigroup over to the standby site. root@deopmn1:~# time geomg switchover -s hillsboro ebs-mg Are you sure you want to switch over multigroup ebs-mg to site hillsboro? (yes no > yes Processing operation... The timeout period for this multigroup operation is seconds... Beginning "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" Beginning "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" Beginning "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" completed successfully "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed successfully "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed successfully "Switchover" operation succeeded for the multigroup "ebs-mg". real 8m12.937s user 0m3.078s sys 0m2.151s root@deopmn1:~# Switchover ebs-mg Multigroup to the Secondary Cluster 93

98 c. Check the status of the multigroup. This step shows that after the switchover, the protection groups in the standby site assume the primary role. geomg status ebs-mg Multigroup: ebs-mg Description : Site name : hillsboro Creation signature Last update Configuration status : dr-opmn-zc Oct 16, :19:33 PM MeST : Oct 16, :29:16 PM MeST Synchronization status with : dr-cm-zc Timeout : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs- Protection group list pg/dr-cm-zc:cm-odg-pg : seconds Status of protection group "dr-cm-zc:cm-odg-pg" : Global status Role Activation state : Primary : Activated Status of protection group "dr-cm-zc:cm-ebs-pg" : Global status Role : Primary Activation state : Activated Status of protection group "dr-opmn-zc:opmn-ebs-pg" : Global status Role : Primary Activation state : Activated root@deopmn1:~# Switchover ebs-mg Multigroup to the Secondary Cluster 94

99 Switchover Timing Information The following figure shows timing information about the switchover operation. Figure 7 - Oracle SuperCluster - E-Business Suite Disaster Recovery Orchestration demo Services Availability Timing Figure 7 provides the following data: Site switchover time <10 minutes including validations at the start and end of the operation. Actual service interruption ~6 minutes. Switchover ebs-mg Multigroup to the Secondary Cluster 95

100 9. Takeover ebs-mg Multigroup from the Secondary Cluster This section describes procedures for performing a takeover of the ebs-mg multigroup by the standby site. The section first describes the steps for the takeover. Then timing information regarding the takeover operation is provided. Takeover Operation On the primary (santaclara site, issue a stop command at each SuperCluster compute node s service processor. The command causes an outage by powering off all the site s nodes. This step simulates an unplanned outage at the site. Stop /SYS f -script On the standby site, issue the command to execute a takeover. root@deopmn1:~# time geomg takeover -f -s hillsboro ebs-mg Processing operation... The timeout period for this multigroup operation is seconds... Beginning "Takeover" operation on protection group "dr-cm-zc:cm-odg-pg" Beginning "Takeover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" Beginning "Takeover" operation on protection group "dr-cm-zc:cm-ebs-pg" "Takeover" operation on protection group "dr-cm-zc:cm-odg-pg" completed successfully "Takeover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed successfully "Takeover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed successfully "Takeover" operation succeeded for the multigroup "ebs-mg". real 10m19.878s user 0m3.759s sys 0m2.756s root@deopmn1:~# Access Oracle Data Guard broker. root@etc18cn1dbiodom1:~# su oracle Oracle Corporation SunOS December 2015 oracle@etc18cn1dbiodom1:~$ oracle@etc18cn1dbiodom1:~$. VIS.env oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password> Takeover ebs-mg Multigroup from the Secondary Cluster 96

101 DGMGRL for Solaris Version bit Production Copyright (c 2000, 2009, Oracle. All Rights reserved. Welcome to DGMGRL, type help for information. Connected. DGMGRL> Check and verify that the standby database has become primary. DGMGRL> show configuration Configuration - DR Orchestration for EBS Protection Mode: MaxAvailability Databases: VISION - Primary database Warning: ORA-16629: database reports a different protection level from the protection mode VIS - Physical standby database (disabled ORA-16661: the standby database needs to be reinstated Fast-Start Failover: DISABLED Configuration Status: WARNING DGMGRL> Check the status of the multigroup. geomg status ebs-mg Takeover ebs-mg Multigroup from the Secondary Cluster 97

102 Takeover Timing Information The following figure shows timing information about the takeover operation. Figure 8 - Oracle SuperCluster E-Business Suite Disaster Recovery Orchestration demo Services Availability Timing Figure 8 provides the following data: Site takeover time ~10 minutes including validations at the start and end of the operation. Actual service interruption ~15 minutes, includes ~5 minutes of decision making time. Takeover ebs-mg Multigroup from the Secondary Cluster 98

103 10. Restore the Primary Site This section provides instructions for the restoration of the failed primary site that has been fixed. Issue the following command. The command powers up the server, and as a consequence, also boots up all the domains. However the software detects that the site has been taken over by the standby site. Thus, application and database services will remain offline. Make sure to issue the command at each of the SuperCluster compute node s service processor. Start /SYS f -script When the primary site is ready for use again, reinstate its database as an active standby database. a. Access Oracle Data Guard broker. root@etc11-cn01:~# su oracle Oracle Corporation SunOS December 2015 oracle@etc11-cn01:~$ oracle@etc11-cn01:~$. VIS.env oracle@etc11-cn01:~$ dgmgrl sys/<password> DGMGRL for Solaris Version bit Production Copyright (c 2000, 2009, Oracle. All Rights reserved. Welcome to DGMGRL, type help for information. Connected. DGMGRL> b. Reinstate the database. DGMGRL> reinstate database 'VIS' Reinstating database "VIS", please wait... Operation requires shutdown of instance "VIS1" on database "VIS" Shutting down instance "VIS1"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "VIS1" on database "VIS" Starting instance "VIS1"... ORA-32004: obsolete or deprecated parameter(s specified for RDBMS instance ORACLE instance started. Restore the Primary Site 99

104 Database mounted. Continuing to reinstate database "VIS"... Reinstatement of database "VIS" succeeded DGMGRL> show configuration Configuration - DR Orchestration for EBS Protection Mode: MaxAvailability Databases: VISION - Primary database VIS - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS DGMGRL> Display the status of the multigroup at santaclara. In the following example, the global status shows that the ebs-mg multigroup is in an error state. This state is expected after the former primary cluster of the protection groups is up and running after a takeover by a secondary cluster. The next three steps in this procedure remedy this error state. root@eopmn1:~# geomg status ebs-mg Multigroup: ebs-mg Description : Site name : santaclara Creation signature : opmn-zc Oct 16, :14:52 PM PDT Last update : Oct 16, :15:50 PM PDT Configuration status Synchronization status with : cm-zc Timeout Protection group list : seconds : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg/cm-zc:cm-odg-pg Status of protection group "opmn-zc:opmn-ebs-pg" : Global status : Error Role : Primary Activation state : Deactivated Status of protection group "cm-zc:cm-ebs-pg" : Global status : Error Role : Primary Activation state : Deactivated Restore the Primary Site 100

105 Status of protection group "cm-zc:cm-odg-pg" : Global status : Error Role : Primary Activation state : Deactivated Update and start the protection group opmn-ebs-pg. root@eopmn1:~# geopg update opmn-ebs-pg WARNING:This operation overwrites the existing local configuration for protection group "opmn-ebs-pg" with the configuration of the remote cluster. Check the configuration for "opmn-ebs-pg" on the remote cluster before proceeding. Do you want to continue (yes no > yes Protection group "opmn-ebs-pg" successfully updated. root@eopmn1:~# root@eopmn1:~# geopg start -e global opmn-ebs-pg Processing operation... The timeout period for this operation on each cluster is 3600 seconds... Protection group "opmn-ebs-pg" successfully started. root@eopmn1:~# geopg status opmn-ebs-pg Protection group "opmn-ebs-pg" Partnership : opmn-ps Synchronization Cluster opmn-zc Role Activation state Configuration Data replication Resource groups : Secondary : Activated : None Cluster dr-opmn-zc Role Activation state Configuration Data replication Resource groups : Primary : Activated : None root@eopmn1:~# Update and start the protection group cm-odg-pg. root@ecm2:~# geopg update cm-odg-pg WARNING:This operation overwrites the existing local configuration Restore the Primary Site 101

106 for protection group "cm-odg-pg" with the configuration of the remote cluster. Check the configuration for "cm-odg-pg" on the remote cluster before proceeding. Do you want to continue (yes no > yes Protection group "cm-odg-pg" successfully updated. geopg start -e global cm-odg-pg Processing operation... The timeout period for this operation on each cluster is 3600 seconds... Protection group "cm-odg-pg" successfully started. geopg status cm-odg-pg Protection group "cm-odg-pg" Partnership : cm-ps Synchronization Cluster cm-zc Role Activation state Configuration Data replication Resource groups : Secondary : Activated Cluster dr-cm-zc Role Activation state Configuration Data replication Resource groups root@ecm2:~# : Primary : Activated Update and start the protection group cm-ebs-pg. The cm-ebs-pg group involves the ZFS SA replication. This step performs a cleanup on the ZFS SA to destroy the ebs-app-cmlogs share. a. On the standby site s ZFS SA, issue the commands as shown. e tc18sn1:shares ebs-apps-cmlogs replication> ls Actions: TARGET STATUS NEXT action-000 etc11-vnic1 idle Sync now etc18sn1:shares ebs-apps-cmlogs replication> select action-000 Restore the Primary Site 102

107 etc18sn1:shares ebs-apps-cmlogs action-000> sendupdate etc18sn1:shares ebs-apps-cmlogs action-000> set continuous=true continuous = true (uncommitted etc18sn1:shares ebs-apps-cmlogs action-000> set enabled=true enabled = true (uncommitted etc18sn1:shares ebs-apps-cmlogs action-000> commit etc18sn1:shares ebs-apps-cmlogs action-000> done etc18sn1:shares ebs-apps-cmlogs replication> ls Actions: TARGET STATUS NEXT action-000 etc11-vnic1 sending Continuous b. On the primary site s ZFS SA, issue the commands as shown. etc11-sn1:shares> ls Properties: pool = ssc Projects: QuorumProject default ebs-apps ebs-apps-cmlogs ebs-appzones ebs-backup ebs-db iodomrpool osc-backup ssc Children: replication => Manage remote replication schema => Define custom property schema etc11-sn1:shares> destroy ebs-apps-cmlogs This will destroy all data in "ebs-apps-cmlogs"! Are you sure? (Y/N Y etc11-sn1:shares> ls Properties: pool = ssc Projects: QuorumProject default ebs-apps ebs-appzones ebs-backup Restore the Primary Site 103

108 ebs-db iodomrpool osc-backup ssc Children: etc11-sn1:shares> replication => Manage remote replication schema => Define custom property schema c. Update and start the cm-ebs-pg group. geopg update cm-ebs-pg WARNING:This operation overwrites the existing local configuration for protection group "cm-ebs-pg" with the configuration of the remote cluster. Check the configuration for "cm-ebs-pg" on the remote cluster before proceeding. Do you want to continue (yes no > yes Protection group "cm-ebs-pg" successfully updated. geopg start -e global cm-ebs-pg Processing operation... The timeout period for this operation on each cluster is 3600 seconds... Protection group "cm-ebs-pg" successfully started. geopg status cm-ebs-pg Protection group "cm-ebs-pg" Partnership : cm-ps Synchronization Cluster cm-zc Role Activation state Configuration Data replication Resource groups : Secondary : Activated Cluster dr-cm-zc Role Activation state Configuration Data replication Resource groups : Primary : Activated root@ecm2:~# Restore the Primary Site 104

109 Verify the status on both clusters to ensure that normal operations have resumed. a. Check the status of the primary site. geomg status Multigroup: ebs-mg Description : Site name : santaclara Creation signature : opmn-zc Oct 16, :14:52 PM PDT Last update : Oct 16, :15:50 PM PDT Configuration status Synchronization status with : cm-zc Timeout : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg/cm- Protection group list zc:cm-odg-pg : seconds Status of protection group "opmn-zc:opmn-ebs-pg" : Global status Role Activation state : Secondary : Activated Status of protection group "cm-zc:cm-ebs-pg" : Global status Role : Secondary Activation state : Activated Status of protection group "cm-zc:cm-odg-pg" : Global status Role Activation state : Secondary : Activated root@eopmn1:~# b. Check the status of the standby site. root@deopmn1:~# geomg status Multigroup: ebs-mg Description : Site name : hillsboro Creation signature : dr-opmn-zc Oct 16, :19:33 PM MeST Restore the Primary Site 105

110 Last update Configuration status : Oct 16, :29:16 PM MeST Synchronization status with : dr-cm-zc Timeout : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs- Protection group list pg/dr-cm-zc:cm-odg-pg : seconds Status of protection group "dr-cm-zc:cm-odg-pg" : Global status Role Activation state : Primary : Activated Status of protection group "dr-cm-zc:cm-ebs-pg" : Global status Role : Primary Activation state : Activated Status of protection group "dr-opmn-zc:opmn-ebs-pg" : Global status Role : Primary Activation state : Activated root@deopmn1:~# (Optional: If the status of both sites indicates OK, switch over the ebs-mg multigroup to the original primary site. The sample output is intentionally truncated. root@deopmn1:~# time geomg switchover -s santaclara ebs-mg Are you sure you want to switch over multigroup ebs-mg to site santaclara? (yes no > yes Processing operation... The timeout period for this multigroup operation is seconds Apply SuperCluster Quarterly Maintenance Updates Periodically the SuperCluster needs to be patched with a Quarterly Maintenance Update (QMU. Every quarter, Oracle provides a SuperCluster Quarterly Full Stack Download Patches (QFSDP. QFSDP is the only mechanism that delivers the proactive patching policy for all SuperClusters. Systems should not be kept out of date for more than 6 months (2 QFSDPs. Oracle makes no guarantees that SuperCluster patches older than 6 months will be maintained. Apply SuperCluster Quarterly Maintenance Updates 106

111 The QFSDP should be applied as a whole unit in a single operation. Partial or selective patching is not recommended. Reactive patching policy does allow individual patches to be applied to individual components to address critical issues with no reasonable workaround. However, before applying patches selectively, seek advice from Oracle Support first. Refer to the README files within the QFSDP for further details. For example, for the project that is the basis of this document s implementation, the Jan 2016 QFSDP was applied to upgrade the OS release from Solaris 11.2 to Solaris 11.3 and Oracle Solaris Cluster from 4.2 to 4.3. Refer to the following MOS notes: Contents of each SuperCluster Quarterly Full Stack Download Patch (QFSDP (Doc ID SuperCluster QFSDP Installation Checklists (Doc ID The following is an extract from the QFSDP Installation checklist for T4-4 SuperCluster, January 2016 that is referenced from Doc ID The intent of this document is to provide a high-level overview of the Quarterly Full Stack Download Patch (QFSDP installation. This document does not replace the requirement to read & follow the instructions in the QFSDP README files and the QFSDP SuperCluster Patching Best Practices document (Oracle Support Document , this document is meant as a navigation help and for tracking progress during the update process. When planning a QFSDP, follow this recommended sequence. Update the standby site first. For example, in this document, the patch update is applied to the hillsboro site first. After the update of the standby site, perform a planned switchover of the multigroup to the standby site. For example, in this project, the ebs-mg multigroup is switched over to the hillsboro site as follows: root@deopmn1:~# geomg switchover -s hillsboro ebs-mg Are you sure you want to switch over multigroup ebs-mg to site hillsboro? (yes no > yes Processing operation... The timeout period for this multigroup operation is seconds... Beginning "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" Beginning "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" Beginning "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" completed successfully "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed successfully "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed successfully "Switchover" operation succeeded for the multigroup "ebs-mg". root@deopmn1:~# Apply SuperCluster Quarterly Maintenance Updates 107

112 After the standby site takes over the services, apply the same patch update to the primary site. As an option, you can perform a load test during the switchover between the sites to monitor any performance changes after the QFSDP application. The following figure shows the load testing graph when OATS was used to perform load testing after the standby site has been updated. No performance degradation observed. Figure 9 E-Business Suite Disaster Recovery Orchestration demo Services Availability Timing Apply SuperCluster Quarterly Maintenance Updates 108

113 ORACLE CORPORATION Worldwide Headquarters 500 Oracle Parkway, Redwood Shores, CA USA Worldwide Inquiries TELE ORACLE1 FAX oracle.com CONNECT WITH US Call ORACLE1 or visit oracle.com. Outside North America, find your local office at oracle.com/contact. blogs.oracle.com/oracle facebook.com/oracle twitter.com/oracle Copyright 2018, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group Oracle E-Business Suite Using Oracle Solaris Cluster Geo Edition Implementation Guide August 2018 Author: Neil Garthwaite, Gia-Khanh Nguyen