Configuring High Availability for the PMG DB

Transcription

1 This chapter describes the process of configuring high availability for the PMG DB. It provides the prerequisites and procedures required to configure and test the hot standby and cold standby for Cisco RMS. It also includes procedures required to back up and restore standby and primary configurations, troubleshoot Data Guard on PMG DB, and upgrade Cisco RMS. The Oracle Data Guard tool is used to ensure high availability and disaster recovery for PMG DB in Cisco RMS. This tool is used to configure one primary database and one or more standby databases. These databases in a Data Guard configuration are connected by Oracle Net and may be dispersed geographically. The primary and standby databases can be managed using the SQL CLIs or the Data Guard Broker interfaces, including a CLI and a GUI that is integrated in the Oracle Enterprise Manager. The Oracle Enterprise Manager GUI or the Data Guard (DGMGRL) CLI is used to enable fast-start failover to fail over automatically when the primary database becomes unavailable. When a fast-start failover is 1

2 Prerequisites enabled, the Data Guard Broker determines if a failover is necessary and initiates the failover to the specified target standby database automatically, without a need for DBA intervention. A failover occurs when the primary database is unavailable. Failover is performed only in the event of a failure of the primary database, and the failover results in a transition of a standby database to the primary role. A switchover is a role reversal between the primary database and one of its standby databases. This is typically done for planned maintenance of the primary system. During a switchover, the primary database transitions to a standby role, and the standby database transitions to the primary role. The following sections describes the prerequisites, process for configuring high availability for the PMG DB, and Cisco RMS upgrade information. Note The output of some of the commands may slightly differ based on the installation setup and time of the execution. Prerequisites, page 2 Configuration Workflow, page 3 Configuring the Standby Server Setup, page 4 Setting Up the Oracle Data Guard Broker, page 16 Enabling Flashback, page 20 Configuring Hot Standby for PMG DB, page 21 Configuring Cold Standby, page 34 Testing Hot Standby, page 35 Testing Cold Standby, page 45 Rolling Back and Cleaning Up Standby and Primary Configurations, page 55 Troubleshooting Data Guard on PMG DB, page 65 Prerequisites Primary server should have a running instance. Standby server should have a software only installation. Operator or user of this procedure should have the following experience: Knowledge of basic database/dba know-how Knowledge of basic Linux/shell commands Ability to edit files with vi or vim Ability to view files (cat, tail, more, less) 2

3 Configuration Workflow ORACLE_HOME directory path should be /u01/app/oracle/product/11.2.0/dbhome_1. If this path is different, respective file paths (for example, data files) would differ. To enable the failover feature for Cisco RMS applications connecting to PMGDB, the Cisco RMS Central node should be configured for PMG DB as per the install guide (see Cisco RAN Management System Installation Guide). Note The installation can be performed on either UCS 5108 Blade Server or UCS 240 Server. There is no specific dependency based type of hardware. Configuration Workflow The following table provides the general flow which can be followed to perform tasks for different types of redundancy setup for PMGDB using the Oracle Data Guard tool. Sl. No Task Configure the Standby Server Add Only Hot Standby to Primary Server Add Only Cold Standby to Primary Server Add Both, Hot and Cold Standby to Primary Server Test Hot Standby Setup Test Cold Standby Setup Procedure Configuring the Standby Server Setup, on page 4 Configuring the Hot Standby Server, on page 10 Configuring Primary With Only Cold Standby, on page 34 Configuring Primary With Hot and Cold Standby, on page 34 Testing Hot Standby, on page 35 Testing Site Failure, on page 46 Recovering Original Primary After Site Failure, on page Troubleshooting PMG DB Rollback and Clean-up Troubleshooting Data Guard on PMG DB, on page 65 Rolling Back and Cleaning Up Standby and Primary Configurations, on page 55 3

4 Configuring the Standby Server Setup Configuring the Standby Server Setup The following procedure describes how to configure a standby server for the primary server. If the PMG DB redundancy setup involves primary with only cold standby, this standby server can be configured as cold standby server. If the redundancy setup involves primary with only hot standby or primary with both hot and cold standby, this standby can be configured as a hot standby. Configuring the Primary Server, on page 4 Configuring the Hot Standby Server, on page 10 Configuring the Primary Server Step 1 Log in to the primary server with the username, oracle. Log in to the sql prompt. $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba Step 3 SQL*Plus: Release Production on Mon Jun 16 17:51: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> Create the pfile as a backup vsphere client. This pfile can be used later when rollback is needed. The specific path for initpmgdb_pre_dgsetup.ora file can be specified if the file is to be stored in some backup directory (for example, /backup/initpmgdb_pre_dgsetup.ora). If the path is not specified the file gets created under the default directory, that is, $ORACLE_HOME/dbs. SQL> CREATE PFILE='initPMGDB_pre_dgsetup.ora' FROM SPFILE; File created. Logging Step 1 Ensure that the primary database is in archivelog mode. SQL> SELECT log_mode FROM v$database; LOG_MODE NOARCHIVELOG 4

5 Configuring the Primary Server If it is noarchivelog mode, switch to archivelog mode: SQL> SHUTDOWN IMMEDIATE; Database closed. Database dismounted. ORACLE instance shut down. SQL> STARTUP MOUNT; ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. SQL> ALTER DATABASE ARCHIVELOG; SQL> ALTER DATABASE OPEN; Step 3 SQL> SELECT log_mode FROM v$database; LOG_MODE ARCHIVELOG Enable forced logging by using the following command. SQL> ALTER DATABASE FORCE LOGGING; Initializing Parameters Step 1 Check the setting for the DB_NAME and DB_UNIQUE_NAME parameters. In this case they are both set to "PMGDB" on the primary database. SQL> show parameter db_name NAME TYPE VALUE db_name string PMGDB SQL> show parameter db_unique_name NAME TYPE VALUE db_unique_name string PMGDB Enter the DB_NAME of the standby database. It should be the same as that of the primary, but it must have a different DB_UNIQUE_NAME value. The DB_UNIQUE_NAME values of the primary and standby database should be used in the DG_CONFIG setting of the LOG_ARCHIVE_CONFIG parameter. For this example, the standby database has the value "PMGDB_STBY". 5

6 Configuring the Primary Server Step 3 SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY)'; Set the suitable remote archive log destinations. In this case, flash recovery area for the local location is used; however, other location can be specified. Note the SERVICE and the DB_UNIQUE_NAME for the remote location reference the standby location. SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=pmgdb_stby NOAFFIRM ASYNC VALID_FOR= (ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY'; LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY)'; Step 4 SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE; Set the LOG_ARCHIVE_FORMAT and LOG_ARCHIVE_MAX_PROCESSES parameters to appropriate values and the REMOTE_LOGIN_PASSWORDFILE must be set to exclusive. SQL> ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE; SQL> ALTER SYSTEM SET LOG_ARCHIVE_MAX_PROCESSES=30; Step 5 SQL> ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE SCOPE=SPFILE; Set the following parameters (in addition to the previous setting) to ensure that the primary server is ready to switch to standby mode. (This setting is recommended.) Adjust the *_CONVERT parameters to account for filename and path differences between the servers. SQL> ALTER SYSTEM SET FAL_SERVER=PMGDB_STBY; SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='PMGDB_STBY','PMGDB' SCOPE=SPFILE; SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT='PMGDB_STBY','PMGDB' SCOPE=SPFILE; Step 6 SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO; Restart the database to implement the modifications made to the parameters. SQL> shutdown immediate Database closed. Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area bytes 6

7 Configuring the Primary Server Step 7 Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. Exit from sql prompt bytes bytes bytes bytes SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production Setting Up the Service Step 1 Provide the values for the primary and standby databases in the "$ORACLE_HOME/network/admin/tnsnames.ora" files on both the servers. The following values are required. The values of database server hostname or IP address, ports, and oracle home should be specified as per the installation setup. PMGDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(port = <port number>) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB) ) ) PMGDB_STBY = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <standby-server-host-address>)(port = <port number>)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB) ) ) Edit "$ORACLE_HOME/network/admin/listener.ora" to add the entry that the Data Guard Broker will refer to. If SID_LIST is already present, add SID_DESC to the list. SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_NAME = PMGDB_DGMGRL) (ORACLE_HOME = <oracle-home-directory-path>) (SID_NAME = PMGDB) ) ) Starting the Listener Ensure that the listener is started on the primary server. 7

8 Configuring the Primary Server Note All of the output is not displayed. $ lsnrctl status If not running: LSNRCTL for Linux: Version Production on 02-JUL :35:37 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<host address>)(port=<port number>)) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 111: Connection refused If running: LSNRCTL for Linux: Version Production on 02-JUL :43:23 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<host address>)(port=<port number>))) STATUS of the LISTENER Services Summary... The command completed successfully 1 If listener is not running (that is, TNS-12541: TNS:no listener), start it. $ lsnrctl start LSNRCTL for Linux: Version Production on 17-JUN :10:31 Copyright (c) 1991, 2009, Oracle. All rights reserved. Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait... The command completed successfully 2 If listener is running, reload it. $ lsnrctl reload LSNRCTL for Linux: Version Production on 17-JUN :12:23 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to The command completed successfully Backing Up the Primary Database Step 1 Back up the primary database, which will be restored on the standby database. To do this, open another console for the primary database server and log in as an oracle user. Set the ORACLE_SID to PMGDB and take a backup. Note The complete output is not displayed. $ export ORACLE_SID=PMGDB $ rman target=/ Recovery Manager: Release Production on Thu Jul 3 18:53: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 8

9 Configuring the Primary Server connected to target database: PMGDB (DBID= ) RMAN> BACKUP DATABASE PLUS ARCHIVELOG; Starting backup at 03-JUL-14 current log archived using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=97 device type=disk channel ORA_DISK_1: starting archived log backup set channel ORA_DISK_1: specifying archived log(s) in backup set input archived log thread=1 sequence=15 RECID=93 STAMP= input archived log thread=1 sequence=16 RECID=95 STAMP= channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00005 name=/u01/app/oracle/oradata/pmgdb/mapinfo_ts.dbf input datafile file number=00006 name=/u01/app/oracle/oradata/pmgdb/pmgdb_ts.dbf input datafile file number=00001 name=/u01/app/oracle/oradata/pmgdb/system01.dbf input datafile file number=00002 name=/u01/app/oracle/oradata/pmgdb/sysaux01.dbf input datafile file number=00003 name=/u01/app/oracle/oradata/pmgdb/undotbs01.dbf input datafile file number=00004 name=/u01/app/oracle/oradata/pmgdb/users01.dbf piece handle=/u01/app/oracle/flash_recovery_area/pmgdb/backupset/2014_07_03/o1_mf_annnn_tag t185830_9vbpmykl_.bkp tag=tag t comment=none channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01 Finished backup at 03-JUL-14 RMAN> exit Recovery Manager complete. Note To check if the backup is completed successfully, check the last statement in the output. For example, Finished backup at 03-JUL-14. Creating Standby Control File and PFILE Step 1 Login to the sql prompt. $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 17:51: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> Create a control file for the standby database using the following command on the primary database. 9

10 Configuring the Hot Standby Server Step 3 SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/pmgdb_stby.ctl'; Create a parameter file for the standby database. Step 4 SQL> CREATE PFILE='/tmp/initPMGDB_stby.ora' FROM SPFILE; File created. Exit from the sql prompt. Step 5 SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production Edit the PFILE generated, that is, /tmp/initpmgdb_stby.ora, making the entries relevant for the standby database. Because this is a replica of the original server, only the following parameters should be modified. Modify the parameter if it exists, otherwise add the parameter with the specified value. *.db_unique_name='pmgdb_stby' *.fal_server='pmgdb' *.log_archive_dest_2='service=pmgdb ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB' Configuring the Hot Standby Server Step 1 Log in to the secondary or standby server with the username oracle. Add the tnsnames entries to "$ORACLE_HOME/network/admin/tnsnames.ora". PMGDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(port = <port number>)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB) ) ) PMGDB_STBY = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <standby-server-host-address>)(port = <port number>)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB) ) ) Step 3 Edit "$ORACLE_HOME/network/admin/listener.ora" to add the entry that the Data Guard Broker will refer to. 10

11 Configuring the Hot Standby Server SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_NAME = PMGDB_STBY_DGMGRL) (ORACLE_HOME = <oracle-home-directory-path>) (SID_NAME = PMGDB) ) ) Copying Files Step 1 Create the necessary directories on the standby server. $ mkdir -p /u01/app/oracle/oradata/pmgdb The system responds with a command prompt. $ mkdir -p /u01/app/oracle/flash_recovery_area/pmgdb The system responds with a command prompt. $ mkdir -p /u01/app/oracle/admin/pmgdb/adump The system responds with a command prompt. Copy the files from the primary to the standby server. Copy the standby control file to all locations. $ scp oracle@<primary-server-host-address>:/tmp/pmgdb_stby.ctl /u01/app/oracle/oradata/pmgdb/control01.ctl <scp-output> Step 3 $ cp /u01/app/oracle/oradata/pmgdb/control01.ctl /u01/app/oracle/flash_recovery_area/pmgdb/control02.ctl The system responds with a command prompt. Copy archivelogs and backups. $ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/pmgdb/ archivelog /u01/app/oracle/flash_recovery_area/pmgdb <scp-output> Step 4 $ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/pmgdb/backupset /u01/app/oracle/flash_recovery_area/pmgdb <scp-output> Copy the parameter file. Step 5 $ scp oracle@<primary-server-host-address>: /tmp/initpmgdb_stby.ora /tmp/initpmgdb_stby.ora <scp-output> Copy the remote login password file. 11

12 Configuring the Hot Standby Server $ scp oracle@<primary-server-host-address>:$oracle_home/dbs/orapwpmgdb $ORACLE_HOME/dbs <scp-output> Note The backups are copied across to the standby server as part of the flash recovery area (FRA) copy. If backups are not held within the FRA, ensure that those are copied to the standby server and make them available from the same path as used on the primary server. Starting the Listener on Standby Server Ensure that the listener is started on the standby server. Note All of the output is not displayed. $ lsnrctl status If not running: LSNRCTL for Linux: Version Production on 02-JUL :35:37 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>)) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 111: Connection refused If running: LSNRCTL for Linux: Version Production on 02-JUL :43:23 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>))) STATUS of the LISTENER Services Summary... The command completed successfully 1 If listener is not running (that is, TNS-12541: TNS:no listener), start it. $ lsnrctl start LSNRCTL for Linux: Version Production on 17-JUN :10:31 Copyright (c) 1991, 2009, Oracle. All rights reserved. Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait... The command completed successfully 12

13 Configuring the Hot Standby Server 2 If listener is running, reload it. $ lsnrctl reload LSNRCTL for Linux: Version Production on 17-JUN :12:23 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to The command completed successfully Restoring Backup Step 1 Create the SPFILE from the modified PFILE. $ export ORACLE_SID=PMGDB The system responds with a command prompt. $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 12:24: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. SQL> CREATE SPFILE FROM PFILE='/tmp/initPMGDB_stby.ora'; File created. Exit from the sql prompt. Step 3 SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production Restore the backup files. Depending on database size, restore time will vary. Note All of the output is not displayed. $ export ORACLE_SID=PMGDB $ rman target=/ Recovery Manager: Release Production on Thu Jul 3 19:23: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. connected to target database (not started) RMAN> STARTUP MOUNT; Oracle instance started database mounted Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers bytes bytes bytes bytes bytes RMAN> RESTORE DATABASE; Starting restore at 03-JUL-14 13

14 Configuring the Hot Standby Server Starting implicit crosscheck backup at 03-JUL-14 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: piece handle=/u01/app/oracle/flash_recovery_area/pmgdb/backupset/2014_07_03/ o1_mf_nnndf_tag t185812_9vbpmf2x_.bkp tag=tag t channel ORA_DISK_1: restored backup piece 1 channel ORA_DISK_1: restore complete, elapsed time: 00:00:26 Finished restore at 03-JUL-14 RMAN> exit Recovery Manager complete. Creating Redo Logs Step 1 Exit from the sqlplus prompt if already logged in and re-login. SQL> exit Disconnected $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 12:24: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. Create online redo logs for the standby. SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL; SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/pmgdb/online_redo01.log') SIZE 50M; SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/pmgdb/online_redo02.log') SIZE 50M; SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/pmgdb/online_redo03.log') SIZE 50M; SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO; Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs should be at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs. In this case, the following standby redo logs must be created on both servers. SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo01.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo02.log') SIZE 50M; 14

15 Configuring the Hot Standby Server SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo03.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo04.log') SIZE 50M; Step 3 Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs should be at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs. In this case, the following standby redo logs must be created on both servers. SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo01.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo02.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo03.log') SIZE 50M; Step 4 SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo04.log') SIZE 50M; Repeat steps 1 to 3 to create standby redo logs on the primary server. After this is complete, the apply process can be started. Starting the Apply Process Start the apply process on the standby server. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION; Checking Status Step 1 Check status on primary database. Note The status LOG SWITCH GAP or RESOLVABLE GAP will change after the Data Guard Broker is set up. SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS PMGDB READ WRITE PMGDB PRIMARY LOG SWITCH GAP Check status on secondary database. Note The switchover status could be TO PRIMARY or SESSIONS ACTIVATE. 15

16 Setting Up the Oracle Data Guard Broker SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS PMGDB MOUNTED PMGDB_STBY PHYSICAL STANDBY TO PRIMARY Setting Up the Oracle Data Guard Broker The Oracle Data Guard Broker is used to create a broker configuration that allows the broker to manage and monitor primary and standby databases together as an integrated unit. Step 1 Log in to the primary server with the username oracle. Check the Data Guard Broker process on the primary database. SQL> sho parameter dg_broker NAME TYPE VALUE dg_broker_config_file1 string /u01/app/oracle/product/ /dbhome_1/dbs/dr1pmgdb.dat dg_broker_config_file2 string /u01/app/oracle/product/ /dbhome_1/dbs/dr2pmgdb.dat Step 3 dg_broker_start boolean FALSE Start the Data Guard Broker process on the primary database. Step 4 SQL> alter system set dg_broker_start=true scope=both; Check the DG_BROKER on the standby database and start it. SQL> sho parameter dg_broker NAME TYPE VALUE dg_broker_start boolean FALSE Step 5 SQL> alter system set dg_broker_start=true scope=both ; Verify the "$ORACLE_HOME/network/admin/listener.ora" file. If required, edit the file that includes the db_unique_name_dgmgrl.db_domain values for the GLOBAL_DBNAME in both primary and standby database. To set the value, check the db_domain value on the primary and standby database. 16

17 Setting Up the Oracle Data Guard Broker SQL> show parameter db_domain NAME TYPE VALUE db_domain string Because the value of the db_domain is null so is the value of GLOBAL_DBNAME = PMGDB_DGMGRL for primary database and for standby GLOBAL_DBNAME = PMGDB_STBY_DGMGRL. The primary "$ORACLE_HOME/network/admin/listener.ora" file is as. Step 6 Step 7 SID_LIST_LISTENER = (SID_DESC = (GLOBAL_NAME = PMGDB_DGMGRL) (ORACLE_HOME = <oracle-home-directory-path>) (SID_NAME = PMGDB) ) ) Verify the "$ORACLE_HOME/network/admin/listener.ora" file on the standby database. Configure the Data Guard configuration. Open another console on primary or standby server to open the DGMGRL CLI. Note DGMGRL CLI can be accessed from any server where Oracle Client Administrator is installed. $ dgmgrl DGMGRL for Linux: Version bit Production Copyright (c) 2000, 2009, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. DGMGRL> connect sys@pmgdb Password:. Connected. Step 8 DGMGRL> create configuration 'dgpmgdb' as primary database is 'PMGDB' connect identifier is PMGDB ; Configuration "dgpmgdb" created with primary database " PMGDB". Check the status of the configuration after the configuration is created. Step 9 DGMGRL> show configuration Configuration - dgpmgdb Protection Mode : MaxPerformance Databases : PMGDB - Primary database Fast-Start Failover : DISABLED Configuration Status : DISABLED Add the standby database to the Data Guard Broker configuration. DGMGRL> add database 'PMGDB_STBY' as connect identifier is PMGDB_STBY maintained as physical ; Database " PMGDB_STBY " added DGMGRL> show configuration Configuration - dgpmgdb Protection Mode : MaxPerformance Databases : PMGDB - Primary database : PMGDB_STBY - Physical standby database 17

18 Setting Up the Oracle Data Guard Broker Step 10 Fast-Start Failover : DISABLED Configuration Status : DISABLED Enable the configuration. DGMGRL> enable configuration Enabled. Step 11 DGMGRL> show configuration Configuration - dgpmgdb Protection Mode : MaxPerformance Databases : PMGDB - Primary database : PMGDB_STBY - Physical standby database Fast-Start Failover : DISABLED Configuration Status : SUCCESS View the primary and standby database properties. Note While copying the command from the document, the single quotes (for example, as in 'PMGDB') may not be copied correctly, the command may have to be entered at the prompt. DGMGRL> show database verbose 'PMGDB' Database - PMGDB Role: PRIMARY Intended State: TRANSPORT-ON Instance(s): PMGDB Properties: DGConnectIdentifier = 'pmgdb' ObserverConnectIdentifier = '' LogXptMode = 'ASYNC' DelayMins = '0' Binding = 'optional' MaxFailure = '0' MaxConnections = '1' ReopenSecs = '300' NetTimeout = '30' RedoCompression = 'DISABLE' LogShipping = 'ON' PreferredApplyInstance = '' ApplyInstanceTimeout = '0' ApplyParallel = 'AUTO' StandbyFileManagement = 'AUTO' ArchiveLagTarget = '0' LogArchiveMaxProcesses = '30' LogArchiveMinSucceedDest = '1' DbFileNameConvert = 'PMGDB_STBY, PMGDB' LogFileNameConvert = 'PMGDB_STBY, PMGDB' FastStartFailoverTarget = '' StatusReport = '(monitor)' InconsistentProperties = '(monitor)' InconsistentLogXptProps = '(monitor)' SendQEntries = '(monitor)' LogXptStatus = '(monitor)' RecvQEntries = '(monitor)' HostName = 'oracle-vm-primary' SidName = 'PMGDB' StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracle-vm-primary)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=PMGDB_DGMGRL)(INSTANCE_NAME=PMGDB)(SERVER=DEDICATED)))' StandbyArchiveLocation AlternateLocation LogArchiveTrace LogArchiveFormat TopWaitEvents = 'USE_DB_RECOVERY_FILE_DEST' = '' = '0' = '%t_%s_%r.arc' = '(monitor)' 18

19 Verifying Log Synchronization on Standby Server Database Status: SUCCESS DGMGRL> show database verbose 'PMGDB_STBY' Database - PMGDB_STBY Role: PHYSICAL STANDBY Intended State: APPLY-ON Transport Lag: 0 seconds Apply Lag: 0 seconds Real Time Query: OFF Instance(s): PMGDB Properties: DGConnectIdentifier = 'pmgdb_stby' ObserverConnectIdentifier = '' LogXptMode = 'ASYNC' DelayMins = '0' Binding = 'OPTIONAL' MaxFailure = '0' MaxConnections = '1' ReopenSecs = '300' NetTimeout = '30' RedoCompression = 'DISABLE' LogShipping = 'ON' PreferredApplyInstance = '' ApplyInstanceTimeout = '0' ApplyParallel = 'AUTO' StandbyFileManagement = 'AUTO' ArchiveLagTarget = '0' LogArchiveMaxProcesses = '30' LogArchiveMinSucceedDest = '1' DbFileNameConvert = 'PMGDB_STBY, PMGDB' LogFileNameConvert = 'PMGDB_STBY, PMGDB' FastStartFailoverTarget = '' StatusReport = '(monitor)' InconsistentProperties = '(monitor)' InconsistentLogXptProps = '(monitor)' SendQEntries = '(monitor)' LogXptStatus = '(monitor)' RecvQEntries = '(monitor)' HostName = 'blr-oracle2-standby' SidName = 'PMGDB' StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=blr-oracle2-standby)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=PMGDB_STBY_DGMGRL)(INSTANCE_NAME=PMGDB)(SERVER=DEDICATED)))' StandbyArchiveLocation AlternateLocation LogArchiveTrace LogArchiveFormat TopWaitEvents Database Status: SUCCESS = 'USE_DB_RECOVERY_FILE_DEST' = '' = '0' = '%t_%s_%r.arc' = '(monitor)' Verifying Log Synchronization on Standby Server Step 1 Identify the existing archived redo log files on the standby server. Note the latest sequence number. Note Sequence number and timestamp in the output will vary depending on the installation setup. 19

20 Enabling Flashback SQL> ALTER SESSION SET nls_date_format='dd-mon-yyyy HH24:MI:SS'; Session altered. SQL> SELECT sequence#, first_time, next_time, applied FROM v$archived_log ORDER BY sequence#; SEQUENCE# FIRST_TIME NEXT_TIME APPLIED JUL :06:27 10-JUL :06:53 YES JUL :06:53 10-JUL :36:09 YES JUL :36:09 10-JUL :51:38 IN-MEMORY Force a log switch to archive the current online redo log file on the primary server. Step 3 SQL> ALTER SYSTEM SWITCH LOGFILE; Verify on the standby, the new redo data that was archived on the standby database. That is, new sequence number with latest timestamp of redo apply is displayed. Note The value of the APPLIED column for the most recently received log file will either be IN-MEMORY or YES if that log file has been applied. SQL> SELECT sequence#, first_time, next_time, applied FROM v$archived_log ORDER BY sequence#; SEQUENCE# FIRST_TIME NEXT_TIME APPLIED JUL :06:27 10-JUL :06:53 YES JUL :06:53 10-JUL :36:09 YES JUL :36:09 10-JUL :51:38 YES JUL :51:38 10-JUL :03:48 IN-MEMORY Enabling Flashback Flashback database is needed to enable fast-start failover for hot standby. In case of cold standby also it is advisable to enable Flashback database because it is helpful in recovering the database. Step 1 Check if flashback is on from the primary database sql prompt. SQL> select flashback_on from v$database; FLASHBACK_ON NO Set flashback "on" if it is not already on. 20

21 Configuring Hot Standby for PMG DB Step 3 Check if the flashback is on from the standby database sql prompt. Step 4 SQL> select flashback_on from v$database; FLASHBACK_ON NO Cancel standby apply process if flashback mode has to be made on. Step 5 SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; Set flashback on if it is not already on. Step 6 SQL> alter database flashback on; Start back apply process. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION; Configuring Hot Standby for PMG DB Hot standby refers to a standby server that becomes active and acts as primary when the original primary server fails. The failover is automatic without manual intervention and with a little or no downtime. Typically hot standby servers are placed on different hosts in the same site as that of the primary server. The following procedure describes how to add a hot standby. Skip this section if there is no need to configure a hot standby. Setting Up the Hot Standby Before You Begin Ensure that you have completed the Configuring the Standby Server Setup, on page 4 to create a standby server. 21

22 Setting Up the Hot Standby Enabling Failover DETAILED STEPS Step 1 Command or Action To enable failover, return to the console where the DGMGRL tool is running and change the LogXptMode property for the PMGDB. Purpose DGMGRL> edit database "PMGDB" set property LogXptMode='SYNC'; Property "logxptmode" updated Change the LogXptMode PMGDB_STBY. Step 3 DGMGRL> edit database "PMGDB_STBY" set property LogXptMode='SYNC'; Property "logxptmode" updated Check the LogXptStatus. Step 4 DGMGRL> show database 'PMGDB' 'LogXptStatus'; LOG TRANSPORT STATUS PRIMARY_INSTANCE_NAME STANDBY_DATABASE_NAME PMGDB PMGDB_STBY Set the protection mode to MaxAvailability STATUS Step 5 DGMGRL> edit configuration set protection mode as MaxAvailability; Succeeded. Return to the DGMGRL console and enable failover. If not connected, connect as sys@pmgdb. Step 6 DGMGRL> enable fast_start failover; Enabled Open another console for the standby server and login as an oracle user. Start the Observer as a background process. The Observer monitors both primary and standby databases and detects failures, if any. $ nohup dgmgrl -silent sys/<password>@pmgdb "start observer" & $ cat nohup.out Observer started Note It is recommended running the Observer on a server host that is separate from the primary and standby servers. This host needs Oracle Client Administrator installed to run DGMGRL tool and the Observer. Otherwise, run the Observer on the standby server. In case of failure, if primary and standby roles are switched, ensure that the observer is running on the new standby server. 22

23 Setting Up the Hot Standby Command or Action Purpose Step 7 Return to the previous DGMGRL console and verify the configuration. DGMGRL> show configuration verbose Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - (*) Physical standby database Warning: ORA-16826: apply service state is inconsistent with the DelayMins property (*) Fast-Start Failover target Step 8 Fast-Start Failover: ENABLED Threshold: 30 seconds Target: PMGDB_STBY Observer: oracle-vm-primary Lag Limit: 30 seconds (not in use) Shutdown Primary: TRUE Auto-reinstate: TRUE Configuration Status: WARNING Start recovery from the current log file on the standby server if the above error (mentioned in the previous step) is observed. SQL> alter database recover managed standby database cancel; Step 9 SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION; Return to the DGMGRL console to check the configuration again. DGMGRL> show configuration verbose Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - (*) Physical standby database (*) Fast-Start Failover target Fast-Start Failover: ENABLED Threshold: 30 seconds Target: PMGDB_STBY Observer: oracle-vm-primary Lag Limit: 30 seconds (not in use) Shutdown Primary: TRUE Auto-reinstate: TRUE Configuration Status: SUCCESS 23

24 Setting Up the Hot Standby Checking Status Step 1 Check status on primary database. Note The status LOG SWITCH GAP or RESOLVABLE GAP will change after the Data Guard Broker is set up. SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS PMGDB READ WRITE PMGDB PRIMARY LOG SWITCH GAP Check status on secondary database. Note The switchover status could be TO PRIMARY or SESSIONS ACTIVATE. SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS PMGDB MOUNTED PMGDB_STBY PHYSICAL STANDBY TO PRIMARY Initializing Parameters for Standby Server Step 1 Add db_unique_name to log_archive_config SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY,PMGDB_STBY2)'; Set and enable LOG_ARCHIVE_DEST_3. Step 3 SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=PMGDB_stby2 NOAFFIRM ASYNC VALID_FOR=( ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY2'; Set the parameters to ensure that the primary is ready to switch roles to become a standby. SQL> ALTER SYSTEM SET FAL_SERVER=PMGDB_STBY,PMGDB_STBY2; SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='PMGDB','PMGDB_STBY', 'PMGDB', 'PMGDB_STBY2' SCOPE=SPFILE; 24

25 Setting Up the Hot Standby Step 4 SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT ='PMGDB','PMGDB_STBY', 'PMGDB', 'PMGDB_STBY2' SCOPE=SPFILE; Restart the database to implement the modifications made to the parameters. SQL> shutdown immediate Database closed. Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Database opened. Setting Up the Service for Standby Server Provide the values for the new standby databases in the "$ORACLE_HOME/network/admin/tnsnames.ora" files on both the servers. The following values are used during the setup. PMGDB_STBY2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <standby2-server-host-address>)(port = <port number>)) ) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB) ) Backing Up the Primary Database Step 1 Back up the primary database, which will be restored on the standby database. To do this, open another console for the primary database server and log in as an oracle user. Set the ORACLE_SID to PMGDB and take a backup. Note The complete output is not displayed. $ export ORACLE_SID=PMGDB $ rman target=/ Recovery Manager: Release Production on Thu Jul 3 18:53: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. connected to target database: PMGDB (DBID= ) 25

26 Setting Up the Hot Standby RMAN> BACKUP DATABASE PLUS ARCHIVELOG; Starting backup at 03-JUL-14 current log archived using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=97 device type=disk channel ORA_DISK_1: starting archived log backup set channel ORA_DISK_1: specifying archived log(s) in backup set input archived log thread=1 sequence=15 RECID=93 STAMP= input archived log thread=1 sequence=16 RECID=95 STAMP= channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00005 name=/u01/app/oracle/oradata/pmgdb/mapinfo_ts.dbf input datafile file number=00006 name=/u01/app/oracle/oradata/pmgdb/pmgdb_ts.dbf input datafile file number=00001 name=/u01/app/oracle/oradata/pmgdb/system01.dbf input datafile file number=00002 name=/u01/app/oracle/oradata/pmgdb/sysaux01.dbf input datafile file number=00003 name=/u01/app/oracle/oradata/pmgdb/undotbs01.dbf input datafile file number=00004 name=/u01/app/oracle/oradata/pmgdb/users01.dbf piece handle=/u01/app/oracle/flash_recovery_area/pmgdb/backupset/2014_07_03/o1_mf_annnn_tag t185830_9vbpmykl_.bkp tag=tag t comment=none channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01 Finished backup at 03-JUL-14 RMAN> exit Recovery Manager complete. Note To check if the backup is completed successfully, check the last statement in the output. For example, Finished backup at 03-JUL-14. Creating the Standby Control File and PFILE for Standby Server Step 1 Return to the sqlplus prompt of the primary database and create a control file for the standby database using the following command on the primary database. SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/pmgdb_stby2.ctl'; Create a parameter file for the standby database. Step 3 SQL> CREATE PFILE='/tmp/initPMGDB_stby2.ora' FROM SPFILE; File created. Modify the PFILE /tmp/initpmgdb_stby2.ora making the entries relevant for the standby database. Because this is a replica of the original server, only the following parameters need to be modified, added, or updated. Modify the parameter if it exists otherwise add the parameter with specified value. 26

27 Setting Up the Hot Standby *.db_unique_name='pmgdb_stby2' *.fal_server='pmgdb' *.log_archive_dest_2='service=pmgdb ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB' *.log_archive_config='dg_config=(pmgdb,pmgdb_stby2)' *.db_file_name_convert='pmgdb_stby2','pmgdb' *.log_file_name_convert='pmgdb_stby2','pmgdb' Also, remove the following parameters: *.log_archive_dest_3='enable' *.log_archive_dest_state_3='enable' Configuring the Standby Server Step 1 Log in to the standby 2 server (to be used as cold standby) with the username oracle. Add tnsnames entries to "$ORACLE_HOME/network/admin/tnsnames.ora". PMGDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(port = <port number>)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB) ) ) PMGDB_STBY2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <standby2-server-host-address>)(port = <port number>)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB) ) ) Step 3 Edit "$ORACLE_HOME/network/admin/listener.ora to add a entry that the Data Guard Broker will refer to. SID_LIST_LISTENER = (SID_DESC = (GLOBAL_NAME = PMGDB_STBY2_DGMGRL) (ORACLE_HOME = <oracle-home-directory-path>) (SID_NAME = PMGDB) ) ) 27

28 Setting Up the Hot Standby Copying Files to the Standby Server Step 1 Create the necessary directories on the standby server. $ mkdir -p /u01/app/oracle/oradata/pmgdb The system responds with a command prompt. $ mkdir -p /u01/app/oracle/flash_recovery_area/pmgdb The system responds with a command prompt. $ mkdir -p /u01/app/oracle/admin/pmgdb/adump The system responds with a command prompt. Copy the files from the primary to the standby server. Copy the standby control file to all locations. $ scp oracle@<primary-server-host-address>:/tmp/pmgdb_stby.ctl /u01/app/oracle/oradata/pmgdb/control01.ctl <scp-output> Step 3 $ cp /u01/app/oracle/oradata/pmgdb/control01.ctl /u01/app/oracle/flash_recovery_area/pmgdb/control02.ctl The system responds with a command prompt. Copy archivelogs and backups. $ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/pmgdb/ archivelog /u01/app/oracle/flash_recovery_area/pmgdb <scp-output> Step 4 $ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/pmgdb/backupset /u01/app/oracle/flash_recovery_area/pmgdb <scp-output> Copy the parameter file. Step 5 $ scp oracle@<primary-server-host-address>: /tmp/initpmgdb_stby.ora /tmp/initpmgdb_stby.ora <scp-output> Copy the remote login password file. $ scp oracle@<primary-server-host-address>:$oracle_home/dbs/orapwpmgdb $ORACLE_HOME/dbs <scp-output> Starting the Listener on Standby Server Ensure that the listener is started on the standby server. 28

29 Setting Up the Hot Standby Note All of the output is not displayed. $ lsnrctl status If not running: LSNRCTL for Linux: Version Production on 02-JUL :35:37 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>)) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 111: Connection refused If running: LSNRCTL for Linux: Version Production on 02-JUL :43:23 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>))) STATUS of the LISTENER Services Summary... The command completed successfully 1 If listener is not running (that is, TNS-12541: TNS:no listener), start it. $ lsnrctl start LSNRCTL for Linux: Version Production on 17-JUN :10:31 Copyright (c) 1991, 2009, Oracle. All rights reserved. Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait... The command completed successfully 2 If listener is running, reload it. $ lsnrctl reload LSNRCTL for Linux: Version Production on 17-JUN :12:23 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to The command completed successfully Restoring Backup on the Standby Server Step 1 Create the SPFILE from the modified PFILE. $ export ORACLE_SID=PMGDB The system responds with a command prompt. $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 12:24:

30 Setting Up the Hot Standby Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. SQL> CREATE SPFILE FROM PFILE='/tmp/initPMGDB_stby.ora'; File created. Exit from the sql prompt. Step 3 SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production Restore the backup files. Depending on database size, restore time will vary. Note All of the output is not displayed. $ export ORACLE_SID=PMGDB $ rman target=/ Recovery Manager: Release Production on Thu Jul 3 19:23: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. connected to target database (not started) RMAN> STARTUP MOUNT; Oracle instance started database mounted Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers bytes bytes bytes bytes bytes RMAN> RESTORE DATABASE; Starting restore at 03-JUL-14 Starting implicit crosscheck backup at 03-JUL-14 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: piece handle=/u01/app/oracle/flash_recovery_area/pmgdb/backupset/2014_07_03/ o1_mf_nnndf_tag t185812_9vbpmf2x_.bkp tag=tag t channel ORA_DISK_1: restored backup piece 1 channel ORA_DISK_1: restore complete, elapsed time: 00:00:26 Finished restore at 03-JUL-14 RMAN> exit Recovery Manager complete. 30

31 Setting Up the Hot Standby Creating Redo Logs for Standby Server Step 1 Exit from the sqlplus prompt if already logged in and re-login. SQL> exit Disconnected $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 12:24: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. Create online redo logs for the standby. SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL; SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/pmgdb/online_redo01.log') SIZE 50M; SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/pmgdb/online_redo02.log') SIZE 50M; SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/pmgdb/online_redo03.log') SIZE 50M; SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO; Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs should be at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs. In this case, the following standby redo logs must be created on both servers. SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo01.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo02.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo03.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo04.log') SIZE 50M; Step 3 Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs should be at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs. In this case, the following standby redo logs must be created on both servers. SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo01.log') SIZE 50M; SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo02.log') SIZE 50M; 31

32 Setting Up the Hot Standby SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo03.log') SIZE 50M; Step 4 SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo04.log') SIZE 50M; Repeat steps 1 to 3 to create standby redo logs on the primary server. After this is complete, the apply process can be started. Starting the Apply Process on Standby Server Start the apply process on the standby server. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION; Configuring the Data Guard Broker on Standby Server Step 1 Start the Data Guard Broker Process on the standby server. SQL> alter system set dg_broker_start=true scope=both; Configure the Data Guard Broker by opening another console on the primary or standby server to invoke the DGMGRL CLI. Note DGMGRL CLI can be accessed from any server where the Oracle Client Administrator is installed. $ dgmgrl DGMGRL for Linux: Version bit Production Copyright (c) 2000, 2009, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Step 3 Step 4 DGMGRL> connect sys@pmgdb Password:. Connected. Add the standby database to the Data Guard Broker configuration. DGMGRL> add database 'PMGDB_STBY2' as connect identifier is PMGDB_STBY2 maintained as physical ; Database " PMGDB_STBY2 " added Update the primary database property and enable the database. DGMGRL> edit database 'PMGDB' set property DbFileNameConvert='PMGDB,PMGDB_STBY, PMGDB,PMGDB_STBY2'; Property "dbfilenameconvert" updated 32

33 Setting Up the Hot Standby DGMGRL> edit database 'PMGDB' set property LogFileNameConvert='PMGDB,PMGDB_STBY,PMGDB, PMGDB_STBY2'; Property "logfilenameconvert" updated Step 5 DGMGRL> enable database 'PMGDB'; Enabled. Enable the standby database. Step 6 DGMGRL> enable database 'PMGDB_STBY2'; Enabled. Verify the configuration. DGMGRL> show configuration Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - (*) Physical standby database PMGDB_STBY2 - Physical standby database Fast-Start Failover: ENABLED Configuration Status: SUCCESS Verifying Log Synchronization on Standby Server Step 1 Identify the existing archived redo log files on the standby server. Note the latest sequence number. Note Sequence number and timestamp in the output will vary depending on the installation setup. SQL> ALTER SESSION SET nls_date_format='dd-mon-yyyy HH24:MI:SS'; Session altered. SQL> SELECT sequence#, first_time, next_time, applied FROM v$archived_log ORDER BY sequence#; SEQUENCE# FIRST_TIME NEXT_TIME APPLIED JUL :06:27 10-JUL :06:53 YES JUL :06:53 10-JUL :36:09 YES JUL :36:09 10-JUL :51:38 IN-MEMORY Force a log switch to archive the current online redo log file on the primary server. Step 3 SQL> ALTER SYSTEM SWITCH LOGFILE; Verify on the standby, the new redo data that was archived on the standby database. That is, new sequence number with latest timestamp of redo apply is displayed. 33

34 Configuring Cold Standby Note The value of the APPLIED column for the most recently received log file will either be IN-MEMORY or YES if that log file has been applied. SQL> SELECT sequence#, first_time, next_time, applied FROM v$archived_log ORDER BY sequence#; SEQUENCE# FIRST_TIME NEXT_TIME APPLIED JUL :06:27 10-JUL :06:53 YES JUL :06:53 10-JUL :36:09 YES JUL :36:09 10-JUL :51:38 YES JUL :51:38 10-JUL :03:48 IN-MEMORY Enabling Flashback on Standby Server Follow the steps in Enabling Flashback, on page 20 to enable flashback. Read the standby server as cold standby server. Configuring Cold Standby Cold standby refers to a standby server that is made active and switched as a primary when the original primary server or site fails due to any unforeseen event. The switchover needs manual intervention and there is some downtime involved. Typically cold standby servers are placed in a different site other than the primary server site. There are two options available to configure cold standby: Configuring Primary With Only Cold Standby, on page 34 Configuring Primary With Hot and Cold Standby, on page 34 Configuring Primary With Only Cold Standby Ensure that you complete the Configuring the Standby Server Setup, on page 4 to create a standby server. No additional steps are required to configure for primary with only cold standby setup. Configuring Primary With Hot and Cold Standby Setting up Hot Standby Complete the Configuring Hot Standby for PMG DB, on page 21 to set up the hot standby server. Setting Up Additional Standby Server as Cold Standby This will be an additional standby database added to the primary and should be used as a cold standby server for disaster recovery such as site failure recovery. Following configuration steps mentioned assume hot standby is already created. 34

35 Testing Hot Standby Testing Hot Standby Testing Failover Process You can test the following two hot standby processes for the database: Testing Failover Process, on page 35 Testing Switchover Process, on page 42 To test the following failover processes, follow these procedures. Testing Failover From Primary Database to Standby Database, on page 35 Testing Failover Revert From New Primary to Original Primary Database, on page 39 Testing Failover From Primary Database to Standby Database Step 1 Log in to the primary server with the username oracle. If you are already logged in, proceed to. Shutdown the primary database and check if standby is implemented. SQL> select db_unique_name from v$database; DB_UNIQUE_NAME PMGDB Step 3 SQL> shut abort ORACLE instance shut down. Check alert logs on the standby server. Note The Oracle base directory path may vary based on your installation. Step 4 $ cd /u01/app/oracle/diag/rdbms/pmgdb_stby/pmgdb/trace $ tail -f alert_pmgdb.log -n100 Failover succeeded. Primary database is now PMGDB_STBY. Check the log status. SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET

36 Testing Failover Process Step 5 PMGDB_STBY PRIMARY READ WRITE REINSTATE REQUIRED PMGDB Synchronize failure status by starting up and mounting the database on the original primary server. SQL> startup mount ORACLE instance started. Step 6 Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Open a new console on the original primary server, and keep tailing the log file drcpmgdb.log. Wait until the tail output stops at "DMON: >> DMON Process Shutdown <<". Step 7 $cd /u01/app/oracle/diag/rdbms/pmgdb/pmgdb/trace $tail -f drcpmgdb.log <tail output> :41: DMON: status from posting instances for Database QUIESCE = ORA :41: INSV: Received message for inter-instance publication :41: req ID , opcode CTL_QUIESCE, phase TEARDOWN, flags :41: DMON: Releasing (convert to NULL) Health Check Master lock :41: DMON: Releasing FSFP HOME lock :41: INSV: Reply received for message with :41: req ID , opcode CTL_QUIESCE, phase TEARDOWN :41: DMON: Entered rfm_release_chief_lock() for CTL_QUIESCE :41: Fore: FSFO shutting down :41: DMON: Data Guard Broker shutting down :41: DMON: Terminating RSM processes :41: RSM0: delete state object for RSM :41: DMON: RSM0 successfully terminated :41: DMON: Terminating NetSlave processes :41: DMON: Freeing all task elements :41: DMON: Terminating Instance Slave process :41: INSV: Shutting down :41: DMON: INSV successfully terminated :41: DMON: Zeroing metadata root pointer :41: DMON: Clearing Primary State :41: DMON: Freeing Broker SGA heap :41: DMON: Freeing PGA heap :41: DMON: Removing DMON's state object :41: DMON: Resetting DMON context structure :41: DMON: >> DMON Process Shutdown << Return to the sql prompt of the original primary server, exit the sql prompt, relogin to the sql prompt, and redo the startup mount. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options $ sqlplus / as sysdba SQL*Plus: Release Production on Thu Jul 3 21:57: Copyright (c) 1982, 2009, Oracle. All rights reserved. 36

37 Testing Failover Process Connected to an idle instance. SQL> startup mount ORACLE instance started. Step 8 Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Connect the DGMGRL to PMGDB_STBY, which is the primary now and reinstate the failover status. Step 9 DGMGRL> connect sys@pmgdb_stby Password: Connected View the configuration. Note that the database roles are changed, however, a warning to reinstate standby is displayed. DGMGRL> show configuration In case of primary with only hot standby: Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB_STBY - Primary database Warning: ORA-16817: unsynchronized fast-start failover configuration PMGDB - (*) Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: ENABLED Configuration Status: WARNING In case of primary with hot and cold standby: Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB_STBY - Primary database Warning: ORA-16817: unsynchronized fast-start failover configuration PMGDB - (*) Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated PMGDB_STBY2 - Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: ENABLED Step 10 Configuration Status: WARNING Reinstate the standby database. Note If Error: ORA-16653: failed to reinstate database is observed in the output, go to the original primary database, exit the sql prompt, re-login to the sql prompt, startup, and mount the database again. Step 11 DGMGRL> REINSTATE DATABASE 'PMGDB' Reinstating database "PMGDB", please wait... Reinstatement of database "PMGDB" succeeded View the configuration again to check if the configuration status is a "SUCCESS" and with no errors. 37

38 Testing Failover Process Note In case of primary with hot and cold standby, the error or warning of cold standby (PMGDB_STBY2) can be ignored. DGMGRL> show configuration In case of primary with only hot standby: Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB_STBY - Primary database PMGDB - (*) Physical standby database Fast-Start Failover: ENABLED Configuration Status: SUCCESS In case of primary with hot and cold standby: Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB_STBY - Primary database PMGDB - (*) Physical standby database PMGDB_STBY2 - Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: ENABLED Step 12 Configuration Status: SUCCESS Check the status on the original primary server. Step 13 SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB PHYSICAL STANDBY MOUNTED SYNCHRONIZED PMGDB Check the status on the new primary server. SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB_STBY PRIMARY READ WRITE SYNCHRONIZED PMGDB 38

39 Testing Failover Process Testing Failover Revert From New Primary to Original Primary Database Step 1 Log in to the primary server with the username oracle. If you are already logged in, proceed to Shutdown the new primary (original hot standby) database and check if failover reverts to the original primary database. SQL> select db_unique_name from v$database; DB_UNIQUE_NAME PMGDB_STBY Step 3 SQL> shut abort ORACLE instance shut down Check the alert logs on the original primary server. Note The Oracle base directory path may vary based on your installation. Step 4 $ cd /u01/app/oracle/diag/rdbms/pmgdb/pmgdb/trace $ tail -f alert_pmgdb.log -n100 Failover succeeded. Primary database is now PMGDB. Check the log status on the original primary server. Step 5 SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB PRIMARY READ WRITE REINSTATE REQUIRED PMGDB_STBY Synchronize the failover status on the original hot standby server, start up, and mount the database. SQL> startup mount ORACLE instance started. Step 6 Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Open a new console on the original hot standby server and keep tailing the log file drcpmgdb.log. Wait until the tail output stops at "DMON: >> DMON Process Shutdown <<". Note The Oracle base directory path may vary based on your installation. $cd /u01/app/oracle/diag/rdbms/pmgdb_stby/pmgdb/trace $tail -f drcpmgdb.log 39

40 Testing Failover Process Step 7 <tail output> :27: DMON: status from posting instances for Database QUIESCE = ORA :27: DMON: Releasing (convert to NULL) Health Check Master lock :27: DMON: Releasing FSFP HOME lock :27: INSV: Reply received for message with :27: req ID , opcode CTL_QUIESCE, phase TEARDOWN :27: DMON: Entered rfm_release_chief_lock() for CTL_QUIESCE :27: Fore: FSFO shutting down :27: DMON: Data Guard Broker shutting down :27: DMON: Terminating RSM processes :27: RSM0: delete state object for RSM :28: DMON: RSM0 successfully terminated :28: DMON: Terminating NetSlave processes :28: DMON: Freeing all task elements :28: DMON: Terminating Instance Slave process :28: INSV: Shutting down :28: DMON: INSV successfully terminated :28: DMON: Zeroing metadata root pointer :28: DMON: Clearing Primary State :28: DMON: Freeing Broker SGA heap :28: DMON: Freeing PGA heap :28: DMON: Removing DMON's state object :28: DMON: Resetting DMON context structure :28: DMON: >> DMON Process Shutdown << Return to the sql prompt of the original hot standby server, exit the sql prompt, relogin to the sql prompt, and redo the startup mount. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options $ sqlplus / as sysdba SQL*Plus: Release Production on Thu Jul 3 21:57: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. SQL> startup mount ORACLE instance started. Step 8 Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Connect the DGMGRL to the PMGDB, which is the primary now and reinstate the failover status. Step 9 DGMGRL> connect sys@pmgdb Password: Connected. View the configuration. Note that the database roles are changed, however, a warning to reinstate standby is displayed. DGMGRL> show configuration In case of primary with only hot standby: Configuration - dgpmgdb 40

41 Testing Failover Process Protection Mode: MaxAvailability Databases: PMGDB - Primary database Warning: ORA-16817: unsynchronized fast-start failover configuration PMGDB_STBY - (*) Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: ENABLED Configuration Status: WARNING In case of primary with hot and cold standby: Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database Warning: ORA-16817: unsynchronized fast-start failover configuration PMGDB_STBY - (*) Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated PMGDB_STBY2 - Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: ENABLED Step 10 Configuration Status: WARNING Reinstate the standby database. Note If Error: ORA-16653: failed to reinstate database is observed in the output, go to the original hot standby database, exit the sql prompt, relogin to the sql prompt, start up, and mount the database again. Step 11 DGMGRL> REINSTATE DATABASE 'PMGDB_STBY' Reinstating database "PMGDB_STBY", please wait... Reinstatement of database "PMGDB_STBY" succeeded View the configuration again to check if the configuration status is a "SUCCESS" and with no errors. Note Sometimes processes from the previous command may not be complete, therefore the show configuration may show a different output. Allow a couple of minutes and execute the command again. In case of primary with hot and cold standby, an error or warning of cold standby (PMGDB_STBY2) can be ignored. DGMGRL> show configuration In case of primary with only hot standby: Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - (*) Physical standby database Fast-Start Failover: ENABLED Configuration Status: SUCCESS In case of primary with hot and cold standby: Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: 41

42 Testing Switchover Process PMGDB - Primary database PMGDB_STBY - (*) Physical standby database PMGDB_STBY2 - Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: ENABLED Step 12 Configuration Status: SUCCESS Check the status on the original hot standby server. Step 13 SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB_STBY PHYSICAL STANDBY MOUNTED SYNCHRONIZED PMGDB_STBY Check the status on the original primary server. SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB PRIMARY READ WRITE SYNCHRONIZED PMGDB_STBY Testing Switchover Process Switchover from the primary to secondary or standby database and vice versa is a manual process typically carried out during planned maintenance or disaster recovery. The following procedures describe the switchover from PMGDB (primary) to PMGDB_STBY (hot standby). Testing Switchover From Primary to Standby Database, on page 42 Testing Switchover Revert From New Primary to Original Primary Database, on page 44 Testing Switchover From Primary to Standby Database Step 1 Start the DGMGRL utility from the primary or standby server (whichever is up) or from another host where the Oracle client is installed. $ dgmgrl DGMGRL for Linux: Version bit Production 42

43 Testing Switchover Process Copyright (c) 2000, 2009, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connect to the PMG DB. Step 3 DGMGRL> connect sys@pmgdb Password: Connected. Switchover to PMGDB_STBY. DGMGRL> SWITCHOVER TO 'PMGDB_STBY' Performing switchover NOW, please wait... New primary database "PMGDB_STBY" is opening... Operation requires shutdown of instance "PMGDB" on database "PMGDB" Shutting down instance "PMGDB"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "PMGDB" on database "PMGDB" Starting instance "PMGDB"... Unable to connect to database ORA-12514: TNS:listener does not currently know of service requested in connect descriptor Failed. Warning: You are no longer connected to ORACLE. Step 4 Please complete the following steps to finish switchover: start up and mount instance "PMGDB" of database "PMGDB" Exit from the previous sql prompt to start up and mount the PMGDB. Step 5 SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options Connect to sqlplus. Step 6 $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 16:50: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. Start up and mount the database. Step 7 SQL> startup mount ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Verify that the roles are switched. a) Verify on the original primary (PMGDB). SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; 43

44 Testing Switchover Process DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB PHYSICAL STANDBY MOUNTED SYNCHRONIZED PMGDB b) Verify on the new primary (PMGDB_STBY). SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB_STBY PRIMARY READ WRITE SYNCHRONIZED PMGDB Testing Switchover Revert From New Primary to Original Primary Database Before You Begin Step 1 Start the DGMGRL utility from the primary or standby server or from another host where the Oracle client is installed. $ dgmgrl DGMGRL for Linux: Version bit Production Copyright (c) 2000, 2009, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connect to PMGDB_STBY. Step 3 DGMGRL> connect sys@pmgdb_stby Password: Connected. Switchover to PMGDB. DGMGRL> SWITCHOVER TO 'PMGDB' Performing switchover NOW, please wait... New primary database "PMGDB" is opening... Operation requires shutdown of instance "PMGDB" on database "PMGDB_STBY" Shutting down instance "PMGDB"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "PMGDB" on database "PMGDB_STBY" Starting instance "PMGDB"... Unable to connect to database ORA-12514: TNS:listener does not currently know of service requested in connect descriptor Failed. Warning: You are no longer connected to ORACLE. 44

45 Testing Cold Standby Step 4 Please complete the following steps to finish switchover: start up and mount instance "PMGDB" of database "PMGDB_STBY" Exit from previous sql prompt to start up and mount the PMGDB_STBY. Step 5 SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options Connect to sqlplus. Step 6 $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 16:50: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. Start up and mount the database. Step 7 SQL> startup mount ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Verify that the roles are switched. a) Verify on the hot standby (PMGDB_STBY). SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB_STBY PHYSICAL STANDBY MOUNTED SYNCHRONIZED PMGDB_STBY b) Verify on the original primary (PMGDB). SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database; DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS FSFO_CURR_TARGET PMGDB PRIMARY READ WRITE SYNCHRONIZED PMGDB_STBY Testing Cold Standby Use these procedures to test cold standby on PMG DB. 45

46 Testing Site Failure Testing Site Failure, on page 46 Recovering Original Primary After Site Failure, on page 48 Testing Site Failure Use the following steps to test unscheduled failure of primary and first hot standby (if available) servers. To prevent applications (PMG, OpsTools) connection failures without any manual intervention, ensure that the PMGDB configuration script (/rms/app/rms/install/pmgdb_configure.sh) for the Central node is executed with the standby server configuration as per the instructions in the installation guide. Usage: pmgdb_configure.sh <Pmgdb_Enabled> <Pmgdb_Primary_Dbserver_Address> <Pmgdb_Primary_Dbserver_Port> [<Pmgdb_Stby1_Dbserver_Address>] [<Pmgdb_Stby1_Dbserver_Port>] [<Pmgdb_Stby2_Dbserver_Address>] [<Pmgdb_Stby2_Dbserver_Port>] If not configured earlier, execute the script again with the appropriate values for database servers and ports. For more details, see the Cisco RAN Management System Installation Guide. Step 1 To test site failure, verify if the applications connecting to the PMGDB database are able to connect. For example, the OpsTools script getareas.sh can be executed. Log in to Central node as admin1 user and execute the script. $ getareas.sh -key 1001 Config files script-props/private/getareas.properties or script-props/public/getareas.properties not found. Continuing with default settings. Execution parameters: key=1001 GetAreas processing can take some time please do not terminate. Received areas, total areas 1 Writing to file: /home/admin1/getareas.csv The report captured in csv file: /home/admin1/getareas.csv **** GetAreas End Script *** Log in to the hot standby server as an oracle user and shut down the database server. Note Perform this step only if the hot standby server is configured. Else, proceed to the next step. Step 3 $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba SQL> shutdown immediate Database closed. Database dismounted. ORACLE instance shut down. Log in to the primary server an oracle user and shut down the database server. $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba SQL> shutdown immediate Database closed. 46

47 Testing Site Failure Step 4 Database dismounted. ORACLE instance shut down. Verify that the applications connecting to the PMGDB database are failing considering that the primary and hot standby are down. Log in to the Central node as admin1 user and execute the getareas.sh script. $ getareas.sh -key 1001 Config files script-props/private/getareas.properties or script-props/public/getareas.properties not found. Continuing with default settings. Execution parameters: key=1001 GetAreas processing can take some time please do not terminate. Jul 01, :45:14 PM org.apache.tomcat.jdbc.pool.connectionpool init SEVERE: Unable to create initial connections of pool..... Failed to Get Areas : Error while connecting to PmgDb, java.sql.sqlexception: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor.... Alternate Output: Failed to Get Areas : Error while connecting to PmgDb, java.sql.sqlrecoverableexception: Io exception: Step 5 The Network Adapter could not establish the connection Log in to the cold standby server as an oracle user and log in to sql prompt. Step 6 $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba Connected to database Stop the redo apply process. Step 7 SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; Finish applying all received redo data. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH; Step 8 SQL> ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE; Switch the physical standby database to the primary role. Step 9 SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN; ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN * ERROR at line 1: ORA-01109: database not open Open the new primary database. This may take some time to complete. 47

48 Recovering Original Primary After Site Failure Step 10 SQL> ALTER DATABASE OPEN; Check the status of the database. The switchover status could be FAILED DESTINATION or NOT ALLOWED. Step 11 SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database; NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS PMGDB READ WRITE PMGDB_STBY2 PRIMARY FAILED DESTINATION Verify that applications connecting to the PMGDB database are getting connected now that the cold standby is up as a new primary. Log in to the Central node as admin1 user and execute the getareas.sh script. $ getareas.sh -key 1001 Config files script-props/private/getareas.properties or script-props/public/getareas.properties not found. Continuing with default settings. Execution parameters: key=1001 GetAreas processing can take some time please do not terminate. Received areas, total areas 1 Writing to file: /home/admin1/getareas.csv The report captured in csv file: /home/admin1/getareas.csv **** GetAreas End Script *** Recovering Original Primary After Site Failure The database can be recovered using either the RMAN backup or Flashback database. Following steps describe recovery using RMAN backup. Converting a Failed Primary into a Standby Database Using RMAN Backups DETAILED STEPS Step 1 Command or Action Determine the SCN at which the old/cold standby database became the primary database. Run the following query on the new primary database to determine the SCN at which the original/cold standby database became the new primary database: Step 3 SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE; TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) Restore and recover the entire database on the original primary. 48

49 Recovering Original Primary After Site Failure Command or Action Step 4 Run the following RMAN commands on the original primary. $ export ORACLE_SID=PMGDB $ rman target=/ Recovery Manager: Release Production on Tue Jul 8 19:30: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. connected to target database (not started) RMAN> startup mount Oracle instance started database mounted Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers bytes bytes bytes bytes bytes RMAN> RUN { SET UNTIL SCN <recovery_scn> ; RESTORE DATABASE; RECOVER DATABASE; } executing command: SET until clause Starting restore at 08-JUL-14 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=11 device type=disk channel ORA_DISK_1: starting datafile backup set restore channel ORA_DISK_1: specifying datafile(s) to restore from backup set channel ORA_DISK_1: restoring datafile to /u01/app/oracle/oradata/pmgdb/system01.dbf channel ORA_DISK_1: restoring datafile to /u01/app/oracle/oradata/pmgdb/sysaux01.dbf channel ORA_DISK_1: restoring datafile to /u01/app/oracle/oradata/pmgdb/undotbs01.dbf channel ORA_DISK_1: restoring datafile to /u01/app/oracle/oradata/pmgdb/users01.dbf channel ORA_DISK_1: restoring datafile to /u01/app/oracle/oradata/pmgdb/mapinfo_ts.dbf channel ORA_DISK_1: restoring datafile to /u01/app/oracle/oradata/pmgdb/pmgdb_ts.dbf channel ORA_DISK_1: reading from backup piece /u01/app/oracle/flash_recovery_area/pmgdb/backupset/2014_07_08/o1_mf_nnndf_tag t162646_9vqmmgsv_.bkp channel ORA_DISK_1: piece handle=/u01/app/oracle/flash_recovery_area/pmgdb/backupset/2014_07_08/o1_mf_nnndf_tag t162646_9vqmmgsv_. tag=tag t channel ORA_DISK_1: restored backup piece 1 channel ORA_DISK_1: restore complete, elapsed time: 00:00:25 49

50 Recovering Original Primary After Site Failure Command or Action Finished restore at 08-JUL-14 Starting recover at 08-JUL-14 using channel ORA_DISK_1 starting media recovery archived log for thread 1 with sequence 24 is already on disk as file /u01/app/oracle/flash_recovery_area/pmgdb/archivelog/2014_07_08/o1_mf_1_24_9vqmn043_.arc archived log for thread 1 with sequence 25 is already on disk as file /u01/app/oracle/flash_recovery_area/pmgdb/archivelog/2014_07_08/o1_mf_1_25_9vqv5j39_.arc archived log for thread 1 with sequence 26 is already on disk as file /u01/app/oracle/flash_recovery_area/pmgdb/archivelog/2014_07_08/o1_mf_1_26_9vqw8v1v_.arc archived log file name=/u01/app/oracle/flash_recovery_area/pmgdb/archivelog/2014_07_08/o1_mf_1_24_9vqmn043_.arc thread=1 sequence=24 media recovery complete, elapsed time: 00:00:00 Finished recover at 08-JUL-14 Step 5 RMAN> exit Recovery Manager complete. Convert original primary database to a physical standby database. Perform the following steps on the original primary database. $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba SQL*Plus: Release Production on Mon Jun 16 17:51: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> Step 6 SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY; Shut down and start up the original primary. SQL> SHUTDOWN IMMEDIATE; ORA-01507: database not mounted ORACLE instance shut down. SQL> STARTUP MOUNT; ORACLE instance started. Step 7 Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Open the database as read-only. SQL> ALTER DATABASE OPEN READ ONLY; 50

51 Recovering Original Primary After Site Failure Command or Action Step 8 Mount the new standby (original primary) again. SQL> SHUTDOWN IMMEDIATE; Database closed. Database dismounted. ORACLE instance shut down. SQL> STARTUP MOUNT; ORACLE instance started. Step 9 Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Log in to the sql prompt on the new primary (old cold standby) and make sure the archive destination is enabled. SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE; Step 10 Start Redo Apply on the new standby (original primary) database. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT; Step 11 Verify status, it could be SWITCHOVER PENDING or SWITCHOVER LATENT or TO PRIMARY. SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE; SWITCHOVER_STATUS SWITCHOVER PENDING Step 12 Verify the status on the new primary as well. A value of TO STANDBY or SESSIONS ACTIVE indicates that the primary database be switched to the standby role. SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE; SWITCHOVER_STATUS TO STANDBY 51

52 Recovering Original Primary After Site Failure Converting Failed Primary Into a Standby Database Using RMAN Backups DETAILED STEPS Step 1 Command or Action Issue the following SQL statement on the primary database to switch it to the standby role: Purpose SQL> select DATABASE_ROLE from v$database; DATABASE_ROLE PRIMARY SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN; Shut down and then mount this current primary database. SQL> SHUTDOWN IMMEDIATE; ORA-01507: database not mounted ORACLE instance shut down. Step 3 SQL> STARTUP MOUNT; ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Check the switchover status on the current primary which now has become cold standby again. A value of TO PRIMARY or SESSIONS ACTIVE indicates that the standby database is ready. If neither of these values is returned, continue to query this column until the value returned is either TO PRIMARY or SESSIONS ACTIVE. Step 4 SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE; SWITCHOVER_STATUS TO_PRIMARY Return to the original primary to revert its database role to the primary role. SQL> RECOVER MANAGED STANDBY DATABASE FINISH; Media recovery complete. SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN; 52

53 Recovering Original Primary After Site Failure Step 5 Command or Action Now open the original primary database. Purpose Step 6 Step 7 SQL> ALTER DATABASE OPEN; Database opened. Return to the new physical standby (cold standby) database and start redo apply. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION; Check status on the original primary and cold standby to make sure that the database role is correct. SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database; NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS PMGDB READ WRITE PMGDB PRIMARY FAILED DESTINATION Note Ignore and continue if error "ORA-01153: an incompatible media recovery is active" is displayed. On original primary: On cold standby: Step 8 SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database; NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS PMGDB MOUNTED PMGDB_STBY2 PHYSICAL STANDBY SESSIONS ACTIVE Start up and mount the hot standby if hot standby is present in the setup. $ sqlplus / as sysdba SQL*Plus: Release Production on Thu Jul 10 11:39: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. SQL> startup mount ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes 53

54 Recovering Original Primary After Site Failure Step 9 Step 10 Command or Action Redo Buffers Database mounted bytes Purpose Verify the dgbroker status, log in to DGMGRL either from primary or standby. Note that in case of hot and cold standby to primary setup, WARNING is displayed The status should be a SUCCESS, which indicates reverting to the original as hot standby server is not enabled. For only cold standby to primary setup, it roles is complete for hot and cold should show SUCCESS which means reverting to the original role is complete. standby to primary setup. $ dgmgrl DGMGRL for Linux: Version bit Production Copyright (c) 2000, 2009, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. DGMGRL> connect sys@pmgdb Password:. Connected. DGMGRL> show configuration Configuration Protection Mode Databases Fast-Start Failover Configuration Status - dgpmgdb : MaxPerformance : PMGDB - Primary database : PMGDB_STBY - Physical standby database : DISABLED : SUCCESS Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - (*) Physical standby database PMGDB_STBY2 - Physical standby database Fast-Start Failover: ENABLED Configuration Status: SUCCESS Enable cold standby if cold standby (PMGDB_STBY2) is showing disabled or with warning. DGMGRL> enable database 'PMGDB_STBY2' Enabled. DGMGRL> show configuration Note If the output of command shows ERROR or WARNING status, see the Troubleshooting Data Guard on PMG DB, on page 65 to resolve the issues. Sometimes processes from the previous command may not be complete, therefore the show configuration command may show different output. Allow a couple of minutes and execute the command again. In case of primary and cold standby setup: In case of primary hot and cold standby setup: In case of primary hot and cold standby setup: Note If the output of command shows ERROR or WARNING status, see the Troubleshooting Data Guard on PMG DB, on page 65 to resolve the issues. Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - (*) Physical standby database PMGDB_STBY2 - Physical standby database Fast-Start Failover: ENABLED 54

55 Rolling Back and Cleaning Up Standby and Primary Configurations Command or Action Configuration Status: SUCCESS Purpose Rolling Back and Cleaning Up Standby and Primary Configurations This section covers common scenarios for rollback. For specific scenarios, like database corruption and so on, it is recommended that you refer to standard DBA practices or Oracle documentation. Note Before starting the rollback, it is recommended that the primary database is backed up. The following procedures describe how to roll back the standby and primary configurations. Removing Data Guard Broker Configuration, on page 55 Removing the Standby Database, on page 60 Removing the Additional Standby Database, on page 61 Removing Data Guard Broker Configuration Step 1 Connect to the primary database from DGMGRL. DGMGRL> connect sys@pmgdb Password: Connected. Disable FAST_START FAILOVER. Note Perform this step only if the setup has hot standby configured. Step 3 DGMGRL> DISABLE FAST_START FAILOVER FORCE Disabled. Change the protection mode. Note Perform this step only if the setup has hot standby configured. 55

56 Removing Data Guard Broker Configuration Step 4 DGMGRL> edit configuration set protection mode as MaxPerformance; Succeeded. Remove the configuration. Step 5 DGMGRL> remove configuration Removed configuration Check if the configuration is removed. DGMGRL> show configuration ORA-16532: Data Guard broker configuration does not exist Configuration details cannot be determined by DGMGRL Alternate Output ORA-16596: database not part of the Data Guard broker configuration Configuration details cannot be determined by DGMGRL Removing Configuration Files from Primary Server Step 1 Log in to the primary server as an oracle user and connect to sqlplus. $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba SQL*Plus: Release Production on Wed Jul 2 20:17: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> Stop the Data Guard Broker process. Step 3 SQL> alter system set dg_broker_start=false; Check the path of the Data Guard Broker configuration files. SQL> select name, value from v$parameter where name like '%dg_broker%'; NAME VALUE dg_broker_start FALSE dg_broker_config_file1 56

57 Removing Data Guard Broker Configuration /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1pmgdb.dat Step 4 dg_broker_config_file2 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2pmgdb.dat Exit from the sql prompt. Step 5 SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production Remove dg_broker_config_file1 and dg_broker_config_file2. Note The Oracle base directory path may vary based on your installation. $ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1pmgdb.dat Step 6 $ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2pmgdb.dat The system responds with a command prompt. Remove the standby control files if they still exist in /tmp directory. Ignore the error if the standby control files do not exist. $ rm /tmp/pmgdb_stby.ctl $ rm /tmp/pmgdb_stby2.ctl The system responds with a command prompt. Removing Configuration Files From Standby Server Note If the setup is only cold standby added to the primary, then this standby is cold standby. If the setup is only hot standby added to the primary, then this standby is hot standby. If the setup is both hot and cold standby added to the primary, this standby is hot standby. DETAILED STEPS Command or Action Step 1 Log in to the first standby server as an oracle user and connect to sqlplus. $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba SQL*Plus: Release Production on Wed Jul 2 20:17: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL 57

58 Removing Data Guard Broker Configuration Command or Action Stop the redo apply process. SQL> alter database recover managed standby database cancel; Step 3 Stop Data Guard Broker process. SQL> alter system set dg_broker_start=false; Step 4 Check the path of the Data Guard Broker configuration files. SQL> select name, value from v$parameter where name like '%dg_broker%'; NAME VALUE dg_broker_start FALSE dg_broker_config_file1 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1pmgdb_stby.dat dg_broker_config_file2 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2pmgdb_stby.dat Step 5 Open the database in read-only mode. SQL> alter database open read only; alter database open read only; Step 6 Exit from the sql prompt. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production Step 7 Remove the dg_broker_config_file1 and dg_broker_config_file2. $ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1pmgdb_stby.dat The system responds with a command prompt. $ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2pmgdb_stby.dat The system responds with a command prompt. 58

59 Removing Data Guard Broker Configuration Removing Configuration Files From Additional Standby Server Note This section is applicable only to the setup with both hot and cold standby added to primary. This additional standby is the cold standby in the setup. DETAILED STEPS Command or Action Step 1 Log in to the standby server as an oracle user and connect to sqlplus. $ export ORACLE_SID=PMGDB $ sqlplus / as sysdba SQL*Plus: Release Production on Wed Jul 2 20:17: Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> Stop the redo apply process. SQL> alter database recover managed standby database cancel; Step 3 Stop the Data Guard Broker process. SQL> alter system set dg_broker_start=false; Step 4 Check the path of the Data Guard Broker configuration files. SQL> select name, value from v$parameter where name like '%dg_broker%'; NAME VALUE dg_broker_start FALSE dg_broker_config_file1 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1pmgdb_stby2.dat dg_broker_config_file2 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2pmgdb_stby2.dat Step 5 Open the database in read-only mode. SQL> alter database open read only; alter database open read only; Step 6 Exit from the sql prompt 59

60 Removing the Standby Database Command or Action SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release bit Production Step 7 Remove the dg_broker_config_file1 and dg_broker_config_file2. $ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1pmgdb_stby2.dat The system responds with a command prompt. $ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2pmgdb_stby2.dat The system responds with a command prompt. Removing the Standby Database Note If the setup is only cold standby added to the primary, then this standby is cold standby. If the setup is only hot standby added to the primary, then this standby is hot standby. If the setup is both hot and cold standby added to the primary, this standby is hot standby. Step 1 From first standby server console, execute the dbca command. $ dbca -silent -deletedatabase -sourcedb PMGDB Connecting to database 100% complete Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/pmgdb.log" for further details. Move the respective directories as backup directories. Note The Oracle base directory path may vary based on your installation. $ mv /u01/app/oracle/oradata/pmgdb /u01/app/oracle/oradata/pmgdb-bak The system responds with a command prompt. $ mv /u01/app/oracle/flash_recovery_area/pmgdb /u01/app/oracle/flash_recovery_area/pmgdb-bak The system responds with a command prompt. $ mv /u01/app/oracle/flash_recovery_area/pmgdb_stby /u01/app/oracle/flash_recovery_area/pmgdb-stby-bak The system responds with a command prompt. 60

61 Cleaning Up the Primary Database $ mv /u01/app/oracle/admin/pmgdb /u01/app/oracle/admin/pmgdb-bak The system responds with a command prompt. Removing the Additional Standby Database DETAILED STEPS This section is applicable only to the setup with both hot and cold standby added to primary. This additional standby is the cold standby in this setup. Step 1 Command or Action From the standby server console, execute the dbca command. Purpose $ dbca -silent -deletedatabase -sourcedb PMGDB Connecting to database 100% complete Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/pmgdb.log" for further details. Move the respective directories as backup directories. $ mv /u01/app/oracle/oradata/pmgdb /u01/app/oracle/oradata/pmgdb-bak The system responds with a command prompt. $ mv /u01/app/oracle/flash_recovery_area/pmgdb /u01/app/oracle/flash_recovery_area/pmgdb-bak The system responds with a command prompt. $ mv /u01/app/oracle/flash_recovery_area/pmgdb_stby2 /u01/app/oracle/flash_recovery_area/pmgdb-stby2-bak The system responds with a command prompt. $ mv /u01/app/oracle/admin/pmgdb /u01/app/oracle/admin/pmgdb-bak The system responds with a command prompt. Cleaning Up the Primary Database Cleaning Up the Redo Log Files Step 1 Drop the standby redo log files from the primary database. SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo01.log'); 61

62 Cleaning Up the Primary Database SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo02.log'); SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo03.log'); SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/pmgdb/standby_redo04.log'); Remove the standby redo log files from the directory. $ rm /u01/app/oracle/oradata/pmgdb/standby_redo01.log The system responds with a command prompt. $ rm /u01/app/oracle/oradata/pmgdb/standby_redo02.log The system responds with a command prompt. $ rm /u01/app/oracle/oradata/pmgdb/standby_redo03.log The system responds with a command prompt. $ rm /u01/app/oracle/oradata/pmgdb/standby_redo04.log The system responds with a command prompt. Cleaning Up Initialization Parameters The initialization parameters can be reset using either one of these methods: Note (Recommended) Create spfile from backup pfile that was generated during initial steps. For more information, see Using the Backup Pfile, on page 62. If the backup pfile is not available, the parameters can be reset using sql statements. For more information, see Using SQL Statements, on page 63. Using the Backup Pfile Use this option to restore the spfile from the backup pfile. Step 1 Shutdown the database. SQL> shutdown immediate Database closed. Database dismounted. ORACLE instance shut down. From the sql prompt, create the spfile from the backup pfile. Note f the specific path for the initpmgdb_pre_dgsetup.ora file (for example, /backup/initpmgdb_pre_dgsetup.ora) was specified when the file was backed up during the initial steps, specify the full path (for example, /backup/initpmgdb_pre_dgsetup.ora). If the path is not specified, the file will be picked up from the default directory, that is, $ORACLE_HOME/dbs. 62

63 Cleaning Up the Primary Database Step 3 SQL> CREATE SPFILE FROM PFILE='initPMGDB_pre_dgsetup.ora'; File created. Start up the database. SQL> startup ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. Database opened. Using SQL Statements If the backup pfile is not available, the parameters can be reset through sql statements. This section can be skipped if spfile is already restored from the backup pfile in the previous section. From the sql prompt, reset the initialization parameters. SQL> ALTER SYSTEM RESET LOG_ARCHIVE_CONFIG; SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_2; SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_STATE_2; SQL> ALTER SYSTEM RESET LOG_ARCHIVE_FORMAT SCOPE=SPFILE; SQL> ALTER SYSTEM RESET LOG_ARCHIVE_MAX_PROCESSES; SQL> ALTER SYSTEM RESET FAL_SERVER; SQL> ALTER SYSTEM RESET DB_FILE_NAME_CONVERT SCOPE=SPFILE; SQL> ALTER SYSTEM RESET LOG_FILE_NAME_CONVERT SCOPE=SPFILE; SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_3; SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_STATE_3; 63

64 Recreating Standby Servers Verifying the Database Before You Begin Step 1 From the sql prompt, check if the database role is Primary. SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database; NAME TYPE VALUE service_names string PMGDB_PRIMARY Ensure that the database service PMGDB_PRIMARY is present. Step 3 SQL> show parameter service NAME TYPE VALUE service_names string PMGDB_PRIMARY Verify that applications connecting to the PMGDB database are getting connected. For example, the OpsTools script getareas.sh can be executed. Log in to the Central node as admin1 user and execute the script. $ getareas.sh -key 1001 Config files script-props/private/getareas.properties or script-props/public/getareas.properties not found. Continuing with default settings. Execution parameters: key=1001 GetAreas processing can take some time please do not terminate. Received areas, total areas 1 Writing to file: /home/admin1/getareas.csv The report captured in csv file: /home/admin1/getareas.csv **** GetAreas End Script *** Recreating Standby Servers Now that the standby server and primary server configurations are cleaned up, reconfigure the Data Guard Broker. For more information, see the Setting Up the Oracle Data Guard Broker procedure on page 58. Deleting the Primary Database The primary database is now rolled back to previous state using steps in the previous sections. However, if there is a need to clean up all configurations, including the primary database, ensure that you do the following: Delete the primary database and create it again using the standard DBA process (for example, using Oracle DBCA utility, and so on). Install the PMG DB schema and populate data as per steps mentioned in the install guide. For more information, see, Cisco RAN Management System Installation Guide. 64

65 Troubleshooting Data Guard on PMG DB To create the standby servers again, follow the tasks listed in the Configuration Workflow, on page 3. Troubleshooting Data Guard on PMG DB This section provides solutions to some of the errors that may be encountered while configuring Data Guard on the PMG DB. Reverting Original Primary Database After Site Failure, on page 65 Verifying the Data Guard Broker Configuration, on page 73 Reverting From Disk Space Issues, on page 73 Reverting Original Primary Database After Site Failure If the Data Guard Broker "show configuration" status is not a SUCCESS, check the error code and try to resolve using steps under respective errors. If not connected to DGMGRL tool, connect as sys@pmgdb. Note The outputs of the steps are from the setup with primary, hot standby, and cold standby. ORA ORA-16825: multiple errors or warnings, including fast-start failover-related errors or warnings, detected for the database If DGMGRL> show configuration is showing this error for primary database, check verbose view to see exact errors related to the database. DGMGRL> show database verbose 'PMGDB' Database - PMGDB Role: Intended State: Instance(s): PMGDB PRIMARY TRANSPORT-ON Database Error(s): ORA-16628: broker protection mode inconsistent with the database setting Database Warning(s): ORA-16817: unsynchronized fast-start failover configuration ORA ORA-16628: broker protection mode inconsistent with the database setting If DGMGRL> show configuration is showing this error for the primary database, change to MaxAvailability mode. Disable Fast-Start Failover, if configuration shows it as ENABLED. DGMGRL> disable fast_start failover Disabled. DGMGRL> edit configuration set protection mode as MaxAvailability; Succeeded. 65

66 Reverting Original Primary Database After Site Failure Check configuration; if it is a SUCCESS enable fast_start failover, otherwise troubleshoot for the error displayed. DGMGRL> show configuration Fast-Start Failover: DISABLED Configuration Status: SUCCESS DGMGRL> enable fast_start failover Enabled. ORA ORA-16810: multiple errors or warnings detected for the database If DGMGRL> show configuration is showing this error for standby database, check verbose view to see exact errors related to the database. DGMGRL> show database verbose 'PMGDB_STBY' Database - PMGDB_STBY Role: PHYSICAL STANDBY Intended State: APPLY-ON Transport Lag: (unknown) Apply Lag: (unknown) Real Time Query: OFF Instance(s): PMGDB Database Error(s): ORA-16700: the standby database has diverged from the primary database ORA-16766: Redo Apply is stopped ORA ORA-16700: the standby database has diverged from the primary database If DGMGRL> show database verbose is showing this error for the standby database it indicates that the the primary and standby server files have a mismatch. View alertpmgdb.log of the standby database to see if it is due to ORA error. If ORA is in the log, follow the trouble steps for ORA Following is sample output of the logs. $ cd /u01/app/oracle/diag/rdbms/pmgdb_stby/pmgdb/trace $ tail -f alert_pmgdb.log -n100 Or $ vi alert_pmgdb.log MRP0 started with pid=66, OS id=24771 MRP0: Background Managed Standby Recovery process started (PMGDB) Serial Media Recovery started Managed Standby Recovery starting Real Time Apply Warning: Recovery target destination is in a sibling branch of the controlfile checkpoint. Recovery will only recover changes to datafiles. MRP0: Detected orphaned datafiles! Recovery will possibly be retried after flashback... Errors in file /u01/app/oracle/diag/rdbms/pmgdb_stby/pmgdb/trace/pmgdb_mrp0_24771.trc: ORA-19909: datafile 1 belongs to an orphan incarnation ORA-01110: data file 1: '/u01/app/oracle/oradata/pmgdb/system01.dbf' 66

67 Reverting Original Primary Database After Site Failure Managed Standby Recovery not using Real Time Apply Note The log directory may vary based on the installation setup. ORA ORA-19909: datafile 1 belongs to an orphan incarnation If alert_pmgdb.log shows ORA error, it indicates that the primary database was opened with RESETLOGS after the database recovery that causes the database incarnation to be different between the primary and standby database. The following steps describe how to avoid re-creating a physical standby database to resolve this issue. Note These steps require FLASHBACK enabled. FLASHBACK is enabled as part of configuring the standby server setup and is enabled for standby databases. However, for any reason if it is not enabled, the standby database needs to be recreated. To do this, see ORA Before proceeding ensure that the fast_start failover in DGMRL configuration is DISABLED. If not disabled, disable it. DGMGRL> disable fast_start failover Disabled. 2 Determine the SCN before the RESETLOGS operation occurred. On the primary database, use the following query to obtain the value of the system change number (SCN) that is two SCNs before the RESETLOGS operation occurred on the primary database. SQL> SELECT TO_CHAR(RESETLOGS_CHANGE# - 2) FROM V$DATABASE; TO_CHAR(RESETLOGS_CHANGE#-2) Obtain the current SCN on the standby database. SQL> SELECT TO_CHAR(CURRENT_SCN) FROM V$DATABASE; TO_CHAR(CURRENT_SCN) If the value of CURRENT_SCN is larger than the value of resetlogs_change# - 2, issue the following statement on the standby database to flash back the standby database. Note If the value of CURRENT_SCN is less than the value of the resetlogs_change# - 2, skip to next step. SQL> FLASHBACK STANDBY DATABASE TO SCN ; Flashback complete. 5 Restart Redo Apply on the standby server. 67

68 Reverting Original Primary Database After Site Failure Note This step may have to be repeated if following step shows configuration status Error or Warning. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT; 6 Connect to DGMGRL as sys@pmgdb if not connected and view the DGMGRL show configuration; if the status is error with error ORA-16766, repeat previous step to restart redo apply and continue with this step till no error is displayed. DGMGRL> show configuration Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - Physical standby database Error: ORA-16766: Redo Apply is stopped PMGDB_STBY2 - Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: DISABLED Configuration Status: ERROR 7 If the previous step, shows Warning: ORA-16826: apply service state is inconsistent with the DelayMins property, use this step on the standby server sql prompt. Else continue with the next step. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION; DGMGRL> show configuration Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - Physical standby database PMGDB_STBY2 - Physical standby database (disabled) ORA-16661: the standby database needs to be reinstated Fast-Start Failover: DISABLED Configuration Status: SUCCESS 8 Now enable fast_start failover and if cold standby is present in the setup, enable cold standby database. DGMGRL> enable fast_start failover Enabled. DGMGRL> enable database 'PMGDB_STBY2' Enabled. 9 Verify that errors are resolved and the setup is reverted to the original state successfully. DGMGRL> show configuration Configuration - dgpmgdb Protection Mode: MaxAvailability Databases: PMGDB - Primary database PMGDB_STBY - (*) Physical standby database PMGDB_STBY2 - Physical standby database 68

69 Reverting Original Primary Database After Site Failure Fast-Start Failover: ENABLED Configuration Status: SUCCESS RMAN RMAN-20208: UNTIL CHANGE is before RESETLOGS change The following command may throw error as shown. RMAN> RUN { SET UNTIL SCN <recovery_scn> ; RESTORE DATABASE; RECOVER DATABASE; } Starting restore at 17-JUL-14 using target database control file instead of recovery catalog RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of restore command at 07/17/2014 RMAN-20208: UNTIL CHANGE is before RESETLOGS change Solution: The error RMAN occurred because the command is trying to go to an SCN (<recovery_scn>) before the current incarnation of the database. Follow the steps to resolve this. 1 List the incarnation of the original primary database in RMAN. $ export ORACLE_SID=PMGDB $ rman target=/ RMAN> list incarnation of database; using target database control file instead of recovery catalog List of Database Incarnations DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time PMGDB PARENT 1 15-AUG PMGDB PARENT JUL PMGDB PARENT JUL PMGDB CURRENT JUL-14 2 Now mark the incarnation having SCN before the <recovery_scn>. In this example, if value of recovery_scn is , select incarnation 2 with SCN Reset the database to this incarnation. RMAN> reset database to incarnation 2; database reset to incarnation 2 3 Now execute the RUN command which was failed; it should show output as expected. RMAN> RUN { SET UNTIL SCN <recovery_scn> ; RESTORE DATABASE; RECOVER DATABASE; } executing command: SET until clause media recovery complete, elapsed time: 00:00:00 Finished recover at 08-JUL-14 ORA ORA-38729: Not enough flashback database log data to do FLASHBACK. 69

70 Reverting Original Primary Database After Site Failure This error may occur while resolving error RMAN If FLASHBACK DATABASE is run when the target SCN is outside the flashback window, then FLASHBACK DATABASE fails with an ORA error. In this case the database will not be changed. The possible reason to get this error could be intentionally or unintentionally removing/moving some flashback database logs. To overcome this error, recreate the standby again, that is, clean up as described in Rolling Back and Cleaning Up Standby and Primary Configurations, on page 55 and create the standby as described in Configuring the Primary Server, on page 4 and Configuring the Hot Standby Server, on page 10. Though most of the steps are same, some additional changes are required. Hence those steps are repeated here. 1 Log in to primary sql prompt and defer LOG_ARCHIVE_DEST_STATE SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=DEFER; 2 Log in to DGMGRL from primary or hot standby; Disable FAST_START FAILOVER is not disabled already. DGMGRL> DISABLE FAST_START FAILOVER FORCE Disabled. 3 Change protection mode in Data Guard Broker configuration. DGMGRL> edit configuration set protection mode as MaxPerformance; Succeeded. 4 Remove hot standby from Data Guard Broker configuration. DGMGRL> REMOVE DATABASE 'PMGDB_STBY' Removed database "PMGDB_STBY" from the configuration 5 To remove the configuration files from the hot standby, see. Note Redo apply process may have already stopped for the hot standby. Ignore the error if the step "Stop redo apply process" throws any error. 6 To remove the standby database, see Removing the Standby Database, on page Return and log in to the primary sql prompt and enable LOG_ARCHIVE_DEST_STATE. SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE; 8 Back up the primary database, see Backing Up the Primary Database, on page 8. 9 Move existing file as backup if pfile and control file already exist in the /tmp file. $ mv /tmp/pmgdb_stby.ctl /tmp/pmgdb_stby.ctl_back $ mv /tmp/initpmgdb_stby.ora /tmp/initpmgdb_stby.ora_back System responds with command prompt. 10 Create the standby control file and PFILE, see the Creating Standby Control File and PFILE, on page Stop the observer by logging in to DGMGRL and connecting as sys@pmgdb if the observer is running. DGMGRL> stop observer Done. 12 Create a hot standby, see Configuring the Hot Standby Server, on page 10. ORA ORA-16820: fast-start failover observer is no longer observing this database This error may be displayed in DGMGRL show configuration output, if the observer is not running or the drcpmgdb.log (under /u01/app/oracle/diag/rdbms/pmgdb/pmgdb/trace) on primary shows warning as ObserverHB indicates lack of Observer pings at Standby. 70