CONTENTS: PREFACE: A. Overview

Transcription

1 Using the Serviceguard Toolkit for Oracle Data Guard in an HP Serviceguard Cluster ========================================================================= ================= Version: B CONTENTS: A. Overview B. Supported versions C. Setting Up Oracle Data Guard D. Setting Up the Toolkit E. Oracle Data Guard Package Configuration Example F. Adding the Package to the Cluster G. Oracle Data Guard Maintenance H. Using Serviceguard Toolkit for Oracle Data Guard in Continentalclusters environment I. Using Serviceguard Toolkit for Oracle Data Guard in Metrocluster and Extended Distance Cluster environments J. Verification of the setup PREFACE: This README file describes the High Availability Serviceguard Toolkit for Oracle Data Guard (also referred to as "Oracle Data Guard Toolkit") designed for use in a Serviceguard environment. This README covers the basic steps for configuring Oracle Data Guard in an HP-UX cluster, and is intended for users who want to integrate in an HP-UX cluster, and is intended for users who want to integrate Oracle Data Guard with HP Serviceguard. This README assumes that its readers are already familiar with Serviceguard configuration, as well as with Oracle Data Guard concepts, including installation and configuration procedures. A. Overview The Serviceguard Toolkit for Oracle Data Guard consists of a set of shell scripts that are used to start, stop, and monitor Oracle Data Guard primary and standby database instances. Oracle Data Guard provides protection to Oracle Database by replicating the database data to one or more standby databases. Integrating Oracle Data Guard with HP Serviceguard has the following advantages: - Provides High Availability for Data Guard processes for both primary and the standby databases.

2 HP Serviceguard includes built in monitoring capabilities to monitor system resources like network, volume groups, filesystems etc and to initiate failover in the instant of their failure. - Oracle Data Guard integration with HP Serviceguard Extended Distance Cluster or other Serviceguard based "Disaster Recovery (DR)" clusters, will provide disaster protection and recovery for Oracle database as well as the applications that use the databases. NOTE: The toolkit also provides HA for Active standby database (Oracle 11g and above) NOTE: This version of the toolkit does t include support for performing role transitions. These tasks must be manually performed. After these tasks are manually performed, the toolkit detects the new role of the database and the monitors the associated processes. This toolkit supports automatic role transitions only in a Continentalclusters environment when the primary cluster fails. NOTE: This version of toolkit will support only single instance Oracle and does t support Oracle RAC This toolkit leverages scripts from the ECMT Oracle Toolkit to start, stop, and monitor the database and the listener. IMPORTANT: Oracle must be installed in the same location on all cluster des that will run the package. B. Supported versions i) Single instance Oracle Database Oracle Data Guard 10gR2,11gR1 10gR2,11gR1 11gR2 ECMT B B B Serviceguard A A A A HP-UX OS 11i v2,11i v3 11i v3 11i v3 Unless otherwise stated, this toolkit supports all storage and volume managers that Serviceguard supports. At the time of publication, this version of the toolkit supports the listed software and OS versions. For the most recent information, see the Compatibility matrix available at: NOTE:

3 VxVM/CVM and CFS will t be supported in this release. NOTE: This version of toolkit supports only modular packages. C. Setting Up Oracle Data Guard: Oracle Data Guard provides the management, monitoring, and automation software to create and maintain one or more standby databases to protect Oracle data from failures, disasters, human error, and data corruptions. Oracle Data Guard is included with the Enterprise Edition and Personal Edition of the Oracle database software. A Data Guard configuration consists of one production database and up to nine standby databases. There are two types of standby databases. i. Physical Standby database ii. Logical Standby database The Active Data Guard option, first available with Oracle Database 11g, enables a physical standby database to be used for read-only applications while simultaneously receiving updates from the primary database. NOTE: This README assumes that Oracle is installed on all the clusters des, by the user 'oracle' and that shared storage is already configured. For information on configuring Oracle Data Guard can be obtained from the Oracle website: D. Setting up the toolkit - Toolkit Overview This toolkit has to be used in combination with the ECMT Oracle module. Hence, ensure that ECMT B has been installed. After installing the ODG toolkit, two scripts (hadg.sh and hadg.conf) and this README file will be in the /opt/cmcluster/toolkit/dataguard directory. Two more scripts (tkit_module.sh and tkit_gen.sh) and one file (dataguard.1) which are used for modular packaging will be installed in the /etc/cmcluster/scripts/tkit/dataguard directory and /etc/cmcluster/modules/tkit/dataguard respectively.

4 These scripts are: - hadg.conf (user configuration file) This script contains a list of pre-defined variables that the user must customize for use with a particular database instance. This is a configuration file which is read by the toolkit script, hadg.sh. The following variables are contained in hadg.conf: TKIT_DIR: and directs directory. dataguard.debug ORACLE_HOME: installed. This directory is synymous to the package directory holds the toolkit configuration file. This parameter "cmapplyconf" to generate the hadg.conf file under this To put toolkit into maintenance mode, create file under this directory. The base directory of Oracle where oracle is ORACLE_ADMIN: User name of the Oracle database administrator. This will be used for starting and stopping of the database. For Example: ORACLE_ADMIN=oracle SID_NAME: oracle START_MODE: Oracle database. "mount", this parameter The Oracle session name. This uniquely identifies an database instance. This parameter determines the startup mode for The default value is "open". It can take the options - "mount" or "open". NOTE: For the ODG toolkit, always specify the value of as "mount". MAINTENANCE_FLAG: Maintenance flag is used to bring this toolkit into maintenance mode. If set to yes, this will enable maintenance feature in the toolkit. for a file exists and if and database Serviceguard Toolkit for Oracle Data Guard will look out "dataguard.debug" in the package directory. If the file if maintenance feature is enabled, monitoring is paused,

5 instance may be brought down for maintenance. The package would t be failed over to the adoptive de even though the instance has been brought down for maintenance. After the maintenance work, make sure that the instance is brought up properly. Delete the file dataguard.debug the package directory. This would enable toolkit to continue monitoring the Database server application. Note if Maintenance flag is set to "" then the above feature would t be available and toolkit cant be brought into maintenance mode. MONITOR_INTERVAL: The time interval, in seconds, this script will wait between checks to make sure that the Oracle instance is running. The default value is 30 seconds. TIME_OUT: The time for which the toolkit waits for a completion of a rmal shutdown before initiating forceful halt of the application. The TIME_OUT variable is used to protect against a worst case scenario where a hung database or ASM instance prevents the halt script from completing, therefore preventing the standby de from starting the instance TIME_OUT variable has effect on package failover times. The default value is 30 seconds. ACTIVE_STANDBY: This parameter determines whether the database instance is an active standby or t. The Active Data Guard Option available with Oracle Database 11g Enterprise Edition enables you to open a physical standby database for read-only access. This parameter can be set to 'yes' or ''. The default value is '' DG_BROKER: Specify whether the Oracle Data Guard broker is to be used or t. Oracle Data Guard Broker management is t supported in this release. Hence this parameter must be set to "". The default value is "".

6 START_STANDBY_AS_PRIMARY: Specify whether the standby database has to be started as the primary database by failing over the primary database. This parameter can be used in a Continentalcluster environment; the value of this parameter has to be 'yes' in the package configuration file of the recovery package. ALERT_MAIL_ID: This parameter is used to specify the address for sending alerts. - Main Script (hadg.sh) This script contains a list of internally used variables and functions that support the starting and stopping of an Oracle data guard instance. This script will be called by tkit_module.sh to perform the following: - On package startup, it starts the data guard instance primary/standby) and launches monitor processes. - On package halt, it stops the data guard instance and monitor process. NOTE: The following three scripts are used only during the modular method of packaging. - Attribute Definition File (dataguard) The ADF is used to generate a package ASCII template file. - Module Script (tkit_module.sh) This script is called by the Master Control Script and acts as an interface between the Master Control Script and the Toolkit interface script (hadg.sh). It is also responsible for calling the Toolkit Configuration File Generator Script (described below). - Toolkit Configuration File Generator Script (tkit_gen.sh) This script is called by the Module Script when the package configuration is applied using 'cmapplyconf' to generate the user configuration file (hadg.conf) in the package directory (TKIT_DIR). E. Oracle Data Guard Package Configuration Example This section explains an Oracle Data Guard package configuration example. Follow the instructions in the chapter "Building an HA Cluster Configuration" in the manual

7 "Managing HP Serviceguard" to create the logical volume infrastructure on shared disks. The disk must be available to all clustered des that will be configured to run ODG toolkit. Create file systems on all logical volumes on the volume groups. The ODG toolkit can be configured in one of the following methods: a) Install directory method: Here the scripts remain in the installation directory. b) configuration directory method: Here the user has to copy the scripts from the installation directory (including contents of subdirectories) to the configuration directory and define this location in the parameter "TKIT_DIR" in the package configuration file. The users can modify the scripts in the configuration directory to add any specific requirements. Serviceguard first tries to use the hadg.sh script in the configuration directory. If the script is t found in the configuration directory, it takes it from the installation directory. The example configuration uses the installation directory mode operation. This example on ODG Package Setup and Configuration is for an Oracle Data Guard configuration using LVM. a. Creating a Package configuration Create two packages one for the primary database on the primary cluster and the other for the standby database on the standby cluster. Create a directory in /etc/cmcluster, for example "dataguard". This directory will eventually become TKIT_DIR and "cd" to this directory. Run the following commands to create the Package configuration file Templates. Note that the ODG toolkit has to be used in combination with the ECMT Oracle toolkit. A combinational package is the one in which two applications are packaged together by combining their respective Serviceguard modules into one package. The ODG Toolkit module can be combined with the ECMT Oracle Toolkit module as follows: cmmakepkg -m ecmt/oracle/oracle - m tkit/dataguard/dataguard dgpkg.conf b. Specifying configuration parameters in the package

8 Oncce the package configuration file has been created, the user must specify various parameter values. The parameters which are to be modified in dgpkg.conf specifically for this configuration are only shown here. Note that the package configuration file shown below contains attributes of both the ECMT Oracle toolkit and the ODG tookit. (NOTE: The following attributes are specific to the Oracle Toolkit in ECMT.) "package_name" is the name that is used to identify the package. Package names must be unique within a cluster. package_name dgpkg "package_description" specifies the application that the package runs. package_description "Serviceguard Package" "package_type" is the type of package. The package_type attribute specifies the behavior for this package. package_type failover "run_script_timeout" is the number of Seconds allowed for package to start. "halt_script_timeout" is the number of Seconds allowed for package to halt. run_script_timeout 300 halt_script_timeout 610 "script_log_file" is the full path name for the package control script log file. script_log_file /etc/cmcluster/dataguard/pkg.log - -- Define package configuration directory ecmt/oracle/oracle/tkit_dir /etc/cmcluster/dataguard Define the instance type

9 ebs ecmt/oracle/oracle/instance_type database Define Oracle home ecmt/oracle/oracle/oracle_home /var/orahome Define user name of Oracle database administrator ecmt/oracle/oracle/oracle_admin oracle Define oracle session name ecmt/oracle/oracle/sid_name ORCL Define Oracle database startup mode ecmt/oracle/oracle/start_mode mount Define whether the database instance is using ASM or t ecmt/oracle/oracle/asm Define ASM disk groups used by the database instance ecmt/oracle/oracle/asm_diskgroup Define the volume groups used in the ASM disk groups for the database instance ecmt/oracle/oracle/asm_volume_group The ASM home ecmt/oracle/oracle/asm_home Define user name of Oracle ASM administrator ecmt/oracle/oracle/asm_user oracle The ASM session name ecmt/oracle/oracle/asm_sid

10 Define whether configured listener has to be started with the server ecmt/oracle/oracle/listener yes Define the Oracle listener name(s) ecmt/oracle/oracle/listener_name LISTENER_ORCL Define the listener password(s) ecmt/oracle/oracle/listener_pass ecmt/oracle/oracle/listener_restart ecmt/oracle/oracle/listener_restart (NOTE: The following are the Service commands for the combinational package) service_name oracle_service_test service_cmd "$SGCONF/scripts/ecmt/oracle/tkit_module.sh oracle_monitor" service_restart ne service_fail_fast_enabled service_halt_timeout 300 service_name oracle_listener_service_test service_cmd "$SGCONF/scripts/ecmt/oracle/tkit_module.sh oracle_monitor_listener" service_restart ne service_fail_fast_enabled service_halt_timeout 300 service_name oracle_hang_service_test service_cmd "$SGCONF/scripts/ecmt/oracle/tkit_module.sh oracle_hang_monitor 30 failover" service_restart ne service_fail_fast_enabled service_halt_timeout 300 service_name dataguard_service_test service_cmd "$SGCONF/scripts/tkit/dataguard/tkit_module.sh dataguard_monitor" service_restart ne service_fail_fast_enabled service_halt_timeout 300 (NOTE: The following attributes are specific to the ODG toolkit)

11 Define ACTIVE_STANDBY tkit/dataguard/dataguard/active_standby Define DG_BROKER tkit/dataguard/dataguard/dg_broker Define START_STANDBY_AS_PRIMARY tkit/dataguard/dataguard/start_standby_as_primary Define address for sending alerts tkit/dataguard/dataguard/alert_mail_id "vg" is used to specify which volume groups are used by this package. vg vgora "fs_name", "fs_directory", "fs_mount_opt", "fs_umount_opt", "fs_fsck_opt", and "fs_type" specify the filesystems which are used by this package. fs_name /dev/vgora/lvol1 fs_directory /oradb fs_type "vxfs" fs_mount_opt "-o rw" fs_umount_opt fs_fsck_opt F. Adding the Package to the Cluster After the setup is complete, add the package to the Serviceguard cluster and start it up. $ cmapplyconf -P dgpkg.conf $ cmmodpkg -e -n <de1> -n <de2> dgpkg $ cmmodpkg -e dgpkg For more information on adding a package to the cluster, see the "Managing HP Serviceguard" document available at: G. Oracle Data Guard Maintenance

12 There might be situations, when the Oracle Data guard database instances have to be taken down for maintenance purposes like changing configuration, without having the instance to migrate to standby de. The following procedure should be followed: The maintenance feature is enabled only when the configuration variable MAINTENANCE_FLAG is set to "yes" in the toolkit configuration file. Note: The example assumes that the package name is dgpkg, package directory is /etc/cmcluster/pkg/dgpkg and the ORACLE_HOME is configured as /orahome. 1. Disable the package switching through cmmodpkg command. $ cmmodpkg -d dgpkg 2. Pause the monitor script Create an empty file /etc/cmcluster/pkg/dgpkg/dataguard.debug as shown below: $ touch /etc/cmcluster/pkg/dgpkg/dataguard.debug Toolkit monitor scripts (both database instance and listener monitoring scripts) which continuously monitor Oracle Data Guard process daemons' processes, would w stop monitoring these daemon processes. The messages, "Serviceguard Toolkit for Oracle Data Guard pausing Dataguard monitoring and entering maintenance mode" appears in the Serviceguard Package Control script log. 3. If required, stop the database instance(s) as shown below: $ cd /etc/cmcluster/pkg/dgpkg/ $ $PWD/hadg.sh stop 4. Perform maintenance actions (Example: changing the configuration parameters in the parameter file of the Oracle instance. If this file is changed, please remember to distribute the new file to all cluster des). 5. Re-start the Oracle database instance again if you manually stopped it prior to maintenance $ cd /etc/cmcluster/pkg/dgpkg/ $ $PWD/hadg.sh start

13 6. Allow monitoring scripts to continue rmally as shown below: $ rm -f /etc/cmcluster/pkg/dgpkg/dataguard.debug The message "Starting Oracle Data Guard monitoring again after maintenance" appears in the Serviceguard Package Control script log. 7. Enable the package failover $ cmmodpkg -e dgpkg Note: If a package failure occurs during maintenance operations, the package does t automatically fail over to an adoptive de. You must manually start the package on the adoptive de. For more information on manually starting the package on an adoptive de, see the "Managing HP Serviceguard" guide available at: H. Using Serviceguard Toolkit for Oracle Data Guard in Continentalclusters environment: A Continental cluster provides an alternative disaster tolerant solution in which two clusters are geographically dispersed. For information about setting up Continentalclusters, see the document "Designing Disaster Recovery HA Clusters Using Metrocluster and Continentalclusters Manual" available at In Continentalclusters environment the user needs to configure a recovery group which consists of 1. Primary package: This package has to be created for the primary database on the primary cluster. Create a combinational package for the primary database with ECMT Oracle toolkit and the ODG toolkit. 2. Data receiver package: This package has to created for the standby database on the recovery cluster. Create a combinational package for the standby darabase using ECMT Oracle toolkit and the ODG toolkit. 3. Recovery package: This package has to be created for the standby database on the recovery cluster. Create a combinational package for the standby database using ECMT Oracle toolkit and the ODG toolkit. Note that the package attribute "START_STANDBY_AS_PRIMARY" has to be set to "yes" in case of a recovey package.

14 When the primary cluster fails, run "cmrecovercl" command on the recovery cluster to bring up the recovery package. I. Using Serviceguard Toolkit for Oracle Data Guard in Metrocluster and Extended Distance Cluster environments: The Primary database can be configured inside a Metrocluster or an Extended Distance Cluster. The standby database can be configured outside the Metrocluster/Extended Distance Cluster as a standalone database or as part of a Serviceguard cluster. For information about setting up Metrocluster, see the document "Designing Disaster Recovery HA Clusters Using Metrocluster and Continentalclusters Manual" available at J. Verification of the setup: This section provides the guidelines to verify if the Oracle Data Guard package has been configured properly or t. The following steps helps the user to troubleshoot the possible causes for the package failure. 1. Verify Oracle Data Guard setup: To verify a Data Guard configuration, check if the standby database is able to receive the redo logs. Also check if the RFS process is running on the standby database by querying the table "V$MANAGED_STANDBY". If the standby site is t receiving the logs, obtain information about the archiving status of the primary database by querying the V$ARCHIVE_DEST view. Check especially for error messages. For example, enter: SQL> SELECT dest_id "ID", 2> status "DB_status", 3> destination "Archive_dest", 4> error "Error" 5> FROM v$archive_dest; ID DB_status Archive_dest Error VALID /vobs/oracle/work/arc_dest/arc 2 ERROR standby1 ORA-16012: Archivelog standby database identifier mismatch 3 INACTIVE 4 INACTIVE 5 INACTIVE

15 5 rows selected If the output of the query does t help, check the following list of possible issues. * The service name for the standby instance is t configured correctly in the tnsnames.ora file at the primary site. * The service name listed in the LOG_ARCHIVE_DEST_n parameter of the primary initialization parameter file is incorrect. * The LOG_ARCHIVE_DEST_STATE_n parameter specifying the state of the standby archiving destination has the value DEFER. * The listener.ora file has t been configured correctly at the standby site. * The listener is t started. * The standby instance is t started. * You have added a standby archiving destination to the primary initialization parameter file, but have t yet enabled the change. * You used an invalid backup as the basis for the standby database (for example, you used a backup from the wrong database, or did t create the standby control file using the correct method). Also, check the Oracle Alert log for errors. 2. Verify the Toolkit setup: Check whether the package configuration attributes specified in the package configuration file are valid or t. If the package fails after the command "cmrunpkg" has been used, then check for errors in the package log file. Copyright 2010 Hewlett-Packard Development Company, L.P. Confidential computer software.valid license from HP required for possession, use, or copying. Consistent with FAR and , Commercial Computer Software, Computer Software Documentation, and Technical Data forcommercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without tice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall t be liable for technical or editorial errors or omissions contained herein.