Leveraging SAN Volume Controller (SVC) Advanced Copy Services for Oracle User-Managed Backup and Recovery Oracle RAC 10g with ASM

Transcription

1 Leveraging SAN Volume Controller (SVC) Advanced Copy Services for Oracle User-Managed Backup and Recovery Oracle RAC 10g with ASM Authors: Bob Gonzalez and Betty Mason IBM Systems and Technology Group Open Systems Lab, San Jose, California July All Rights Reserved. Version July 25, 2008 All trademarks or registered trademarks mentioned herein are the property of their respective holders

2 Table of contents Abstract...1 Introduction...1 Assumptions... 1 Technology...2 SVC virtualization overview... 2 Oracle components overview... 3 Oracle ASMLib for LINUX...3 Oracle Automatic Storage Management (ASM)...3 Oracle Clusterware (required by Oracle RAC)...4 Overview of Oracle Database Backup and Recovery...5 Components and commands for Backup and Recovery scenarios... 6 Configuration and setup...7 Lab hardware and software configurations... 7 Oracle RAC host nodes...7 SVC nodes...8 SVC VDisk configuration...8 Oracle software installation...8 Oracle Clusterware disks...9 Configuration of primary Oracle RAC servers...9 Oracle ASM and Oracle Database instances...9 RMAN and FlashCopy Configuration of Clone or Disaster Recovery Oracle RAC nodes...12 Oracle ASM diskgroup configuration...13 Preparation of Clone or Disaster Recovery Oracle RAC...13 Clone or Disaster Recovery Oracle ASM setup...13 Clone or Disaster Recovery database instance requirements...14 SVC Advanced Copy Services overview and configuration SVC FlashCopy characteristics...14 Examples of Advanced Copy Services FlashCopy commands...16 SVC Metro Mirror characteristics...19 Examples of Advanced Copy Services Metro Mirror commands...20 Creating SVC intercluster partnership...20 Backup and Recovery by example...24 Backup and Recovery procedures and validations...24 Scenario 1: Loss of DATA_DG containing data files Scenario 2: Loss of the entire database Scenario 3: FlashCopy Clone of an Oracle RAC 10g with ASM database Scenario 4: Metro Mirror Clone of a Oracle RAC 10g database Scenario 5: Backup and restore the OCR and voting disks Summary...43 Appendix A: Oracle terms, components and definitions...44 Appendix B: Resources...47 Trademarks and special notices

3 Abstract This white paper explains the use of IBM System Storage SAN Volume Controller (SVC) Advanced Copy Services for backing up and recovering Oracle Real Application Clusters 10g (RAC) databases that use Oracle Database 10g Automatic Storage Management (ASM). Both Oracle RAC and Oracle ASM technologies introduce some differences with regard to Oracle Backup and Recovery that uses IBM FlashCopy technologies when compared to singleinstance, file system-based Oracle configurations. You can leverage SVC Advanced Copy Services in any production shop that runs Oracle RAC 10g databases on Oracle ASM. This white paper shows the robustness, speed, and ease with which you can back up, restore, and recover databases, no matter their size. Introduction The procedures in this white paper are based on an Oracle User Group white paper that is titled Backup and Recovery Generic Guidelines with Point-in-Time Copy Solutions and Automatic Storage Management (located at: This white paper shows the usage of IBM FlashCopy and Metro Mirror to do the following tasks: Backup and restore an Oracle Real Application Clusters (RAC) database with Oracle Database 10g Automatic Storage Management (ASM) Clone an Oracle RAC database with Oracle ASM to a local Oracle RAC Clone an Oracle RAC database with Oracle ASM to a remote IBM System Storage SAN Volume Controller (SVC) and Oracle RAC Backup and recover Oracle RAC cluster services information For demonstration purposes, four-node Oracle RACs were used. The process was repeated on various operating systems (OSs) to show the OS-independence of the procedures. On all of the test beds, Oracle Clusterware, Oracle ASM and Oracle Database binaries were installed in separate local directories. On the SVC cluster, VDisks were allocated for the Oracle Cluster Registry, Voting disks and the Oracle ASM disk groups. The VDisks were created by using a consistent naming convention for understanding the relationships between SVC VDisk, SVC copy relationships and the Oracle ASM disk groups. One VDisk was allocated for each Oracle RAC to serve as the mountable backup disk. Consistent naming allows the user to know at a glance what portion of the database is acted upon in the backup-recovery scenarios. Assumptions The intended audience for this white paper is any technical leader, system administrator, storage administrator or Oracle database administrator (DBA) in a production environment who is experienced with backup and recovery concepts and procedures. After reading this white paper, the technical staff will understand how to back up and recover an Oracle RAC 10g database with Oracle ASM on a storage area network (SAN) Volume Controller by using Advanced Copy Services. The Backup and Recovery by example section of this white paper is written in a step-by-step manner to explain the process. Thus, you can incorporate it into any backup-recovery strategy. 1

4 Technology In this section, you will find an overview of the concepts related to SVC virtualization, relevant Oracle components and the Oracle commands for the backup and recovery scenarios used in this text. SVC virtualization overview The IBM System Storage SAN Volume Controller nodes (also called engines) are the hardware elements of the SVC. The SAN Volume Controller combines pairs of nodes into a high-availability cluster. Each of the nodes in the cluster contains high-speed memory that serves as the cluster cache. You can only install the SVC nodes in pairs for high availability. The SVC solution is designed to reduce both the complexity and costs of managing your SAN-based storage. The SAN Volume Controller helps you to achieve the following goals: Simplify management and increase administrator productivity by consolidating storagemanagement intelligence from disparate systems into a single view. Improve Disaster Recovery and business continuance by applying and managing replication services across disk systems within the SAN. Simplify device-driver configuration on hosts, such that all hosts within your network use the same IBM device driver to access all supported storage systems through the SAN Volume Controller. Backup: FlashCopy does not impact your backup time, but it allows you to create a point-in-time copy (PiT) that is a consistent backup across VDisks, with a minimum of downtime for your source host. You can then mount the FlashCopy target on a different host or the backup server. Using this procedure, the backup speed becomes less important. The reason for this is that the backup time does not require downtime for the host, which is dependent on the source VDisks. Restore: You can keep periodically created FlashCopy targets online to provide very fast restoration of a PiT-consistent backup. When a background copy process has finished and a complete restoration is needed, it is possible to delete the FlashCopy mappings and create corresponding FlashCopy mappings in the opposite direction. You can use this procedure to very quickly restore the PiTconsistent backup that was obtained from the preceding FlashCopy process. Move and migrate data: When you need to move a consistent dataset from one host to another, FlashCopy can facilitate this action with a minimum of downtime for the host application that is dependent on the source VDisk. The dataset that has been created on the FlashCopy target is immediately available, as is the dataset on the source VDisk. 2

5 FlashCopy: FlashCopy is also known as point-in-time copy (PiT). You can use this technique to help solve the difficult problem of making a consistent copy of a dataset that is constantly updated. When you invoke FlashCopy, the resulting data at the target appears as if the copy were made instantaneously. Consistency for FlashCopy: After a FlashCopy mapping starts, the source and target VDisks are, by definition, consistent. The mapping represents a single point in time and every FlashCopy mapping in a consistency group represents the same point in time. Metro Mirror: Metro Mirror works by establishing a synchronous copy relationship between two VDisks of equal size. Intercluster Metro Mirror operation requires a pair of SVC clusters that are separated by a number of moderately high bandwidth links. Consistency for Metro Mirror: When you start Metro Mirror relationships, they are inconsistent. They must go through a synchronization stage before they become consistent. After this process is complete, the relationships stay consistent as long as the links between the clusters remain operational. Oracle components overview This section provides a summary of the Oracle components that are used for this white paper. Oracle ASMLib for LINUX The Oracle Automatic Storage Management library (ASMLIB) driver simplifies the configuration and management of the disk devices by eliminating the need to rebind the raw devices that are used with Oracle ASM each time the system restarts. Oracle Automatic Storage Management (ASM) Oracle ASM is a volume manager and a file system for Oracle database files that supports singleinstance Oracle Database and Oracle Real Application Clusters (Oracle RAC) configurations. Oracle ASM is the Oracle recommended storage-management solution that provides an alternative to conventional volume managers, file systems and raw devices. Oracle ASM disk groups Oracle ASM uses disk groups to store data files. An Oracle ASM disk group is a collection of disks that Oracle ASM manages as a unit. Within a disk group, Oracle ASM exposes a file-system interface for Oracle database files. The content of files that are stored in a disk group are evenly distributed, or striped, to prevent hot spots and to provide uniform performance across the disks. 3

6 Oracle Clusterware (required by Oracle RAC) This section provides a summary of the Oracle Clusterware that is required by Oracle RAC. Oracle Real Application Clusters (RAC): Multiple Instance Systems Some hardware architectures (that is, shared-disk systems) enable multiple computers to share access to data, software or peripheral devices. Oracle RAC takes advantage of such architectures by running multiple instances that share a single physical database. Oracle Clusterware is required to run Oracle RAC. Oracle Cluster Ready Services (CRS) CRS is a component of Oracle Clusterware and is the primary program for managing highavailability operations within a cluster. Anything that the CRS process manages is known as a cluster resource. The CRS process manages cluster resources based on the resource's configuration information that is stored in the Oracle Cluster Registry (OCR). This includes start, stop, monitor and failover operations. The CRS process generates events when a resource status changes. CRS monitors the Oracle instance, listener and so on, and automatically restarts these components when a failure occurs. By default, the CRS process makes five attempts to restart a resource and then does not make further attempts if the resource does not restart. Event management (EVM) EVM is a background process that publishes events that CRS creates. Oracle Notification Service (ONS) ONS is a publish-and-subscribe service that Oracle Clusterware uses for publishing fast application-notification (FAN) events. RACG RACG extends Clusterware to support Oracle-specific requirements and complex resources. It runs server-callout scripts when FAN events occur. Process Monitor Daemon (OPROCD) This process is locked in memory to monitor the cluster and to provide I/O fencing. OPROCD performs its check, stops running and, if the wake up is beyond the expected time, resets the processor and reboots the node. An OPROCD failure results in Oracle Clusterware restarting the node. OPROCD uses the hangcheck timer on Linux platforms. Oracle Cluster Synchronization Services (CSS) CSS is the component of Oracle Clusterware that handles group membership for the cluster. Oracle ASM requires the use of CSS to enable synchronization between an Oracle ASM instance and the database instances that rely on it for database file storage. 4

7 Overview of Oracle Database Backup and Recovery Oracle provides various mechanisms for the following processes: Database recovery that is required because of different types of failures Flexible recovery operations that suit any situation Availability of data during backup and recovery operations that allow users of the system to continue working Types of database failures Several circumstances can halt the operation of an Oracle database. The most common types of failure are described in Table 1. Recovering from media failure is the subject matter that is covered in this white paper. Failure User error Statement failure Process failure Instance failure Media (disk) failure Description This failure requires data to be recovered to a point in time before the error occurred. For example, if a user accidentally drops a table, the data file can be recovered to the instant in time before the table was dropped. This failure occurs when there is a logical failure in the handling of a statement in an Oracle program. When a statement failure occurs, any effects of the statement are automatically undone by Oracle and control is returned to the user. This results from a failure in a user process that accesses the Oracle database, such as an abnormal disconnection or process termination. The PMON background process automatically detects the failed user process, rolls back the uncommitted transaction of the user process, and releases any resources that the process was using. This failure occurs when a problem arises that prevents an instance from continuing work. Instance failure can result from a hardware problem (such as a power outage) or a software problem (such as an OS failure). After an instance failure, Oracle automatically performs instance recovery. An error can occur when trying to write or read a file on disk that is required to operate the database. For example, a disk failure requires you to restore database files from backup media and then to perform media recovery. The user must initiate media recovery. Table 1. Types of database failures Oracle provides for complete media recovery from all possible types of hardware failures, including disk failures. Options are provided so that a database can be completely or partially recovered to a specific point in time. 5

8 Components and commands for Backup and Recovery scenarios The following structures are required for complete recovery from an instance or disk failure: the database data files, online redo logs, archived redo logs, control files and database backups. Database datafiles: These are structures that contain the actual data that the database manages. Redo Log: The redo log is a set of files that contain changes that are made to the data within the database. The redo log can consist of the online redo log and the archived redo log. Archived redo log: The filled online redo log files are archived to the archived redo logs before they are reused. If running in Archivelog mode, the database can be backed up while it is open and available for use. However, additional administrative operations are required to maintain the archived redo log. Control files: The control files include information about the file structure of the database and the current log-sequence number. During normal recovery procedures, the information in a control file guides the automatic progression of the recovery operation. Database backups: Because one or more files can be physically damaged as the result of a disk failure, media recovery requires the restoration of the damaged files from the most recent OS backup of a database. Parameter files: Parameter files contain a list of configuration parameters for that instance and database. The parameter file can be either a text init.ora file or a binary-format server-parameter file (spfile). Hot-backup mode: Data files are marked with indicator that shows recovery is needed from x point forward. Transactions cause entire database blocks to be written, in full, to the redo logs. Putting a database in hot-backup mode allows the use of FlashCopy to capture the point-in-time copy. Archive log current: In an Oracle RAC environment, this writes current redo logs for all threads to the archived logs. oracleasm: The oracleasm-support package provides the utilities that are used to get the ASM driver up and running. Its functions include scandisk, listdisk and createdisk. srvctl: This is an Oracle-supplied tool to control resources that are registered with Oracle Clusterware. Only the database owner (oracle userid) should run it. ocrconfig: This is the command that is supplied by Oracle to export, import or restore backups of the OCR disks that contain cluster information about registered services and Oracle RAC nodes. crs_stat: This is the utility that is supplied by Oracle o show the status of the cluster resources. RMAN: Oracle Recovery Manager is a utility that is supplied by Oracle for backup and recovery. RMAN is the preferred Oracle method for backup and recovery. SQL*Plus: This is the command-line tool that is used to interface with the instance, the database and all the components that are contained in the database. asmcmd: This is a utility that is supplied by Oracle to manage ASM disks. 6

9 Configuration and setup This section describes the configuration and setup procedures for these tests. Lab hardware and software configurations This section explains the hardware and software configurations that were used in the lab for this white paper. Oracle RAC host nodes Table 2 describes the three 4-host node clusters that were used in the Backup and Recovery exercises. Oracle RAC server type Processor 4 x IBM System x x IBM System x3550 IBM eserver pseries p690 2 x Dual-Core AMD 2 x Dual-Core Intel Xeon 4 Opteron processor 2222 S processor 3.00 GHz MHz Memory 4 GB 12 GB 4 GB Host bus QLE2462 Emulex 4G Model IBM Emulex LP9002 HBA adapter (HBA) model 42C port PCIe FC HBA for System x Operating Red Hat Enterprise Linux SUSE Enterprise Linux IBM AIX 5.3 system (RHEL) AS4 U4 Server 9 SP3 Kernel version ELsmp smp AIX 5.3 ML6 Multipath software device-mapper-multipath RHEL4 HBA driver HBAAP(I) v2.1.c, device-mapper Subsystem device driver path control module (SDD-PCM) IBM HBA firmware A6 (Z2F2.50A6) 02E01974 Cluster file None None None system Oracle software Oracle Database, ASM and Oracle Clusterware Oracle Database, ASM and Oracle Clusterware Oracle Database, ASM and Oracle Clusterware Logical configure 4-node RAC members ALPHA, BETA, GAMMA, DELTA Table 2. Oracle RAC host nodes 4-node RAC members ALPHA, BETA, GAMMA, DELTA 4-node RAC members ALPHA, BETA, GAMMA, DELTA 7

10 SVC nodes Table 3 describes the two SVC clusters that were used in the Backup and Recovery exercises. Role Production cluster Disaster and recovery cluster Storage name OSL_SVC_11 ARC_SVC_JMT1 Storage IP Model F F4 Micro code level (build ) (build ) Logical configuration 2-node SVC cluster 2-node SVC cluster Table 3. SVC clusters SVC VDisk configuration This section of the white paper explains the configuration of the SVC VDisk. Production SVC cluster OSL_SVC_11 Create VDisks by using the following naming conventions and assigning them to Oracle RAC nodes: PRIME_DATA_ASM1, PRIME_DATA_ASM2, PRIME_DATA_ASM3 PRIME_ARCH_ASM4, PRIME_ARCH_ASM5, PRIME_ARCH_ASM6 PRIME_LOGS _ASM7, PRIME_LOGS _ASM8, PRIME_LOGS _ASM9 Disaster Recovery cluster ARC_SVC_JMT1 Create VDisks by using the following naming conventions and assigning them to remote Oracle RAC nodes: PRIME_DATA_ASM1, PRIME_DATA_ASM2, PRIME_DATA_ASM3 PRIME_ARCH_ASM4, PRIME_ARCH_ASM5, PRIME_ARCH_ASM6 PRIME_LOGS _ASM7, PRIME_LOGS _ASM8, PRIME_LOGS _ASM9 For this demonstration, the same VDisk names are used on both the production cluster, as well as the Disaster Recovery cluster. Oracle software installation As shown in Table 4, Oracle binaries are installed in three Oracle homes. Clusterware home ASM home Database home CRS_HOME=/u01/crs/product/10.2/crs ORACLE_HOME=/u01/asm/product/10.2/asm ORACLE_HOME=/u01/db/product/10.2/db Table 4. Directories for Oracle binaries 8

11 Oracle Clusterware disks When installing Oracle Clusterware, follow the Oracle installation documentation for Clusterware and define five 1 GB VDisks: two for OCR and three for Vote. Ensure that each host node has the shared VDisk identified consistently on each. These are labeled Shared-cluster services disks in Table 5. Configuration of primary Oracle RAC servers Table 5 shows the storage allocations for each of the host cluster nodes. Host-node name ALPHA BETA GAMMA DELTA Shared-cluster services disks 2 OCR 3 Vote 2 OCR 3 Vote 2 OCR 3 Vote 2 OCR 3 Vote Shared-data disks Backup disk mounted on one node at a time Multipath driver (same driver on all nodes) Device mapper, or SDD PCM Device mapper, or SDD PCM Device mapper, or SDD PCM Device mapper, or SDD PCM Oracle Stack configured on local host disk CRS_HOME/u01/crs/product/10.2/crs ASM HOME /u01/asm/product/10.2/asm DATABASE_HOME /u01/db/product/10.2/db +ASM Instance name per host +ASM1 +ASM2 +ASM3 +ASM4 Database name on host cluster PRIME Database instance name per host PRIME1 PRIME2 PRIME3 PRIME4 Table 5. Server nodes for four-node cluster Oracle ASM and Oracle Database instances Create ASM disk groups DATA_DG, LOGS_DG, ARCH_DG. Assign the corresponding disks to the ASM disk groups and populate them with database components, as shown in Table 6. ASM disk group ASM disk name (alias on host ) Contains database components DATA_DG DATA_ASM_1 Data files DATA_ASM_2 DATA_ASM_3 ARCH_DG ARCH_ASM_4 Archived redo logs ARCH_ASM_5 Spfile ARCH_ASM_6 Control file (1) LOGS_DG LOGS_ASM_7 redo logs LOGS_ASM_8 control file (2,3) LOGS_ASM_9 Table 6. ASM disk-group configuration and contents Additional directories +ARCH_DG/PRIME/rman_backups 9

12 RMAN and FlashCopy This white paper describes how to use FlashCopy and Metro Mirror for backing up the database. RMAN is only used to make backup copies of the archive logs, control files and spfiles to either a disk that is mountable to any node on the Oracle RAC cluster or to a directory in an Oracle ASM disk group. The cloning scenario does provide the ability to also back up the database to other media through RMAN. This is done by bringing the database up to the mount state on the Clone or Disaster Recovery site by using a backup control file, connecting to RMAN, and backing up the database (for example: database files, control files, spfiles, and archived logs). This white paper does not describe this particular process. RMAN Recovery Catalog Create an RMAN recovery catalog. Then, register the database in the catalog. For this white paper, the recovery catalog database is called utils. User rman (with the password of opensezme) owns the recovery catalog. Use the following RMAN commands to connect to the catalog and register the database. Example 1: Connecting to RMAN [oracle@alpha ~]$ rman target / catalog rman/opensezme@utils; RMAN> register database; After registering the database, run the report schema command to view the registration information in the RMAN catalog output, as shown in Example 2. Example 2: RMAN report schema [oracle@alpha ~]$ rman target / catalog rman/opensezme@utils; Recovery Manager: Release Production on Thu Feb 21 14:24: Copyright (c) 1982, 2005, Oracle. All rights reserved. connected to target database: PRIME (DBID= ) connected to recovery catalog database RMAN> report schema; starting full resync of recovery catalog full resync complete Report of database schema List of Permanent Datafiles =========================== File Size(MB) Tablespace RB segs Datafile Name SYSTEM YES +DATA_DG/prime/datafile/system UNDOTBS1 YES +DATA_DG/prime/datafile/undotbs SYSAUX NO +DATA_DG/prime/datafile/sysaux USERS NO +DATA_DG/prime/datafile/users EXAMPLE NO +DATA_DG/prime/datafile/example UNDOTBS2 YES +DATA_DG/prime/datafile/undotbs UNDOTBS3 YES +DATA_DG/prime/datafile/undotbs UNDOTBS4 YES +DATA_DG/prime/datafile/undotbs SOE NO +DATA_DG/prime/datafile/soe SOEINDEX NO +DATA_DG/prime/datafile/soeindex List of Temporary Files ======================= File Size(MB) Tablespace Maxsize(MB) Tempfile Name TEMP DATA_DG/prime/tempfile/temp

13 RMAN backup directories For this set of scenarios, the choice was to mount a disk that contained the backup directory to the Oracle RAC node that performs the backup. When recovering a database, it is important to mount the backup disk to the cluster server that performs the recovery. This example uses a disk that is mounted on the server that performs the backup: /u01/backup/prime/ - spfile, control file and archived logs This example also uses a directory in an ASM disk group: +ARCH_DG/PRIME/rman_backups - spfile, control file Example 3 shows the creation of a backup directory in an ASM disk group. Example 3: Create backup directory in ASM disk group [oracle@alpha ~]$ export ORACLE_HOME=/u01/asm/product/10.2/asm [oracle@alpha ~]$ asmcmd p ASMCMD [+] > mkdir +ARCH_DG/PRIME/rman_backups RMAN backup control file This white paper uses three types of control-file backups: trace, backup copy and RMAN dataset. For a trace file, use the following command: sql 'alter database backup controlfile to trace'; For a control-file backup file, use the following command: backup as copy format '/u01/backup/prime/ctl_copy_t%t_s%s_p%p.ctl' current controlfile; For an RMAN dataset, use the following command: backup format '/u01/backup/prime/cf_t%t_s%s_p%p' current controlfile; For an RMAN dataset on ASM, use the following command: backup format '+ARCH_DG/PRIME/rman_backups/cf_t%t_s%s_p%p' current controlfile; RMAN backup spfile Two spfile backups are used in this white paper: init.ora and RMAN dataset. For an init.ora file, use the following command: sql "create pfile=''/u01/backup/prime/init.ora.back'' from spfile"; For an RMAN dataset, use the following command: backup format '/u01/backup/prime/spfile_t%t_s%s_p%p' spfile; 11

14 For an RMAN dataset on ASM, use the following command: backup format '+ARCH_DG/PRIME/rman_backups/spfile_t%t_s%s_p%p' spfile; RMAN archive log backup To create an RMAN archive log backup in this white paper, use the following command: backup filesperset 10 format '/u01/backup/prime/al_t%t_s%s_p%p' (archivelog all); Configuration of Clone or Disaster Recovery Oracle RAC nodes Preparation of Clone or Disaster Recovery Oracle RAC requires the following tasks: 1. Oracle RAC nodes should have the same OS level. 2. You need to allocate SVC VDisks for Oracle Clusterware before the installation of binaries. 3. You must install Oracle binaries at the same patch level and in the same directory paths as the primary Oracle RAC. 4. You need to configure cluster services. 5. You must allocate storage disks from SVC for data in the same number and size as the primary cluster. Table 7 shows the component requirements for the Clone and Disaster Recovery cluster configuration reflecting the same setup as the primary cluster. Host-node name CL_ALPHA CL_BETA CL_GAMMA CL_DELTA Shared-cluster services disks 2 OCR 3 Vote 2 OCR 3 Vote 2 OCR 3 Vote 2 OCR 3 Vote Shared-data disks that are equal in number and size to a primary cluster Backup disk mounted on one node at a time that is equal in size to the primary cluster backup disk Multipath driver (same driver on all nodes) Device mapper, or SDD PCM Device mapper, or SDD PCM Device mapper, or SDD PCM Device mapper, or SDD PCM Oracle stack configured on local host disk CRS_HOME /u01/crs/product/10.2/crs ASM HOME /u01/asm/product/10.2/asm DATABASE_HOME /u01/db/product/10.2/db +ASM Instance name per host +ASM1 +ASM2 +ASM3 +ASM4 Database name Database instance name per host Table 7. Clone and Disaster Recovery Oracle RAC node configuration (server nodes for four-node cluster-software stack) 12

15 Oracle ASM diskgroup configuration Oracle ASM diskgroup configuration must be defined with the same number of disks and disk size as on the primary cluster (see Table 8). ASM diskgroup configuration Clone or Disaster Recovery cluster ASM disk group ASM disk name Contents DATA_DG DATA_ASM_1 DATA_ASM_2 ARCH_DG DATA_ASM_3 ARCH_ASM_4 ARCH_ASM_5 LOGS_DG ARCH_ASM_6 LOGS_ASM_7 LOGS_ASM_8 LOGS_ASM_9 Table 8. Oracle ASM disk group configuration for Clone or Disaster Recovery Oracle RAC Preparation of Clone or Disaster Recovery Oracle RAC Set up the cluster by duplicating the Oracle binaries at the same version and patch level as is the case at the primary site. The directory paths for the Oracle binary installations should be the same as at the primary site. SVC VDisks need to be allocated and configured for the Shared Oracle Clusterware disks, as specified in Table 7. Data disks must be allocated such that they are equal in number and size to the primary Oracle RAC. Clone or Disaster Recovery Oracle ASM setup This section of the white paper explains how to set up the Clone and Disaster Recovery Oracle ASM disk groups. Use Oracle Database Configuration Assistant (DBCA) to set up the ASM instances on the Clone or Disaster Recovery Oracle RAC. Do not define ASM disk groups. Use the Oracle srvctl tool to stop ASM instances on all target Oracle RAC nodes. On node 1 of the Clone or Disaster Recovery cluster, set the asm_diskgroups parameter such that it is equal to the primary cluster pfile asm_diskgroups parameter. For example, on this setup, the parameter is set as: asm_diskgroups='data_dg','logs_dg','arch_dg'. Copy this init.ora to the remaining Clone or Disaster Recovery nodes. 13

16 Clone or Disaster Recovery database instance requirements There are four important files to copy or edit on the Clone or Disaster Recovery Oracle RAC nodes to migrate the database from the primary: initprimex.ora, orapwprimex, oratab file and tnsnames.ora: Copy the initprimex.ora file (found at /u01/db/product/10.2/db/dbs/initprimex.ora), which contains a pointer to the spfile in the Oracle ASM. Copy the orapwprimex file (found at /u01/db/product/10.2/dbs/orapwprimex), which is the database password file. Check the oratab file and add the following (the syntax of +ASMx is dependent on the node): +ASMx:/u01/asm/product/10.2/asm:N PRIME:/u01/db/product/10.2/db:N Copy the tnsnames.ora file from the primary nodes subdirectory /u01/db/product/10.2/db/network/admin/. Then, edit it by replacing the original host-node names with the Clone or Disaster Recovery Oracle RAC host names. Create subdirectories for database logs and trace files under /u01/admin/prime (that is, adump, bdump, cdump, dpdump, hdump, pfile and udump). After the editing of the files for the first node is complete, copy initprimex.ora, orapwprimex, tnsnames.ora and the oratab files to the remaining nodes in the Clone or Disaster Recovery Oracle RAC being sure to also change the x value to the appropriate node number. SVC Advanced Copy Services overview and configuration This section provides the technical details for using SAN Volume Controller Advanced Copy Services functions. SVC FlashCopy characteristics SVC FlashCopy has many features and functions (see Table 9 for configuration limitations): The FlashCopy process involves two VDisks that are associated with a FlashCopy mapping. After being used in a FlashCopy mapping, a VDisk becomes a source or target VDisk. The FlashCopy process copies a whole source VDisk to a target VDisk within one cluster. The VDisks that you choose as the source and target VDisks can be provided by different I/O groups. A VDisk can be the source VDisk in multiple FlashCopy mappings. A VDisk can be the target VDisk in one FlashCopy mapping. A VDisk can be the source in one mapping and the target in another at the same time. The VDisks that you use for one FlashCopy mapping must be of the same size. You cannot change the size of VDisks that are members in a FlashCopy mapping. The FlashCopy process does not change the content of the source VDisk. 14

17 The FlashCopy process destroys the original content of the target VDisk. After initiating the FlashCopy, the target VDisk represents a clone of the source VDisk. After establishing the FlashCopy, you can access the target VDisk as read-write. Until you copy all the data from the source to the target, the target VDisk presents the data as long as the mapping exists. After you copy all the data, delete the mapping and the VDisks are independent again. After you copy all the data, the former target VDisk holds the data, even without the mapping. FlashCopy property Maximum Comment FlashCopy targets per source 16 This is the maximum number of FlashCopy mappings that can exist with the same source VDisk. FlashCopy mappings per SVC cluster FlashCopy consistency groups per SVC cluster FlashCopy VDisk space per I/O group FlashCopy mappings per consistency group Table 9. Configuration limits for FlashCopy, in SVC To use FlashCopy, follow this sequence of events: 3855 You calculate the maximum number of FlashCopy mappings by noticing that 240 source VDisks with 16 FlashCopy mappings plus one more with 15 mappings uses up all 4096 VDisks per cluster. 128 The SVC software sets this TB This is a limit on the quantity of FlashCopy mappings that use bitmap space from this I/O group. This maximum configuration consumes all 512 MB of bitmap space for the I/O group and allows no Metro and Global Mirror bitmap space. The default is 40 TB Plan and associate the source dataset with a target location (one or more source and target VDisks). 2. Create a new FlashCopy mapping between the specified source VDisk and target VDisk. 3. Prepare (pretrigger) the FlashCopy. Direct the prepare command to either a consistency group of FlashCopy mappings or to a standalone FlashCopy mapping. The prepare command places the FlashCopy mappings in the preparing state. (The prepare command also flushes the cached data for the source disk). 4. After all of the FlashCopy mapping or mappings in a consistency group are in the prepared state, you can start the FlashCopy mappings. This is often referred to as triggering the FlashCopy. 5. After copying the source to the target, there should be no dependent mappings the source and target are independent and the status of the mapping shows as copied. If you use the autodelete option when creating or changing the FlashCopy mapping, the mapping is automatically deleted when the copy completes. Without the autodelete option, the mapping remains and you can reactivate it by preparing and starting again. At this point, both the source and target are available for read-write activity. 15

18 Background copy rate, parameter copyrate: A FlashCopy mapping has a background copy rate property. This is expressed as a percentage and can take values between 0 and 100. You can change the background copy rate when the FlashCopy mapping is in any state. If you specify a value of 0, then background copy is disabled. This is equivalent to a NOCOPY option, which is suitable for shortlived FlashCopy mappings that are used for backup purposes only. Because the source dataset is not expected to change much during the lifetime of the FlashCopy mapping, it is more efficient in terms of managed disk I/O processes not to perform a background copy. Consistency for a group of mappings: A single mapping is consistent if the dataset of the target VDisk matches the dataset of the source VDisk at some point in past time. In the same way, a group of mappings is consistent if each of the mappings in the group are themselves consistent and each mapping represents the same point in time as all of the others. Figure 1 shows an example of a consistency group for the mappings of DATA_DG. FlashCopy consistency group FC_PR_DATA_VDG PRIME_DATA_ASM1 FlashCopy mapping name PRIME MAP ASM1 FC_PR_DATA_ASM1 PRIME_DATA_ASM2 FlashCopy mapping name PRIME MAP ASM2 FC_PR_DATA_ASM2 PRIME_DATA_ASM3 FlashCopy mapping name PRIME MAP ASM3 FC_PR_DATA_ASM3 Figure 1. Example of a FlashCopy consistency group Examples of Advanced Copy Services FlashCopy commands This section of the white paper provides examples of how you can use Advanced Copy Services FlashCopy commands. Creating FlashCopy consistency groups Make three consistency groups corresponding to each of the Oracle ASM disk groups DATA_DG, LOGS_DG, and ARCH_DG. These consistency groups ensure that all FlashCopy member pairs are in synchronization when taking a backup. Example 4: svctask mkfcconsistgrp IBM_2145:OSL_SVC_11:admin> svctask mkfcconsistgrp -name FC_PR_DATA_VDG IBM_2145:OSL_SVC_11:admin> svctask mkfcconsistgrp -name FC_PR_ARCH_VDG IBM_2145:OSL_SVC_11:admin> svctask mkfcconsistgrp -name FC_PR_LOGS_VDG 16

19 Creating FlashCopy mappings Table 10 shows the FlashCopy consistency groups and related mappings. FCCONSISTENCYGRP SVC disk name FlashCopy mapping name SVC target-disk name FC_PR_DATA_VDG PRIME_DATA_ASM1 PRIME_DATA_ASM2 PRIME_DATA_ASM3 PRIME_MAP_ASM1 PRIME_MAP_ASM2 PRIME_MAP_ASM3 FC_PR_DATA_ASM1 FC_PR_DATA_ASM2 FC_PR_DATA_ASM3 FC_PR_ARCH_VDG FC_PR_LOGS_VDG PRIME_ARCH_ASM4 PRIME_ARCH_ASM5 PRIME_ARCH_ASM6 PRIME_LOGS _ASM7 PRIME_LOGS _ASM8 PRIME_LOGS _ASM9 Table 10. Mappings for FlashCopy by consistency group PRIME_MAP_ASM4 PRIME_MAP_ASM5 PRIME_MAP_ASM6 PRIME_MAP_ASM7 PRIME_MAP_ASM8 PRIME_MAP_ASM9 FC_PR_ARCH_ASM4 FC_PR_ARCH_ASM5 FC_PR_ARCH_ASM6 FC_PR_LOGS_ASM7 FC_PR_LOGS_ASM8 FC_PR_LOGS_ASM9 1. Create the FlashCopy mappings and add them to the appropriate consistency group. 2. With SVC or later, use the incremental option. With this option, the first FlashCopy for a mapping copies all blocks from one VDisk to another, but subsequent triggering of the same mapping only copies changed blocks. This can significantly speed up all subsequent VDisk synchronizations. If the target VDisks have the database brought online, you must recreate the FlashCopy, rather than rely on the incremental FlashCopy parameter. 3. Using PRIME_DATA_ASM1 VDisk as an example, you must make mappings on a VDiskby-VDisk basis and place the mappings in the appropriate consistency groups. Example 5: svctask mkfcmap IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -source PRIME_DATA_ASM1 -target FC_PR_DATA_ASM1 name PRIME_MAP_ASM1 -consistgrp FC_PR_DATA_VDG -copyrate 100 -incremental Listing FlashCopy mappings Here is an example of listing the mappings that belong to the FC_PR_DATA_VDG consistency group: Example 6: svcinfo lsfcmap IBM_2145:OSL_SVC_11:admin> svcinfo lsfcmap -delim : -filtervalue group_name=fc_pr_data_vdg id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name: group_id:group_name:status:progress:copy_rate:clean_progress:incremental 6:PRIME_MAP_ASM1:5:PRIME_DATA_ASM1:45:FC_PR_DATA_ASM1:1:FC_PR_DATA_VDG:idle_ or_copied:0:100:100:on 7:PRIME_MAP_ASM2:6:PRIME_DATA_ASM2:46:FC_PR_DATA_ASM2:1:FC_PR_DATA_VDG:idle_ or_copied:0:100:100:on 8:PRIME_MAP_ASM3:7:PRIME_DATA_ASM3:47:FC_PR_DATA_ASM3:1:FC_PR_DATA_VDG:idle_ or_copied:0:100:100:on 17

20 Creating reversed FlashCopy mappings When making the reversed FlashCopy mappings, ensure that the primary mappings are idle or 100 percent copied before removing them to prepare for the reversed FlashCopy. Create the mappings by making the former target VDisks into the source and the original source into the new target. Use the autodelete parameter to automatically remove the reversed mappings when the copy is complete (see Table 11). FCCONSISTENCYGRP SVC target-disk name FC_PR_DATA_VDG FC_PR_ARCH_VDG FC_PR_LOGS_VDG FC_PR_DATA_ASM1 FC_PR_DATA_ASM2 FC_PR_DATA_ASM3 FC_PR_ARCH_ASM4 FC_PR_ARCH_ASM5 FC_PR_ARCH_ASM6 FC_PR_LOGS_ASM7 FC_PR_LOGS_ASM8 FC_PR_LOGS_ASM9 Reverse FlashCopy name SVC disk name REV_MAP_ASM1 REV_MAP_ASM2 REV_MAP_ASM3 REV_MAP_ASM4 REV_MAP_ASM5 REV_MAP_ASM6 REV_MAP_ASM7 REV_MAP_ASM8 REV_MAP_ASM9 PRIME_DATA_ASM1 PRIME_DATA_ASM2 PRIME_DATA_ASM3 PRIME_ARCH_ASM4 PRIME_ARCH_ASM5 PRIME_ARCH_ASM6 PRIME_LOGS_ASM7 PRIME_LOGS_ASM8 PRIME_LOGS_ASM9 Table 11. Reverse FlashCopy mappings by consistency group Example 7: svctask mkfcmap reverse relationship IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -source FC_PR_DATA_ASM1 -target PRIME_DATA_ASM1 -name REV_PR_MAP_ASM1 -consistgrp PRIME_DATA_VDG -copyrate 100 -autodelete Creating cloned FlashCopy mappings Create the cloned FlashCopy mappings by consistency group. (see Table 12). FCCONSISTENCYGRP SVC disk name FlashCopy mapping name SVC target-disk name FC_PR_DATA_VDG PRIME_DATA_ASM1 PRIME_DATA_ASM2 PRIME_DATA_ASM3 CL_MAP_ASM1 CL_MAP_ASM2 CL_MAP_ASM3 CL_DATA_ASM1 CL_DATA_ASM2 CL_DATA_ASM3 FC_PR_ARCH_VDG FC_PR_LOGS_VDG PRIME_ARCH_ASM4 PRIME_ARCH_ASM5 PRIME_ARCH_ASM6 PRIME_LOGS_ASM7 PRIME_LOGS_ASM8 PRIME_LOGS_ASM9 Table 12. Clone FlashCopy mappings by consistency group Example 8: svctask mkfcmap CL_MAP_ASM4 CL_MAP_ASM5 CL_MAP_ASM6 CL_MAP_ASM7 CL_MAP_ASM8 CL_MAP_ASM9 CL_ARCH_ASM4 CL_ARCH_ASM5 CL_ARCH_ASM6 CL_LOGS_ASM7 CL_LOGS_ASM8 CL_LOGS_ASM9 IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -source PRIME_DATA_ASM1 -target CL_DATA_ASM1 -name CL_MAP_ASM1 -consistgrp FC_PR_DATA_VDG -copyrate incremental Starting consistency groups Start the consistency group by using the following command sequences. Example 9: svctask startfcconsistgrp IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_DATA_VDG IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_LOGS_VDG IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_ARCH_VDG 18

21 SVC Metro Mirror characteristics There are certain characteristics of SVC Metro Mirror that you need to understand, as follows (also see Table 13): FlashCopy targets cannot be in a Metro Mirror or Global Mirror relationship, only FlashCopy sources can be in either of these mirrored relationships. You cannot move Metro Mirror or Global Mirror source or target VDisks to different I/O groups. You cannot resize Metro Mirror or Global Mirror VDisks. Intracluster Metro Mirror or Global Mirror can only mirror between VDisks in the same I/O group. The primary VDisks must be the same size as the secondary VDisks. However, it can be a different type (image, striped or sequential mode), or have different cache settings (cacheenabled or cache-disabled). Metro Mirror configuration limits Number of Metro Mirror consistency groups Number of Metro Mirror relationships Total VDisk size per I/O group Table 13. Metro Mirror limits in SVC Parameter value 256 per SVC cluster 1024 per SVC cluster 40 TB is the limit per I/O group on the quantity of primary and secondary VDisk address spaces that can participate in a Metro Mirror relationship. Sequence of events in creating Metro Mirror relationships The following tasks show the sequence of events for creating Metro Mirror relationships: 1. Create partnership between clusters. 2. Make remote-copy consistency groups. 3. Create relationships. 4. Trigger remote-copy groups. 5. When the status is consistent, stop the relationship for a point-in-time copy. Metro Mirror consistency Metro Mirror is a fully synchronous remote-copy technique; it is designed to ensure that updates are committed at both primary and secondary VDisks before the application is given completion to an update task. Figure 2 shows how a write task to the primary VDisk, which is mirrored to the cache for the secondary VDisk before an acknowledgement of the write task is sent back to the host that has issued the write task. This ensures that the secondary is real-time synchronized, in case it is needed in a failover situation. However, this means that the application is fully exposed to the latency and bandwidth limitations of the communication link to the secondary site. This is the reason for the distance limitations when applying Metro Mirror. 19

22 Consistency for a group of relationships A single relationship is consistent if the dataset of the target VDisk matches the source Vdisk s dataset at some point in past time. Similarly, a group of relationships is consistent if each relationship in the group is itself consistent and represents the same point in time as all others. 1. Write 4. ACK Cache 3. Acknowledge write 2. Mirror write Cache Primary VDisk Metro Mirror relationship Secondary VDisk Figure 2. Metro Mirror consistency Examples of Advanced Copy Services Metro Mirror commands This section provides examples of using Advanced Copy Services Metro Mirror commands. Creating SVC intercluster partnership Metro Mirror requires that you configure an intercluster partnership. You can establish a partnership after the appropriate connections and zoning are complete. The SVC clusters that are used in this partnership are described in Table 14. Role Production SVC cluster Disaster Recovery SVC cluster Storage Name OSL_SVC_11 ARC_SVC_JMT1 Storage IP Table 14. SVC partner clusters Listing cluster candidates Check for a cluster candidate for the partnership, see the following example. Example 10: svcinfo lsclustercandidate IBM_2145:OSL_SVC_11:admin> svcinfo lsclustercandidate id configured cluster_name A42 no ARC_SVC_JMT1 The ARC_SVC_JMT1 cluster candidate is not configured as a partner. 20

23 Creating cluster partnership from production to Disaster Recovery The mkpartnership command creates a one-way partnership between the production cluster (OSL_SVC_11) and the Disaster Recovery cluster (ARC_SVC_JMT1). To complete the two-way partnership, issue the equivalent mkpartnership command on the Disaster Recovery cluster. Example 11: svctask mkpartnership IBM_2145:OSL_SVC_11:admin> svctask mkpartnership ARC_SVC_JMT1 Now, the partnership is partially created. Example 12: svcinfo lscluster IBM_2145:OSL_SVC_11:admin> svcinfo lscluster -delim : ARC_SVC_JMT1 id: a09064 name:arc_svc_jmt1 location:remote partnership:partially_configured_local bandwidth:50 Creating cluster partnership from Disaster Recovery to production Connect to the remote cluster (ARC_SVC_JMT1) and complete the creation of the partnership. Example 13: svctask mkpartnership IBM_2145:ARC_SVC_JMT_1:admin> svctask mkpartnership OSL_SVC_11 Verify that the status of the remote-to-primary partnership shows that the relationship is fully configured. Example 14: svcinfo lscluster IBM_2145:ARC_SVC_JMT_1:admin> svcinfo lscluster -delim : OSL_SVC_11 id: a4 name:osl_svc_11 location:remote partnership:fully_configured bandwidth:50 Now, the partnership between OSL_SVC_11 and ARC_SVC_JMT1is fully configured with a bandwidth of 50 MBps. Creating remote-copy consistency groups In this example, a consistency group is created between OSL_SVC_11 and ARC_SVC_JMT1 and the copy relationship is implemented with that consistency group. (Note: If you create a Metro Mirror consistency group without using the cluster [the name or ID of the remote cluster], the consistency group is only created on the local cluster.) Make three consistency groups: MM_PR_DATA_VDG, MM_PR_ARCH_VDG and MM_PR_LOGS_VDG. Example 15: svctask mkrcconsistgrp IBM_2145:OSL_SVC_11:admin> svctask mkrcconsistgrp -cluster ARC_SVC_JMT1 - name MM_PR_DATA_VDG IBM_2145:OSL_SVC_11:admin> svctask mkrcconsistgrp -cluster ARC_SVC_JMT1 - name MM_PR_ARCH_VDG IBM_2145:OSL_SVC_11:admin> svctask mkrcconsistgrp -cluster ARC_SVC_JMT1 - name MM_PR_LOGS_VDG 21

24 Example 16: svcinfo lsrcconsistgrp IBM_2145:OSL_SVC_11:admin> svcinfo lsrcconsistgrp name MM_PR_DATA_VDG id name master_cluster_id master_cluster_name aux_cluster_id aux_cluster_name primary state relationship_count copy_type 248 MM_PR_DATA_VDG A4 OSL_SVC_ A09064 ARC_SVC_JMT1 empty 0 empty_group 249 MM_PR_ARCH_VDG A4 OSL_SVC_ A09064 ARC_SVC_JMT1 empty 0 empty_group 250 MM_PR_LOGS_VDG A4 OSL_SVC_ A09064 ARC_SVC_JMT1 empty 0 empty_group Creating a remote-copy relationship Create VDisks on the remote cluster (SVC_JMT_1) with the same size and name as the VDisks on the primary cluster (OSL_SVC_11). This is required to create a copy relationship. Create the remote-copy relationships and add them to the appropriate consistency groups (see Table 15). OSL_SVC_11 SVC_JMT_1 RCCONSISTENCYGRP VDisk name(source) RELATIONSHIP VDisk name(target) -consistgrp -master -name -aux MM_PR_DATA_VDG PRIME_DATA_ASM1 PRIME_DATA_ASM2 PRIME_DATA_ASM3 PRIME_MM_ASM1 PRIME_MM_ASM2 PRIME_MM_ASM3 PRIME_DATA_ASM1 PRIME_DATA_ASM2 PRIME_DATA_ASM3 MM_PR_ARCH_VDG MM_PR_LOGS_VDG PRIME_ARCH_ASM4 PRIME_ARCH_ASM5 PRIME_ARCH_ASM6 PRIME_LOGS_ASM7 PRIME_LOGS_ASM8 PRIME_LOGS_ASM9 Table 15. Metro Mirror relationships by consistency group Creating remote-copy relationships PRIME_MM_ASM4 PRIME_MM_ASM5 PRIME_MM_ASM6 PRIME_MM_ASM7 PRIME_MM_ASM8 PRIME_MM_ASM9 The following example shows how to create remote-copy relationships. Example 17: svctask mkrcrelationship IBM_2145:OSL_SVC_11:admin> svctask mkrcrelationship -master PRIME_DATA_ASM1 -aux PRIME_DATA_ASM1 -cluster ARC_SVC_JMT1 -name PRIME_MM_ASM1 -consistgrp MM_PR_DATA_VDG Listing remote-copy relationships The following example shows how to list the remote-copy relationships. Example 18: svcinfo lsrcrelationship PRIME_ARCH_ASM4 PRIME_ARCH_ASM5 PRIME_ARCH_ASM6 PRIME_LOGS_ASM7 PRIME_LOGS_ASM8 PRIME_LOGS_ASM9 IBM_2145:OSL_SVC_11:admin> svcinfo lsrcrelationship -delim : -filtervalue group_name= MM_PR_* Starting the consistency group When a consistency group is started or stopped, it affects all the relationships in that group. The startrcconsistgrp command has the following parameters: 22

25 -primary: Which VDisk will become the primary VDisk in this relationship? Is it the master or the auxiliary that needs to be the active disk? -force: This forces the copy process to start. -clean: Any changes that are made at the secondary are ignored. Only changes that are made at the primary are considered when synchronizing the primary and secondary VDisks. The consistency group is started and the master VDisk in the relationship is designated as the primary (source) VDisk. Example 19: svctask startrcconsistgrp IBM_2145:OSL_SVC_11:admin> svctask startrcconsistgrp -primary master MM_PR_DATA_VDG Listing the consistency group IBM_2145:OSL_SVC_11:admin> svcinfo lsrcconsistgrp -delim : -filtervalue name= MM_PR_DATA_VDG id:249 name:mm_pr_data_vdg master_cluster_id: a4 master_cluster_name:osl_svc_11 aux_cluster_id: a09064 aux_cluster_name:arc_svc_jmt1 primary: state:idling relationship_count:3 freeze_time: status: sync:out_of_sync copy_type:metro RC_rel_id:5 RC_rel_name:PRIME_MM_ASM1 RC_rel_id:6 RC_rel_name:PRIME_MM_ASM2 RC_rel_id:7 RC_rel_name:PRIME_MM_ASM3 Stopping the consistency group When the relationships are in the consistent_synchronized state, issuing the stoprcconsistgrp command changes the state to consistent_stopped, which creates a read-only point-in-time copy. By using the -access parameter, the consistency group is stopped and the state idles. Both VDisks are available for read-write activities. Example 20: svctask stoprcconsistgrp IBM_2145:OSL_SVC_11:admin> svctask stoprcconsistgrp -access MM_PR_DATA_VDG 23

26 Backup and Recovery by example This section is a step-by-step description of using SVC Copy Services to back up and recover Oracle RAC 10g databases on Oracle ASM. It does not describe the use of tape backups, which is part of the backup strategy that is left up to the individual enterprise to incorporate in its Backup and Recovery strategy. Backup and Recovery procedures and validations The procedures discussed here are presented in a form that describes in detail the steps to complete the task. Then, the reasoning behind these steps is explained. Finally, you will learn how to validate the correctness of this approach. This section lists the most commonly encountered scenarios when backing up and recovering Oracle databases and, then, presents a detailed checklist for this purpose. The general approach is as follows: Back up a running Oracle RAC 10g database on Oracle ASM by using Advanced Copy Services. Restore the database by using SVC Advanced Copy Services. Recover the database by using standard Oracle database recovery procedures. Verify that the database is up, running and functioning normally. For the examples that are included here, the commands are displayed after each step, as follows: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp prep FC_PR_DATA_VDG 24

27 Scenario 1: Loss of DATA_DG containing data files This scenario demonstrates database recovery by creating a FlashCopy mapping to restore a previous backup of the DATA_DG Oracle ASM disk group. This assumes that the database is recoverable with the recover database command and that it can then be opened with the alter database open command. In other words, it assumes that the control files, the online redo logs and spfile are not lost. Prerequisites Here are the prerequisites for scenario 1: You must have already configured the database and have it running, as described in the Configuration and setup section. You must have already created the SVC VDisk and Oracle ASM mappings. You must have already created the FlashCopy mappings. The following represents an outline of the steps that scenario 1 follows: 1. Back up the DATA_DG disk group. 2. Back up the parameter and control files. 3. Shut down the database. 4. Reverse the FlashCopy process. 5. Recover the database on one node of the cluster. 6. After the recovery is complete, start the cluster resources for the database. Recovering from the loss of ASM DATA_DG (explained in detail) This scenario explains the case where it is necessary to restore and recover database data files. Further, it assumes that only the DATA_DG disk group is restored. 1. Start a SQL*Plus session as SYSDBA and put the database in hot-backup mode: [oracle@alpha ~]$ export ORACLE_HOME=/u01/db/product/10.2/db [oracle@alpha ~]$ export ORACLE_SID=PRIME [oracle@alpha ~]$ sqlplus / as sysdba SQL> alter database begin backup; 2. Start the FlashCopy process for the FC_PR_DATA_VDG consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_DATA_VDG 3. Take the database out of hot-backup mode. Force the archiving of the online redo logs for all Oracle RAC threads: SQL> alter database end backup; SQL> alter system archive log current; 4. This database recovery uses the archived logs that are stored in the ARCH_DG Oracle ASM disk group. However, as a standard practice, you should back up the archived logs: RMAN> backup archivelog all format'/u01/backup/prime/al_t%t_s%s_p%p'; 25

28 5. Back up the control file: RMAN> sql 'alter database backup controlfile to trace'; RMAN> backup as copy format '/u01/backup/prime/ctl_copy_t%t_s%s_p%p.ctl' current controlfile; RMAN> backup format '/u01/backup/prime/cf_t%t_s%s_p%p' current controlfile; 6. Back up the spfile: RMAN> sql "create pfile=''/u01/backup/prime/init.ora.back'' from spfile"; RMAN> backup format '/u01/backup/prime/spfile_t%t_s%s_p%p' spfile; 7. Monitor the progress of the copy of the consistency-group mappings: IBM_2145:OSL_SVC_11:admin> svcinfo lsfcmap -delim : -filtervalue group_name=fc_pr_data_vdg id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name :group_id:group_name:status:progress:copy_rate:clean_progress:incremental 6:PRIME_MAP_ASM1:5:PRIME_DATA_ASM1:45:FC_PR_DATA_ASM1:1:FC_PR_DATA_VDG:copy ing:29:100:100:off 7:PRIME_MAP_ASM2:6:PRIME_DATA_ASM2:46:FC_PR_DATA_ASM2:1:FC_PR_DATA_VDG:copy ing:15:100:100:off 8:PRIME_MAP_ASM3:7:PRIME_DATA_ASM3:47:FC_PR_DATA_ASM3:1:FC_PR_DATA_VDG:copy ing:16:100:100:off Recovery starts from this point: 8. You can remove the relationships to prepare for the restore operation (only when the consistency-group mappings show that the background copy is complete). Remove all of the original FlashCopy mappings that belong to the FC_PR_DATA_VDG consistency group: IBM_2145:OSL_SVC_11:admin> svctask rmfcmap PRIME_MAP_ASM1 IBM_2145:OSL_SVC_11:admin> svctask rmfcmap PRIME_MAP_ASM2 IBM_2145:OSL_SVC_11:admin> svctask rmfcmap PRIME_MAP_ASM3 9. Use the srvctl command on node ALPHA to shut down the database and all ASM instances: [oracle@alpha ~]$ srvctl stop database d PRIME -o abort [oracle@alpha ~]$ srvctl stop asm -n ALPHA -o abort [oracle@alpha ~]$ srvctl stop asm -n BETA -o abort [oracle@alpha ~]$ srvctl stop asm -n GAMMA -o abort [oracle@alpha ~]$ srvctl stop asm -n DELTA o abort 10. Create new FlashCopy mappings that have the source and destination VDisks in reverse order from step 2 (also known as SVC flashback relationships). That is, you can create the mappings to restore the database backup. Use the -autodelete option: IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -target PRIME_DATA_ASM1 -source FC_PR_DATA_ASM1 -name PRIME_MAP_ASM1 -consistgrp FC_PR_DATA_VDG -copyrate 100 -autodelete IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -target PRIME_DATA_ASM2 -source FC_PR_DATA_ASM2 -name PRIME_MAP_ASM2 -consistgrp FC_PR_DATA_VDG -copyrate 100 -autodelete 26

29 IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -target PRIME_DATA_ASM3 -source FC_PR_DATA_ASM3 -name PRIME_MAP_ASM3 -consistgrp FC_PR_DATA_VDG -copyrate 100 -autodelete 11. Start the FlashCopy process for the FC_PR_DATA_VDG consistency group (with the reversed FlashCopy relationships in place): IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp prep FC_PR_DATA_VDG 12. If your system runs the Linux operating system, use the Linux asmlib command to rescan the disks in order to refresh the disk information on all server nodes: ~]$ /etc/init.d/oracleasm scandisks ~]$ /etc/init.d/oracleasm scandisks ~]$ /etc/init.d/oracleasm scandisks ~]$ /etc/init.d/oracleasm scandisks 13. Start the ASM instance on node ALPHA: ~]$ srvctl start asm -n ALPHA 14. Set the environment to the +ASM1 instance. Start SQL*Plus and check the status of the ASM disk groups. Mount any disk groups that are not already mounted: ~]$ export ORACLE_HOME=u01/asm/product/10.2/asm ~]$ export ORACLE_SID=ASM1 ~]$ sqlplus / as sysdba SQL> select name, state from v$asm_diskgroup; NAME STATE DATA_DG DISMOUNTED LOGS_DG MOUNTED ARCH_DG MOUNTED SQL> alter diskgroup DATA_DG mount; 15. Start up and mount the database instance on node ALPHA by using the srvctl command: [oracle@alpha ~]$ srvctl start instance -i PRIME1 -d PRIME -o mount 16. Set the environment to the PRIME1 instance. Start SQL*Plus and check the status of the data files. Bring any offline data files online: [oracle@alpha ~]$ export ORACLE_HOME=u01/db/product/10.2/db [oracle@alpha ~]$ export ORACLE_SID=PRIME1 [oracle@alpha ~]$ sqlplus / as sysdba SQL> select name, status from v$datafile where status not in ('ONLINE', 'SYSTEM'); NAME STATUS DATA_DG/prime/datafile/soe OFFLINE SQL> alter database datafile +DATA_DG/prime/datafile/soe online; 17. Begin the database-recovery process and specify automatic mode: SQL> recover database; ORA-00279: change generated at 01/25/ :47:07 needed for thread 1 ORA-00289: suggestion : +ARCH_DG/prime/1_193_ arc ORA-00280: change for thread 1 is in sequence #193 Specify log: {<RET>=suggested filename AUTO CANCEL} auto 27

30 18. When recovery is complete, open the database: SQL> alter database open; 19. Use the srvctl command to start the remaining ASM instances, then start the remaining database instances: ~]$ srvctl start asm -n BETA ~]$ srvctl start asm -n GAMMA ~]$ srvctl start asm -n DELTA ~]$ srvctl start database d PRIME 20. Use the crs_stat t command to verify that all cluster resources have returned to online status: [root@alpha ~]$ crs_stat -t 21. Check the status of the FlashCopy restore mappings for FC_PR_DATA_VDG: IBM_2145:OSL_SVC_11:admin> svcinfo lsfcconsistgrp -filtervalue name=fc_pr_data_vdg 22. When no FlashCopy mappings are present, the system is ready to re-create the regular FlashCopy relationships: IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -source PRIME_DATA_ASM1 -target FC_PR_DATA_ASM1 -name PRIME_MAP_ASM1 -consistgrp FC_PR_DATA_VDG -copyrate 100 -incremental 23. Normal backups can now resume. 28

31 Scenario 2: Loss of the entire database The second exercise shows how to back up all Oracle ASM disk groups to make it possible to recover not only from the loss of the database data files but also to be able to recover the control files, the online redo logs and the spfile. This is an incomplete recovery to a point in time before the loss of the Oracle ASM disk groups. Prerequisites Here are the prerequisites for scenario 2: You must have already configured the database and have it running, as described in the Configuration and setup section. You must have already created the SVC VDisk and Oracle ASM mappings. You must have already created the FlashCopy mappings. The following represents an outline of the steps that scenario 2 follows: 1. Back up the LOGS_DG ASM disk group. 2. Back up the DATA_DG ASM disk group. 3. Back up the parameter and control files to the directory in ARCH_DG. 4. Back up the ARCH_DG ASM disk group. 5. Shut down the database. 6. Reverse the FlashCopy process. 7. Recover the database on one node of the cluster. 8. After the recovery is complete, start the cluster resources for the database. Recovering from the loss of all ASM disk groups (explained in detail) This scenario explains the case where it is necessary to recover from the loss of all ASM disk groups. 1. Start the FlashCopy process for the FC_PR_LOGS_VDG consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_LOGS_VDG 2. Start a SQL*Plus session as SYSDBA to put the database in hot-backup mode, which makes it possible to use FlashCopy to capture the point-in-time copy while any changes that occur during this process are sent to the archived redo logs: [oracle@alpha ~]$ export ORACLE_HOME=/u01/db/product/10.2/db [oracle@alpha ~]$ export ORACLE_SID=PRIME1 [oracle@alpha ~]$ sqlplus / as sysdba SQL> alter database begin backup; 3. After putting the database in backup mode, start the FlashCopy process for the PRIME_DATA_DG consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_DATA_VDG 29

32 4. Verify that the consistency group has started before proceeding. You will see the copying process: IBM_2145:OSL_SVC_11:admin> svcinfo lsfcmap -delim : -filtervalue group_name=fc_pr_data_vdg id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name :group_id:group_name:status:progress:copy_rate:clean_progress:incremental 0:PRIME_MAP_ASM1:5:PRIME_DATA_ASM1:45:FC_PR_DATA_ASM1:1:FC_PR_DATA_VDG:copy ing:45:100:100:on 1:PRIME_MAP_ASM2:6:PRIME_DATA_ASM2:46:FC_PR_DATA_ASM2:1:FC_PR_DATA_VDG:copy ing:14:100:100:on 2:PRIME_MAP_ASM3:7:PRIME_DATA_ASM3:47:FC_PR_DATA_ASM3:1:FC_PR_DATA_VDG:copy ing:20:100:100:on 5. Take the database out of hot-backup mode. Force the archiving of the online redo logs from all redo log threads: SQL> alter database end backup; SQL> alter system archive log current; 6. Start the FlashCopy process for the FC_PR_ARCH_VDG consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_ARCH_VDG 7. This database recovery uses the archived logs that are stored in the ARCH_DG ASM disk group. However, as a standard practice, it is best to back up the archived logs: RMAN> backup archivelog all format'/u01/backup/prime/al_t%t_s%s_p%p'; 8. Back up the control file: RMAN> sql 'alter database backup controlfile to trace'; RMAN> backup as copy format '/u01/backup/prime/ctl_copy_t%t_s%s_p%p.ctl' current controlfile; RMAN> backup format '/u01/backup/prime/cf_t%t_s%s_p%p' current controlfile; 9. Back up the spfile: RMAN> sql "create pfile=''/u01/backup/prime/init.ora.back'' from spfile"; RMAN> backup format '/u01/backup/prime/spfile_t%t_s%s_p%p' spfile; 10. Check the status of the consistency-group copies: IBM_2145:OSL_SVC_11:admin> svcinfo lsfcmap -delim : -filtervalue group_name=fc_pr_* id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name :group_id:group_name:status:progress:copy_rate:clean_progress:incremental 0:PRIME_MAP_ASM1:5:PRIME_DATA_ASM1:45:FC_PR_DATA_ASM1:1:FC_PR_DATA_VDG:copy ing:54:100:100:on Recovery starts from this point: 11. You can remove the relationships to prepare for the restore operation (only when the consistency-group mappings show that the background copy is complete). Remove all of the 30

33 original FlashCopy mappings that belong to consistency groups FC_PR_DATA_VDG, FC_PR_LOGS_VDG and FC_PR_ARCH_VDG: IBM_2145:OSL_SVC_11:admin> svctask rmfcmap PRIME_MAP_ASM1 12. Use the srvctl command on node ALPHA to shut down the database and all ASM instances: ~]$ srvctl stop database -d PRIME -o abort ~]$ srvctl stop asm -n ALPHA -o abort ~]$ srvctl stop asm -n BETA -o abort ~]$ srvctl stop asm -n GAMMA -o abort ~]$ srvctl stop asm -n DELTA o abort 13. Create new FlashCopy mappings that have the source and destination VDisks in reverse order from the original mappings; that is, create the mappings to restore the database backup. Use the -autodelete option: IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -source FC_PR_DATA_ASM1 -target PRIME_DATA_ASM1 -name REV_PR_MAP_ASM1 -consistgrp FC_PR_DATA_VDG -copyrate 100 -autodelete 14. Start the FlashCopy process for the FC_PR_DATA_VDG, FC_PR LOGS_VDG and FC_PR_ARCH_VDG consistency groups (with reversed FlashCopy relationships in place): IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_DATA_VDG IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_ARCH_VDG IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_LOGS_VDG 15. If you are using the Linux operating system: After the FlashCopy mappings have started, on the server nodes with asmlib on Linux, rescan the disks to refresh the disk information on all server nodes: [root@alpha ~]$ /etc/init.d/oracleasm scandisks [root@beta ~]$ /etc/init.d/oracleasm scandisks [root@gamma ~]$ /etc/init.d/oracleasm scandisks [root@delta ~]$ /etc/init.d/oracleasm scandisks 16. Start the ASM instance on node ALPHA: [oracle@alpha ~]$ srvctl start asm -n ALPHA 17. As the Oracle database administrator on node ALPHA, change your environment to the +ASM1 instance. Log in to SQL*Plus as SYSDBA and check the status of the ASM disk groups. Mount any disk groups that are not already mounted: [oracle@alpha ~]$ export ORACLE_HOME=u01/asm/product/10.2/asm [oracle@alpha ~]$ export ORACLE_SID=ASM1 [oracle@alpha ~]$ sqlplus / as sysdba SQL> select name, state from v$asm_diskgroup; NAME STATE DATA_DG DISMOUNTED LOGS_DG MOUNTED ARCH_DG MOUNTED SQL> alter diskgroup DATA_DG mount; 18. Start up and nomount the database instance on the ALPHA node (using the srvctl command): [oracle@alpha ~]$ srvctl start instance -i PRIME1 -d PRIME -o nomount 31

34 19. Restore the control file: RMAN>restore controlfile; 20. Change your environment to the PRIME1 instance. Log in to SQL*Plus as SYSDBA and mount the database: ~]$ export ORACLE_HOME=u01/db/product/10.2/db ~]$ export ORACLE_SID=PRIME1 ~]$ sqlplus / as sysdba SQL> alter database mount; 21. Check the status of the data files. Bring any offline data files online: SQL> select name, status from v$datafile where status not in ('ONLINE', 'SYSTEM'); NAME STATUS DATA_DG/prime/datafile/soe OFFLINE SQL> alter database datafile +DATA_DG/prime/datafile/soe online; 22. Begin the database recovery process and specify automatic mode: SQL> recover database until cancel using backup controlfile; ORA-00279: change generated at 01/25/ :47:07 needed for thread 1 ORA-00289: suggestion : +ARCH_DG/prime/1_193_ arc ORA-00280: change for thread 1 is in sequence #193 Specify log: {<RET>=suggested filename AUTO CANCEL} auto 23. When recovery is complete, use the alter database open resetlogs command to open the database instance on node ALPHA: SQL> alter database open resetlogs; 24. Use Oracle Clusterware to start the remaining ASM instances, then start the remaining database instances: [oracle@alpha ~]$ srvctl start asm -n BETA [oracle@alpha ~]$ srvctl start asm -n GAMMA [oracle@alpha ~]$ srvctl start asm -n DELTA [oracle@alpha ~]$ srvctl start database -d PRIME 25. Verify that all cluster services return to online status by using the crs_stat t command: [root@alpha ~]$ crs_stat -t 26. Verify that the FlashCopy restore has finished and then re-create the original mappings that are used for regular database backups: IBM_2145:OSL_SVC_11:admin> svcinfo lsfcconsistgrp -filtervalue name=fc_pr_* 27. After the FlashCopy restore completes, the reversed mappings are gone. The system is now ready to re-create the regular FlashCopy mappings for the backup that is required immediately after recovery: IBM_2145:OSL_SVC_11:admin> svctask mkfcmap -source PRIME_DATA_ASM1 -target FC_PR_DATA_ASM1 -name PRIME_MAP_ASM1 -consistgrp FC_PR_DATA_VDG -copyrate 100 -incremental 28. Resume regularly scheduled backups. 32

35 Scenario 3: FlashCopy Clone of an Oracle RAC 10g with ASM database This exercise shows how to make a copy (clone) of an Oracle RAC 10g database on ASM to another set of Oracle RAC nodes. It is expected that the target Oracle RAC nodes are prepared with the same Oracle binaries that are installed on the source Oracle RAC. This example does not rename the database. You can use this process to set up a database for testing or development (which you later need to modify after it is cloned). Prerequisites Here are the prerequisites for scenario 3: You must have already configured the source database and have it running, as described in the Configuration of primary Oracle RAC servers section. You must have already set up the target Oracle RAC, as described in Table 7. You must have already configured the VDisks to be visible on the target Oracle RAC, as described in Table 8. The ASM setup must be complete, as explained in the Clone or Disaster Recovery Oracle ASM setup section. The database instance config files must be complete, as explained in the Clone or Disaster Recovery database instance requirements section. You must have already prepared the FlashCopy mappings, as explained in Table 12. The following represents an outline of the steps that Scenario 3 follows: 1. Trigger FlashCopy consistency groups to create a consistent backup. 2. Register the resources for the copied database with the target CRS. 3. Recover the database on one node of the target cluster. 4. After recovery is complete, start the remaining cluster resources for the database. Cloning Oracle RAC 10g with ASM (explained in detail) This scenario explains the case where it is necessary to clone Oracle RAC 10g with ASM. 1. Start the FlashCopy process for the PRIME_LOGS_DG consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_LOGS_VDG 2. Start a SQL*Plus session as SYSDBA and put the database in hot-backup mode: [oracle@alpha ~]$ export ORACLE_HOME=/u01/db/product/10.2/db [oracle@alpha ~]$ export ORACLE_SID=PRIME1 [oracle@alpha ~]$ sqlplus / as sysdba SQL> alter database begin backup; 3. Start the FlashCopy process for the PRIME_DATA_DG consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_DATA_VDG 33

36 4. Take the database out of backup mode. Force the archiving of the online redo logs: SQL> alter database end backup; SQL> alter system archive log current; 5. This database recovery uses the archived logs that are stored in the ARCH_DG ASM disk group: RMAN>backup archivelog all format'/u01/backup/prime/al_t%t_s%s_p%p'; 6. Back up the control file: RMAN> sql 'alter database backup controlfile to trace'; RMAN> backup format '/u01/admin/prime/pfile/ctl_copy_t%t_s%s_p%p.ctl' current controlfile; RMAN> backup format '+ARCH_DG/PRIME/rman_backups/cf_t%t_s%s_p%p' current controlfile; 7. Back up the spfile: RMAN> sql "create pfile=''/u01/admin/prime/pfile/init.ora.back'' from spfile"; RMAN> backup format '+ARCH_DG/PRIME/rman_backups/spfile_t%t_s%s_p%p' spfile; 8. Start the FlashCopy relationship for the FC_PR_ARCH_VDG consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_PR_ARCH_VDG 9. Monitor the copying progress for the consistency group mappings: IBM_2145:OSL_SVC_11:admin> svcinfo lsfcmap -delim : -filtervalue group_name=fc_pr_* 10. If you are using the Linux operating system: With ASMLib on Linux, rescan the disks to refresh the disk information on all clone server nodes: [root@cl_alpha ~]$ /etc/init.d/oracleasm scandisks [root@cl_beta ~]$ /etc/init.d/oracleasm scandisks [root@cl_gamma ~]$ /etc/init.d/oracleasm scandisks [root@cl_delta ~]$ /etc/init.d/oracleasm scandisks 11. Start the ASM instance on clone node1: [oracle@cl_alpha ~]$ srvctl start asm -n CL_ALPHA 34

37 12. As the Oracle database administrator on node CL_ALPHA, check the status of the ASM disk groups. Mount any disk groups that are not already mounted: ~]$ export ORACLE_HOME=u01/asm/product/10.2/asm ~]$ export ORACLE_SID=+ASM1 ~]$ sqlplus / as sysdba SQL> select name, state from v$asm_diskgroup; NAME STATE DATA_DG DISMOUNTED LOGS_DG MOUNTED ARCH_DG MOUNTED SQL> alter diskgroup DATA_DG mount; 13. Add the database to the cluster registry on the clone cluster: [root@cl_alpha ~]$export ORACLE_HOME=/u01/db/product/10.2/db [oracle@cl_alpha ~]$ srvctl add database -d PRIME -o /u01/db/product/10.2/db 14. Add the instances to the database on the clone cluster: [oracle@cl_alpha ~]$ srvctl add instance -d PRIME -i PRIME1 -n CL_ALPHA [oracle@cl_alpha ~]$ srvctl add instance -d PRIME -i PRIME2 -n CL_BETA [oracle@cl_alpha ~]$ srvctl add instance -d PRIME -i PRIME3 -n CL_GAMMA [oracle@cl_alpha ~]$ srvctl add instance -d PRIME -i PRIME4 -n CL_DELTA 15. Start up and nomount the database instance on the CL_ALPHA cluster node: [oracle@cl_alpha ~]$ srvctl start instance -i PRIME1 -d PRIME -o nomount 16. On the CL_ALPHA node, restore the control file: RMAN>restore controlfile; 17. On the CL_ALPHA node, mount the database: [oracle@cl_alpha ~]$ export ORACLE_HOME=u01/db/product/10.2/db [oracle@cl_alpha ~]$ export ORACLE_SID=PRIME1 [oracle@cl_alpha ~]$ sqlplus / as sysdba SQL> alter database mount; 18. Check the status of the data files. Bring any offline data files online: SQL> select name, status from v$datafile where status not in ('ONLINE', 'SYSTEM'); NAME STATUS DATA_DG/prime/datafile/soe OFFLINE SQL> alter database datafile +DATA_DG/prime/datafile/soe online; 19. Begin the database recovery and specify automatic mode: SQL> recover database until cancel using backup controlfile; ORA-00279: change generated at 01/25/ :47:07 needed for thread 1 ORA-00289: suggestion : +ARCH_DG/prime/1_193_ arc ORA-00280: change for thread 1 is in sequence #193 Specify log: {<RET>=suggested filename AUTO CANCEL} auto 35

38 20. When recovery is complete, use the alter database open resetlogs command to open the database instance on clone node CL_ALPHA: SQL> alter database open resetlogs; 21. Use Oracle Clusterware to start the remaining ASM instances on the clone nodes, then start the remaining database instances on the clone nodes: ~]$ srvctl start asm -n CL_BETA ~]$ srvctl start asm -n CL_GAMMA ~]$ srvctl start asm -n CL_DELTA ~]$ srvctl start database -d PRIME 22. Verify that all cluster resources return to online status by using the crs_stat t command: [root@cl_alpha ~]$ crs_stat -t 36

39 Scenario 4: Metro Mirror Clone of a Oracle RAC 10g database This exercise will show how to clone an Oracle RAC 10g database on ASM to a remote SVC and Oracle RAC using Metro Mirror. It is expected that the recipient RAC nodes have been prepared ahead of time with the same Oracle binaries installed on the target Oracle RAC. Prerequisites Here are some prerequisites for scenario 4: You must have already established the SVC cluster partnership. (See the Creating SVC intercluster partnership section.) You must have already configured the source database and have it running, as described in the Configuration of primary Oracle RAC servers section. You must have already set up the target Oracle RAC, as described in Table 7. You need to have already configured the VDisks so that they are visible on the target Oracle RAC, as described in Table 8. The ASM setup must be complete, as explained in the Clone or Disaster Recovery Oracle ASM setup section. The database instance config files must be complete, as explained in the Clone or Disaster Recovery database instance requirements section. You must have already created the Metro Mirror relationships, as described in Table 15. The following represents an outline of the steps that Scenario 4 follows: 1. Trigger Metro Mirror consistency groups to start the synchronization. 2. Stop the Metro Mirror consistency groups when the state is consistent_synchronized. 3. Register the resources for the copied database with the target CRS. 4. Recover the database on one node of the target Oracle RAC. 5. After recovery is complete, start the remaining cluster resources for the database. Metro Mirror clone of an Oracle RAC 10g database (explained in detail) This scenario explains the case where it is necessary to remotely clone Oracle RAC 10g with ASM. 1. Start the copy for the MM_PR_DATA_VDG, MM_PR_LOGS_VDG and MM_PR_ARCH_VDG Metro Mirror consistency groups: IBM_2145:OSL_SVC_11:admin> svctask startrcconsistgrp -primary master MM_PR_DATA_VDG IBM_2145:OSL_SVC_11:admin> svctask startrcconsistgrp -primary master MM_PR_ARCH_VDG IBM_2145:OSL_SVC_11:admin> svctask startrcconsistgrp -primary master MM_PR_LOGS_VDG 37

40 2. Wait until all of the Metro Mirror consistency groups show a status of consistent_synchronized and then stop the MM_PR_LOGS_VDG consistency group by using the access parameter: IBM_2145:OSL_SVC_11:admin> svcinfo lsrcconsistgrp -delim : IBM_2145:OSL_SVC_11:admin> svctask stoprcconsistgrp access MM_PR_LOGS_VDG 3. On the source Oracle RAC node, start a SQL*Plus session as SYSDBA and put the database in hot backup mode: [oracle@alpha ~]$ export $ORACLE_HOME=/u01/db/product/10.2/db [oracle@alpha ~]$ export $ORACLE-SID=PRIME [oracle@alpha ~]$ sqlplus / as sysdba SQL> alter database begin backup; 4. Stop the MM_PR_DATA_VDG Metro Mirror consistency group: IBM_2145:OSL_SVC_11:admin> svctask stoprcconsistgrp -access MM_PR_DATA_VDG 5. From the source Oracle RAC, take the database out of backup mode and force the archiving of the online redo logs: SQL> alter database end backup; SQL> alter system archive log current; 6. Back up the archived logs to shared storage. This scenario does not require these backed up logs. However, as a standard practice, you should back up the logs: RMAN>backup archivelog all format'/u01/backup/prime/al_t%t_s%s_p%p'; 7. Back up the control file: RMAN> sql 'alter database backup controlfile to trace'; RMAN>backup format '/u01/admin/prime/pfile/ctl_copy_t%t_s%s_p%p.ctl' current controlfile; RMAN>backup format '+ARCH_DG/PRIME/rman_backups/cf_t%t_s%s_p%p' current controlfile; 8. Back up the spfile: RMAN> sql "create pfile=''/u01/admin/prime/pfile/init.ora.back'' from spfile"; RMAN> backup format '+ARCH_DG/PRIME/rman_backups/spfile_t%t_s%s_p%p' spfile; 9. Stop the MM_PR_ARCH_VDG Metro Mirror consistency group: IBM_2145:OSL_SVC_11:admin> svctask stoprcconsistgrp access MM_PR_ARCH_VDG 10. Check the status of the Metro Mirror consistency group mappings. They should all show a status of idling: IBM_2145:OSL_SVC_11:admin> svcinfo lsrcrelationship -delim : 38

41 11. If you are using the Linux operating system: With ASMLib on Linux, rescan the disks to refresh the disk information on all clone Oracle RAC nodes: ~]$ /etc/init.d/oracleasm scandisks ~]$ /etc/init.d/oracleasm scandisks ~]$ /etc/init.d/oracleasm scandisks ~]$ /etc/init.d/oracleasm scandisks 12. Start the ASM instance on clone Oracle RAC node1: ~]$ srvctl start asm -n CL_ALPHA 13. As the Oracle database administrator on node CL_ALPHA, check the status of the ASM disk groups. Mount any disk groups that are not already mounted. If the disk groups are all dismounted, make sure that you remove the relationships: ~]$ export ORACLE_HOME=u01/asm/product/10.2/asm ~]$ export ORACLE_SID=+ASM1 ~]$ sqlplus / as sysdba SQL> select name, state from v$asm_diskgroup; NAME STATE DATA_DG DISMOUNTED LOGS_DG MOUNTED ARCH_DG MOUNTED SQL> alter diskgroup DATA_DG mount; 14. Add the database to the cluster registry on the clone cluster: [oracle@cl_alpha ~]$ srvctl add database -d PRIME -o /u01/db/product/10.2/db 15. Add the instances to the database on the clone cluster: [oracle@cl_alpha ~]$ srvctl add instance -d PRIME -i PRIME1 -n CL_ALPHA [oracle@cl_alpha ~]$ srvctl add instance d PRIME i PRIME2 n CL_BETA [oracle@cl_alpha ~]$ srvctl add instance d PRIME i PRIME3 n CL_GAMMA [oracle@cl_alpha ~]$ srvctl add instance d PRIME i PRIME4 n CL_DELTA 16. Start up and nomount the database instance on cluster node CL_ALPHA: [oracle@cl_alpha ~]$ srvctl start instance -i PRIME1 -d PRIME -o nomount 17. On node CL_ALPHA, restore the control file. (RMAN uses the backup directory on ASM disk group +ARCH_DG.) RMAN> restore controlfile; 18. On clone Oracle RAC node CL_ALPHA, mount the database: [oracle@cl_alpha ~]$ export ORACLE_HOME=u01/db/product/10.2/db [oracle@cl_alpha ~]$ export ORACLE_SID=PRIME1 [oracle@cl_alpha ~]$ sqlplus / as sysdba SQL> alter database mount; 19. Check the status of the data files. Bring any offline data files online: SQL> select name, status from v$datafile where status not in ('ONLINE', 'SYSTEM'); NAME STATUS DATA_DG/prime/datafile/soe OFFLINE 39

42 SQL> alter database datafile +DATA_DG/prime/datafile/soe online; 20. Begin the database recovery and specify automatic mode: SQL> recover database until cancel using backup controlfile; ORA-00279: change generated at 01/25/ :47:07 needed for thread 1 ORA-00289: suggestion : +ARCH_DG/prime/1_193_ arc ORA-00280: change for thread 1 is in sequence #193 Specify log: {<RET>=suggested filename AUTO CANCEL} auto 21. When recovery is complete, use the alter database open resetlogs command to open the database instance on clone node CL_ALPHA.: SQL> alter database open resetlogs; 22. Use Oracle Clusterware to start the remaining ASM instances on the clone nodes, then start the remaining database instances on the clone nodes: [oracle@cl_alpha ~]$ srvctl start asm -n CL_BETA [oracle@cl_alpha ~]$ srvctl start asm -n CL_GAMMA [oracle@cl_alpha ~]$ srvctl start asm -n CL_DELTA [oracle@cl_alpha ~]$ srvctl start database -d PRIME 23. Verify that all cluster resources have returned to online status by using the crs_stat t command: [root@cl_alpha ~]$ crs_stat t 40

43 Scenario 5: Backup and restore the OCR and voting disks This scenario tests the scenario where the Oracle Cluster Registry and voting disks are lost. FlashCopy is used to backup and restore the voting disks The OCR cannot be backed up with FlashCopy while the services are running. The OCR backup and recovery is handled by Oracle Clusterware. Oracle Clusterware will run properly after the recovery. Prerequisites Here are the prerequisites for scenario 5: You need to ensure that automatic cluster-registry backups are working properly. Oracle Clusterware must be taking OCR exports are taken on a regularly scheduled basis. You must have already set up the FlashCopy relationships for the voting disks, as found in Table 16. FCCONSISTENCYGRP SVC VDisk name FlashCopy mapping name SVC target VDisk name FC_VOTE_VDG PRIME_VOTE1 PRIME_VOTE2 PRIME_VOTE3 PRIME_MAP_VOTE1 PRIME_MAP_VOTE2 PRIME_MAP_VOTE3 FC_PRIME_VOTE1 FC_PRIME_VOTE2 FC_PRIME_VOTE3 Table 16. Setting up FlashCopy relationships for voting disks The following represents an outline of the steps that Scenario 5 follows: 1. Verify that the automatic OCR backups are taking place. 2. Run explicit backups of the OCR. 3. Run regularly scheduled FlashCopy backups of the voting disks. 4. Stop autostart of Oracle Clusterware and reboot all nodes. 5. Recover the OCR by using one of the automatic backups or by using one of the explicitly performed backups. 6. Restore the FlashCopy backups of the voting disks. 7. Restart Oracle Clusterware. Backing up and restoring the OCR and vote disks (explained in detail) 1. On a regular basis, display the location, timestamp and originating node name of the backup files that Oracle created in the past 4, 8 and 12 hours, as well as during the last day and week. You need to save this output because you cannot obtain the location of the backups from the OCR if the disks are lost: [root@alpha ~]$ ocrconfig -showbackup 2. Export the OCR explicitly on a regularly scheduled basis: [root@alpha ~]$ ocrconfig -export ocr_export_$(date +'%Y_%m_%d_%H_%M') 3. Start the PRIME_VOTE consistency group: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp -prep FC_VOTE_VDG 41

44 4. When it is time to start recovery of the OCR and Vote disks, disable the starting of Oracle Clusterware during system reboot: ~]$ /etc/init.d/init.crs disable ~]$ /etc/init.d/init.crs disable ~]$ /etc/init.d/init.crs disable ~]$ /etc/init.d/init.crs disable 5. Reboot all Oracle RAC nodes: ~]$ shutdown now -r ~]$ shutdown now -r ~]$ shutdown now -r ~]$ shutdown now -r 6. Restore the OCR by using either of the backups that were automatically taken or by using one of the explicitly taken exports: If you restore one of the automatic backups (recommended), use the following command: ~]$ ocrconfig -restore /u01/crs/product/10.2/crs/cdata/prime_crs/backup00.ocr If you restore one of the explicit exports, use the following command: ~]$ ocrconfig -import ocr_export_2008_03_31_13_25 7. Verify that the OCR disks are now available: ~]$ ocrcheck 8. Recover the voting disks by starting the reverse FlashCopy mapping: IBM_2145:OSL_SVC_11:admin> svctask startfcconsistgrp FC_VOTE_VDG 9. Verify that the voting disks are now available: ~]$ crsctl query css votedisk 10. Re-enable the autostart of cluster services on all nodes: ~]$ /etc/init.d/init.crs enable ~]$ /etc/init.d/init.crs enable ~]$ /etc/init.d/init.crs enable ~]$ /etc/init.d/init.crs enable 11. Start Cluster Services on all nodes: ~]$ /etc/init.d/init.crs start ~]$ /etc/init.d/init.crs start ~]$ /etc/init.d/init.crs start ~]$ /etc/init.d/init.crs start 42

45 Summary This paper explains a method for using IBM SAN Volume Controller (SVC) Advanced Copy Services to back up and recover an Oracle Real Application Clusters 10g database when the database uses Automatic Storage Management (ASM). The paper gives an overview of some fundamental SVC and Oracle concepts. The reader is advised to understand these concepts before implementing this strategy. Also included is a detailed description of the infrastructure that is required to implement this architecture, along with detailed steps for implementing backup and recovery by using both FlashCopy, for local backups and cloning, and Metro Mirror, for remote cloning or disaster recovery. 43

46 Appendix A: Oracle terms, components and definitions The information provided in this appendix is taken from Oracle Database Concepts 10g Release 2 (10.2) Part Number B ( Figure 3. Example of Oracle memory, process and file structures relationships 44